As someone who worked in the ML infra space: Google, Meta, XAI, Oracle, Microsoft, Amazon have clusters that perform better than the highest performing cluster on Top500. They don't submit because there's no reason to, and some want to keep the size of their clusters a secret. They're all running Nvidia. (Except Google, who uses TPUs and Nvidia.)
> El Capitan – we don’t yet know how big of a portion yet as we write this – with 43,808 of AMD’s “Antares-A” Instinct MI300A devices
By comparison XAI announced that they have 100k H100s. MI300A and H100s have roughly similar performance. Meta says they're training on more than 100k H100s for Llama-4, and have the equivalent of 600k H100s worth of compute. (Note that compute and networking can be orthogonal).
Also, Nvidia B200s are rolling out now. They offer 2-3x the performance of H100s.
> Nvidia B200s ... offer 2-3x the performance of H100s
For ML, not for HPC. ML and HPC are two completely different, only loosely related fields.
ML tasks are doing great with low precision, 16 and 8 bit precision is fine, arguably good results can be achieved even with 4 bit precision [0][1]. That won't do for HPC tasks, like predicting global weather, computational biology, etc. -- one would need 64 to 128 bit precision for that.
Nvidia needs to decide how to divide the billions of transistors on their new silicon. Greatly oversimplifying, they can choose to make one of the following:
* Card A with *n* FP64 cores, or
* Card B with *2n* FP32 cores, or
* Card C with *4n* FP16 cores, or
* Card D with *8n* FP8 cores, or (theoretically)
* Card E with *16n* FP4 cores (not sure if FP4 is a thing).
Their new offerings are arguably worse than their older offerings for HPC tasks, and the feeling with the HPC crowd is that "Nvidia and AMD are in the process of abandoning this market".
[0] https://papers.nips.cc/paper/2020/file/13b919438259814cd5be8...
With the B100 somehow announced to have lower scalar FP64 throughput than the H100 (did they remove the DP tensor cores ?), one will have to rely on Ozaki schemes (dgemm with int8 tensor cores) and lots of the recent body of work on mixed-precision linear algebra show there's a lot of computing power to be harnessed from Tensor Cores. One of the problems of HPC now is a level of ossification of some codebases (or the lack of availability of porting/coding/optimizing people). You shouldn't have to rewrite everything every 5 years but the hardware constructors go where they go and we still haven't found the right level of abstraction to avoid big porting efforts.
Yes, that's a great point that I missed. From anecdotal evidence, it seems more people are using supercomputers for ML use cases, that would have been traditionally done by HPC. (eg training models for weather forecasts)
The Top500 list is useful as a public, standardized baseline that is straightforward, with a predicted periodicity for more than 30 years. It is trickier to compare cloud infras due to their heterogeneity, fast pace, and more importantly, due the lack of standardized tests, although the MLCommons [1] have been very keen on helping with that.
If I understand your comment correctly, we're taking a stable but not that relevant metric, because the real players of the market are too secretive, fast and far ahead to allow for simple comparisons.
From a distance, it kinda sounds like listening to kids brag about their allowance while the adults don't want to talk about their salary, and try to draw wider conclusions from there.
Even the DoE posts top 500 results when they commission a supercomputer.
B200s have an incremental increase in FP64 and FP32 performance over H100s. That is the number format that HPC people care about.
The MI300A can get to 150% the FP64 peak performance that B200 devices can get, although AMD GPUs have historically underperformed their spec more than Nvidia GPUs. It's possible that B200 devices are actually behind for HPC.
Top line comparison numbers for reference: https://www.theregister.com/2024/03/18/nvidia_turns_up_the_a...
It does seem like Nvidia is prioritizing int8 / fp8 performance over FP64, which given the current state of the ML marketplace is a great idea.
MI300 also have decent performance in FP16 (~108 TFLOPS). Not as good as NVIDIA, but it's getting there. Anyone has experience using these on JAX? Support is said to be decent, but no idea if it's good enough for research-oriented tasks, i.e. stable enough for training and inference.
Google is running its own TPU hardware for internal workloads. I believe Nvidia is just resold for cloud customers.
Not true. Apple trained some models on their TPU
Apologies, to be clear what I meant was that to my knowledge Google doesn't use GPUs for it's own stuff, but does sell both TPUs and GPUs to others on Cloud.
Also, to be clear, I have no internal info about this, I'm going based on external stuff I've seen.
Nvidia GPUs are also used for inference on Google products. It just depends on availability.
Interesting, do you have a source for this? I've not been able to find one.
GCP plans offer access to high-end NVIDIA GPUs, as well as TPUs. I thought Google products use the same pool of resources that is also resold to customers?
Generally HPC compute has lower margins similar to consoles. It makes sense that AMD would fight for that contract more than NVIDIA similar to IBM stopped doing this. Its sort of comparing Apples to Raspberry Pis.
Hey now I compare Apples to Raspberry Pi's regularly :)
A cluster is not a super computer.
The whole point of a super computer is that it act as much as a single machine as it is possible while a cluster is a soup of nearly independent machines.
> soup of nearly independent machines
that does a serious disservice to hyperscaler clusters.
Sure but it's closer to the truth than saying they have similar or more raw compute than a super computer.
B200 is very much not rolling out because NVIDIA, after the respin, doesn't have the thermals under control (yet).
Your other points may be valid.
Source?
China has been absent from TOP500 for years as well.
Ya exactly - no one cares about top500 outside of academia (literally have never heard it come up at work). So this is like the gold star (participation award) of DCGPU competition.
After skimming the article, I'm confused -- where exactly is the headline being pulled from?
If you look at the table toward the bottom, no matter how you slice it, Nvidia has 50% of the total cores, 50% of the total flops, and 90% of the total systems among the Top 500, while AMD has 26% of the total cores, 27.5% of the total flops, and 7% of the total systems.
Is it a matter of newly-added compute?
> This time around, on the November 2024 Top500 rankings, AMD is the big winner in terms of adding capacity to the HPC base.
> AMD GPUs drove 72.1 percent of the new performance added for the November 2024 rankings
Yes, I saw that, but that doesn't justify the title as written. Had it said "AMD Now Has More New Compute" I wouldn't have said anything.
I'm sure there is also a lot not on the Top500. I've got enough AMD MI300x compute for about 140th position, but haven't submitted numbers.
layperson with no industry knowledge, but it seems like nvidia's CUDA moat will fall in the next 2-5 years. It seems impossible to sustain those margins without competition coming in and getting a decent slice of the pie
But how will AMD or anyone else push in? CUDA is actually a whole virtualization layer on top of the hardware and isn't easily replicable, Nvidia has been at it for 17 years.
You are right, eventually something's gotta give. The path for this next leg isn't yet apparent to me.
P.s. how much is an exaflop or petaflop, and how significant is it? The numbers thrown around in this article don't mean anything to me. Is this new cluster way more powerful than the last top?
The API part isn't thaaat hard. Indeed HIP already works pretty well at getting existing CUDA code to work unmodified on AMD HW. The bigger challenge is that the AMD and Nvidia architectures are so different that the optimization choices for what the kernels would look like are more different between Nvidia and AMD than they would be between Intel and AMD in CPU land even including SIMD.
CUDA is the assembly to Torch's high-level language; for most, it's a very good intermediary, but an intermediary nonetheless, as it is between the actual code they are interested in, and the hardware that runs it.
Most customers care about cost-effectiveness more than best-in-class raw-performance, a fact that AMD has ruthlessly exploited over the past 8 years. It helps that AMD products are occasionally both.
> P.s. how much is an exaflop or petaflop, and how significant is it? The numbers thrown around in this article don't mean anything to me. Is this new cluster way more powerful than the last top?
Nominally, a measurement in "flops" is how many (typically 32-bit) FLoating-point Operations Per Second the hardware is capable of performing, so it's an approximate measure of total available computing power.
A high-end consumer-grade CPU can achieve on the order of a few hundred gigaflops (let's say 250, just for a nice round number). https://boinc.bakerlab.org/rosetta/cpu_list.php
A petaflop is therefore about four thousand of those; multiply by another thousand to get an exaflop.
For another point of comparison, a high-end GPU might be on the order of 40-80 teraflops. https://www.tomshardware.com/reviews/gpu-hierarchy,4388-2.ht...
Anybody spending tens of billions annually on Nvidia hardware is going to be willing to spend millions to port their software away from CUDA.
To slower hardware? What are they supposed to port to, ASICs?
if the hardware is 30% slower and 2x cheaper, that's a pretty great deal.
Power density tends to be the limiting factor for this stuff, not money. If it's 30 percent slower per watt, it's useless.
The ratio between power usage and GPU cost is very, very different than with CPUs, though. If you could save e.g. 20-30% of the purchase price that might make it worth it.
e.g. you could run a H100 at 100% utilization 24/7 for 1 years at $0.4 per kWh (so assuming significant overhead for infrastructure etc.) and that would only cost ~10% of the purchase price of the GPU itself.
For the average non-FAANG company, there's nothing to port to yet. We don't all have the luxury of custom TPUs.
A high grade consumer gpu a (a 4090) is about 80 teraflops. So rounding up to 100, an exaflop is about 10,000 consumer grade cards worth of compute, and a petaflop is about 10.
Which doesn’t help with understanding how much more impressive these are than the last clusters, but does to me at least put the amount of compute these clusters have into focus.
You're off by three orders of magnitude.
My point of reference is that back in undergrad (~10-15 years ago), I recall a class assignment where we had to optimize matrix multiplication on a CPU; typical good parallel implementations achieved about 100-130 gigaflops (on a... Nehalem or Westmere Xeon, I think?).
You are 100% correct, I lost a full prefix of performance there. Edited my message.
Which does make the clusters a fair bit less impressive, but also a lot more sensibly sized.
4090 tensor performance (FP8): 660 teraflops, 1320 "with sparsity" (i.e. max theoretical with zeroes in the right places).
https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvid...
But at these levels of compute, the memory/interconnect bandwidth becomes the bottleneck.
According to Wikipedia the previous #1 was from 2022 with a peak petaflops of 2,055. This system is rated at 2,746. So about 33% faster than the old #1.
Also, of the top 10, AMD has 5 systems.
> P.s. how much is an exaflop or petaflop
1 petaflop = 10^15 flops = 1,000,000,000,000,000 flops.
1 exaflop = 10^18 flops = 1,000,000,000,000,000,000 flops.
Note that these are simply powers of 10, not powers of 2, which are used for storage for example.
Isn't porting software to the next generation supercomputer pretty standard for HPC?
Maybe the DOJ will come in and call it anti-trust shenanigans.
Not that I would want this...
its possible. Just look at Apples GPU, its mostly supported by torch, what's left are mostly edge-cases. Apple should make a datacenter GPU :D that would be insanely funny. It's actually somewhat well positioned as, due to the MacBooks, the support is already there. I assume here that most things translate to linux, as I don't think you can sell MacOS in the cloud :D
I know a lot developing on apples silicon and just pushing it to clusters for bigger runs. So why not run it on an apple GPU there?
> Apple should make a datacenter GPU
Aren't their GPUs pretty slow, though? Not even remotely close to Nvidia's consumer GPU with only (significant) upside being the much higher memory capacity.
> what's left are mostly edge-cases.
For everything that isn't machine learning, I frankly feel like it's the other way around. Apple's "solution" to these edge cases is telling people to write compute shaders that you could write in Vulkan or DirectX instead. What sets CUDA apart is an integration with a complex acceleration pipeline that Apple gave up trying to replicate years ago.
When cryptocurrency mining was king-for-a-day, everyone rushed out to buy Nvidia hardware because it supported accelerated crypto well from the start. The same thing happened with the AI and machine learning boom. Apple and AMD were both late to the party and wrongly assumed that NPU hardware would provide a comparable solution. Without a CUDA competitor, Apple would struggle more than AMD to find market fit.
well, but machine learning is the major reason we use GPUs in the datacenter (not talking about consumer GPUs here). The others are edge-cases for data-centre applications! Apple is uniquely positioned exactly because it is already solved due to a significant part of the ML-engineers using MacBooks to develop locally.
The code to run these things on apples GPUs exist and is used every day! I don't know anyone using AMD GPUs, but pretty often its nvidia on the cluster and Apple on the laptop. So if nvidia is making these juicy profits, i think apple could seriously think about moving to the cluster if it wants to.
People have been chipping away at this for a while. HIP allows source-level translation, and libraries like Jax provide a HIP version.
> But how will AMD or anyone else push in? CUDA is actually a whole virtualization layer on top of the hardware and isn't easily replicable, Nvidia has been at it for 17 years.
NVidia currently has 80-90% gross margins on their LLM GPUs, that’s all the incentive another company needs to invest money into a CUDA alternative.
At this scale cuda is quite useless.
You need to develop your own in house solution to distributing workloads.
The difference to regular clusters is that all the memory is globally visible, so machine 0023 can access and modify address 0x0123456789abcdef0123456789abcdef which happens to be on machine 0999.
We donated one of our MI300x systems to the SCALE team. The moat-less future is coming more quickly than you think.
CUDA moat is highly overrated for AI in the first place and sold as the reason for the failure of AMD. Almost no one in AI uses CUDA. They only use pytorch or Triton. TPUs didn't face lot of hurdle due to CUDA because they were initially better in terms of price to performance and supported pytorch, tensorflow and jax.
The reason why AMD is behind is that it is behind in hardware. MI300x is more pricey per hour in all the cloud I can find compared to H100, and the MFU is order of magnitude lower compared to NVIDIA for transformers, even though transformers are fully supported. And I get same 40-50% MFU in TPU for the same code. If anyone is investing >10 million dollar for hardware, they sure can invest a million dollar to rewrite everything in whatever language AMD asks them to if it is cheaper.
Why the focus on AMD and Nvidia? It really isn't that hard to design a large number of ALU blocks into some silicon IP block and make them work together efficiently.
The real accomplishment is fabricating them.
> It really isn't that hard to design a large number of ALU blocks into some silicon IP block and make them work together efficiently.
It really is that hard, and the fabrication side of the issue the easy part from Nvidia's perspective - you just pay TSMC a shitload of money. Nvidia's real victory (besides leading on performance-per-watt) is that their software stack doesn't suck. They invested in complex shader units and tensor accelerators that scale with the size of the card rather than being restrained in puny and limited NPUs. CUDA unified this featureset and was industry-entrenched for almost a decade, which gave it pretty much any feature you could want be it crypto acceleration or AI/ML primitives.
The ultimate tragedy is that there was a potential future where a Free and Open Source CUDA alternative existed. Apple wrote the OpenCL spec for exactly that purpose and gave it to Khronos, but later abandoned it to focus on... checks clipboard MLX and Metal Performance Shaders. Oh, what could have been if the industry weren't so stingy and shortsighted.
> It really is that hard
YES!! Thank you!
> Nvidia's real victory (besides leading on performance-per-watt) is that their software stack doesn't suck
YES! And it's not just CUDA and CUDA-adjacent tools, but also their cuDNN/cuBLAS/etc. libraries. They invest a massive amount of staffing into squeezingt the last drop of performance out of their hardware, identifying areas for improvement and feeding that back to the architects.
> Apple wrote the OpenCL spec for exactly that purpose and gave it to Khronos
Nitpick: Affie Munshi from Apple wrote down a draft and convinced his management to offer it to Khronos, where it was significantly modified over... was it a year or so?... by a number of representatives from a dozen companies or so. A ton of smart people contributed a ton of work into what became the 1.0 version.
And let me tell you that the discussions were often tense, both during the official meetings as well as what happened behind the scenes. The end result was as good as you can expect from a large committee composed of representatives from competing companies.
But, in summary, you get it, unlike so many commenters in HN.
> you just pay TSMC a shitload of money
I guess with money you can win any argument ...
Sure, Apple did the same thing with TSMC's 5nm node. They still lost in performance-per-watt in direct comparison with Nvidia GPUs using Samsung's 8nm node. Money isn't everything, even when you have so much of it that you can deny your competitors access to the tech you use.
Nvidia's lead is not only cemented by dense silicon. Their designs are extremely competitive, perhaps even a generational leap over what their competitors offer.
Let me phrase it differently.
If Nvidia pulls the plug we can still go to AMD and have a reasonable alternative.
If TSMC pulls the plug, however ...
Samsung's fabrication is about as good as TSMC. Or at least it was when I retired a few years ago.
But not the profits.
It does not matter. AMD is shit when it comes to low-level processing, their algos are stuck that go nowhere. Nvidia is killing it. There is a reason why Zookerberg ordered billions in GPUs from Nvidia and not from AMD.
AMD GPUs handle all inference for Llama3 at Meta btw.
This comment is somewhat more insightful:
Why is AMD shit at low-level processing? What does it mean "their algos are stuck"? Having watched "the industry" for a few decades now, the appeal for NV smells heavily like the old appeal for Xeons, and Big Blue before them. The moat appears (to me, an unknowledgeable outsider) to be just cultural, not necessarily technical.
This is just silly fanboyism, there are pros and cons to each.