Every GPU launch cycle, the headline number is always compute: petaflops, TOPS, some new precision format squeezing out another 2x. NVIDIA's Rubin platform and AMD's MI455X both just went through this dance in the last few weeks, and the FP4 numbers are genuinely eye-catching — 50 petaflops here, 40 petaflops there.

But if you've ever actually run inference on a large model, you know the number that decides whether your GPU feels fast or feels like it's wading through mud isn't TFLOPS. It's memory bandwidth. And 2026 is the year that becomes impossible to ignore, because HBM4 just changed the math.

Training is compute-heavy. Inference — especially the autoregressive decode phase, where a model generates one token at a time — is a different animal entirely. Every single token requires reading the model's full weight tensor from VRAM. The GPU's compute cores can sit mostly idle, waiting on memory, because the read has to finish before the next token can be produced.

This is why the jump from H100 (HBM3, 3.35 TB/s) to H200 (HBM3e, 4.8 TB/s) raised single-GPU throughput on a 70B-parameter model from roughly 24 tokens/sec to about 34 — with no change to the compute cores at all. Moving to B200's 8 TB/s pushed that further, to around 57 tokens/sec. None of that came from more FLOPS. It came from the GPU spending less time waiting.

That's the core idea worth internalising: for inference at low batch sizes, memory bandwidth is the throughput ceiling, not compute.

HBM4 isn't a clock-speed bump on the old design — it's a structural change. The interface width doubles from 1,024 bits to 2,048 bits per stack, standardised under JEDEC's JESD270-4 spec. Per-pin speed stays roughly where HBM3 left it (~6.4–11 Gbps depending on vendor tuning), so the bandwidth gain comes almost entirely from that wider bus and added channel parallelism, not faster signalling.

Depending on vendor and configuration, that works out to roughly 2–3.3 TB/s per stack; Micron's shipping HBM4 lands around 2.8 TB/s per stack at 12-high; SK Hynix reports similar doubling from the wider I/O. Stack multiple HBM4 modules around a GPU die the way Rubin and MI455X do, and you get to the 22–23 TB/s aggregate bandwidth figures both companies are now advertising.

One counterintuitive wrinkle worth knowing if you're speccing hardware: first-wave 12-high HBM4 parts don't automatically give you more capacity — many ship at the same 36GB-per-stack ceiling as HBM3e, trading the generational leap entirely into bandwidth and channel count rather than size. The 64GB-per-stack, 16-high parts come later. If someone's pitching you an HBM4 system on the promise of "more VRAM," check the stack height on the actual part number.

Doubling channels from 1,024 to 2,048 isn't just "more lanes on the highway" — it changes how many independent memory transactions the accelerator can issue at once. That disproportionately helps workloads with irregular, unpredictable access patterns, which is exactly what inference on mixture-of-experts and long-context models looks like. Training tends to access memory in large, predictable sweeps; agentic and MoE inference workloads jump around far more, and that's where extra parallel channels pay off hardest.

There's also a quieter win: HBM4's lower voltage rails cut power draw per bit moved. In a liquid-cooled rack where power density is already the limiting factor on how many accelerators you can pack in — a constraint the industry is talking about more than GPU supply itself these days — that efficiency gain is arguably as important as the raw bandwidth number.

Yes, for training, and for compute-bound batch inference where you're running large batch sizes and the GPU genuinely stays busy between memory reads. If your workload is throughput-at-scale batch processing, FLOPS numbers are still the right thing to compare.

But if you're evaluating hardware for serving models to real users — low-latency, low-batch, "how fast does the first token come back" territory — bandwidth is the number that predicts your actual user experience, and it's the number most spec sheets bury below the compute headline.

When you're comparing accelerators this year, don't stop at the FP4/FP8 petaflop figure at the top of the slide. Find the memory bandwidth line, check whether it's HBM3e or HBM4, and check the stack configuration if capacity matters to you. That's the number that tells you whether the GPU will actually feel fast for the workload you're planning to run on it — not the number that looks best in a press release.

Independent third-party benchmarks comparing HBM3e and HBM4 systems head-to-head are still thin on the ground as of mid-2026, since HBM4-based platforms only started shipping in volume this year. Treat vendor-published comparisons as directional for now, and re-check real-world numbers once Rubin and MI455X systems have been in the field a few months.