Opinion

Memory Costs Jump 500%: The Math Behind AI Infrastructure Just Broke

DRAM and NAND spot prices have quintupled in 12 months, pushing memory cost-performance back to 2007 levels and dismantling the 'cheaper compute every year' assumption that AI companies built their economics on.

Font size

Source: Latent Space, "Memory prices up 500% in 12 months"

500% Isn't Hype—It's Spot Price

When I first saw that number, my instinct was to check whether the source had misplaced a decimal.

It hadn't. According to Latent Space's spot-market data, DRAM and NAND flash have risen 500% over the past 12 months. Spot price—not futures expectations, not analyst forecasts, not vendor guidance. Actual transactions from last week.

The starker comparison is cost-per-performance. At this trajectory, memory price-performance by year-end will revert to 2007 levels. That was the year the first iPhone launched.

For over a decade, storage quietly got cheaper under Moore's law. Nobody expected it to reverse. Nobody treated it as a meaningful variable in TCO calculations.

That assumption is now wrong.

Why It's Exploding Now

Memory price spikes aren't new—they happen every few years. But this cycle hits differently. Three forces have collided.

First, HBM is eating capacity. HBM (High Bandwidth Memory) ships with AI accelerators. Every H100, H200, and B200 carries several stacks of HBM3e. Manufacturing HBM requires advanced packaging and complex die-stacking, consuming roughly 3x the wafer capacity of standard DRAM. The same fab line can't make HBM and DDR5 simultaneously. All three major suppliers (Samsung, SK Hynix, Micron) are shifting production from DDR5 to HBM, draining standard-DRAM supply and pushing prices up first.

Second, the AI capex cycle is in full swing. OpenAI, Anthropic, Meta, and Google are all expanding clusters. xAI's 100,000-GPU cluster is operational. OpenAI's Stargate project commits $1.8 trillion. Each GPU in these clusters needs 80GB or even 192GB of HBM. The demand is real and immediate.

Third, the three major suppliers are throttling output. Samsung, SK Hynix, and Micron have collectively held back capex for two years. Capacity expansion is deliberately constrained. The supply curve has been artificially flattened.

HBM3e and HBM4 capacity is locked in for Nvidia and a handful of tier-one buyers. Mid-tier customers are stuck in queue. This isn't a cyclical blip—it's a structural shortage.

The GPU-Stacking Economics Collapse

For the past two years, AI companies have run training costs through a simplified model:

Total Cost ≈ GPU Purchase + Power + Datacenter + Cooling

Memory was treated as a GPU accessory—bundled into hardware cost with a small weight.

That model no longer works. HBM's share of AI training hardware cost has risen from roughly 20% to nearly 40%. A single B200 spends over $10,000 on HBM alone. The BOM structure has fundamentally changed.

The bandwidth wall compounds the problem. GPU FLOPS double every generation, but HBM bandwidth can't keep pace. During inference, parameters must move from memory to compute units. When bandwidth is insufficient, GPUs idle. Nvidia admitted this at GTC 2025: on many inference workloads, B200 can't hit peak FLOPS because HBM bandwidth is the bottleneck.

Players like OpenAI and Anthropic have rebuilt their TCO spreadsheets multiple times over recent months. Anthropic CEO Dario Amodei previously shared a chart showing frontier-model training costs reaching the $1 billion range—and that was before this memory-price surge.

MoE architecture's real value is only now becoming clear: total parameters can scale massively, but each inference activates only a fraction, directly offsetting rising per-unit memory costs. This is the fundamental reason every major player bet on MoE from 2024 onward.

MoE and Sparsification as Lifelines

Here's the counterintuitive twist: in this memory cycle, MoE models actually benefit.

Traditional dense models have a fixed memory footprint—70B parameters means 70B parameters loaded every inference. DeepSeek-V3's 671B total parameters look intimidating, but the MoE architecture activates only 37B per pass—about 5.5%. Mixtral 8x7B follows the same logic with 12.5% activation.

The result: the same hardware serves more users, or the same user count requires less than half the hardware investment. DeepSeek disclosed V3 training cost at $5.58 million, compared to an estimated ~$100 million for Llama 3.1 405B. Comparable capability, two orders of magnitude apart in cost.

The core logic:

Dense model: hardware cost ∝ total parameters
MoE model:    hardware cost ∝ activated parameters, weakly correlated with total parameters

The more expensive memory gets, the larger MoE's relative advantage. This is why Nvidia itself now pushes sparsification approaches like Minitron—even hardware vendors are using software to hedge hardware constraints.

Who Gets Squeezed Out

The pain isn't limited to giants.

  • Mid-size model companies: Those training their own models and selling API access. Hardware cost has shifted from a manageable line item to the dominant expense. Companies whose funding cycles can't keep pace with price hikes will either raise prices or shut down.
  • Startups building their own clusters: They assumed securing GPUs solved everything, not realizing GPUs are now the least constrained part of the stack.
  • Thin-margin inference applications: Bots, Agents, vertical SaaS—all built on low unit price and high concurrency. Memory inflation eats margins directly. A chat session that ran at 30% margin might now run below 10%.

Players at Anthropic's scale can absorb costs by raising prices. Products like Cursor can pass costs to developers. Mid-tier players lack that pricing power.

What to Watch Over the Next 12 Months

Three variables will determine whether this cycle lasts 6 months or 3 years.

HBM4 capacity. SK Hynix and Samsung are racing for HBM4 mass-production leadership, with Micron close behind. HBM4 boosts per-stack capacity from 24GB to 36–48GB and doubles bandwidth. If ramp goes smoothly, HBM cost-per-GB drops by late 2025—relief, but not reversal.

Chinese DRAM expansion. CXMT is scaling DDR5 capacity to 60,000 wafers/month in 2025. YMTC is adding 3D NAND lines. If they can deliver stable supply, they become a real variable on the global supply side. The catch: their advanced-node yield and yield stability still trail the big three by 1–2 generations.

ASIC substitution pace. Google TPU, AWS Trainium, and Microsoft Maia use custom memory subsystems decoupled from the commodity HBM market. If the ASIC path proves out, Nvidia's pricing power on general-purpose GPUs weakens—indirectly softening HBM pricing leverage.

My read: the tail of this memory cycle runs at least through Q4 2025. If all three conditions hit—smooth HBM4 ramp, Chinese capacity coming online, and ASICs scaling—relief could peak by mid-2026. Miss any two, and it drags into 2027.

AI companies spent two years running on the implicit assumption that compute would keep getting cheaper. That assumption is gone.

From here on, memory is the biggest line item in the TCO spreadsheet.

Source: https://www.latent.space/p/ainews-memory-prices-up-500-in-12