Case Studies

Can Decentralized Compute Beat Big Clusters? Prime Intellect's NanoGPT Speedrun

Prime Intellect hit the Hacker News front page with a 3:33 NanoGPT Speedrun result. The real question: can a handful of scattered consumer-grade GPUs actually outperform CoreWeave or Modal?

Font size

127 points, 31 comments. A technical blog post holding that spot on Hacker News means it either struck a nerve or the data was genuinely interesting. Prime Intellect's NanoGPT Speedrun hit both notes: whether decentralized training can beat a single large cluster, an argument that hasn't cooled since INTELLECT-2 dropped last year, finally got a small-scale live-fire test.

What NanoGPT Speedrun Actually Is

Andrej Karpathy's NanoGPT from 2022 is one of the most underrated toys in the LLM world. Around 300 lines of PyTorch, training a GPT-2-scale model from an empty checkpoint until it generates halfway-decent text. The community has been running training-speed competitions on it for over three years, with a fixed objective, a fixed token budget, and a single question: who can compress wall-clock time the most?

The format is uncompromising:

  • Fixed dataset (FineWeb), fixed step count, fixed val loss threshold
  • The only variables are code and hardware scheduling
  • Anyone who can get it running can submit a result

The value isn't about producing a good model. NanoGPT is too small for that; once GPT-2 124M val loss drops below 3.28, the marginal gains collapse fast. What it actually does is separate optimization skill from raw compute scale. You can't solve it by buying more GPUs; you have to rewrite code, rework scheduling, or tune kernels. The people who've held records over the years (Kevin/Franke, @Grad62304977, and others) didn't change the model — they changed attention implementations, activation checkpointing strategies, and parameter initialization schemes.

The current record line is 3 minutes 28 seconds, set by Keller Jordan with support from the unsloth team. @_c_b_std_ previously hit something close on Modal. In other words, this leaderboard represents the ceiling for single-cluster engineering optimization.

Prime Intellect's Run

Prime Intellect's result was 3 minutes 33 seconds, just shy of the record line.

A few numbers from their blog post:

  • 8x H100 (not 8 nodes, but 8 individual cards spread across different geographic regions)
  • Total training cost roughly $560
  • Full GPT-2 training pipeline completed end-to-end in 3:33
  • Val loss hit the NanoGPT Speedrun standard threshold

For comparison: the CoreWeave record used a considerably larger cluster (exact numbers weren't fully disclosed in the community, but density-wise clearly exceeded 8 cards). Prime Intellect reached within 5 seconds of the record using roughly 1/10 to 1/20 of the total compute.

What's that gap worth? At market cloud GPU rates, 8x H100 for three and a half minutes lands around $500–700. Prime Intellect pushed it to $560, essentially publishing the invoice.

The technical core is their PRIME-RL and SYNTHETIC-1-style co-processing architecture. Communication across the 8 H100s runs through a custom bandwidth-aware scheduler; the goal isn't to cram all cards into the same network fabric, but to coordinate cards that live in different data centers. This shares the same underlying codebase with INTELLECT-2 (their decentralized RL training project).

The Continuation from INTELLECT-2

INTELLECT-2 was Prime Intellect's project from the second half of last year, aiming to let people worldwide lend their idle GPUs to a large model training run and get paid proportionally. It went the RL route (inspired by DeepSeek-R1), producing the 32B model INTELLECT-2.

This NanoGPT run is a validation of that same decentralized infrastructure applied to base pretraining. Three layers got reused:

  1. Asynchronous weight sync — each local card doesn't have to wait for everyone else before syncing
  2. Bandwidth-aware task allocation — cards with worse network conditions get more local compute to compensate
  3. Fault-tolerant scheduling — a card dropping out won't deadlock the whole training run

Moving this stack from RL to pretraining isn't a straight engineering copy, since the optimization targets differ (RL prioritizes rollout throughput; pretraining cares about gradient synchronization). The fact that it works suggests the underlying abstractions are reasonably clean.

What the HN Comments Are Arguing About

The 31 comments split roughly into three camps:

Camp 1: "This isn't real decentralization."
The most common objection. Commenters like @tlsigator pointed out that coordinating 8x H100 across 8 cloud regions is fundamentally different from "gamers pooling their RTX 4090s worldwide." The former is still a big company's private infrastructure deployed across multiple sites; the latter is what INTELLECT-2 originally sold.

Camp 2: "3:33 didn't break the record."
The more technical critique. Voices like @adamjstevenson argued that on this leaderboard you have to win for it to count; being 5 seconds back is "almost," not a real surpass. CoreWeave/Modal's accumulated code optimization sits there, and as long as the decentralized overhead remains unresolved, it'll always be a notch behind.

Camp 3: "Cost is the real story."
Folks in the @metasim thread did the math: if you spent that same $560 on CoreWeave, the wall-clock would probably get compressed to under a minute by sheer cluster size. In other words, you lose on single-job latency but win on per-dollar output. That trade-off matters for small teams and academia; it doesn't matter for top-tier labs.

There's another question I didn't see raised on HN but worth flagging: reproducibility. The NanoGPT Speedrun leaderboard has historically required open-source training scripts and precise environment descriptions. Prime Intellect's blog post links to a code repo, but some commenters pointed out that reproducing 3:33 from the public code still requires a few key parameters that weren't disclosed (specifically, the optimizer's warmup schedule and gradient clipping threshold). If the authors don't fill that in, the record's validity takes a hit.

Editorial Take

What's genuinely worth paying attention to here isn't the five digits of 3:33 — it's what this reveals about the boundary conditions of decentralized training.

Over the past year, the decentralized training narrative has consistently been "we can organize scattered compute to compete with mega-clusters." INTELLECT-2 proved this works for RL, because rollouts are naturally parallel and not very sync-sensitive. This NanoGPT run tells us: once you hit tasks that demand high sync precision (pretraining, especially optimizations requiring large-batch sync), the network overhead of decentralization becomes a hard ceiling. 3:33 vs 3:28, a 5-second gap — that 5 seconds is almost certainly inter-node network round-trip latency, not algorithmic.

My prediction for the next 6 months: two kinds of follow-ups.

  1. The "semi-decentralized" approach, like Prime Intellect's here: keep core synchronization inside a well-connected network and let only a few heterogeneous nodes participate. A compromise between performance and decentralization.
  2. Someone will prove that fully decentralized pretraining can never beat a single cluster, and pivot toward hybrid architectures like "decentralized data labeling + centralized training."

Open questions:

  • Prime Intellect didn't publish full environment details; the reproduction bar is higher than the leaderboard's historical standard. Strictly speaking, this result doesn't qualify as a record — it's a capability demonstration.
  • INTELLECT-2's actual economics haven't been externally audited. How much decentralized contributors actually get paid, how equipment wear-and-tear is accounted for, whether it's net profitable or net loss-making — none of that has public data.
  • The "decentralized" framing of 8x H100 is a bit watered down. Renting cards from 8 different AWS regions is a very different thing from letting players contribute RTX 4090s from their homes.

The community reaction won't be "decentralization won" — but it will make the question of "what can decentralization win, and what can't it" quantifiable. That's more valuable than the record itself.


Source: Prime Intellect NanoGPT Speedrun

Source: https://www.primeintellect.ai/research/nanogpt-speedrun