A Hacker News post titled I trained a small transformer in 1.5hrs and it beats many LLMs hit 333 points and drew 89 comments. The author, mvakde, didn't rent a GPU cluster or burn through months of compute. He wrote a small transformer targeting ARC-AGI 1, trained it in 1.5 hours, and posted a score that beats most LLMs on the market.
The result is worth unpacking.
What Happened
mvakde's target was ARC-AGI 1, an abstract reasoning benchmark. His setup:
- A small transformer implemented from scratch (parameter count not disclosed at an extreme scale)
- Training data: ARC-AGI 1's official training set plus self-generated augmentations
- Training time: 1.5 hours on a consumer GPU
- Evaluation: ARC-AGI 1's private test set
The result: beats most mainstream LLMs on ARC-AGI 1. The qualifier matters — "most," not "all." Top closed-source models like GPT-4 and Claude still hold an edge on certain subtasks.
Source: mvakde's original blog post.
What ARC-AGI 1 Actually Is
ARC stands for Abstraction and Reasoning Corpus, introduced by François Chollet in 2019. Chollet, the creator of Keras, is also the leading advocate of the hypothesis that "intelligence = program efficiency."
Tasks look like this: given 3–5 input-output grid pairs, infer the transformation rule and apply it to a test grid. Example:
- Input: a 3×3 grid with a red dot in the center
- Output: a 5×5 grid with a symmetric pattern radiating out from the red dot
- Test grid: another 3×3 with a red dot — produce the 5×5 expansion
Humans grasp it instantly. LLMs have long struggled here. The reasons:
- The task demands compositional generalization, not linguistic priors
- Solutions are short, enumerable programs — not statistical pattern matches
- Test-time patterns are almost always novel relative to the training distribution
Chollet once put up a $1 million bet that no one would crack ARC-AGI through pure scaling. By late 2024, the o3 family had crept close — but only by burning through massive synthetic datasets and inference compute.
What 1.5 Hours Means
Putting the number in perspective:
| Model | Training cost (estimate) | Training duration |
|---|---|---|
| GPT-4 class | Tens to hundreds of millions USD | Months |
| Llama 3 405B | ~$30M + 10k-GPU cluster | Weeks |
| DeepSeek V3 | ~$5–6M | Weeks |
| mvakde's small transformer | Few dollars in electricity | 1.5 hours |
Six to seven orders of magnitude apart.
The easy misread is "small models just beat big models." That's not quite right. A more accurate framing: on highly structured, narrow tasks, a purpose-built model with good data and the right inductive biases is far more cost-effective than a general-purpose LLM.
Why a Small Model Wins
Three design choices make the difference:
1. Program-Synthesis Inductive Bias
ARC solutions can be expressed as short symbolic programs (a DSL). mvakde's model doesn't output grids directly — it outputs steps like "grab the center point → mirror it → color the result." That's effectively baking an "I know this is a program" assumption into the architecture.
LLMs don't have this bias. They translate the task into a token sequence and predict the next cell — a fundamentally harder problem.
2. The Augmentation Trick
The ARC training set contains only a few hundred tasks — not enough to train anything. mvakde generated hundreds of thousands of synthetic tasks through geometric transformations (rotation, reflection, color remapping, shape recombination). The key detail: the augmentations preserve solution structure. If you rotate a shape, the solution rotates with it — preserving the dataset's underlying symmetries.
3. Training Signal
The supervision is program-level (are the steps correct?) rather than pixel-level (does the final grid match?). Intermediate supervision has dramatically better sample efficiency than end-to-end supervision.
The same flavor of approach shows up in AlphaCode and FunSearch, but ARC's task space is smaller and more structured, so a small model can cover it.
What It Means
This isn't an isolated result. Put it alongside:
- AlphaProof / AlphaGeometry: DeepMind's neuro-symbolic hybrid for math olympiad problems — specialized models outperform general LLMs
- FunSearch: DeepMind's small-model-plus-evaluator loop for math discovery outperforms GPT-4 generating answers directly
- Cursor's Tab model: a small model trained specifically for code completion delivers a better experience than general-purpose models on that task
The trend is clear: "scale is all you need" has never held for structured reasoning tasks — scaling-law success just papered over that for a while.
A more grounded take:
- General LLMs remain the most valuable Swiss Army knife: chat, writing, agents
- But for any vertical, strongly structured, enumerable task, a purpose-built small model with modest compute may be enough
- Teams building AI products should first ask whether their task is narrow and structured — if yes, try the small-model route first
Editorial Note
Why does mvakde's experiment deserve its own writeup? Because it puts a concrete number behind a debate that's been rehashed without clean benchmarks — 1.5 hours, a few dollars of electricity, beating most LLMs.
This differs from past "small model approaches GPT-4 on task X" stories. Those usually cherry-picked benchmarks or picked inherently fuzzy tasks. ARC-AGI 1 is a recognized hard problem — Chollet's own litmus test for general reasoning. mvakde didn't take shortcuts; he ran it straight and let the score speak.
A contrarian observation: the industry has systematically underestimated what "training cost" means. Media loves reporting "Team X trained Model Z for $Y," but that's just pretraining cost. What actually determines whether a model is usable is total lifecycle cost — data cleaning, iteration, evaluation, alignment, and inference. Once inference is included, the specialized-model advantage grows: a small transformer's per-query electricity cost can be 1/1000 of a GPT-4 API call.
A question to leave readers with: for the AI tasks you're actually working on, is "general LLM + aggressive prompting" genuinely more cost-effective — or would "training a small specialized model" come out ahead? Most teams have never seriously run the numbers.
Reference: mvakde's original blog post