Opinion

A 1.5-Hour Transformer Beats Most LLMs

A developer trained a small transformer in 90 minutes on a consumer GPU and outperformed most LLMs on ARC-AGI 1. The result challenges the 'scale is all you need' narrative for structured reasoning tasks.

Font size

A Hacker News post titled I trained a small transformer in 1.5hrs and it beats many LLMs hit 333 points and drew 89 comments. The author, mvakde, didn't rent a GPU cluster or burn through months of compute. He wrote a small transformer targeting ARC-AGI 1, trained it in 1.5 hours, and posted a score that beats most LLMs on the market.

The result is worth unpacking.

What Happened

mvakde's target was ARC-AGI 1, an abstract reasoning benchmark. His setup:

  • A small transformer implemented from scratch (parameter count not disclosed at an extreme scale)
  • Training data: ARC-AGI 1's official training set plus self-generated augmentations
  • Training time: 1.5 hours on a consumer GPU
  • Evaluation: ARC-AGI 1's private test set

The result: beats most mainstream LLMs on ARC-AGI 1. The qualifier matters — "most," not "all." Top closed-source models like GPT-4 and Claude still hold an edge on certain subtasks.

Source: mvakde's original blog post.

What ARC-AGI 1 Actually Is

ARC stands for Abstraction and Reasoning Corpus, introduced by François Chollet in 2019. Chollet, the creator of Keras, is also the leading advocate of the hypothesis that "intelligence = program efficiency."

Tasks look like this: given 3–5 input-output grid pairs, infer the transformation rule and apply it to a test grid. Example:

  • Input: a 3×3 grid with a red dot in the center
  • Output: a 5×5 grid with a symmetric pattern radiating out from the red dot
  • Test grid: another 3×3 with a red dot — produce the 5×5 expansion

Humans grasp it instantly. LLMs have long struggled here. The reasons:

  • The task demands compositional generalization, not linguistic priors
  • Solutions are short, enumerable programs — not statistical pattern matches
  • Test-time patterns are almost always novel relative to the training distribution

Chollet once put up a $1 million bet that no one would crack ARC-AGI through pure scaling. By late 2024, the o3 family had crept close — but only by burning through massive synthetic datasets and inference compute.

What 1.5 Hours Means

Putting the number in perspective:

ModelTraining cost (estimate)Training duration
GPT-4 classTens to hundreds of millions USDMonths
Llama 3 405B~$30M + 10k-GPU clusterWeeks
DeepSeek V3~$5–6MWeeks
mvakde's small transformerFew dollars in electricity1.5 hours

Six to seven orders of magnitude apart.

The easy misread is "small models just beat big models." That's not quite right. A more accurate framing: on highly structured, narrow tasks, a purpose-built model with good data and the right inductive biases is far more cost-effective than a general-purpose LLM.

Why a Small Model Wins

Three design choices make the difference:

1. Program-Synthesis Inductive Bias

ARC solutions can be expressed as short symbolic programs (a DSL). mvakde's model doesn't output grids directly — it outputs steps like "grab the center point → mirror it → color the result." That's effectively baking an "I know this is a program" assumption into the architecture.

LLMs don't have this bias. They translate the task into a token sequence and predict the next cell — a fundamentally harder problem.

2. The Augmentation Trick

The ARC training set contains only a few hundred tasks — not enough to train anything. mvakde generated hundreds of thousands of synthetic tasks through geometric transformations (rotation, reflection, color remapping, shape recombination). The key detail: the augmentations preserve solution structure. If you rotate a shape, the solution rotates with it — preserving the dataset's underlying symmetries.

3. Training Signal

The supervision is program-level (are the steps correct?) rather than pixel-level (does the final grid match?). Intermediate supervision has dramatically better sample efficiency than end-to-end supervision.

The same flavor of approach shows up in AlphaCode and FunSearch, but ARC's task space is smaller and more structured, so a small model can cover it.

What It Means

This isn't an isolated result. Put it alongside:

  • AlphaProof / AlphaGeometry: DeepMind's neuro-symbolic hybrid for math olympiad problems — specialized models outperform general LLMs
  • FunSearch: DeepMind's small-model-plus-evaluator loop for math discovery outperforms GPT-4 generating answers directly
  • Cursor's Tab model: a small model trained specifically for code completion delivers a better experience than general-purpose models on that task

The trend is clear: "scale is all you need" has never held for structured reasoning tasks — scaling-law success just papered over that for a while.

A more grounded take:

  • General LLMs remain the most valuable Swiss Army knife: chat, writing, agents
  • But for any vertical, strongly structured, enumerable task, a purpose-built small model with modest compute may be enough
  • Teams building AI products should first ask whether their task is narrow and structured — if yes, try the small-model route first

Editorial Note

Why does mvakde's experiment deserve its own writeup? Because it puts a concrete number behind a debate that's been rehashed without clean benchmarks — 1.5 hours, a few dollars of electricity, beating most LLMs.

This differs from past "small model approaches GPT-4 on task X" stories. Those usually cherry-picked benchmarks or picked inherently fuzzy tasks. ARC-AGI 1 is a recognized hard problem — Chollet's own litmus test for general reasoning. mvakde didn't take shortcuts; he ran it straight and let the score speak.

A contrarian observation: the industry has systematically underestimated what "training cost" means. Media loves reporting "Team X trained Model Z for $Y," but that's just pretraining cost. What actually determines whether a model is usable is total lifecycle cost — data cleaning, iteration, evaluation, alignment, and inference. Once inference is included, the specialized-model advantage grows: a small transformer's per-query electricity cost can be 1/1000 of a GPT-4 API call.

A question to leave readers with: for the AI tasks you're actually working on, is "general LLM + aggressive prompting" genuinely more cost-effective — or would "training a small specialized model" come out ahead? Most teams have never seriously run the numbers.

Reference: mvakde's original blog post

Source: https://mvakde.github.io/blog/44-on-arc-1/