Case Studies

Training a 3.8B Model for Under $1,000: A Look at little-lm-3-8b

Hugo Vergnes trained a 3.8B-parameter LLM from scratch for $998, reaching a CORE score of 0.384. The real takeaway isn't the low cost but what it reveals about how far small-budget training can actually go.

Font size

Hugo Vergnes recently published an experiment on his personal blog: training a 3.8B-parameter LLM from scratch for $998, hitting a CORE score of 0.384. It made the Hacker News front page with a score of 74 and 13 comments (HN thread), making it one of the more-discussed small-scale training experiments.

What caught my attention isn't the "thousand-dollar model" headline but the way it puts hard numbers on how capable a small model can be when budget is tight. Here's the breakdown.

Where the $998 went

Vergnes published a fairly detailed cost breakdown. Here's the cleaned-up version:

ItemCost
GPU rental (cloud spot instances, ~2 days)~$900
Data preparation / cleaning~$50
Experimental runs + failed retries~$48
Total$998

GPU spend dominates at 90%. Vergnes didn't name the hardware, but working backward from the price and runtime, it's almost certainly H100 80GB or A100 80GB spot instances running for roughly two days. Spot pricing is volatile, so getting this under $900 required some luck.

The training config he shared:

# From the author's blog
model:
  params: 3.8B
  layers: 32
  hidden_size: 2560
  heads: 32
  seq_len: 4096

training:
  tokens: 100B
  batch_size: 1024
  lr: 3e-4
  optimizer: AdamW
  schedule: cosine
  precision: bf16

100B tokens on 3.8B parameters is actually slightly above the Chinchilla optimal ratio (~20x parameters), so the model isn't undertrained by that standard, though it's still small in absolute terms compared to trillion-token runs. That choice matters; see the "why this experiment is worth dissecting" section below.

What 0.384 CORE actually means

CORE (Compositional Reasoning Evaluation) benchmarks reasoning and compositional ability. Here's how the number stacks up against other 3–4B models:

ModelParametersCORE Score
little-lm-3-8b3.8B0.384
Phi-3.5-mini3.8B~0.50
Qwen2.5-3B3.0B~0.46
Llama-3.2-3B3.2B~0.42
SmolLM2-1.7B1.7B~0.38

(CORE scores for the mainstream models are approximate, aggregated from multiple sources rather than a single paper.)

So 0.384 is roughly in line with SmolLM2-1.7B and about 25% below Phi-3.5-mini. Vergnes doesn't shy away from this. From his blog:

"The model is competitive with similarly sized models trained on similar budgets, but it doesn't reach the performance of models trained on trillion-token datasets."

The verdict: not SOTA, not a failure, just a reasonable baseline. Within a $998 budget, 0.384 is probably close to the practical ceiling.

Open source code, closed weights: what that actually means

Vergnes published the training scripts and config publicly; the repo lives at hugovergnes/little-lm (see the blog for the actual repo link).

But the weights are not released. That's the key detail.

  • ✅ Training code (nanoGPT-style, modified)
  • ✅ Config files, data-processing scripts
  • ✅ Training logs and loss curves
  • ❌ Model weights
  • ❌ Full training dataset (only a deduplicated sample)

Without weights, nobody can use the model directly; only reproduce it or continue pre-training from the code. Vergnes explained in the HN thread that handling hosting and licensing for weight files is more work than he can take on alone.

Community response: HN commenters suggested uploading to HuggingFace, using GitHub LFS, or even crowdfunding hosting costs. As of his last update in early August, no weights have been released.

Why this experiment is worth dissecting

$998 isn't the destination; it's the floor.

If Vergnes's run is the starting point, the real question is: at the same budget, how much headroom is left from changing architecture, data, or training strategy? A few angles:

1. Marginal returns from data quality

Vergnes used deduplicated web data (similar to FineWeb's pipeline). Swapping in higher-quality synthetic data plus curated web data could plausibly push CORE to 0.42–0.45 on the same 100B tokens. The community has already validated this pattern over the past year.

2. The token-count sweet spot

He trained on 100B tokens, already slightly past Chinchilla-optimal for 3.8B. Returns diminish past this point; a 3.8B model saturates around 200B tokens. But doubling the budget to $2000 for another 100B tokens might still push CORE up to ~0.40.

3. Architecture choice

Standard dense transformer. Switching to MoE (e.g., 8B total / 2B active parameters) would likely gain 10–20% at the same FLOPs, though engineering complexity goes up substantially.

4. Multi-stage training + SFT

$998 covers pre-training only. Adding even $200 for supervised fine-tuning (ShareGPT-style data) would meaningfully improve downstream usefulness, though that's outside Vergnes's original experiment.

Practical guidance for anyone considering a self-training run:

  • Budget < $1,000: 1–3B model + 50–100B tokens → 0.35–0.40 CORE
  • Budget $1,000–$5,000: 3–7B model + 200B tokens + basic SFT → 0.45–0.50 CORE
  • Budget > $10,000: start competing seriously with "semi-industrial" models like Phi-3.5-mini

Vergnes's run confirms the first tier is real and achievable.

Why this matters

The real signal here isn't "cheap training." It's that the marginal cost of pre-training is collapsing.

A year ago, $998 would have bought a single ablation run, a few hyperparameter tweaks and a loss curve. Today it buys a full 3.8B model trained end-to-end. A CORE score of 0.384 doesn't sound impressive, but it represents a complete training pipeline, not a toy benchmark.

For small teams and independent researchers, this means pre-training is no longer exclusive to large labs. The moat behind "small" models like Phi-3, Qwen2.5, and Llama-3.2 comes down to data quality and engineering, not raw compute. A patient small team with a $5k–$10k budget, public datasets, and an open-source training framework can realistically build something approaching Phi-3.5-mini quality within six months.

That said, a few caveats: this experiment has no released weights, which suggests Vergnes himself doesn't view it as production-ready. For downstream tasks, most people would still reach for Phi-3 or Qwen. In other words, pre-training costs have collapsed, but post-training (instruction tuning, RLHF, DPO) hasn't; that's where the real compute bill still lives.

One uncertainty: does the $998 figure include Vergnes's time? If you account for his labor (probably 2–4 weeks), the true cost is closer to $3,000–$5,000. Even so, those numbers were unthinkable in 2023.

My prediction: 2026 will see more 1–7B pre-trained models from individuals and small teams. They won't replace Qwen or Llama, but they'll become viable foundations for vertical domains (legal, medical, code). Vergnes's experiment is one early point on that curve.


References:

Source: https://hugovergnes.github.io/little-lm-3-8b/