Case Studies

LLMs-from-scratch: Building a GPT From the Ground Up

Sebastian Raschka's open-source repo walks through every layer of a GPT in pure PyTorch—from token embedding to SFT and DPO—making it one of the most thorough LLM teaching projects available.

Font size

GitHub has no shortage of LLM repositories, but few go as deep as rasbt/LLMs-from-scratch. Authored by Sebastian Raschka, a research scientist at Lightning AI and author of Machine Learning with PyTorch and Scikit-Learn, the repo accompanies his newer Manning book Build a Large Language Model (From Scratch) (ISBN 9781633437166) and covers the full GPT pipeline—from forward pass to DPO.

What Problem It Solves

Anyone trying to understand how LLMs actually work hits the same walls:

  • HuggingFace tutorials lean on AutoModelForCausalLM.from_pretrained—one line and you're done, but you have no idea what's happening inside the matrices.
  • Paper-summary repos copy the attention formula but never show how it lands in code.

This repo takes a more direct route. Roughly 400 lines of PyTorch take you from token embedding through multi-head attention, LayerNorm, pretraining, and SFT. Everything lives in Jupyter notebooks, every nn.Module is broken out, and no abstraction layer is hidden.

After working through it, you should be able to explain: (1) why d_out % num_heads == 0 must hold; (2) how a causal mask is implemented as an upper-triangular matrix; (3) what separates the SFT loss from the pretraining loss; and (4) how DPO extracts an implicit reward from a reference model's log-probabilities.

How to Use It

Clone the repo and read the Jupyter notebooks in order:

  • ch02_text_data.ipynb — tokenizer + sliding-window dataset
  • ch03_coding_attention_mechanisms.ipynb — four self-attention variants
  • ch04_implementing_gpt_model.ipynb — TransformerBlock + GPTModel
  • ch05_pretraining_on_unlabeled_data.ipynb — pretraining
  • ch06_finetuning_for_classification.ipynb — classification fine-tuning
  • ch07_instruction_finetuning.ipynb — SFT
  • ch08_aligning_models_with_human_feedback.ipynb — DPO/PPO

Each chapter ends with exercises; solutions live in exercise-solutions.ipynb. The README recommends uv for environment setup.

Fastest path to a working demo: open Colab on a free GPU, run ch04_implementing_gpt_model.ipynb, and train a 124M-parameter GPT-2 skeleton. A few hours gets you a model that can complete text. A full pass through every chapter takes roughly a week.

How It's Structured

The codebase is organized in four layers, built bottom-up:

  1. ch02 — Data layer: tiktoken handles BPE tokenization; a sliding window chops long texts into context_length-sized training samples.
  2. ch03 — Attention layer: progresses from naive self-attention to causal-masked attention to multi-head attention, finally swapping in PyTorch's built-in torch.nn.functional.scaled_dot_product_attention. Each version runs on the same toy input so you can compare outputs side by side.
  3. ch04 — Model layer: implements TransformerBlock (LayerNorm + MHA + FFN + residual connections) and GPTModel. Configurations match GPT-2 124M exactly—d_model=768, n_heads=12, n_layers=12—so you can load OpenAI's official weights directly in ch05.
  4. ch07 + ch08 — Alignment layer: SFT uses instruction-response data; DPO derives an implicit reward from the log-ratio difference between a reference model and a policy model.

Key design choice: every matrix dimension lives in a single GPT_CONFIG_124M dict. If a shape is wrong, the code simply won't run, which makes it easy to cross-check against Table 2 in the GPT-2 paper.

Here's the core of MultiHeadAttention from ch04_implementing_gpt_model.ipynb—under 30 lines that handle Q/K/V projection, scaled dot-product attention, the causal mask, and dropout:

import torch
import torch.nn as nn

class MultiHeadAttention(nn.Module):
    def __init__(self, d_in, d_out, context_length, dropout, num_heads, qkv_bias=False):
        super().__init__()
        assert d_out % num_heads == 0, "d_out must be divisible by num_heads"
        self.d_out = d_out
        self.num_heads = num_heads
        self.head_dim = d_out // num_heads
        self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias)
        self.W_key   = nn.Linear(d_in, d_out, bias=qkv_bias)
        self.W_value = nn.Linear(d_in, d_out, bias=qkv_bias)
        self.out_proj = nn.Linear(d_out, d_out)
        self.dropout = nn.Dropout(dropout)
        self.register_buffer(
            "mask", torch.triu(torch.ones(context_length, context_length), diagonal=1)
        )

    def forward(self, x):
        b, num_tokens, d_in = x.shape
        keys = self.W_key(x).view(b, num_tokens, self.num_heads, self.head_dim).transpose(1, 2)
        queries = self.W_query(x).view(b, num_tokens, self.num_heads, self.head_dim).transpose(1, 2)
        values = self.W_value(x).view(b, num_tokens, self.num_heads, self.head_dim).transpose(1, 2)
        attn_scores = queries @ keys.transpose(2, 3)
        attn_scores.masked_fill_(self.mask.bool()[:num_tokens, :num_tokens], -torch.inf)
        attn_weights = torch.softmax(attn_scores / self.head_dim**0.5, dim=-1)
        attn_weights = self.dropout(attn_weights)
        context_vec = (attn_weights @ values).transpose(1, 2).contiguous().view(b, num_tokens, self.d_out)
        return self.out_proj(context_vec)

Before running this cell, prepare an input tensor:

x = torch.randn(2, 6, 768)  # batch=2, tokens=6, d_model=768
mha = MultiHeadAttention(768, 768, context_length=1024, dropout=0.1, num_heads=12)
mha(x).shape
# torch.Size([2, 6, 768])

If the shape matches, your tensor plumbing is correct. Reshaping Q/K/V from (b, tokens, d_out) to (b, tokens, num_heads, head_dim) and then transposing along dimensions 1 and 2 is the key move that lets multiple heads compute in parallel. Staring at the diagram in the paper ten times doesn't compare to running this cell once.

Strengths and Weaknesses

What works well:

  1. Zero hidden abstractions—no nn.TransformerEncoder, no AutoModel. Every layer is implemented by hand.
  2. The companion Manning book is unusually rigorous for an open-source repo: standardized chapter endings with "Summary" and "Exercises" sections in proper academic style.
  3. Recent additions cover Qwen3 and DeepSeek weight loading, keeping pace with the last two years of model releases. LoRA and KV-cache implementations are included too.

Where it falls short:

  1. Notebook-first format means you can't pip install it as a library—there's no from llms_from_scratch import GPTModel. You have to copy the code into your own project.
  2. A 124M model is small by modern standards. It won't produce ChatGPT-quality output; the project is explicitly pedagogical, not a path to production-scale training.
  3. The RLHF/PPO chapter (ch08) is noticeably shallower than the earlier material. DPO gets working code, but the theoretical derivation is thin. For serious alignment work, supplement with the HuggingFace TRL docs or the original Christiano et al. paper.

Hands-On Notes

After cloning, I opened gpt_train.ipynb from ch04 and trained the 124M model for 5k steps on a Colab T4. Loss dropped from around 11 to about 3.5. The generated samples are grammatically coherent—"Once upon a time" produces a recognizable fairy-tale opening—but every fact is invented, which is normal at this scale.

The attention visualizations at the end of ch03 are particularly illuminating. The first layer learns local token alignment ("the" attending to nearby nouns like "cat"), and longer-range dependencies only emerge in deeper layers—consistent with observations in BERTology papers. Reading about it isn't the same as seeing the attention map light up: that's when the transformer architecture clicks.

One other thing worth noting: ch07's SFT uses instruction-data.json in Alpaca format. The dataset is small, but it's enough to teach a 124M model the instruction-response pattern. The loss falls faster than during pretraining and stabilizes within 2k steps—a textbook case of a small model benefiting from high-quality data.

Who Should Use It

Good fit for:

  • Working engineers who want to understand what transformers actually compute. After working through it, you can code-review any attention implementation.
  • Anyone who wants to derive the attention formula cold during an interview. This repo is your lifeline.
  • ML practitioners with PyTorch basics who haven't yet looked under the hood of an LLM.

Not a good fit for:

  • Anyone who needs a production-grade model—head straight to HuggingFace.
  • Anyone planning to train 7B+ parameter runs. The repo covers pedagogy; FSDP, ZeRO, and tensor parallelism are out of scope.
  • Complete beginners to PyTorch. Work through the official 60-minute intro first.

Dependencies are minimal: PyTorch, Python, and three third-party packages—tiktoken, datasets, and wandb.

Link: github.com/rasbt/LLMs-from-scratch

Source: https://github.com/rasbt/LLMs-from-scratch