Tools

How Maple Fits Gemma 4 Into 6GB of VRAM

Maple applies Sub-2-BPW Moe Pruning to Gemma 4 26B, with claims of a 6GB GPU footprint and 45 tokens per second. The headline figures are intriguing, but independent replication and pruning-quality data remain limited.

Font size

The open-source Maple project uses Sub-2-BPW Moe Pruning to compress Gemma 4 26B, claiming that the model can fit on a GPU with just 6GB of VRAM and generate 45 tokens per second. The numbers look impressive, but third-party reproduction and the quality loss caused by pruning still need to be confirmed.

How Maple Fits a 26B Model Into 6GB

Maple is an open-source GitHub project. Its repository is available at https://github.com/AmSach/maple.

It uses Sub-2-BPW Moe Pruning, a sparse pruning method designed to keep the average number of bits per weight below 2. The project claims that Gemma 4 26B can fit on a GPU with only 6GB of VRAM and generate at 45 tokens per second.

The combination of figures is hard to ignore: a 26B model, 6GB of VRAM, and 45 TPS.

That does not necessarily mean it is time to replace a graphics card.

Maple demonstrates an engineering approach rather than a fully validated conclusion. Its real contribution to test is how it sidesteps the assumption that all 26B parameters must remain intact.

How a 26B Model Fits in 6GB

A common misconception about conventional low-bit quantization is that reducing weight precision means fewer weights participate in computation and that memory usage falls in direct proportion to the bit width.

Inference also has to leave room for:

  • KV cache
  • Activations and temporary buffers
  • Framework and operator workspaces
  • Additional overhead from batch size and context length

Maple's central trade-off is not to shrink every weight to an extreme degree. Instead, it uses the sparse structure of a MoE model to reduce the parameters activated and computed for each token.

Two different quantities need to be distinguished:

  1. Total model parameters: Gemma 4 26B remains a model with 26 billion parameters.
  2. Active parameters per token: The MoE router selects only a subset of experts for each pass, and Maple further sparsifies the computation around that subset.

“Sub-2-BPW” does not simply mean that every weight averages fewer than 2 bits. It describes the post-pruning bit budget per weight, with a requirement that the computational path impose sufficiently low overhead from the pruned portions. “Moe” in the title refers to Gemma's mixture-of-experts architecture, not to relabeling a dense model.

This explains how 26 billion parameters and 6GB of memory can coexist, but it does not mean every 26B model can run in the same way.

Results will vary with the model architecture, expert routing, kernels, and memory allocation strategy.

45 TPS Does Not Mean All 26B Parameters Remain on the GPU

“Fewer than 2B active parameters” is often presented as “a 26B model reduced to fewer than 2B parameters.”

That wording is too loose.

The 26B figure represents the total parameter count, while “fewer than 2B” more likely refers to the subset actually activated or computed during a given inference pass. This distinction is what allows MoE models to reduce inference costs.

Low-memory inference still has to answer a concrete question: where are the retained expert weights stored?

If most weights still have to be mapped into GPU memory, computation may be sparse without memory usage falling to 6GB. Conversely, if some experts remain on the CPU, 45 TPS may be constrained by PCIe transfers and expert-routing latency.

Anyone evaluating a 45 tokens-per-second claim should check at least the following conditions:

  • GPU model and VRAM capacity
  • CUDA, driver, and inference-framework versions
  • Batch size
  • Prompt length
  • Number of tokens generated per iteration
  • Whether prefill and decode are reported separately
  • Whether streaming is enabled
  • Whether time to first token is included
  • CPU offload, expert offloading, and caching strategies
  • Whether the timing script excludes model-loading time

Tokens per second is not a complete metric by itself.

Short prompts, small outputs, and a single concurrent request tend to produce more flattering numbers. Long contexts, concurrent requests, and sustained generation expose KV-cache and bandwidth bottlenecks.

The project claims that Gemma 4 26B can reach 45 TPS on a GPU with 6GB of VRAM. Until more evidence is available, the result should be recorded in the project's own terms:

Maple claims that Gemma 4 26B can run on a GPU with only 6GB of VRAM at 45 tokens per second.

Whether it reaches that figure needs to be established through independent reproduction.

Evaluate Pruning Loss Before Speed Records

Speed tests are usually easier than capability tests.

With the same card, script, and prompt, a few timed runs can produce a TPS figure. Determining whether the model has become “less capable,” however, requires a more comprehensive evaluation.

Pruning is not lossless compression. Even if weights average fewer than 2 bits, removing critical connections can reduce output quality. The problem becomes more complex for MoE models:

  • Has an individual expert been pruned too aggressively?
  • Can the router still send requests to the appropriate experts?
  • Do rare tasks degrade because critical experts have been weakened?
  • Does combining quantization and pruning amplify errors?
  • Can errors accumulate over long contexts?

At least three categories of results are needed to assess Maple.

The first is perplexity.

This metric measures language-modeling loss and is useful for comparing compression approaches. It can still be misleading if the test data overlaps heavily with the model's training data.

The second is task accuracy.

Examples include MMLU, ARC, HellaSwag, or the practical tasks used by the project. The most useful presentation compares the unpruned model with the Maple model rather than providing only an isolated score.

The third is an end-to-end benchmark.

It should cover realistic workflows such as code generation, question answering, summarization, and structured output. Failed examples are also needed, rather than just an average score.

Based on the available information, Maple's most eye-catching claims concern 6GB of VRAM and 45 TPS. Speed figures are relatively easy to reproduce, but post-pruning model quality is the deciding factor for deployment.

If the README does not later provide a reproducible evaluation configuration, the project is better described as “an engineering prototype worth testing” than as a mature solution.

Hacker News Shows Only 4 Upvotes and 1 Comment

The project reached the Hacker News front page, but the page had recorded only 4 upvotes and 1 comment.

That is a weak signal.

It does not prove the project has no value. When an open-source inference project is first released, discussion on Hacker News also depends on timing, community structure, and distribution channels. A niche project for uncommon GPUs, edge devices, or quantization can naturally attract little discussion.

It does show that broad validation has not yet occurred.

Without many developers publishing hardware reproductions, speed comparisons across GPU models, loss curves, and task-accuracy results, 45 TPS remains primarily a single data point supplied by the project.

Maple can be viewed at three levels:

  1. Established facts: It is an open-source GitHub project that uses Sub-2-BPW Moe Pruning and targets sparse pruning and low-memory inference for MoE models.
  2. Project claims: Gemma 4 26B can run at 45 TPS on a GPU with 6GB of VRAM.
  3. Awaiting validation: Whether the speed can be reproduced across hardware, how much quality the pruning costs, and whether the model remains usable for everyday tasks.

This separation is more reliable than declaring that “26B models have now entered the 6GB GPU era.”

Editorial Perspective

The competition in low-memory inference may no longer hinge on squeezing every group of weights into fewer bits. It may instead hinge on changing which parameters need to participate in computation. Maple offers a practical approach: sparsely prune an MoE model, first reducing the active computation and then controlling weight storage and runtime overhead. If its results can be reproduced reliably, consumers may begin to ask which tasks are worth running locally rather than simply whether to buy a 24GB or 48GB GPU.

Maple differs from a conventional quantization release because it combines pruning, MoE routing, and 6GB operation in one package. That also introduces more potential bottlenecks. Kernel compatibility, frequent expert offloading, and acceptable quality loss all remain unclear. Neither the 45 TPS figure nor the pruning quality has received independent validation, so the project is better treated as a prototype for stress-testing with real workflows than as a production-ready conclusion.

Source

Source: https://github.com/AmSach/maple