Anyone running LLMs locally on Apple silicon has hit this wall: small models, fast Mac, plenty of theoretical bandwidth on paper, yet token throughput refuses to climb. A reverse-engineering effort on M3 traced the issue to its source — a hardware erratum baked into the RTL, slicing the ANE's effective bandwidth to roughly one-third of spec.
The result is striking. After a one-line change to the prefetch path, Llama 3.2 1B jumped from 10 tokens/s to 24.3 tokens/s.
Source: eiln.github.io/posts/ane-dma.html (142 points, 27 comments on HN)
The Problem: ANE Bandwidth at One-Third of Spec
M3 ANE's advertised DRAM streaming bandwidth is 45–60 GB/s — enough headroom on paper to push a 1B model past 40 tokens/s.
What eiln measured: 17–19 GB/s.
That caps Llama 3.2 1B at 10.0 tokens/s on a Mac Studio M3, where doubling that figure should have been trivial.
This isn't a tuning exercise — it's a microarchitectural root-cause analysis. The conclusion: this isn't a misconfigured Metal API, isn't a driver bug. It's an RTL-level hardware erratum — a performance bug frozen into the silicon.
Not the Driver — an RTL Erratum
Background first. ANE weight loading is streaming: the model is too large for on-chip SRAM, so tiles get fed from DRAM into the ANE's local buffer in chunks.
This is covered by a speculative prefetch ring — each slot holds a candidate line address, and the ANE guesses the next line based on the current access pattern, fetching ahead into the internal buffer to hide DRAM latency.
Here's the catch: certain paths on the ring carry an RTL bug on M3 silicon. When a prefetch crosses a specific address-aligned boundary, the state machine enters an invalid transition. The data from that prefetch sits unused until the next ring wraparound — latency multiplies by the ring depth.
Impact on end-to-end throughput:
- Advertised bandwidth: 45–60 GB/s
- Measured bandwidth: 17–19 GB/s
- Loss: ~60%
Neither the driver nor the Metal API can rescue this — the path is locked in hardware. Apple hasn't documented the issue anywhere public, but eiln's microbenchmarks pin it down to a specific RTL state-machine transition.
The Workaround: Keep Prefetch Off the Broken Path
Can't fix the silicon — route around it.
eiln's approach disables the path that triggers the bad transition in the DMA engine's prefetch ring. A single prefetch-hint line was changed: DMA address alignment shifted from "natural alignment" to "forced 4KB alignment + skip a specific stride."
Results:
- Llama 3.2 1B throughput: 10.0 tokens/s → 24.3 tokens/s
- ANE effective DRAM bandwidth: 17–19 GB/s → ~40 GB/s
- Speedup: 1× → 2.43×
Code-side, eiln provides a minimal kernel stub (simplified):
// 关键改动:强制 prefetch hint 走避开 RTL bug 的路径
ane_dma_descriptor_t desc = {
.src_addr = weight_ptr,
.length = tile_bytes,
.stride = ALIGN_TO_4KB, // 强制 4KB 对齐
.prefetch_mode = ANE_PF_SAFE_PATH, // 关键:禁用跨边界预取
.ring_slot_mask = 0x0E, // 跳过触发 bug 的 slot
};
ane_submit_dma(&desc);
(Actual register definitions depend on ANE's private SDK; the full microbench code is in eiln's original post.)
This affects every inference workload that streams through ANE — including anything using Core ML or MLX with the ANE backend. Translation: most local-LLM benchmarks run on M3 today are measured under this throttled bandwidth.
What This Means for On-Device LLM Deployment
Three layers of fallout.
First, public microarchitectural performance analysis is vanishingly rare. Consumer SoC AI accelerators don't get the PTX/SASS-level scrutiny NVIDIA GPUs do. ANE is a black box — Apple has never published a microarchitecture whitepaper. eiln's writeup is the first deep RTL behavioral analysis of the ANE DMA subsystem.
Second, the on-device LLM deployment bottleneck usually isn't the model. Debates center on "1B or 3B," "int4 or int8 quantization" — nobody discusses what the ANE actually delivers. If the hardware itself is shaving off 60%, no amount of model-side tuning recovers it.
Third, the community needs a standard recipe for routing around this bug. Apple almost certainly won't acknowledge the erratum (correctness is unaffected, only performance) and won't ship a patch. If MLX, Core ML, and llama.cpp's ANE backend want to squeeze M3/M4 dry, they have to implement their own "safe path" logic.
Editorial Take
Hardware errata typically surface on datacenter GPUs — H100's HBM3 channel issues, A100's PCIe link downclocking. Big customers running big workloads catch them, and Intel/AMD/NVIDIA eventually acknowledge and patch.
This one is different. It's a consumer SoC AI accelerator, and the victim workload is the hottest area in inference right now: on-device LLMs. This isn't some hyperscaler's training job — it's every developer running a local model on a MacBook.
Deeper still: this exposes an under-appreciated fact — on-device AI accelerators are far less mature than datacenter GPUs. Cloud GPUs have armies of engineers doing kernel optimization, performance analysis, and benchmark reproduction. ANE has essentially only Apple. Even with open frameworks like MLX, the microarchitectural optimization surface remains largely unexplored.
Prediction: the next six months will see more cases of "hardware performance bugs exposed under on-device AI workloads." If M4 has anything similar, Apple will almost certainly stay silent — but the community will dig it out on its own. eiln's post is the template.
Caveats: eiln's microbench ran on M3 Max. Whether base M3 / M3 Pro share the issue, and the exact address-stride of the buggy path, aren't fully spelled out in the original. For reproduction, read the original post.