On an NVIDIA GPU you can read the instruction-set manual and disassemble what ran. Apple publishes no equivalent. The instruction set, the latencies, the register file, the occupancy rules, the cache behaviour: all undocumented. A Metal kernel compiles to an intermediate form, and the driver turns that into machine code you never see. What the public knows comes from community reverse engineering, which runs two to three generations behind the silicon because every generation re-breaks the encodings.
Two traditions work against that void. Driver projects decode the encodings, because a compiler backend must emit machine code: applegpu and Asahi Linux, a multi-year effort repeated every generation. The other decodes nothing and recovers what it needs by measurement: metal-benchmarks, Rigel on the M4 tensor path, the A19 and M5 Neural Accelerator benchmarks.
This work belongs to the second, and its claim is narrow. To beat the software running on undocumented hardware you need two trustworthy measurements, not a specification. An honest ceiling to divide by, and a fitness signal that survives being wrong. Given those, the chip stops being a black box to decode and becomes an oracle to query. Nothing here required knowing how the hardware works inside. Two of the numbers below could be checked against an independent measurement, and both agree: the matrix ceiling to 3%, the shader ceiling to 5%.
What that bought is a map of where Apple’s own inference stack sits against its measured ceilings, and a working evolutionary kernel search. The headline is not a kernel. It is that Apple’s matrix-multiply kernels are already excellent and its attention kernels are not. Reaching a number worth trusting was the hard part: three audit rounds, each overturning the one before it, and eight ways a plausible number can describe something that is not happening does.
Every "% of roofline" here divides by a number we measured. Apple publishes 153 GB/s for this chip. It sustains 120.5, so the spec overstates by 27%. Taking the published figure on trust would have inflated every gap on this page and aimed the project at the wrong target.
The M5’s Neural Accelerators are real matrix hardware rather than a shader trick. The matmul path measures 15.4 TFLOPS fp16 against 6.1 on the general-purpose path, a 2.5× gap that only dedicated silicon explains. That shader figure uses packed operands. An earlier probe issued scalar half instructions, read 4.3, and would have flattered the gap to 3.6×. int8 measures 18.6 TFLOPS, above the fp16 number. M4 and earlier have no such units, so every "% of NA ceiling" here belongs to this generation alone.
Scattered reads collapse to 7.4 GB/s, 16× below streaming. That is physics rather than an MLX defect: a random 4-byte read wastes most of its cache line. It is also why SSD-streamed MoE decode was ruled out here on measurement instead of on argument.
The MoE “banked” verdict above is regime-limited, and the correction is the largest single revision to the map. Those 90–97% readings are decode. At prefill, the routed-expert kernel re-runs its entire inner loop once per distinct expert inside a fixed 64-row tile, so useful arithmetic is ≈ min(1, R/64) where R = chunk × top-k / experts, a caller-side parameter. A 256-expert top-8 model prefilling 512-token chunks presents R=16 and runs at 21%. Measured 5%→87% across R, and the law predicts it to within 0.02 everywhere. The control that makes it causal: identical runs aligned to the tile boundary vs shifted by half a tile, same arithmetic and same bytes, measure 11.94 vs 6.85 TF. Only tile phase differs. Fix (BM=32 dispatch keyed on R) measured 1.3–1.45× at kernel level, 1.066× end-to-end, and is offered upstream as ml-explore/mlx#3925.
The hd=256 prefill pothole has since been fixed locally: a fused flash kernel on the Neural Accelerators, found by the evolutionary search seeded from a hand-written kernel, runs at 80% of the ceiling: 2.3× the fallback this page measured at 34%. With it integrated (quality-gated against garbage output) and the chunk size it unblocks, the 35B hybrid’s time-to-first-token at 32k went 70.2 s → 35.5 s across the week, every factor a same-day paired measurement. That story is Part 2: what the search found, what it cost, and what analysis did versus what the search did.
The last unexplained seconds now have names. The time-to-first-token budget closes at 94%: the largest remaining term is the gated-delta-net recurrence (4.05 s), a kernel that is latency-bound: 1024 sequential dependent steps per thread, running at 3% of both the compute and bandwidth ceilings simultaneously. No amount of tuning fixes a formulation; that one needs the chunked delta-rule algorithm, and it is open.
| rule | because |
|---|---|
| Dependent-chain timing | decode is a dependency chain; independent calls overlap and hide latency |
| Rotating buffers | so the cache cannot serve what 48 distinct layers would miss |
| Measured ceilings | never the spec sheet, which overstates by 21% here |
| σ reported per probe | promotion needs to beat the incumbent by >max(2σ, 1%) |
| Proof the code ran | invocation counters; a silent no-op is indistinguishable from parity |
| Product memory layout | strided views, not fresh contiguous tensors |
| Held-out shapes | plus bit-determinism and a fresh-random adversarial verifier |