All posts

A Megakernel per Model每个模型一个 Megakernel

Claude wrote a megakernel for each of eight open models and built an inference engine around it. How much faster these engines are than vLLM and SGLang, how we measured it, and how the agents got there.Claude 为八个开源模型各写了一个 Megakernel,并以此 build inference engine。这些 engine 比 vLLM 和 SGLang 快多少,我们怎么测的,以及 agent 是怎么一步步做到的。

Project Link: github.com/dangxingyu/megakernel-zoo项目链接:github.com/dangxingyu/megakernel-zoo

TL;DR

General inference engines such as vLLM1 and SGLang2 serve every model through one scheduler and one library of kernels. We let a coding agent write the engine for one checkpoint on one GPU instead, as a megakernel: one GPU program that runs the whole decode loop without returning to the host between tokens.vLLM1 和 SGLang2 这样的通用 inference engine 用同一个 scheduler 和同一套 kernel 库来服务所有模型。我们反过来,让一个 coding agent 为一块 GPU 上的一个 checkpoint 写专用的 engine,形式是一个 megakernel:一个 GPU 程序跑完整个 decode loop,token 之间不回到 host。

Steady decoding. With the same data flow as the reference implementation (track A), the engines are 1.13–1.77× as fast as the fastest general engine with one sequence, on all eight models from 8 B to 594 B parameters. When the storage of the model may change but its function may not, numerically (track B), the factor reaches 2.05×. Serving a production trace on Inkling-2-Small and 4 × B200, at 1 to 8 requests per second, 99–100% of requests stay within the service levels against 82–85% for SGLang, and the longest stall is 0.08–0.15 s against 2.3–4.4 s.Steady decoding。 在数据流与参考实现相同的前提下(track A),单序列时 engine 的速度是最快通用 engine 的 1.13–1.77 倍,覆盖从 8 B 到 594 B 参数的全部八个模型。如果允许改变模型的存储方式、但不改变它在数值上的函数(track B),倍数可达 2.05×。在生产 trace 上做 serving:Inkling-2-Small 跑在 4 × B200 上,每秒 1 到 8 个请求时,99–100% 的请求满足 service level,SGLang 是 82–85%;最长卡顿是 0.08–0.15 s,SGLang 是 2.3–4.4 s。

Validation. We built a framework in which agents design megakernels on their own. Its core is a test suite the agent cannot change: an engine is timed only after it agrees with a high-precision reference in its numbers, on inputs it has never seen and while it actually runs, and after an independent audit. We ran it with Claude Opus 5.5 and Fable 5.2. The engines, a server that puts them behind the OpenAI API, the test suite and the tools to run it on another model are in megakernel-zoo.验证。 我们搭了一个让 agent 自主设计 megakernel 的框架,核心是一套 agent 改不了的 test suite:engine 必须在数值上与高精度 reference 一致——包括它从没见过的输入、包括它真正运行时的输出——并通过独立审计之后,才会被计时。我们用 Claude Opus 5.5 和 Fable 5.2 跑了这套框架。engine、把它们包装成 OpenAI API 的 server、test suite,以及在别的模型上复现的工具,都在 megakernel-zoo 里。

Recorded answers, replayed at the speed they arrived录下来的回答,按到达时的速度回放

Model

Both servers ran on the same GPU, one after the other, and received the same question through the streaming chat API; text appears at its recorded arrival time. Our engine is served by mkserve, the server in the repository. Its first token arrives later because the steady-decode engine consumes the prompt one token per pass.两个 server 先后跑在同一块 GPU 上,通过 streaming chat API 收到同一个问题;文字按记录下来的到达时间出现。我们的 engine 由 mkserve 提供服务,即 repository 里的 server。它的第一个 token 来得更晚,因为 steady-decode engine 每个 pass 只读入 prompt 的一个 token。

A megakernel per model每个模型一个 megakernel

vLLM and SGLang are designed to run any model. Each decoding step is assembled from a library of general-purpose kernels, one per operation (attention, matrix multiplication, normalization), which run one after another from a CUDA graph, while the CPU schedules every step. We take the other extreme: an agent writes a single kernel for one checkpoint on one GPU type, and that kernel executes every operation of every step, and the loop over tokens, by itself.vLLM 和 SGLang 的设计目标是能跑任何模型。每个 decoding step 由一个通用 kernel 库拼成,每种运算一个 kernel(attention、矩阵乘、normalization),这些 kernel 从一个 CUDA graph 里依次执行,由 CPU 调度每一个 step。我们走另一个极端:由 agent 为一种 GPU 上的一个 checkpoint 写一个 kernel,这个 kernel 自己执行每个 step 的所有运算,以及对 token 的循环。

Where the decode loop runsdecode loop 在哪里运行

CUDA graphvLLM, SGLang: one graph per tokentoken 1token 2token 3a gap at every kernel boundaryhost between tokensMegakernelone launch for many tokenstoken 1token 2token 3grid barriers instead of kernel boundaries
Schematic. vLLM and SGLang replay a CUDA graph for each decode step, so the host launches once per token. The step's operators are separate kernels: each one finishes before the next starts, so the GPU drains and refills at every boundary, and between tokens the host reads the result and schedules the next step (overlapped scheduling hides part of this). A steady-decode megakernel is launched once for many tokens; grid barriers separate its phases, and the choice of the next token and the loop over tokens run inside it.示意图。vLLM 和 SGLang 每个 decode step 回放一次 CUDA graph,所以 host 每个 token 发起一次 launch。step 里的各个算子是独立的 kernel:一个结束后下一个才开始,所以在每个边界上 GPU 都要排空再填满;token 之间,host 读取结果并调度下一个 step(overlapped scheduling 能掩盖其中一部分)。steady-decode megakernel 一次 launch 跑很多个 token;它的各个阶段之间用 grid barrier 隔开,选下一个 token 和对 token 的循环都在 kernel 内部完成。

With the whole decoding step in one kernel, the agent can schedule operators freely: prefetch the next layer's weights while the current layer computes, fuse across operator boundaries without round-tripping intermediates through memory, and start idle thread blocks on the next phase early. Separate kernel launches make each of these hard.整个 decoding step 都在一个 kernel 里,agent 就能自由地调度算子:当前层计算时预取下一层的权重,跨算子边界融合而不必让中间结果经过显存往返,让空闲的 thread block 提前开始下一阶段。拆成多次 kernel launch 时,这些都很难做到。

The price is why megakernels are rare: they are hard to maintain and debug, they are tied to one model and one kind of GPU, and they need retuning when the batch or context changes, so until now they have been written by hand for a few models (Hazy Research's for Llama-1B3, Cohere's for North Mini Code4) or compiled for a set of supported architectures (MPK5). Coding agents now have the capability to do this work, so that cost is gone.代价正是 megakernel 少见的原因:它难维护、难调试,绑定在一个模型和一种 GPU 上,batch 或 context 一变就要重新调优,所以至今只有少数几个模型有手写版本(Hazy Research 为 Llama-1B 写的3、Cohere 为 North Mini Code 写的4),或者由编译器为一组支持的架构生成(MPK5)。现在 coding agent 有 capability 来完成这个事情,所以没有这个成本了。

We report two tracks.我们分两个 track 报告。

  • Track A keeps the data flow of the reference implementation unchanged: weights in the checkpoint's own formats, activations and KV cache in the formats the general engines use; only how the computation is organized on the GPU changes.Track A 要求数据流与参考实现保持一致:权重保持 checkpoint 自己的格式,activation 和 KV cache 用通用 engine 的格式;改变的只是计算在 GPU 上如何组织。
  • Track B is looser: it only requires the function to be unchanged in exact arithmetic, and lets the agent change how the model is stored and what is computed at load time. Lossy quantization or pruning does not qualify, since it changes the function.Track B 更宽松,只要求函数在精确算术下不变,允许 agent 改变模型的存储方式和 load 时预先计算的内容。有损的 quantization 或 pruning 不算,因为它们改变了函数。

We evaluate eight models from 8 B to 594 B parameters, dense and mixture-of-experts, on H100 and B200 GPUs, against vLLM and SGLang and, for North Mini Code, Cohere's megakernel. Versions and flags are on the data page.我们评测了八个模型,从 8 B 到 594 B 参数,有 dense 也有 mixture-of-experts,跑在 H100 和 B200 GPU 上,对比 vLLM 和 SGLang;North Mini Code 还对比了 Cohere 的 megakernel。版本和参数见 数据页。

Steady decodingSteady decoding

We use the standard decode benchmark: a 64-token prompt, 1,024 generated tokens, greedy decoding, batch size 1 or 8, every system served over HTTP on the same GPU.我们用标准的 decode benchmark:64-token 的 prompt,生成 1,024 个 token,greedy decoding,batch size 为 1 或 8,所有系统都通过 HTTP、在同一块 GPU 上提供服务。

Figure 1. Speed relative to SGLang on the same GPU图 1. 相对同一块 GPU 上 SGLang 的速度

SGLang, the baseline (vLLM where SGLang is missing)SGLang,即基准(没有 SGLang 的模型用 vLLM)other general engines其他通用 enginetrack A: the checkpoint's number formatstrack A:checkpoint 自己的数值格式track B, outputs bit-identical to track Atrack B,输出与 track A 逐位相同track B, not bit-identical to track A (outputs differ in the last bits)track B,与 track A 非逐位相同(输出在最后几位不同)
batch size = 100.5×1.0×1.5×2.0×SGLang = 1.0×batch size = 800.5×1.0×1.5×2.0×SGLang = 1.0×Qwen3-8BH100Qwen3-8B, SGLang, batch size 1: 6.09 ms per token (1.00× SGLang)SGLang1.00×Qwen3-8B, vLLM, batch size 1: 6.75 ms per token (0.90× SGLang)vLLM0.90×Qwen3-8B, track A, batch size 1: 5.13 ms per token (1.19× SGLang)track A1.19×Qwen3-8B, track B, batch size 1: 3.83 ms per token (1.59× SGLang)track B1.59×Qwen3-8B, SGLang, batch size 8: 6.85 ms per token (1.00× SGLang)SGLang1.00×Qwen3-8B, vLLM, batch size 8: 6.95 ms per token (0.99× SGLang)vLLM0.99×Qwen3-8B, track A, batch size 8: 5.48 ms per token (1.25× SGLang)track A1.25×Qwen3-8B, track B, batch size 8: 4.25 ms per token (1.61× SGLang)track B1.61×North Mini CodeH100relative to vLLMNorth Mini Code, vLLM, batch size 1: 5.05 ms per token (1.00× vLLM)vLLM1.00×North Mini Code, Cohere, batch size 1: 3.62 ms per token (1.39× vLLM)Cohere1.39×North Mini Code, track A, batch size 1: 2.54 ms per token (1.99× vLLM)track A1.99×North Mini Code, track B, batch size 1: 2.15 ms per token (2.35× vLLM)track B2.35×North Mini Code, vLLM, batch size 8: 9.46 ms per token (1.00× vLLM)vLLM1.00×North Mini Code, Cohere, batch size 8: 7.07 ms per token (1.34× vLLM)Cohere1.34×North Mini Code, track A, batch size 8: 6.90 ms per token (1.37× vLLM)track A1.37×North Mini Code, track B, batch size 8: 6.05 ms per token (1.56× vLLM)track B1.56×gpt-oss-20bH100gpt-oss-20b, SGLang, batch size 1: 3.50 ms per token (1.00× SGLang)SGLang1.00×gpt-oss-20b, vLLM, batch size 1: 3.31 ms per token (1.06× SGLang)vLLM1.06×gpt-oss-20b, track A, batch size 1: 1.87 ms per token (1.87× SGLang)track A1.87×gpt-oss-20b, track B, batch size 1: 1.62 ms per token (2.17× SGLang)track B2.17×gpt-oss-20b, SGLang, batch size 8: 5.22 ms per token (1.00× SGLang)SGLang1.00×gpt-oss-20b, vLLM, batch size 8: 4.93 ms per token (1.06× SGLang)vLLM1.06×gpt-oss-20b, track A, batch size 8: 3.39 ms per token (1.54× SGLang)track A1.54×gpt-oss-20b, track B, batch size 8: 3.39 ms per token (1.54× SGLang)track B1.54×flintB200relative to vLLMflint, vLLM, batch size 1: 1.99 ms per token (1.00× vLLM)vLLM1.00×flint, track A, batch size 1: 1.39 ms per token (1.43× vLLM)track A1.43×flint, track B, batch size 1: 1.36 ms per token (1.47× vLLM)track B1.47×flint, vLLM, batch size 8: 3.79 ms per token (1.00× vLLM)vLLM1.00×flint, track A, batch size 8: 3.35 ms per token (1.13× vLLM)track A1.13×flint, track B, batch size 8: 3.35 ms per token (1.13× vLLM)track B1.13×Qwen3.8-27BB200Qwen3.8-27B, SGLang, batch size 1: 9.53 ms per token (1.00× SGLang)SGLang1.00×Qwen3.8-27B, vLLM, batch size 1: 10.19 ms per token (0.93× SGLang)vLLM0.93×Qwen3.8-27B, track A, batch size 1: 8.45 ms per token (1.13× SGLang)track A1.13×Qwen3.8-27B, track B, batch size 1: 7.26 ms per token (1.31× SGLang)track B1.31×Qwen3.8-27B, SGLang, batch size 8: 10.30 ms per token (1.00× SGLang)SGLang1.00×Qwen3.8-27B, vLLM, batch size 8: 10.74 ms per token (0.96× SGLang)vLLM0.96×Qwen3.8-27B, track A, batch size 8: 8.89 ms per token (1.16× SGLang)track A1.16×Qwen3.8-27B, track B, batch size 8: 7.72 ms per token (1.33× SGLang)track B1.33×Qwen3-32BB200Qwen3-32B, SGLang, batch size 1: 11.43 ms per token (1.00× SGLang)SGLang1.00×Qwen3-32B, vLLM, batch size 1: 12.25 ms per token (0.93× SGLang)vLLM0.93×Qwen3-32B, track A, batch size 1: 10.11 ms per token (1.13× SGLang)track A1.13×Qwen3-32B, track B, batch size 1: 8.99 ms per token (1.27× SGLang)track B1.27×Qwen3-32B, SGLang, batch size 8: 11.68 ms per token (1.00× SGLang)SGLang1.00×Qwen3-32B, vLLM, batch size 8: 12.23 ms per token (0.96× SGLang)vLLM0.96×Qwen3-32B, track A, batch size 8: 10.43 ms per token (1.12× SGLang)track A1.12×Qwen3-32B, track B, batch size 8: 9.44 ms per token (1.24× SGLang)track B1.24×gpt-oss-120bB200gpt-oss-120b, SGLang, batch size 1: 2.22 ms per token (1.00× SGLang)SGLang1.00×gpt-oss-120b, vLLM, batch size 1: 2.52 ms per token (0.88× SGLang)vLLM0.88×gpt-oss-120b, track A, batch size 1: 1.78 ms per token (1.25× SGLang)track A1.25×gpt-oss-120b, track B, batch size 1: 1.69 ms per token (1.31× SGLang)track B1.31×gpt-oss-120b, SGLang, batch size 8: 3.33 ms per token (1.00× SGLang)SGLang1.00×gpt-oss-120b, vLLM, batch size 8: 3.77 ms per token (0.88× SGLang)vLLM0.88×gpt-oss-120b, track A, batch size 8: 3.07 ms per token (1.08× SGLang)track A1.08×gpt-oss-120b, track B, batch size 8: 3.03 ms per token (1.10× SGLang)track B1.10×Inkling-2-Small4 × B200Inkling-2-Small, SGLang, batch size 1: 4.74 ms per token (1.00× SGLang)SGLang1.00×Inkling-2-Small, track A, batch size 1: 3.30 ms per token (1.43× SGLang)track A1.43×Inkling-2-Small, track B, batch size 1: 3.29 ms per token (1.44× SGLang)track B1.44×Inkling-2-Small, SGLang, batch size 8: 6.55 ms per token (1.00× SGLang)SGLang1.00×Inkling-2-Small, track A, batch size 8: 5.39 ms per token (1.22× SGLang)track A1.22×Inkling-2-Small, track B, batch size 8: 5.38 ms per token (1.22× SGLang)track B1.22×
SGLang's time per token divided by each system's: SGLang is 1.0×, and a longer bar is faster. SGLang cannot load flint and was not measured on North Mini Code; those two rows are relative to vLLM. Gray bars are the general engines measured in the same job (for North Mini Code also the megakernel Cohere wrote for it), colored bars are ours. The factors in the text are against the fastest general engine of each model, which is vLLM on gpt-oss-20b and Cohere's megakernel on North Mini Code. A 64-token prompt, 1,024 generated tokens, greedy decoding, all systems served over HTTP on one GPU in the same job, three rounds. Hover a bar for the times in milliseconds.SGLang 的每 token 时间除以各系统的:SGLang 是 1.0×,柱子越长越快。SGLang 加载不了 flint,North Mini Code 上也没有测 SGLang,这两行以 vLLM 为基准。灰色柱子是同一个 job 里测的通用 engine(North Mini Code 还有 Cohere 为它写的 megakernel),彩色柱子是我们的。正文里的倍数是相对每个模型最快的通用 engine:gpt-oss-20b 上是 vLLM,North Mini Code 上是 Cohere 的 megakernel。64-token 的 prompt,生成 1,024 个 token,greedy decoding,所有系统都通过 HTTP、在同一个 job 里跑在同一块 GPU 上,三轮。鼠标悬停在柱子上可以看到以毫秒计的时间。

At batch size 1, track A is 1.13 to 1.77 times as fast as the fastest general engine on every model. At batch size 8 the speedups are smaller on six of the eight models, lowest on North Mini Code (1.02×) and gpt-oss-120b (1.08×). On North Mini Code the fastest general engine is Cohere's megakernel; against vLLM, track A is 1.99× and 1.37×. On the engines also timed with prose and code prompts and with sampling, the speedups stay within 0.05 of the greedy random-token ones (data page).batch size 为 1 时,track A 在每个模型上都是最快通用 engine 的 1.13 到 1.77 倍。batch size 为 8 时,八个模型中有六个的加速比更小,最低的是 North Mini Code(1.02×)和 gpt-oss-120b(1.08×)。North Mini Code 上最快的通用 engine 是 Cohere 的 megakernel;对比 vLLM,track A 是 1.99× 和 1.37×。另外用散文和代码 prompt、以及 sampling 测过的 engine,加速比与 greedy 随机 token 的结果相差不超过 0.05(数据页)。

Batch-1 decoding is bound by HBM bandwidth. On Qwen3-8B and an H100, reading the 15.1 GB of BF16 weights once at peak bandwidth (3.35 TB/s) takes 4.52 ms; the track-A engine runs at 5.13 ms per token, 88% of that bound.batch size 为 1 的 decoding 受 HBM 带宽限制。在 Qwen3-8B 和 H100 上,以峰值带宽(3.35 TB/s)把 15.1 GB 的 BF16 权重读一遍要 4.52 ms;track-A engine 每个 token 5.13 ms,达到这个上界的 88%。

What track B usedTrack B 用了什么

In track B, the agent can speed up the engine by observing features of this checkpoint's weights. The clearest example is the weight format: BF16 spends 8 bits on the exponent, yet the weights in one row of a trained matrix span only a few binades, a redundancy that lossless formats such as DFloat116 also exploit. The Qwen3-8B agent measured this, encoded each weight losslessly in 12 bits and decoded it in registers ahead of the tensor cores, cutting weight traffic per token by a quarter. Two other techniques recur in Table 1: a certified low-precision sketch of the LM head that shortlists the next token, and per-token lookup tables for layer 0, whose input depends only on the token's embedding. Concurrent work7 found the same format: SpecPilot, an agentic loop that also runs Claude Opus 5.5 in Claude Code, optimizes speculative decoding for each deployment and lists this lossless 12-bit format among its discoveries. Our Claude Opus 5.5 agent found it independently, inside a full decode megakernel. Later runs adapted the method to Qwen3.8-27B and Qwen3-32B, where every product is bit-identical to BF16 arithmetic, and a Claude Fable 5.2 run applied it to the LM head of Inkling-2-Small.在 track B 中,agent 可以通过观察这个 checkpoint 的 weight 的一些 feature 来 speedup。最明显的例子是权重格式:BF16 用 8 个 bit 表示 exponent,但训练好的矩阵里同一行的权重只跨几个 binade,DFloat116 这类无损格式利用的也是这种冗余。Qwen3-8B 上的 agent 测出了这一点,把每个权重无损地编码成 12 bit,并在送往 tensor core 之前于寄存器内解码,每个 token 的权重流量减少四分之一。表 1 里还反复出现两种技术:LM head 的 certified 低精度 sketch,用来筛出下一个 token 的候选;以及第 0 层的 per-token 查找表,因为这一层的输入只取决于 token 的 embedding。同期工作7发现了同样的格式:SpecPilot 是一个同样在 Claude Code 里运行 Claude Opus 5.5 的 agentic loop,它为每个部署优化 speculative decoding,并把这种无损 12-bit 格式列为它的发现之一。我们的 Claude Opus 5.5 agent 也在完整的 decode megakernel 内部独立发现了它。后面的 run 把这个 method adapt 到了 Qwen3.8-27B 和 Qwen3-32B 上,结果上每个乘积都与 BF16 算术逐位相同;一个 Claude Fable 5.2 的 run 也把它用到了 Inkling-2-Small 的 LM head 上。

Table 1. Track B on each model表 1. 各模型上的 track B

Model模型Effect in the runrun 中测得的效果Logits against track A与 track A 的 logits 对比
flintflint−2.9% (head sketch)−2.9%(head sketch)bit-identical逐位相同
Qwen3.8-27BQwen3.8-27B−9% (12-bit weights), then −5.6%−9%(12-bit 权重),之后 −5.6%bit-identical逐位相同
Qwen3-32BQwen3-32B−11% (10.12 → 9.00 ms), 12-bit weights−11%(10.12 → 9.00 ms),12-bit 权重bit-identical逐位相同
gpt-oss-120bgpt-oss-120b−5% (head sketch)−5%(head sketch)bit-identical逐位相同
Inkling-2-SmallInkling-2-Small−0.4%bit-identical逐位相同
Qwen3-8BQwen3-8B−15% (5.34 → 4.52 ms), 12-bit weights−15%(5.34 → 4.52 ms),12-bit 权重not bit-identical: the residual stream is kept in FP32 (BF16 in track A), so it is more precise非逐位相同:residual stream 保持 FP32(track A 是 BF16),因此精度更高
North Mini CodeNorth Mini Code−10% (12-bit weights), −4% (head sketch)−10%(12-bit 权重),−4%(head sketch)not bit-identical: the rare weights outside the 12-bit range are added in a separate pass, same values in a different rounding order非逐位相同:少数超出 12-bit 范围的权重单独补算,数值相同,舍入顺序不同
gpt-oss-20bgpt-oss-20b−14% (1.87 → 1.61 ms)−14%(1.87 → 1.61 ms)not bit-identical: the router weights are pre-multiplied by the norm weights in FP32, so the normalized input is never rounded to BF16 and the router logits differ in the last bits非逐位相同:router 权重预先乘上 norm 权重并以 FP32 存储,归一化后的输入不再舍入到 BF16,router logits 在最后几位不同
Before and after scores, from the run's own evaluator, of the submission that introduced each change. A head sketch is a low-precision copy of the LM head with a provable error bound: it only shortlists candidate tokens, and every reported logit comes from the exact head. What each engine does is on the data page.引入每处改动的那次 submission 在 run 自己的 evaluator 下前后的分数。head sketch 是 LM head 的低精度副本,带有可证明的误差上界:它只用来筛选候选 token,所有报告的 logit 都来自精确的 head。每个 engine 具体做了什么见 数据页。

Where the checkpoint is already compressed (the 4-bit experts of Inkling-2-Small and gpt-oss), track B gains little.如果 checkpoint 本身已经压缩过(Inkling-2-Small 和 gpt-oss 的 4-bit expert),track B 的收益很小。

Serving live traffic服务真实流量

The engines above decode a batch whose sequences start together; a server must admit new requests while others are decoding. For Qwen3-8B and Inkling-2-Small, a second run moved scheduling onto the GPU: continuous batching, chunked prefill inside the kernel, a paged KV cache and prefix caching.上面的 engine 解码的是同时开始的一个 batch;而 server 必须在其他请求 decode 的同时接收新请求。对 Qwen3-8B 和 Inkling-2-Small,第二个 run 把调度搬到了 GPU 上:continuous batching、kernel 内的 chunked prefill、paged KV cache 和 prefix caching。

The serving engine: the scheduling loop runs on the GPUserving engine:调度循环在 GPU 上运行

HOSTmkserveHTTP, tokenizerrequest checksHOST-MAPPED MEMORYrequest ringtoken ringsubmitpollGPU, ONE CUDA GRAPHkernel scheduleradmit, page K/V, prefix cache, plan the passone passdecode rows + prefill chunks, every layertoken committokens out, loop conditionwhileworkeverything is here
The serving loop runs on the GPU: one CUDA graph admits requests, runs a pass over the model and writes tokens back. mkserve is a thin host-side wrapper that writes requests into a ring buffer in host-mapped memory and reads tokens from another.serving loop 在 GPU 上运行:一个 CUDA graph 负责接收请求、跑一个 pass、把 token 写回。mkserve 只是一层很薄的 host 端包装,把请求写进 host-mapped memory 里的一个 ring buffer,再从另一个 ring buffer 读出 token。

We replay the Mooncake tool-and-agent trace8 (mean prompt length about 9,000 tokens), with every system served over HTTP on the same node.我们回放 Mooncake 的 tool-and-agent trace8(prompt 平均约 9,000 个 token),所有系统都通过 HTTP 在同一个节点上提供服务。

Figure 2. Inkling-2-Small on 4 × B200 under a production request trace图 2. Inkling-2-Small 在 4 × B200 上处理生产请求 trace

SLO attainmentpercent0501001248requests per secondours: 100 at 1ours: 100 at 2ours: 100 at 4ours: 99.2 at 8SGLang: 83.6 at 1SGLang: 84.6 at 2SGLang: 83.1 at 4SGLang: 81.7 at 8oursSGLang
Longest stall of a decoding requestseconds, log scale0.10.3131248requests per secondours: 0.08 at 1ours: 0.08 at 2ours: 0.09 at 4ours: 0.15 at 8SGLang: 2.48 at 1SGLang: 2.35 at 2SGLang: 2.6 at 4SGLang: 4.45 at 8SGLangours
Time between a request's tokensmilliseconds, log scale10301003001,0001248requests per secondservice level 100 msours, median: 27.9 at 1ours, median: 29.3 at 2ours, median: 45.4 at 4ours, median: 60.2 at 8ours, p99: 68.8 at 1ours, p99: 68.7 at 2ours, p99: 73.1 at 4ours, p99: 80.5 at 8SGLang, median: 9.15 at 1SGLang, median: 9.36 at 2SGLang, median: 10.1 at 4SGLang, median: 18 at 8SGLang, p99: 637 at 1SGLang, p99: 732 at 2SGLang, p99: 755 at 4SGLang, p99: 789 at 8SGLang, p99ours, p99ours, medianSGLang, median
Time to the first tokenseconds, log scale0.3131248requests per secondservice level 5 sours, median: 0.19 at 1ours, median: 0.17 at 2ours, median: 0.18 at 4ours, median: 0.2 at 8ours, p99: 3.98 at 1ours, p99: 3.41 at 2ours, p99: 3.43 at 4ours, p99: 4.63 at 8SGLang, median: 0.56 at 1SGLang, median: 0.59 at 2SGLang, median: 0.61 at 4SGLang, median: 0.62 at 8SGLang, p99: 2.21 at 1SGLang, p99: 2.18 at 2SGLang, p99: 2.16 at 4SGLang, p99: 2.01 at 8ours, p99SGLang, p99SGLang, medianours, median
Replay of the first 300 seconds of the Mooncake tool-and-agent trace at four arrival rates. Both systems are served over HTTP on the same node; three rounds per rate, the first a warm-up. A request meets its SLOs when its time to first token is at most 5 s and its mean time between tokens at most 0.1 s.以四种到达速率回放 Mooncake tool-and-agent trace 的前 300 秒。两个系统都通过 HTTP 在同一个节点上提供服务;每个速率三轮,第一轮是 warm-up。一个请求的首 token 时间不超过 5 s、token 间平均间隔不超过 0.1 s,就算满足 SLO。

At 1, 2 and 4 requests per second our engine reaches 100% SLO attainment, and 99.2% at 8; SGLang reaches 82 to 85% (a request meets its SLOs when its time to first token is at most 5 s and its mean time between tokens at most 0.1 s; attainment counted as in DistServe9). The gap comes from stalls: when a long prompt arrives, SGLang's decoding requests wait behind its prefill for up to 2.3 to 4.4 s; our megakernel's longest stall is 0.08 to 0.15 s.每秒 1、2 和 4 个请求时,我们的 engine 的 SLO 达成率是 100%,每秒 8 个时是 99.2%;SGLang 是 82 到 85%(首 token 时间不超过 5 s、token 间平均间隔不超过 0.1 s 即算满足,达成率按 DistServe9 的方式统计)。差距来自卡顿:长 prompt 到来时,SGLang 中正在 decode 的请求要排在它的 prefill 后面,最长等 2.3 到 4.4 s;我们的 megakernel 最长的卡顿是 0.08 到 0.15 s。

On Qwen3-8B and 1 × H100 the pattern is the same: SLO attainment is 95 to 97% versus 68 to 75% for SGLang and vLLM at 1 and 2 requests per second, and 90 to 93% versus 59 to 68% at 3; the longest stall is 0.09 to 0.13 s.在 Qwen3-8B 和 1 × H100 上规律相同:每秒 1 和 2 个请求时,SLO 达成率是 95 到 97%,SGLang 和 vLLM 是 68 到 75%;每秒 3 个时是 90 到 93%,对方是 59 到 68%;最长卡顿是 0.09 到 0.13 s。

How we validate an engine我们如何验证一个 engine

For autoresearch the test suite matters more than anything else: the agent optimizes whatever the evaluator rewards, so every gap in the evaluator becomes a shortcut. Agent-written kernels have passed tests by reusing the reference's output memory, by special-casing the test inputs and by timing the wrong stream101112, and open kernel competitions now routinely filter out reward-hacked submissions13. The agent cannot modify the suite, and an engine is timed only after it passes.对 autoresearch 来说,test suite 比什么都重要:agent 优化的是 evaluator 奖励的东西,evaluator 的每个漏洞都会变成捷径。agent 写的 kernel 曾经靠复用 reference 的输出显存、针对测试输入写特例、计时测错 stream 而通过测试101112,公开的 kernel 比赛现在也会例行过滤 reward hacking 的 submission13。agent 无法修改这套 suite,engine 通过之后才会被计时。

Reference and tolerances. The reference is the model evaluated in FP64 (FP32 where FP64 does not fit in memory). Tolerances are calibrated on the standard BF16 implementation (Hugging Face Transformers14): we measure its deviation from the reference at every position and allow the engine twice that on the median and the 99th percentile of the logit error, as FlashAttention's own tests do15; four times that on the KL divergence of the next-token distribution, which tracks how many answers a change flips better than accuracy does16; and top-1 agreement within one point. vLLM and SGLang, measured the same way, deviate from the reference about as much as the BF16 implementation, so the tolerance is the error a general engine already has.Reference 与容差。 reference 是用 FP64 计算的模型(显存放不下时用 FP32)。容差以标准的 BF16 实现(Hugging Face Transformers14)为基准:我们在每个位置测量它与 reference 的偏差,logit 误差的中位数和 99th percentile 允许 engine 达到它的两倍,与 FlashAttention 自己的测试做法相同15;next-token 分布的 KL divergence 允许四倍——它比 accuracy 更能反映一个改动会让多少答案翻转16;top-1 agreement 与它相差不超过一个百分点。按同样方式测量,vLLM 和 SGLang 与 reference 的偏差和 BF16 实现相当,所以这个容差就是通用 engine 本来就有的误差。

Inputs. A public set covers every position up to 128 and every eighth position up to 1,152. A hidden set of natural text, code and math word problems extends to position 2,000; the run never sees it and only learns whether it passed.输入。 公开集覆盖 128 以内的每个位置,以及到 1,152 为止每隔八个的位置。一个由自然文本、代码和数学应用题组成的 hidden set 一直延伸到位置 2,000;run 看不到它,只知道自己是否通过。

The engine as it runs. The engine's own decode loop must reproduce its step-by-step greedy tokens; the logits produced during the timed run are checked afterwards; and rerunning the batch with the sequences in reverse order must give bit-identical logits (in general engines a sequence's logits can depend on the batch it runs in17). Before release, each engine is also tested at every batch size, at full context and after a restart.运行中的 engine。 engine 自己的 decode loop 必须复现它逐步 greedy 生成的 token;计时运行中产生的 logits 会在事后检查;把 batch 里的序列倒序后重跑,logits 必须逐位相同(在通用 engine 里,一个序列的 logits 可能取决于它所在的 batch17)。发布前,每个 engine 还要在每个 batch size、满 context 以及重启之后各测一遍。

Table 2. What an engine has to pass表 2. engine 必须通过的检查

Check检查Compared with对比对象Passes when通过条件Runs运行时机
Logits at every position每个位置的 logitsthe model in FP64 (FP32 if it does not fit), on public and hidden sequencesFP64 计算的模型(放不下时用 FP32),公开序列和 hidden 序列median and 99th-percentile relative error ≤ 2× the BF16 implementation's相对误差的中位数和 99th percentile ≤ BF16 实现的 2×every submission每次 submission
Distribution分布the same reference同一个 referenceKL divergence median and p99 ≤ 4× the BF16 implementation's; top-1 agreement within one point of itKL divergence 的中位数和 p99 ≤ BF16 实现的 4×;top-1 agreement 与它相差不超过一个百分点every submission每次 submission
Hidden setHidden setnatural text, code and word problems to position 2,000, never shown to the run自然文本、代码和应用题,到位置 2,000,从不给 run 看all of the above; only pass or fail is reported以上全部;只报告通过或失败every submission每次 submission
As it runs运行中its own step-by-step greedy tokens它自己逐步 greedy 生成的 tokenits decode loop gives the same tokens; logits taken during the timed run match the referencedecode loop 给出相同的 token;计时运行中取出的 logits 与 reference 吻合every submission每次 submission
Slot order and repetitionSlot 顺序与重复the same batch with the sequences reversed序列倒序的同一个 batchlogits bit-identicallogits 逐位相同every submission每次 submission
Batch sizes, full context, restartbatch size、满 context、重启fresh engines新启动的 engine1 to 8 sequences within the limits; bit-identical after a full context and after a restart1 到 8 个序列都在阈值内;满 context 之后和重启之后逐位相同before release发布前
Audit审计an independent agent reading the source一个独立的 agent 阅读源码no rule broken; its own experiments reproduce the scores没有违反任何规则;它自己的实验能复现分数before release发布前
Tolerances are calibrated per model on the standard BF16 implementation (Hugging Face) at the same positions. vLLM and SGLang, measured the same way, land within about one point of its top-1 agreement and within 1.5× of its KL.容差按模型以标准 BF16 实现(Hugging Face)在相同位置上的偏差为基准。按同样方式测量,vLLM 和 SGLang 的 top-1 agreement 与它相差约一个百分点以内,KL 在它的 1.5× 以内。

Every engine shown in this post except the three Inkling-2-Small engines was re-tested with the current suite: 20 of 20 pass every check.除三个 Inkling-2-Small engine 外,本文展示的每个 engine 都用当前的 suite 重新测过:20 个中有 20 个通过全部检查。

Audits. Every final engine was reviewed by an independent agent that read the source for rule violations and ran its own experiments. No engine calls another engine, caches outputs, special-cases the evaluator's inputs or skips part of the model. The audits did find problems that the evaluator of the time had passed; each led to a fix in the engine, in what we report, or in the suite.审计。 每个最终 engine 都由一个独立的 agent 审查:它阅读源码寻找违规之处,并跑自己的实验。没有 engine 调用别的 engine、缓存输出、针对 evaluator 的输入写特例或跳过模型的某一部分。但审计确实发现了当时的 evaluator 放过的问题;每一个都促成了一处修正:在 engine 里、在我们报告的内容里,或在 suite 里。

Table 3. What the audits found that the evaluator had passed表 3. 审计发现、而当时 evaluator 放过的问题

Finding发现Effect影响What was done处理Caught now by现在由什么检查拦住
Two of the three timed prompts drive Qwen3.8-27B into a repetition loop that the drafts predict三个计时 prompt 中有两个会让 Qwen3.8-27B 陷入 draft 能预测的重复循环multi-token decoding scored 0.415 ms per token, 3.71 ms on natural textmulti-token decoding 得分 0.415 ms/token,在自然文本上是 3.71 msthe natural-text figure is the one reported报告的是自然文本上的数字hidden natural-text set; natural-text timing自然文本 hidden set;自然文本计时
Three track-B engines labeled bit-identical were not三个标为逐位相同的 track-B engine 其实不是flint's engine picked a different top-1 token at 10.8% of positionsflint 的 engine 在 10.8% 的位置选了不同的 top-1 tokenflint re-run, now bit-identical; the other two relabeledflint 重新跑过,现在逐位相同;另外两个改了标注bit-for-bit check against track A与 track A 的逐位比对
No error past the context limit超过 context 上限时不报错Inkling-2-Small overwrote other sequences' KV cache; flint returned empty outputInkling-2-Small 覆盖了其他序列的 KV cache;flint 返回空输出both runs extended; the engines refuse such requests两个 run 都延长了;engine 现在拒绝这类请求requests at and past the limit达到和超过上限的请求
A check added after the first Qwen3-8B run第一个 Qwen3-8B run 之后加的一项检查one of 1,024 positions over the tolerance: attention scores were rounded to BF16 twice1,024 个位置中有一个超出容差:attention score 被两次舍入到 BF16a 3-hour run fixed it at the same speed一个 3 小时的 run 修好了它,速度不变median and p99 instead of the maximum用中位数和 p99 代替最大值
Serving engines did not validate their inputsserving engine 不校验输入prefixes matched by a 64-bit hash without comparing tokens; no bounds checks前缀用 64-bit hash 匹配、不比较 token;没有越界检查Inkling-2-Small fixed in an extended run; Qwen3-8B being fixedInkling-2-Small 在延长的 run 中修好;Qwen3-8B 正在修mkserve input checks; collision testmkserve 输入检查;hash 碰撞测试
Qwen3-8B track-A serving under heavy load高负载下的 Qwen3-8B track-A servingtokens differed from the same request run alone in 12 of 17 trials17 次试验中有 12 次,token 与同一请求单独运行时不同under investigation with the fix above与上面的修复一起排查中slot-order and load testsslot 顺序测试和负载测试

Timing.计时。

  • All systems of a comparison run in the same job on the same GPU, one after another.同一组对比的所有系统在同一个 job 里、同一块 GPU 上依次运行。
  • Three rounds; each restarts every server and begins with a different system.跑三轮;每轮重启所有 server,并由不同的系统先跑。
  • Every system is behind an OpenAI-compatible streaming endpoint and timed by the same client.每个系统都放在 OpenAI 兼容的 streaming endpoint 后面,由同一个 client 计时。
  • Prompts are token ids, and every request generates exactly 1,024 tokens.prompt 以 token id 给出,每个请求恰好生成 1,024 个 token。
  • vLLM and SGLang are released versions with their recommended flags, or a faster configuration where we found one.vLLM 和 SGLang 用发布版本和官方推荐的参数,我们找到更快配置的地方就用更快的。
  • An engine is timed only after it passes the correctness checks.engine 通过正确性检查之后才会被计时。

How the agents got thereagent 是怎么做到的

A run gives Claude (Opus 5.5 or Fable 5.2, in Claude Code) the checkpoint, a GPU pod and a budget of agent-hours. The agent profiles and experiments as it sees fit and submits source code whenever it wants a score; each submission is logged with the hypothesis the agent wrote for it. The prompt states the goal, the rules and the tools, and nothing else. The engines in this post took 19 runs, 458 agent-hours and 675 submissions (every run is on the data page).一个 run 给 Claude(Opus 5.5 或 Fable 5.2,在 Claude Code 中)一个 checkpoint、一个 GPU pod,以及一份 agent-hour 预算。agent 自行决定怎么 profile、怎么做实验,想要分数时随时提交源代码;每次 submission 都连同 agent 为它写下的 hypothesis 一起记录。prompt 只说明目标、规则和工具,别的什么都不给。本文中的 engine 用了 19 个 run、458 个 agent-hour 和 675 次 submission(每个 run 都在 数据页 上)。

Figure 3. How four of the engines got faster during their runs图 3. 四个 engine 在各自 run 中如何变快

track Atrack Atrack Btrack Bgeneral engine通用 enginea submission named below下方列出的 submission
Qwen3-8B · H100track A21.05 → 5.13 ms in 30 stepstrack B4.52 → 3.81 ms in 25 stepsOpus 5.5 39 h · Fable 5.2 2 h · Opus 5.5 13 h4680204060agent-hoursms per tokens10, s11SGLang 6.07A 5.13B 3.81The whole decode step in one kernel: its 181 operator boundaries become in-kernel grid barriers (atomic counter and polling) instead of kernel launches.1RMSNorm fused into the preceding residual-add epilogue, which emits w·x in BF16 and per-tile partial sums of x²; the next GEMV applies 1/rms to its outputs (the GEMV is linear), so no SM re-reads the FP32 residual stream.2Lossless 12-bit weights (sign, 4-bit exponent code, 7-bit mantissa; rows pre-scaled by exact powers of two) decoded in registers before the MMA, the ~10⁻⁴ of weights outside the window patched by extra MMAs: HBM bytes per step 15.1 → 11.4 GB.3
  1. 1The whole decode step in one kernel: its 181 operator boundaries become in-kernel grid barriers (atomic counter and polling) instead of kernel launches. 7.37 → 6.46 ms1整个 decode step 放进一个 kernel:181 个算子边界从 kernel launch 变成 kernel 内的 grid barrier(atomic counter 加轮询)。7.37 → 6.46 ms
  2. 2RMSNorm fused into the preceding residual-add epilogue, which emits w·x in BF16 and per-tile partial sums of x²; the next GEMV applies 1/rms to its outputs (the GEMV is linear), so no SM re-reads the FP32 residual stream. 6.14 → 5.95 ms2RMSNorm 融合进前一个 residual-add 的 epilogue:它直接输出 BF16 的 w·x 和每个 tile 的 x² 部分和,下一个 GEMV 在输出上再乘 1/rms(GEMV 是线性的),各个 SM 不必再重读 FP32 residual stream。6.14 → 5.95 ms
  3. 3Lossless 12-bit weights (sign, 4-bit exponent code, 7-bit mantissa; rows pre-scaled by exact powers of two) decoded in registers before the MMA, the ~10⁻⁴ of weights outside the window patched by extra MMAs: HBM bytes per step 15.1 → 11.4 GB. 5.34 → 4.52 ms3无损 12-bit 权重(符号位、4-bit 指数码、7-bit 尾数;每行先乘一个精确的 2 的幂),在 MMA 之前于寄存器内解码;约 10⁻⁴ 超出窗口的权重用额外的 MMA 补算。每步读取的 HBM 字节从 15.1 GB 降到 11.4 GB。5.34 → 4.52 ms
flint · B200track A3.00 → 1.39 ms in 31 stepstrack B2.19 → 1.35 ms in 26 stepsFable 5.2 36 h · Opus 5.5 20 h1.522.502040agent-hoursms per tokens11vLLM 1.97A 1.39B 1.35Attention V accumulation re-partitioned so each warp owns one head-dim chunk across key groups (partials in 8 KB instead of 64 KB of shared memory), softmax weights passed as MMA B operands; router top-6 for all sequences in parallel; LM head over all 152 SMs.1Shared-memory carve-out per thread block cut from 224 KB to 128 KB (V no longer staged, K staging from 20 to 12 key groups); L1 and shared memory share one pool, and a batch of 16 weight-tile loads takes 0.76 µs instead of 2.27 µs.2Softmax normalization: one reciprocal per thread plus an exact FMA correction (Markstein's division step) eliminates most of the IEEE divisions.3Track B: certified int8 sketch of the LM head with per-block error norms on the greedy path; only rows that can hold the argmax are recomputed with the BF16 head, so the logits stay bit-identical to track A.4
  1. 1Attention V accumulation re-partitioned so each warp owns one head-dim chunk across key groups (partials in 8 KB instead of 64 KB of shared memory), softmax weights passed as MMA B operands; router top-6 for all sequences in parallel; LM head over all 152 SMs. 3.00 → 1.90 ms1重新划分 attention 的 V 累加:每个 warp 负责一个 head-dim 分块、跨所有 key group(部分和占 8 KB 而不是 64 KB 的 shared memory),softmax 权重直接作为 MMA 的 B 操作数;所有序列的 router top-6 并行计算;LM head 分到全部 152 个 SM 上。3.00 → 1.90 ms
  2. 2Shared-memory carve-out per thread block cut from 224 KB to 128 KB (V no longer staged, K staging from 20 to 12 key groups); L1 and shared memory share one pool, and a batch of 16 weight-tile loads takes 0.76 µs instead of 2.27 µs. 1.90 → 1.68 ms2每个 thread block 的 shared memory carve-out 从 224 KB 减到 128 KB(V 不再暂存,K 的暂存从 20 个 key group 减到 12 个)。L1 和 shared memory 共用一块存储,16 个权重 tile 一批 load 的时间从 2.27 µs 降到 0.76 µs。1.90 → 1.68 ms
  3. 3Softmax normalization: one reciprocal per thread plus an exact FMA correction (Markstein's division step) eliminates most of the IEEE divisions. 1.61 → 1.55 ms3softmax 归一化:每个 thread 只求一次倒数,再用一次 FMA 做精确修正(Markstein 除法步),省掉了大部分 IEEE 除法。1.61 → 1.55 ms
  4. 4Track B: certified int8 sketch of the LM head with per-block error norms on the greedy path; only rows that can hold the argmax are recomputed with the BF16 head, so the logits stay bit-identical to track A. 1.47 → 1.40 ms4Track B:greedy 路径上用带 per-block 误差范数的 certified int8 LM-head sketch,只有可能包含 argmax 的行才用 BF16 head 重算,logits 与 track A 保持逐位相同。1.47 → 1.40 ms
Qwen3.8-27B · B200track A16.99 → 8.44 ms in 29 stepstrack B10.69 → 7.17 ms in 35 stepsOpus 5.5 24 h · Opus 5.5 6 h · Fable 5.2 7 h · Opus 5.5 26 h810120204060agent-hoursms per tokens5s9s11SGLang 9.52A 8.44B 7.17Three measured costs removed: per-thread stack traffic for operator arguments (5.2 → 0.8 µs per operator), the shared-memory copy of activations before every GEMM (now L1 loads), and per-tile call overhead (one streaming loop per warp).1Split-K tile hand-off without 64-bit integer division: partition boundaries written to shared memory before streaming, removing 1.5–2.2 µs of integer work per hand-off.2Exact 12-bit weights, one 15-exponent window per output row, out-of-window weights patched into the MMA operand registers: every logit bit-identical to BF16 arithmetic.3
  1. 1Three measured costs removed: per-thread stack traffic for operator arguments (5.2 → 0.8 µs per operator), the shared-memory copy of activations before every GEMM (now L1 loads), and per-tile call overhead (one streaming loop per warp). 16.99 → 10.83 ms1去掉三处实测的开销:算子参数在每个 thread stack 上的读写(每个算子 5.2 → 0.8 µs)、每次 GEMM 前把 activation 拷进 shared memory(改为经 L1 读取)、以及每个 tile 一次函数调用的开销(每个 warp 一个 streaming 循环)。16.99 → 10.83 ms
  2. 2Split-K tile hand-off without 64-bit integer division: partition boundaries written to shared memory before streaming, removing 1.5–2.2 µs of integer work per hand-off. 9.87 → 9.50 ms2split-K tile 交接不再做 64-bit 整数除法:分区边界在 streaming 之前写进 shared memory,每次交接省去 1.5–2.2 µs 的整数运算。9.87 → 9.50 ms
  3. 3Exact 12-bit weights, one 15-exponent window per output row, out-of-window weights patched into the MMA operand registers: every logit bit-identical to BF16 arithmetic. 10.69 → 8.67 ms3精确的 12-bit 权重:每个输出行一个 15-exponent 窗口,窗口外的权重直接补进 MMA 操作数寄存器,每个 logit 都与 BF16 算术逐位相同。10.69 → 8.67 ms
Inkling-2-Small · 4 × B200Opus 5.5 24 h · Fable 5.2 12 h (A), 15 h (B)345605101520agent-hoursms per tokenSGLang 4.71A, Opus 3.62A, Fable 3.3B, Fable 3.29768-thread blocks (80 registers, half the spills), contiguous GEMV ranges per block, one block per strip for qkv, o_proj and w2, 8-wide vectorized residual updates.1Tensor-parallel partial sums of o_proj and w2 exchanged across the four GPUs in BF16 instead of FP32 (summed in FP32): halves the NVLink transfer bytes.2Shared-memory carve-out per thread block cut from 222 KB to 121 KB, returning the difference to L1 (the two share 256 KB per SM): the FP4 GEMV runs about 15% faster.3Output-gate projection (40% of the attention-input GEMV's bytes) computed during the attention phase by thread blocks that are idle in it.4Track B: LM head in the exact 12-bit code, and layer 0's projections precomputed per token id at load time.5Track B: the same two changes applied to the final track-A kernel.6
  1. 1768-thread blocks (80 registers, half the spills), contiguous GEMV ranges per block, one block per strip for qkv, o_proj and w2, 8-wide vectorized residual updates. 6.56 → 5.32 ms1768-thread 的 block(80 个寄存器,spill 减半),每个 block 的 GEMV 范围连续,qkv、o_proj 和 w2 每个 strip 一个 block,residual 更新改为 8 路向量化。6.56 → 5.32 ms
  2. 2Tensor-parallel partial sums of o_proj and w2 exchanged across the four GPUs in BF16 instead of FP32 (summed in FP32): halves the NVLink transfer bytes. 3.96 → 3.62 ms2o_proj 和 w2 的 tensor-parallel 部分和在四块 GPU 之间以 BF16 而非 FP32 交换(仍以 FP32 求和):NVLink 传输的字节减半。3.96 → 3.62 ms
  3. 3Shared-memory carve-out per thread block cut from 222 KB to 121 KB, returning the difference to L1 (the two share 256 KB per SM): the FP4 GEMV runs about 15% faster. 3.89 → 3.65 ms3每个 thread block 的 shared memory carve-out 从 222 KB 减到 121 KB,差额还给 L1(两者共用每个 SM 的 256 KB):FP4 GEMV 快约 15%。3.89 → 3.65 ms
  4. 4Output-gate projection (40% of the attention-input GEMV's bytes) computed during the attention phase by thread blocks that are idle in it. 3.40 → 3.31 ms4output-gate projection(占 attention 输入 GEMV 字节的 40%)挪到 attention 阶段,由这一阶段空闲的 thread block 计算。3.40 → 3.31 ms
  5. 5Track B: LM head in the exact 12-bit code, and layer 0's projections precomputed per token id at load time. 4.05 → 3.99 ms5Track B:LM head 用精确的 12-bit 编码;第 0 层的 projection 在 load 时按 token id 预先算好。4.05 → 3.99 ms
  6. 6Track B: the same two changes applied to the final track-A kernel. 3.97 → 3.30 ms6Track B:把同样两处改动用到最终的 track-A kernel 上。3.97 → 3.30 ms
The best valid submission so far, in milliseconds per token with one sequence, as scored by the run's evaluator; gray lines are the general engines, and in the shaded area ours is faster. Each numbered submission is a step of at least 3% (all of them for Inkling-2-Small), listed next to its panel with what it changed and the time per token before and after (against the best earlier submission on the same track); under each title, each track's first valid time and its best, and in how many steps (new bests) it got there. A dotted line marks where a later run continued the same engine. The other four models' panels are on the data page.到当时为止最好的有效 submission,单序列每 token 毫秒数,按 run 自己的 evaluator 计分;灰线是通用 engine,阴影区域内我们的更快。编号的 submission 都是至少 3% 的一步(Inkling-2-Small 列出全部),列在对应面板旁,写明改了什么,以及前后的每 token 时间(与同一 track 上之前最好的 submission 比较);每个标题下写的是各 track 第一个有效时间和最好成绩,以及经过多少步(新的最好成绩)达到。虚线标出后续 run 接着优化同一个 engine 的位置。另外四个模型的面板见 数据页。

The runs follow a common trajectory. The first engine to pass the tests is slower than the fastest general engine. The first large gain comes from fusing the whole decoding step into one kernel and laying the weights out in the order the tensor cores consume them. Most later work targets the memory system: which loads go through which cache, what is prefetched in which phase, and what idle thread blocks do while attention runs. The engines overtake the general engines within about three hours on Qwen3-8B, North Mini Code and flint, and after seven and nineteen hours on Inkling-2-Small and Qwen3.8-27B. On Inkling-2-Small, a Fable 5.2 run starting from the Opus 5.5 run's engine at 5.5 hours surpassed, within four hours, what the Opus run reached in 24.各个 run 的轨迹相似。第一个通过测试的 engine 比最快的通用 engine 慢。第一次大的提升来自把整个 decoding step 融合进一个 kernel,并把权重按 tensor core 读取的顺序排布。之后的大部分工作针对内存系统:哪些 load 走哪一级 cache,哪个阶段预取什么,attention 运行时空闲的 thread block 做什么。在 Qwen3-8B、North Mini Code 和 flint 上,engine 大约三小时内就超过了通用 engine;在 Inkling-2-Small 和 Qwen3.8-27B 上分别用了七小时和十九小时。在 Inkling-2-Small 上,一个 Fable 5.2 run 从 Opus 5.5 run 第 5.5 小时的 engine 出发,四小时内就超过了 Opus run 用 24 小时达到的水平。

Limits and next局限与下一步

The steady-decode engines are narrow: at most 8 sequences and a 2,048-token context, the prompt consumed one token per pass, and sampling done outside the kernel (1 to 11% slower). Moving one setting at a time away from the benchmark on Inkling-2-Small: with 32 concurrent requests, SGLang delivers 3,338 tokens per second versus 1,427 for the steady-decode engine, which stays at its 8-sequence throughput; a 16,384-token generation or a 32,000-token prompt requires the serving engine, whose time to first token on a 32,000-token prompt is 0.70 s versus 0.41 s for SGLang. The serving engines handle long prompts and contexts, but under the production trace their tail time to first token and median time between tokens exceed SGLang's: on Inkling-2-Small at 1 to 8 requests per second, p99 time to first token is 3.4 to 4.6 s versus 2.0 to 2.2 s, and median time between tokens 28 to 60 ms versus 9 to 18 ms.steady-decode engine 的适用面很窄:最多 8 个序列和 2,048-token 的 context,每个 pass 只读入 prompt 的一个 token,sampling 在 kernel 外面做(慢 1 到 11%)。在 Inkling-2-Small 上每次只让一个设置偏离 benchmark:有 32 个并发请求时,SGLang 每秒输出 3,338 个 token,steady-decode engine 是 1,427,停在它 8 个序列时的吞吐;16,384-token 的生成或 32,000-token 的 prompt 都需要 serving engine,它在 32,000-token prompt 上的首 token 时间是 0.70 s,SGLang 是 0.41 s。serving engine 能处理长 prompt 和长 context,但在生产 trace 下,它的首 token 尾部时间和 token 间隔中位数都比 SGLang 长:Inkling-2-Small 在每秒 1 到 8 个请求时,首 token 时间的 p99 是 3.4 到 4.6 s,SGLang 是 2.0 到 2.2 s;token 间隔中位数是 28 到 60 ms,SGLang 是 9 到 18 ms。

The Qwen3-8B serving engines still match prompt prefixes by hash and rely on mkserve to reject invalid requests. All numbers come from single nodes and two to three rounds.Qwen3-8B 的 serving engine 目前仍用 hash 匹配 prompt 前缀,并依赖 mkserve 拒绝无效请求。所有数字都来自单个节点,跑了两到三轮。

Next. Anyone can run this autoresearch with their own agent on the model they care about: megakernel-zoo has the test suite and the tools to start a run on a new checkpoint, and an engine that passes the suite can be contributed back to the zoo. Over time, whenever someone wants to run a model, the fastest megakernel for it and their GPU can already be there, on demand. On our side, the next step is a serving engine whose small passes are as fast as the steady-decode kernel, so that one engine does both jobs. That engine needs the steady-decode speed while one to eight requests are decoding, prompt chunks fast enough that under load the first token and the gaps between tokens are at SGLang's level or better, and sampling inside the kernel.下一步。 任何人都可以用自己的 agent,在自己关心的模型上跑这套 autoresearch:megakernel-zoo 提供了 test suite 和在新 checkpoint 上启动 run 的工具,通过 suite 的 engine 可以贡献回 zoo。久而久之,有人想跑某个模型时,适合这个模型和他那块 GPU 的最快 megakernel 可能已经在那里,随取随用。我们这边的下一步,是做一个小 pass 和 steady-decode kernel 一样快的 serving engine,让一个 engine 同时胜任两件事。这个 engine 需要:一到八个请求在 decode 时达到 steady-decode 的速度;prompt chunk 处理得足够快,使负载下第一个 token 的时间和 token 之间的间隔不差于 SGLang;以及在 kernel 内部做 sampling。

Acknowledgements致谢

We thank Wentao Guo for sharing intuitions and a philosophy of megakernel design and for helpful discussions and suggestions, and Wenhao Chai for suggestions on the presentation.感谢 Wentao Guo 分享对 megakernel 设计的直觉和理念,以及有益的讨论和建议;感谢 Wenhao Chai 对呈现方式的建议。

References

  1. 1
  2. 2
  3. 3
    Spector, Juravsky, Sul, Dugan, Lim, Fu, Arora and Ré. Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B. Hazy Research blog, 2025.
  4. 4
    Tong, Zhu and Wang. Inside the megakernel serving engine for North Mini Code. Cohere blog, 2026.
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
    Lange et al. The AI CUDA Engineer: Agentic CUDA Kernel Discovery, Optimization and Composition. Sakana AI, 2025, and its correction of February 2025.
  12. 12
  13. 13
  14. 14
    Wolf et al. Transformers: State-of-the-Art Natural Language Processing. EMNLP 2020 (System Demonstrations).
  15. 15
  16. 16
    Dutta, Krishnan, Kwatra and Ramjee. Accuracy is Not All You Need. NeurIPS 2024.
  17. 17
    He and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Connectionism, 2025.