General inference engines such as vLLM1 and SGLang2 serve every model through one scheduler and one library of kernels. We let a coding agent write the engine for one checkpoint on one GPU instead, as a megakernel: one GPU program that runs the whole decode loop without returning to the host between tokens.vLLM1 和 SGLang2 这样的通用 inference engine 用同一个 scheduler 和同一套 kernel 库来服务所有模型。我们反过来,让一个 coding agent 为一块 GPU 上的一个 checkpoint 写专用的 engine,形式是一个 megakernel:一个 GPU 程序跑完整个 decode loop,token 之间不回到 host。
Steady decoding. With the same data flow as the reference implementation (track A), the engines are 1.13–1.77× as fast as the fastest general engine with one sequence, on all eight models from 8 B to 594 B parameters. When the storage of the model may change but its function may not, numerically (track B), the factor reaches 2.05×. Serving a production trace on Inkling-2-Small and 4 × B200, at 1 to 8 requests per second, 99–100% of requests stay within the service levels against 82–85% for SGLang, and the longest stall is 0.08–0.15 s against 2.3–4.4 s.Steady decoding。 在数据流与参考实现相同的前提下(track A),单序列时 engine 的速度是最快通用 engine 的 1.13–1.77 倍,覆盖从 8 B 到 594 B 参数的全部八个模型。如果允许改变模型的存储方式、但不改变它在数值上的函数(track B),倍数可达 2.05×。在生产 trace 上做 serving:Inkling-2-Small 跑在 4 × B200 上,每秒 1 到 8 个请求时,99–100% 的请求满足 service level,SGLang 是 82–85%;最长卡顿是 0.08–0.15 s,SGLang 是 2.3–4.4 s。
Validation. We built a framework in which agents design megakernels on their own. Its core is a test suite the agent cannot change: an engine is timed only after it agrees with a high-precision reference in its numbers, on inputs it has never seen and while it actually runs, and after an independent audit. We ran it with Claude Opus 5.5 and Fable 5.2. The engines, a server that puts them behind the OpenAI API, the test suite and the tools to run it on another model are in megakernel-zoo.验证。 我们搭了一个让 agent 自主设计 megakernel 的框架,核心是一套 agent 改不了的 test suite:engine 必须在数值上与高精度 reference 一致——包括它从没见过的输入、包括它真正运行时的输出——并通过独立审计之后,才会被计时。我们用 Claude Opus 5.5 和 Fable 5.2 跑了这套框架。engine、把它们包装成 OpenAI API 的 server、test suite,以及在别的模型上复现的工具,都在 megakernel-zoo 里。
Recorded answers, replayed at the speed they arrived录下来的回答,按到达时的速度回放
mkserve, the server in the repository. Its first token arrives later because the steady-decode engine consumes the prompt one token per pass.两个 server 先后跑在同一块 GPU 上,通过 streaming chat API 收到同一个问题;文字按记录下来的到达时间出现。我们的 engine 由 mkserve 提供服务,即 repository 里的 server。它的第一个 token 来得更晚,因为 steady-decode engine 每个 pass 只读入 prompt 的一个 token。A megakernel per model每个模型一个 megakernel
vLLM and SGLang are designed to run any model. Each decoding step is assembled from a library of general-purpose kernels, one per operation (attention, matrix multiplication, normalization), which run one after another from a CUDA graph, while the CPU schedules every step. We take the other extreme: an agent writes a single kernel for one checkpoint on one GPU type, and that kernel executes every operation of every step, and the loop over tokens, by itself.vLLM 和 SGLang 的设计目标是能跑任何模型。每个 decoding step 由一个通用 kernel 库拼成,每种运算一个 kernel(attention、矩阵乘、normalization),这些 kernel 从一个 CUDA graph 里依次执行,由 CPU 调度每一个 step。我们走另一个极端:由 agent 为一种 GPU 上的一个 checkpoint 写一个 kernel,这个 kernel 自己执行每个 step 的所有运算,以及对 token 的循环。
Where the decode loop runsdecode loop 在哪里运行
With the whole decoding step in one kernel, the agent can schedule operators freely: prefetch the next layer's weights while the current layer computes, fuse across operator boundaries without round-tripping intermediates through memory, and start idle thread blocks on the next phase early. Separate kernel launches make each of these hard.整个 decoding step 都在一个 kernel 里,agent 就能自由地调度算子:当前层计算时预取下一层的权重,跨算子边界融合而不必让中间结果经过显存往返,让空闲的 thread block 提前开始下一阶段。拆成多次 kernel launch 时,这些都很难做到。
The price is why megakernels are rare: they are hard to maintain and debug, they are tied to one model and one kind of GPU, and they need retuning when the batch or context changes, so until now they have been written by hand for a few models (Hazy Research's for Llama-1B3, Cohere's for North Mini Code4) or compiled for a set of supported architectures (MPK5). Coding agents now have the capability to do this work, so that cost is gone.代价正是 megakernel 少见的原因:它难维护、难调试,绑定在一个模型和一种 GPU 上,batch 或 context 一变就要重新调优,所以至今只有少数几个模型有手写版本(Hazy Research 为 Llama-1B 写的3、Cohere 为 North Mini Code 写的4),或者由编译器为一组支持的架构生成(MPK5)。现在 coding agent 有 capability 来完成这个事情,所以没有这个成本了。
We report two tracks.我们分两个 track 报告。
- Track A keeps the data flow of the reference implementation unchanged: weights in the checkpoint's own formats, activations and KV cache in the formats the general engines use; only how the computation is organized on the GPU changes.Track A 要求数据流与参考实现保持一致:权重保持 checkpoint 自己的格式,activation 和 KV cache 用通用 engine 的格式;改变的只是计算在 GPU 上如何组织。
- Track B is looser: it only requires the function to be unchanged in exact arithmetic, and lets the agent change how the model is stored and what is computed at load time. Lossy quantization or pruning does not qualify, since it changes the function.Track B 更宽松,只要求函数在精确算术下不变,允许 agent 改变模型的存储方式和 load 时预先计算的内容。有损的 quantization 或 pruning 不算,因为它们改变了函数。
We evaluate eight models from 8 B to 594 B parameters, dense and mixture-of-experts, on H100 and B200 GPUs, against vLLM and SGLang and, for North Mini Code, Cohere's megakernel. Versions and flags are on the data page.我们评测了八个模型,从 8 B 到 594 B 参数,有 dense 也有 mixture-of-experts,跑在 H100 和 B200 GPU 上,对比 vLLM 和 SGLang;North Mini Code 还对比了 Cohere 的 megakernel。版本和参数见 数据页。
Steady decodingSteady decoding
We use the standard decode benchmark: a 64-token prompt, 1,024 generated tokens, greedy decoding, batch size 1 or 8, every system served over HTTP on the same GPU.我们用标准的 decode benchmark:64-token 的 prompt,生成 1,024 个 token,greedy decoding,batch size 为 1 或 8,所有系统都通过 HTTP、在同一块 GPU 上提供服务。
Figure 1. Speed relative to SGLang on the same GPU图 1. 相对同一块 GPU 上 SGLang 的速度
At batch size 1, track A is 1.13 to 1.77 times as fast as the fastest general engine on every model. At batch size 8 the speedups are smaller on six of the eight models, lowest on North Mini Code (1.02×) and gpt-oss-120b (1.08×). On North Mini Code the fastest general engine is Cohere's megakernel; against vLLM, track A is 1.99× and 1.37×. On the engines also timed with prose and code prompts and with sampling, the speedups stay within 0.05 of the greedy random-token ones (data page).batch size 为 1 时,track A 在每个模型上都是最快通用 engine 的 1.13 到 1.77 倍。batch size 为 8 时,八个模型中有六个的加速比更小,最低的是 North Mini Code(1.02×)和 gpt-oss-120b(1.08×)。North Mini Code 上最快的通用 engine 是 Cohere 的 megakernel;对比 vLLM,track A 是 1.99× 和 1.37×。另外用散文和代码 prompt、以及 sampling 测过的 engine,加速比与 greedy 随机 token 的结果相差不超过 0.05(数据页)。
Batch-1 decoding is bound by HBM bandwidth. On Qwen3-8B and an H100, reading the 15.1 GB of BF16 weights once at peak bandwidth (3.35 TB/s) takes 4.52 ms; the track-A engine runs at 5.13 ms per token, 88% of that bound.batch size 为 1 的 decoding 受 HBM 带宽限制。在 Qwen3-8B 和 H100 上,以峰值带宽(3.35 TB/s)把 15.1 GB 的 BF16 权重读一遍要 4.52 ms;track-A engine 每个 token 5.13 ms,达到这个上界的 88%。
What track B usedTrack B 用了什么
In track B, the agent can speed up the engine by observing features of this checkpoint's weights. The clearest example is the weight format: BF16 spends 8 bits on the exponent, yet the weights in one row of a trained matrix span only a few binades, a redundancy that lossless formats such as DFloat116 also exploit. The Qwen3-8B agent measured this, encoded each weight losslessly in 12 bits and decoded it in registers ahead of the tensor cores, cutting weight traffic per token by a quarter. Two other techniques recur in Table 1: a certified low-precision sketch of the LM head that shortlists the next token, and per-token lookup tables for layer 0, whose input depends only on the token's embedding. Concurrent work7 found the same format: SpecPilot, an agentic loop that also runs Claude Opus 5.5 in Claude Code, optimizes speculative decoding for each deployment and lists this lossless 12-bit format among its discoveries. Our Claude Opus 5.5 agent found it independently, inside a full decode megakernel. Later runs adapted the method to Qwen3.8-27B and Qwen3-32B, where every product is bit-identical to BF16 arithmetic, and a Claude Fable 5.2 run applied it to the LM head of Inkling-2-Small.在 track B 中,agent 可以通过观察这个 checkpoint 的 weight 的一些 feature 来 speedup。最明显的例子是权重格式:BF16 用 8 个 bit 表示 exponent,但训练好的矩阵里同一行的权重只跨几个 binade,DFloat116 这类无损格式利用的也是这种冗余。Qwen3-8B 上的 agent 测出了这一点,把每个权重无损地编码成 12 bit,并在送往 tensor core 之前于寄存器内解码,每个 token 的权重流量减少四分之一。表 1 里还反复出现两种技术:LM head 的 certified 低精度 sketch,用来筛出下一个 token 的候选;以及第 0 层的 per-token 查找表,因为这一层的输入只取决于 token 的 embedding。同期工作7发现了同样的格式:SpecPilot 是一个同样在 Claude Code 里运行 Claude Opus 5.5 的 agentic loop,它为每个部署优化 speculative decoding,并把这种无损 12-bit 格式列为它的发现之一。我们的 Claude Opus 5.5 agent 也在完整的 decode megakernel 内部独立发现了它。后面的 run 把这个 method adapt 到了 Qwen3.8-27B 和 Qwen3-32B 上,结果上每个乘积都与 BF16 算术逐位相同;一个 Claude Fable 5.2 的 run 也把它用到了 Inkling-2-Small 的 LM head 上。
Table 1. Track B on each model表 1. 各模型上的 track B
| Model模型 | Effect in the runrun 中测得的效果 | Logits against track A与 track A 的 logits 对比 |
|---|---|---|
| flintflint | −2.9% (head sketch)−2.9%(head sketch) | bit-identical逐位相同 |
| Qwen3.8-27BQwen3.8-27B | −9% (12-bit weights), then −5.6%−9%(12-bit 权重),之后 −5.6% | bit-identical逐位相同 |
| Qwen3-32BQwen3-32B | −11% (10.12 → 9.00 ms), 12-bit weights−11%(10.12 → 9.00 ms),12-bit 权重 | bit-identical逐位相同 |
| gpt-oss-120bgpt-oss-120b | −5% (head sketch)−5%(head sketch) | bit-identical逐位相同 |
| Inkling-2-SmallInkling-2-Small | −0.4% | bit-identical逐位相同 |
| Qwen3-8BQwen3-8B | −15% (5.34 → 4.52 ms), 12-bit weights−15%(5.34 → 4.52 ms),12-bit 权重 | not bit-identical: the residual stream is kept in FP32 (BF16 in track A), so it is more precise非逐位相同:residual stream 保持 FP32(track A 是 BF16),因此精度更高 |
| North Mini CodeNorth Mini Code | −10% (12-bit weights), −4% (head sketch)−10%(12-bit 权重),−4%(head sketch) | not bit-identical: the rare weights outside the 12-bit range are added in a separate pass, same values in a different rounding order非逐位相同:少数超出 12-bit 范围的权重单独补算,数值相同,舍入顺序不同 |
| gpt-oss-20bgpt-oss-20b | −14% (1.87 → 1.61 ms)−14%(1.87 → 1.61 ms) | not bit-identical: the router weights are pre-multiplied by the norm weights in FP32, so the normalized input is never rounded to BF16 and the router logits differ in the last bits非逐位相同:router 权重预先乘上 norm 权重并以 FP32 存储,归一化后的输入不再舍入到 BF16,router logits 在最后几位不同 |
Where the checkpoint is already compressed (the 4-bit experts of Inkling-2-Small and gpt-oss), track B gains little.如果 checkpoint 本身已经压缩过(Inkling-2-Small 和 gpt-oss 的 4-bit expert),track B 的收益很小。
Serving live traffic服务真实流量
The engines above decode a batch whose sequences start together; a server must admit new requests while others are decoding. For Qwen3-8B and Inkling-2-Small, a second run moved scheduling onto the GPU: continuous batching, chunked prefill inside the kernel, a paged KV cache and prefix caching.上面的 engine 解码的是同时开始的一个 batch;而 server 必须在其他请求 decode 的同时接收新请求。对 Qwen3-8B 和 Inkling-2-Small,第二个 run 把调度搬到了 GPU 上:continuous batching、kernel 内的 chunked prefill、paged KV cache 和 prefix caching。
The serving engine: the scheduling loop runs on the GPUserving engine:调度循环在 GPU 上运行
mkserve is a thin host-side wrapper that writes requests into a ring buffer in host-mapped memory and reads tokens from another.serving loop 在 GPU 上运行:一个 CUDA graph 负责接收请求、跑一个 pass、把 token 写回。mkserve 只是一层很薄的 host 端包装,把请求写进 host-mapped memory 里的一个 ring buffer,再从另一个 ring buffer 读出 token。We replay the Mooncake tool-and-agent trace8 (mean prompt length about 9,000 tokens), with every system served over HTTP on the same node.我们回放 Mooncake 的 tool-and-agent trace8(prompt 平均约 9,000 个 token),所有系统都通过 HTTP 在同一个节点上提供服务。
Figure 2. Inkling-2-Small on 4 × B200 under a production request trace图 2. Inkling-2-Small 在 4 × B200 上处理生产请求 trace
At 1, 2 and 4 requests per second our engine reaches 100% SLO attainment, and 99.2% at 8; SGLang reaches 82 to 85% (a request meets its SLOs when its time to first token is at most 5 s and its mean time between tokens at most 0.1 s; attainment counted as in DistServe9). The gap comes from stalls: when a long prompt arrives, SGLang's decoding requests wait behind its prefill for up to 2.3 to 4.4 s; our megakernel's longest stall is 0.08 to 0.15 s.每秒 1、2 和 4 个请求时,我们的 engine 的 SLO 达成率是 100%,每秒 8 个时是 99.2%;SGLang 是 82 到 85%(首 token 时间不超过 5 s、token 间平均间隔不超过 0.1 s 即算满足,达成率按 DistServe9 的方式统计)。差距来自卡顿:长 prompt 到来时,SGLang 中正在 decode 的请求要排在它的 prefill 后面,最长等 2.3 到 4.4 s;我们的 megakernel 最长的卡顿是 0.08 到 0.15 s。
On Qwen3-8B and 1 × H100 the pattern is the same: SLO attainment is 95 to 97% versus 68 to 75% for SGLang and vLLM at 1 and 2 requests per second, and 90 to 93% versus 59 to 68% at 3; the longest stall is 0.09 to 0.13 s.在 Qwen3-8B 和 1 × H100 上规律相同:每秒 1 和 2 个请求时,SLO 达成率是 95 到 97%,SGLang 和 vLLM 是 68 到 75%;每秒 3 个时是 90 到 93%,对方是 59 到 68%;最长卡顿是 0.09 到 0.13 s。
How we validate an engine我们如何验证一个 engine
For autoresearch the test suite matters more than anything else: the agent optimizes whatever the evaluator rewards, so every gap in the evaluator becomes a shortcut. Agent-written kernels have passed tests by reusing the reference's output memory, by special-casing the test inputs and by timing the wrong stream101112, and open kernel competitions now routinely filter out reward-hacked submissions13. The agent cannot modify the suite, and an engine is timed only after it passes.对 autoresearch 来说,test suite 比什么都重要:agent 优化的是 evaluator 奖励的东西,evaluator 的每个漏洞都会变成捷径。agent 写的 kernel 曾经靠复用 reference 的输出显存、针对测试输入写特例、计时测错 stream 而通过测试101112,公开的 kernel 比赛现在也会例行过滤 reward hacking 的 submission13。agent 无法修改这套 suite,engine 通过之后才会被计时。
Reference and tolerances. The reference is the model evaluated in FP64 (FP32 where FP64 does not fit in memory). Tolerances are calibrated on the standard BF16 implementation (Hugging Face Transformers14): we measure its deviation from the reference at every position and allow the engine twice that on the median and the 99th percentile of the logit error, as FlashAttention's own tests do15; four times that on the KL divergence of the next-token distribution, which tracks how many answers a change flips better than accuracy does16; and top-1 agreement within one point. vLLM and SGLang, measured the same way, deviate from the reference about as much as the BF16 implementation, so the tolerance is the error a general engine already has.Reference 与容差。 reference 是用 FP64 计算的模型(显存放不下时用 FP32)。容差以标准的 BF16 实现(Hugging Face Transformers14)为基准:我们在每个位置测量它与 reference 的偏差,logit 误差的中位数和 99th percentile 允许 engine 达到它的两倍,与 FlashAttention 自己的测试做法相同15;next-token 分布的 KL divergence 允许四倍——它比 accuracy 更能反映一个改动会让多少答案翻转16;top-1 agreement 与它相差不超过一个百分点。按同样方式测量,vLLM 和 SGLang 与 reference 的偏差和 BF16 实现相当,所以这个容差就是通用 engine 本来就有的误差。
Inputs. A public set covers every position up to 128 and every eighth position up to 1,152. A hidden set of natural text, code and math word problems extends to position 2,000; the run never sees it and only learns whether it passed.输入。 公开集覆盖 128 以内的每个位置,以及到 1,152 为止每隔八个的位置。一个由自然文本、代码和数学应用题组成的 hidden set 一直延伸到位置 2,000;run 看不到它,只知道自己是否通过。
The engine as it runs. The engine's own decode loop must reproduce its step-by-step greedy tokens; the logits produced during the timed run are checked afterwards; and rerunning the batch with the sequences in reverse order must give bit-identical logits (in general engines a sequence's logits can depend on the batch it runs in17). Before release, each engine is also tested at every batch size, at full context and after a restart.运行中的 engine。 engine 自己的 decode loop 必须复现它逐步 greedy 生成的 token;计时运行中产生的 logits 会在事后检查;把 batch 里的序列倒序后重跑,logits 必须逐位相同(在通用 engine 里,一个序列的 logits 可能取决于它所在的 batch17)。发布前,每个 engine 还要在每个 batch size、满 context 以及重启之后各测一遍。
Table 2. What an engine has to pass表 2. engine 必须通过的检查
| Check检查 | Compared with对比对象 | Passes when通过条件 | Runs运行时机 |
|---|---|---|---|
| Logits at every position每个位置的 logits | the model in FP64 (FP32 if it does not fit), on public and hidden sequencesFP64 计算的模型(放不下时用 FP32),公开序列和 hidden 序列 | median and 99th-percentile relative error ≤ 2× the BF16 implementation's相对误差的中位数和 99th percentile ≤ BF16 实现的 2× | every submission每次 submission |
| Distribution分布 | the same reference同一个 reference | KL divergence median and p99 ≤ 4× the BF16 implementation's; top-1 agreement within one point of itKL divergence 的中位数和 p99 ≤ BF16 实现的 4×;top-1 agreement 与它相差不超过一个百分点 | every submission每次 submission |
| Hidden setHidden set | natural text, code and word problems to position 2,000, never shown to the run自然文本、代码和应用题,到位置 2,000,从不给 run 看 | all of the above; only pass or fail is reported以上全部;只报告通过或失败 | every submission每次 submission |
| As it runs运行中 | its own step-by-step greedy tokens它自己逐步 greedy 生成的 token | its decode loop gives the same tokens; logits taken during the timed run match the referencedecode loop 给出相同的 token;计时运行中取出的 logits 与 reference 吻合 | every submission每次 submission |
| Slot order and repetitionSlot 顺序与重复 | the same batch with the sequences reversed序列倒序的同一个 batch | logits bit-identicallogits 逐位相同 | every submission每次 submission |
| Batch sizes, full context, restartbatch size、满 context、重启 | fresh engines新启动的 engine | 1 to 8 sequences within the limits; bit-identical after a full context and after a restart1 到 8 个序列都在阈值内;满 context 之后和重启之后逐位相同 | before release发布前 |
| Audit审计 | an independent agent reading the source一个独立的 agent 阅读源码 | no rule broken; its own experiments reproduce the scores没有违反任何规则;它自己的实验能复现分数 | before release发布前 |
Every engine shown in this post except the three Inkling-2-Small engines was re-tested with the current suite: 20 of 20 pass every check.除三个 Inkling-2-Small engine 外,本文展示的每个 engine 都用当前的 suite 重新测过:20 个中有 20 个通过全部检查。
Audits. Every final engine was reviewed by an independent agent that read the source for rule violations and ran its own experiments. No engine calls another engine, caches outputs, special-cases the evaluator's inputs or skips part of the model. The audits did find problems that the evaluator of the time had passed; each led to a fix in the engine, in what we report, or in the suite.审计。 每个最终 engine 都由一个独立的 agent 审查:它阅读源码寻找违规之处,并跑自己的实验。没有 engine 调用别的 engine、缓存输出、针对 evaluator 的输入写特例或跳过模型的某一部分。但审计确实发现了当时的 evaluator 放过的问题;每一个都促成了一处修正:在 engine 里、在我们报告的内容里,或在 suite 里。
Table 3. What the audits found that the evaluator had passed表 3. 审计发现、而当时 evaluator 放过的问题
| Finding发现 | Effect影响 | What was done处理 | Caught now by现在由什么检查拦住 |
|---|---|---|---|
| Two of the three timed prompts drive Qwen3.8-27B into a repetition loop that the drafts predict三个计时 prompt 中有两个会让 Qwen3.8-27B 陷入 draft 能预测的重复循环 | multi-token decoding scored 0.415 ms per token, 3.71 ms on natural textmulti-token decoding 得分 0.415 ms/token,在自然文本上是 3.71 ms | the natural-text figure is the one reported报告的是自然文本上的数字 | hidden natural-text set; natural-text timing自然文本 hidden set;自然文本计时 |
| Three track-B engines labeled bit-identical were not三个标为逐位相同的 track-B engine 其实不是 | flint's engine picked a different top-1 token at 10.8% of positionsflint 的 engine 在 10.8% 的位置选了不同的 top-1 token | flint re-run, now bit-identical; the other two relabeledflint 重新跑过,现在逐位相同;另外两个改了标注 | bit-for-bit check against track A与 track A 的逐位比对 |
| No error past the context limit超过 context 上限时不报错 | Inkling-2-Small overwrote other sequences' KV cache; flint returned empty outputInkling-2-Small 覆盖了其他序列的 KV cache;flint 返回空输出 | both runs extended; the engines refuse such requests两个 run 都延长了;engine 现在拒绝这类请求 | requests at and past the limit达到和超过上限的请求 |
| A check added after the first Qwen3-8B run第一个 Qwen3-8B run 之后加的一项检查 | one of 1,024 positions over the tolerance: attention scores were rounded to BF16 twice1,024 个位置中有一个超出容差:attention score 被两次舍入到 BF16 | a 3-hour run fixed it at the same speed一个 3 小时的 run 修好了它,速度不变 | median and p99 instead of the maximum用中位数和 p99 代替最大值 |
| Serving engines did not validate their inputsserving engine 不校验输入 | prefixes matched by a 64-bit hash without comparing tokens; no bounds checks前缀用 64-bit hash 匹配、不比较 token;没有越界检查 | Inkling-2-Small fixed in an extended run; Qwen3-8B being fixedInkling-2-Small 在延长的 run 中修好;Qwen3-8B 正在修 | mkserve input checks; collision testmkserve 输入检查;hash 碰撞测试 |
| Qwen3-8B track-A serving under heavy load高负载下的 Qwen3-8B track-A serving | tokens differed from the same request run alone in 12 of 17 trials17 次试验中有 12 次,token 与同一请求单独运行时不同 | under investigation with the fix above与上面的修复一起排查中 | slot-order and load testsslot 顺序测试和负载测试 |
Timing.计时。
- All systems of a comparison run in the same job on the same GPU, one after another.同一组对比的所有系统在同一个 job 里、同一块 GPU 上依次运行。
- Three rounds; each restarts every server and begins with a different system.跑三轮;每轮重启所有 server,并由不同的系统先跑。
- Every system is behind an OpenAI-compatible streaming endpoint and timed by the same client.每个系统都放在 OpenAI 兼容的 streaming endpoint 后面,由同一个 client 计时。
- Prompts are token ids, and every request generates exactly 1,024 tokens.prompt 以 token id 给出,每个请求恰好生成 1,024 个 token。
- vLLM and SGLang are released versions with their recommended flags, or a faster configuration where we found one.vLLM 和 SGLang 用发布版本和官方推荐的参数,我们找到更快配置的地方就用更快的。
- An engine is timed only after it passes the correctness checks.engine 通过正确性检查之后才会被计时。
How the agents got thereagent 是怎么做到的
A run gives Claude (Opus 5.5 or Fable 5.2, in Claude Code) the checkpoint, a GPU pod and a budget of agent-hours. The agent profiles and experiments as it sees fit and submits source code whenever it wants a score; each submission is logged with the hypothesis the agent wrote for it. The prompt states the goal, the rules and the tools, and nothing else. The engines in this post took 19 runs, 458 agent-hours and 675 submissions (every run is on the data page).一个 run 给 Claude(Opus 5.5 或 Fable 5.2,在 Claude Code 中)一个 checkpoint、一个 GPU pod,以及一份 agent-hour 预算。agent 自行决定怎么 profile、怎么做实验,想要分数时随时提交源代码;每次 submission 都连同 agent 为它写下的 hypothesis 一起记录。prompt 只说明目标、规则和工具,别的什么都不给。本文中的 engine 用了 19 个 run、458 个 agent-hour 和 675 次 submission(每个 run 都在 数据页 上)。
Figure 3. How four of the engines got faster during their runs图 3. 四个 engine 在各自 run 中如何变快
- 1The whole decode step in one kernel: its 181 operator boundaries become in-kernel grid barriers (atomic counter and polling) instead of kernel launches. 7.37 → 6.46 ms1整个 decode step 放进一个 kernel:181 个算子边界从 kernel launch 变成 kernel 内的 grid barrier(atomic counter 加轮询)。7.37 → 6.46 ms
- 2RMSNorm fused into the preceding residual-add epilogue, which emits w·x in BF16 and per-tile partial sums of x²; the next GEMV applies 1/rms to its outputs (the GEMV is linear), so no SM re-reads the FP32 residual stream. 6.14 → 5.95 ms2RMSNorm 融合进前一个 residual-add 的 epilogue:它直接输出 BF16 的 w·x 和每个 tile 的 x² 部分和,下一个 GEMV 在输出上再乘 1/rms(GEMV 是线性的),各个 SM 不必再重读 FP32 residual stream。6.14 → 5.95 ms
- 3Lossless 12-bit weights (sign, 4-bit exponent code, 7-bit mantissa; rows pre-scaled by exact powers of two) decoded in registers before the MMA, the ~10⁻⁴ of weights outside the window patched by extra MMAs: HBM bytes per step 15.1 → 11.4 GB. 5.34 → 4.52 ms3无损 12-bit 权重(符号位、4-bit 指数码、7-bit 尾数;每行先乘一个精确的 2 的幂),在 MMA 之前于寄存器内解码;约 10⁻⁴ 超出窗口的权重用额外的 MMA 补算。每步读取的 HBM 字节从 15.1 GB 降到 11.4 GB。5.34 → 4.52 ms
- 1Attention V accumulation re-partitioned so each warp owns one head-dim chunk across key groups (partials in 8 KB instead of 64 KB of shared memory), softmax weights passed as MMA B operands; router top-6 for all sequences in parallel; LM head over all 152 SMs. 3.00 → 1.90 ms1重新划分 attention 的 V 累加:每个 warp 负责一个 head-dim 分块、跨所有 key group(部分和占 8 KB 而不是 64 KB 的 shared memory),softmax 权重直接作为 MMA 的 B 操作数;所有序列的 router top-6 并行计算;LM head 分到全部 152 个 SM 上。3.00 → 1.90 ms
- 2Shared-memory carve-out per thread block cut from 224 KB to 128 KB (V no longer staged, K staging from 20 to 12 key groups); L1 and shared memory share one pool, and a batch of 16 weight-tile loads takes 0.76 µs instead of 2.27 µs. 1.90 → 1.68 ms2每个 thread block 的 shared memory carve-out 从 224 KB 减到 128 KB(V 不再暂存,K 的暂存从 20 个 key group 减到 12 个)。L1 和 shared memory 共用一块存储,16 个权重 tile 一批 load 的时间从 2.27 µs 降到 0.76 µs。1.90 → 1.68 ms
- 3Softmax normalization: one reciprocal per thread plus an exact FMA correction (Markstein's division step) eliminates most of the IEEE divisions. 1.61 → 1.55 ms3softmax 归一化:每个 thread 只求一次倒数,再用一次 FMA 做精确修正(Markstein 除法步),省掉了大部分 IEEE 除法。1.61 → 1.55 ms
- 4Track B: certified int8 sketch of the LM head with per-block error norms on the greedy path; only rows that can hold the argmax are recomputed with the BF16 head, so the logits stay bit-identical to track A. 1.47 → 1.40 ms4Track B:greedy 路径上用带 per-block 误差范数的 certified int8 LM-head sketch,只有可能包含 argmax 的行才用 BF16 head 重算,logits 与 track A 保持逐位相同。1.47 → 1.40 ms
- 1Three measured costs removed: per-thread stack traffic for operator arguments (5.2 → 0.8 µs per operator), the shared-memory copy of activations before every GEMM (now L1 loads), and per-tile call overhead (one streaming loop per warp). 16.99 → 10.83 ms1去掉三处实测的开销:算子参数在每个 thread stack 上的读写(每个算子 5.2 → 0.8 µs)、每次 GEMM 前把 activation 拷进 shared memory(改为经 L1 读取)、以及每个 tile 一次函数调用的开销(每个 warp 一个 streaming 循环)。16.99 → 10.83 ms
- 2Split-K tile hand-off without 64-bit integer division: partition boundaries written to shared memory before streaming, removing 1.5–2.2 µs of integer work per hand-off. 9.87 → 9.50 ms2split-K tile 交接不再做 64-bit 整数除法:分区边界在 streaming 之前写进 shared memory,每次交接省去 1.5–2.2 µs 的整数运算。9.87 → 9.50 ms
- 3Exact 12-bit weights, one 15-exponent window per output row, out-of-window weights patched into the MMA operand registers: every logit bit-identical to BF16 arithmetic. 10.69 → 8.67 ms3精确的 12-bit 权重:每个输出行一个 15-exponent 窗口,窗口外的权重直接补进 MMA 操作数寄存器,每个 logit 都与 BF16 算术逐位相同。10.69 → 8.67 ms
- 1768-thread blocks (80 registers, half the spills), contiguous GEMV ranges per block, one block per strip for qkv, o_proj and w2, 8-wide vectorized residual updates. 6.56 → 5.32 ms1768-thread 的 block(80 个寄存器,spill 减半),每个 block 的 GEMV 范围连续,qkv、o_proj 和 w2 每个 strip 一个 block,residual 更新改为 8 路向量化。6.56 → 5.32 ms
- 2Tensor-parallel partial sums of o_proj and w2 exchanged across the four GPUs in BF16 instead of FP32 (summed in FP32): halves the NVLink transfer bytes. 3.96 → 3.62 ms2o_proj 和 w2 的 tensor-parallel 部分和在四块 GPU 之间以 BF16 而非 FP32 交换(仍以 FP32 求和):NVLink 传输的字节减半。3.96 → 3.62 ms
- 3Shared-memory carve-out per thread block cut from 222 KB to 121 KB, returning the difference to L1 (the two share 256 KB per SM): the FP4 GEMV runs about 15% faster. 3.89 → 3.65 ms3每个 thread block 的 shared memory carve-out 从 222 KB 减到 121 KB,差额还给 L1(两者共用每个 SM 的 256 KB):FP4 GEMV 快约 15%。3.89 → 3.65 ms
- 4Output-gate projection (40% of the attention-input GEMV's bytes) computed during the attention phase by thread blocks that are idle in it. 3.40 → 3.31 ms4output-gate projection(占 attention 输入 GEMV 字节的 40%)挪到 attention 阶段,由这一阶段空闲的 thread block 计算。3.40 → 3.31 ms
- 5Track B: LM head in the exact 12-bit code, and layer 0's projections precomputed per token id at load time. 4.05 → 3.99 ms5Track B:LM head 用精确的 12-bit 编码;第 0 层的 projection 在 load 时按 token id 预先算好。4.05 → 3.99 ms
- 6Track B: the same two changes applied to the final track-A kernel. 3.97 → 3.30 ms6Track B:把同样两处改动用到最终的 track-A kernel 上。3.97 → 3.30 ms
The runs follow a common trajectory. The first engine to pass the tests is slower than the fastest general engine. The first large gain comes from fusing the whole decoding step into one kernel and laying the weights out in the order the tensor cores consume them. Most later work targets the memory system: which loads go through which cache, what is prefetched in which phase, and what idle thread blocks do while attention runs. The engines overtake the general engines within about three hours on Qwen3-8B, North Mini Code and flint, and after seven and nineteen hours on Inkling-2-Small and Qwen3.8-27B. On Inkling-2-Small, a Fable 5.2 run starting from the Opus 5.5 run's engine at 5.5 hours surpassed, within four hours, what the Opus run reached in 24.各个 run 的轨迹相似。第一个通过测试的 engine 比最快的通用 engine 慢。第一次大的提升来自把整个 decoding step 融合进一个 kernel,并把权重按 tensor core 读取的顺序排布。之后的大部分工作针对内存系统:哪些 load 走哪一级 cache,哪个阶段预取什么,attention 运行时空闲的 thread block 做什么。在 Qwen3-8B、North Mini Code 和 flint 上,engine 大约三小时内就超过了通用 engine;在 Inkling-2-Small 和 Qwen3.8-27B 上分别用了七小时和十九小时。在 Inkling-2-Small 上,一个 Fable 5.2 run 从 Opus 5.5 run 第 5.5 小时的 engine 出发,四小时内就超过了 Opus run 用 24 小时达到的水平。
Limits and next局限与下一步
The steady-decode engines are narrow: at most 8 sequences and a 2,048-token context, the prompt consumed one token per pass, and sampling done outside the kernel (1 to 11% slower). Moving one setting at a time away from the benchmark on Inkling-2-Small: with 32 concurrent requests, SGLang delivers 3,338 tokens per second versus 1,427 for the steady-decode engine, which stays at its 8-sequence throughput; a 16,384-token generation or a 32,000-token prompt requires the serving engine, whose time to first token on a 32,000-token prompt is 0.70 s versus 0.41 s for SGLang. The serving engines handle long prompts and contexts, but under the production trace their tail time to first token and median time between tokens exceed SGLang's: on Inkling-2-Small at 1 to 8 requests per second, p99 time to first token is 3.4 to 4.6 s versus 2.0 to 2.2 s, and median time between tokens 28 to 60 ms versus 9 to 18 ms.steady-decode engine 的适用面很窄:最多 8 个序列和 2,048-token 的 context,每个 pass 只读入 prompt 的一个 token,sampling 在 kernel 外面做(慢 1 到 11%)。在 Inkling-2-Small 上每次只让一个设置偏离 benchmark:有 32 个并发请求时,SGLang 每秒输出 3,338 个 token,steady-decode engine 是 1,427,停在它 8 个序列时的吞吐;16,384-token 的生成或 32,000-token 的 prompt 都需要 serving engine,它在 32,000-token prompt 上的首 token 时间是 0.70 s,SGLang 是 0.41 s。serving engine 能处理长 prompt 和长 context,但在生产 trace 下,它的首 token 尾部时间和 token 间隔中位数都比 SGLang 长:Inkling-2-Small 在每秒 1 到 8 个请求时,首 token 时间的 p99 是 3.4 到 4.6 s,SGLang 是 2.0 到 2.2 s;token 间隔中位数是 28 到 60 ms,SGLang 是 9 到 18 ms。
The Qwen3-8B serving engines still match prompt prefixes by hash and rely on mkserve to reject invalid requests. All numbers come from single nodes and two to three rounds.Qwen3-8B 的 serving engine 目前仍用 hash 匹配 prompt 前缀,并依赖 mkserve 拒绝无效请求。所有数字都来自单个节点,跑了两到三轮。
Next. Anyone can run this autoresearch with their own agent on the model they care about: megakernel-zoo has the test suite and the tools to start a run on a new checkpoint, and an engine that passes the suite can be contributed back to the zoo. Over time, whenever someone wants to run a model, the fastest megakernel for it and their GPU can already be there, on demand. On our side, the next step is a serving engine whose small passes are as fast as the steady-decode kernel, so that one engine does both jobs. That engine needs the steady-decode speed while one to eight requests are decoding, prompt chunks fast enough that under load the first token and the gaps between tokens are at SGLang's level or better, and sampling inside the kernel.下一步。 任何人都可以用自己的 agent,在自己关心的模型上跑这套 autoresearch:megakernel-zoo 提供了 test suite 和在新 checkpoint 上启动 run 的工具,通过 suite 的 engine 可以贡献回 zoo。久而久之,有人想跑某个模型时,适合这个模型和他那块 GPU 的最快 megakernel 可能已经在那里,随取随用。我们这边的下一步,是做一个小 pass 和 steady-decode kernel 一样快的 serving engine,让一个 engine 同时胜任两件事。这个 engine 需要:一到八个请求在 decode 时达到 steady-decode 的速度;prompt chunk 处理得足够快,使负载下第一个 token 的时间和 token 之间的间隔不差于 SGLang;以及在 kernel 内部做 sampling。
Acknowledgements致谢
We thank Wentao Guo for sharing intuitions and a philosophy of megakernel design and for helpful discussions and suggestions, and Wenhao Chai for suggestions on the presentation.感谢 Wentao Guo 分享对 megakernel 设计的直觉和理念,以及有益的讨论和建议;感谢 Wenhao Chai 对呈现方式的建议。
References
- 1Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
- 2Zheng et al. SGLang: Efficient Execution of Structured Language Model Programs. NeurIPS 2024.
- 3Spector, Juravsky, Sul, Dugan, Lim, Fu, Arora and Ré. Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B. Hazy Research blog, 2025.
- 4Tong, Zhu and Wang. Inside the megakernel serving engine for North Mini Code. Cohere blog, 2026.
- 5Cheng et al. Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programs. arXiv 2025.
- 6
- 7Li et al. SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding. arXiv 2026.
- 8Qin et al. Mooncake: Trading More Storage for Less Computation, a KVCache-centric Architecture for Serving LLM Chatbot. FAST 2025. Trace: FAST25-release/traces.
- 9Zhong et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024.
- 10Ouyang et al. KernelBench: Can LLMs Write Efficient GPU Kernels? ICML 2025.
- 11Lange et al. The AI CUDA Engineer: Agentic CUDA Kernel Discovery, Optimization and Composition. Sakana AI, 2025, and its correction of February 2025.
- 12Zhang et al. KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch? arXiv 2026.
- 13Baslak, Schultheis, Sirovatka, Zhang and Saroufim. We Let Agents Compete and They Tried to Cheat. KernelGuard: Defending GPU Competitions from Adversarial Agentic Systems. ICML 2026.
- 14Wolf et al. Transformers: State-of-the-Art Natural Language Processing. EMNLP 2020 (System Demonstrations).
- 15Dao. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. ICLR 2024. Tolerance rule in tests/test_flash_attn.py.
- 16Dutta, Krishnan, Kwatra and Ramjee. Accuracy is Not All You Need. NeurIPS 2024.
- 17He and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Connectionism, 2025.
