The measurements behind A Megakernel per Model. Code, engines and run records: megakernel-zoo. Generated from job logs on Nov 18, 2026, 20:53 Pacific time; a row names the job it was measured in. General engines are in gray. The same cells are in this page as JSON (<script id="data-tables">).
The engines of megakernel-zoo, as registered for its server mkserve. Sequences and context are the limits the engine is served with. Extra checks: logits against float64 at every batch size from 1 to 8; bit-identical under a permutation of the slots; a sequence among other neighbors; bit-identical on repetition; bit-identical after a full context and a reset; identical after a process restart (each in two processes). Wrapper checks: scripts/mkserve_check.py against the served engine. Audit: the conclusion of an independent agent that read the source and ran its own jobs.
| Model | Engine | Track | Kind | Run and submission | Seq. | Context | Extra checks | Wrapper | Audit |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | online-b (default) | B | online | s3-q8b-online-a1 · s0039 | 64 | 34,816 | 6 of 6 | 12 of 12 | no violation found; not bit-identical to track A; same tokens alone and under load (24 of 24 runs); the engine does not enforce its own limits |
| Qwen3-8B | online-a | A | online | s3-q8b-online-a1 · s0038 | 64 | 34,816 | 6 of 6 | 12 of 12 | no violation found; chunked prompts, paging and prefix reuse bit-identical to stepping; the engine does not enforce its own limits; under load, tokens can differ from the request alone at near-ties |
| Qwen3-8B | offline-b | B | offline | s11-q8b-h100-b1 · s0020 | 8 | 2,048 | – | – | not audited |
| Qwen3-8B | offline-a | A | offline | s11-q8b-h100-b1 · s0016 | 8 | 2,048 | – | – | not audited |
| flint | offline-a (default) | A | offline | s11-flint-b200-b1 · s0026 | 8 | 2,048 | – | – | not audited |
| flint | offline-b | B | offline | s11-flint-b200-b1 · s0027 | 8 | 2,048 | – | – | not audited |
| Qwen3.8-27B | offline-a (default) | A | offline | s9-qwen38-27b-b200-f1 · s0022 | 8 | 2,048 | 6 of 6 | 12 of 12 | no violation found; logits bit-identical to the engine audited first at 18,700 positions; RMSNorm scale applied at a different rounding point than Hugging Face; the engine does not check invalid requests |
| Qwen3.8-27B | offline-b | B | offline | s11-q27-b200-b1 · s0020 | 8 | 2,048 | – | – | not audited |
| North Mini Code 1.0 | offline-a (default) | A | offline | s4-north-h100-a1 · s0022 | 8 | 2,048 | 6 of 6 | 12 of 12 | no violation found |
| North Mini Code 1.0 | offline-b | B | offline | s4-north-h100-a1 · s0021 | 8 | 2,048 | 6 of 6 | 12 of 12 | no violation found; outputs not bit-identical to track A |
| Inkling-2-Small | offline-a (default) | A | offline | s7-inkling2s-b200x4-fb1 · s0034 | 8 | 2,048 | 6 of 6 | 12 of 12 | no violation found; logits bit-identical to the audited kernel, with the context limit now enforced |
| Inkling-2-Small | offline-b | B | offline | s7-inkling2s-b200x4-fb1 · s0036 | 8 | 2,048 | 6 of 6 | 12 of 12 | no violation found; logits bit-identical to track A at every position checked |
| Inkling-2-Small | online-a | A | online | s8-inkling2s-b200x4-on1 · s0043 | 64 | 34,816 | 6 of 6 | 12 of 12 | no violation found; the model's arithmetic is that of the engine audited first (same tokens); the first audit's defects are fixed: no out-of-bounds write at small batch, prefixes compared token by token, requests beyond the limits refused; 130 concurrent requests give the tokens of each request alone |
| gpt-oss-20b | offline-a (default) | A | offline | s12-oss20-h100-a1 · s0047 | 8 | 2,048 | – | – | not audited |
| gpt-oss-20b | offline-b | B | offline | s12-oss20-h100-a1 · s0050 | 8 | 2,048 | – | – | not audited |
| Qwen3-32B | offline-a (default) | A | offline | s12-q32-b200-a1 · s0033 | 8 | 2,048 | – | – | not audited |
| Qwen3-32B | offline-b | B | offline | s12-q32-b200-a1 · s0036 | 8 | 2,048 | – | – | not audited |
| gpt-oss-120b | offline-a (default) | A | offline | s12-oss120-b200-a1 · s0049 | 8 | 2,048 | – | – | not audited |
| gpt-oss-120b | offline-b | B | offline | s12-oss120-b200-a1 · s0051 | 8 | 2,048 | – | – | not audited |
64-token prompt, 1,024 generated tokens, greedy, every system behind streaming HTTP and timed by the same client; all systems of a table ran in the job named, one after another, three rounds. Time per token is taken over the window in which all sequences decode; the range is the lowest and highest round. × is the fastest general engine's time over the row's.
| 1 sequence, per token | 8 sequences, per step | × fastest general | tokens/s | TTFT, s | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| System | ms | range | ms | range | 1 seq. | 8 seq. | 1 seq. | 8 seq. | 1 seq. | Job |
| SGLang | 6.086 | 6.085–6.087 | 6.853 | 6.850–6.859 | 1.00× | 1.00× | 164.2 | 1,163.5 | 0.014 | rel-q8-head-k3 |
| vLLM | 6.753 | 6.718–6.772 | 6.947 | 6.881–7.076 | 0.90× | 0.99× | 148.4 | 1,149.2 | 0.013 | rel-q8-head-k3 |
| megakernel, track A (s11 s0016) | 5.133 | 5.132–5.135 | 5.478 | 5.459–5.504 | 1.19× | 1.25× | 182.0 | 1,380.6 | 0.377 | rel-q8-head-k3 |
| megakernel, track B (s11 s0020) | 3.827 | 3.820–3.839 | 4.254 | 4.254–4.255 | 1.59× | 1.61× | 243.0 | 1,777.3 | 0.297 | rel-q8-head-k3 |
| SGLang | 6.130 | 6.129–6.131 | 6.839 | 6.829–6.852 | 1.00× | 1.00× | 163.0 | 1,166.1 | 0.012 | rel-q8-head-f |
| vLLM | 6.670 | 6.601–6.786 | 6.957 | 6.891–7.089 | 0.92× | 0.98× | 149.5 | 1,147.7 | 0.012 | rel-q8-head-f |
| megakernel, track A, earlier build (s10 s0003) | 5.377 | 5.374–5.380 | 5.636 | 5.634–5.637 | 1.14× | 1.21× | 173.9 | 1,342.6 | 0.386 | rel-q8-head-f |
| megakernel, track B, earlier build (s10 s0002) | 4.158 | 4.139–4.173 | 4.469 | 4.460–4.474 | 1.47× | 1.53× | 224.5 | 1,694.4 | 0.308 | rel-q8-head-f |
| SGLang | 6.140 | 6.136–6.144 | 6.847 | 6.844–6.849 | 1.00× | 1.00× | 162.7 | 1,165.1 | 0.013 | rel-q8-axes-1 |
| vLLM | 6.796 | 6.786–6.805 | 7.093 | 7.090–7.096 | 0.90× | 0.97× | 147.0 | 1,125.6 | 0.013 | rel-q8-axes-1 |
| megakernel, serving, track A (s3 s0038) | 6.096 | 6.289 | 1.01× | 1.09× | 164.0 | 1,269.3 | 0.009 | rel-q8-axes-1 | ||
| megakernel, serving, track B (s3 s0039) | 5.624 | 5.834 | 1.09× | 1.17× | 177.7 | 1,368.5 | 0.008 | rel-q8-axes-1 | ||
| 1 sequence, per token | 8 sequences, per step | × fastest general | tokens/s | TTFT, s | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| System | ms | range | ms | range | 1 seq. | 8 seq. | 1 seq. | 8 seq. | 1 seq. | Job |
| Cohere megakernel (vendor's own) | 3.619 | 7.066 | 1.00× | 1.00× | 276.3 | 1,132.2 | 0.002 | compare-final-north-2 | ||
| vLLM | 5.047 | 9.459 | 0.72× | 0.75× | 198.1 | 845.7 | 0.022 | compare-final-north-2 | ||
| megakernel, track A (s4 s0022) | 2.538 | 6.897 | 1.43× | 1.02× | 394.0 | 1,159.9 | 0.171 | compare-final-north-2 | ||
| megakernel, track B (s4 s0021) | 2.150 | 6.051 | 1.68× | 1.17× | 465.0 | 1,322.1 | 0.155 | compare-final-north-2 | ||
| 1 sequence, per token | 8 sequences, per step | × fastest general | tokens/s | TTFT, s | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| System | ms | range | ms | range | 1 seq. | 8 seq. | 1 seq. | 8 seq. | 1 seq. | Job |
| SGLang | 3.504 | 3.502–3.506 | 5.217 | 5.214–5.220 | 0.95× | 0.94× | 284.6 | 1,520.3 | 0.013 | rel-oss20-head-k3 |
| vLLM | 3.312 | 3.200–3.379 | 4.929 | 4.810–4.994 | 1.00× | 1.00× | 300.6 | 1,627.4 | 0.013 | rel-oss20-head-k3 |
| megakernel, track A (s12 s0047) | 1.874 | 1.872–1.875 | 3.388 | 3.387–3.389 | 1.77× | 1.45× | 496.9 | 2,270.7 | 0.144 | rel-oss20-head-k3 |
| megakernel, track B (s12 s0050) | 1.615 | 1.615–1.616 | 3.388 | 3.387–3.389 | 2.05× | 1.46× | 571.0 | 2,267.5 | 0.141 | rel-oss20-head-k3 |
| 1 sequence, per token | 8 sequences, per step | × fastest general | tokens/s | TTFT, s | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| System | ms | range | ms | range | 1 seq. | 8 seq. | 1 seq. | 8 seq. | 1 seq. | Job |
| vLLM | 1.989 | 1.986–1.991 | 3.785 | 3.783–3.786 | 1.00× | 1.00× | 486.3 | 2,050.2 | 0.063 | rel-flint-head-k2 |
| megakernel, track A (s11 s0026) | 1.392 | 1.392–1.393 | 3.354 | 3.353–3.355 | 1.43× | 1.13× | 659.8 | 2,290.9 | 0.127 | rel-flint-head-k2 |
| megakernel, track B (s11 s0027) | 1.356 | 1.356–1.357 | 3.349 | 3.348–3.349 | 1.47× | 1.13× | 676.2 | 2,294.5 | 0.126 | rel-flint-head-k2 |
| vLLM | 1.989 | 3.791 | 3.783–3.808 | 1.00× | 1.00× | 485.8 | 2,040.2 | 0.066 | rel-flint-head-f | |
| megakernel, track A, earlier build (s4 s0077) | 1.478 | 1.478–1.479 | 3.398 | 3.397–3.398 | 1.35× | 1.12× | 622.0 | 2,261.0 | 0.133 | rel-flint-head-f |
| megakernel, track B, earlier build (s4 s0055) | 1.477 | 1.476–1.478 | 3.953 | 3.950–3.955 | 1.35× | 0.96× | 623.0 | 1,935.4 | 0.133 | rel-flint-head-f |
| vLLM | 1.996 | 1.973–2.019 | 1.00× | 485.7 | 0.066 | rel-flint-axes-2 | ||||
| megakernel, track A, earlier build (s4 s0054) | 1.525 | 1.521–1.528 | 3.881 | 3.877–3.888 | 1.31× | 604.1 | 1,975.1 | 0.135 | rel-flint-axes-2 | |
| 1 sequence, per token | 8 sequences, per step | × fastest general | tokens/s | TTFT, s | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| System | ms | range | ms | range | 1 seq. | 8 seq. | 1 seq. | 8 seq. | 1 seq. | Job |
| SGLang | 9.531 | 9.529–9.533 | 10.300 | 10.236–10.429 | 1.00× | 1.00× | 104.1 | 770.4 | 0.060 | rel-q27-head-k2 |
| vLLM | 10.194 | 10.166–10.228 | 10.739 | 10.734–10.743 | 0.93× | 0.96× | 97.6 | 738.9 | 0.043 | rel-q27-head-k2 |
| megakernel, track A (s9 s0022) | 8.450 | 8.447–8.451 | 8.886 | 8.883–8.888 | 1.13× | 1.16× | 117.3 | 882.3 | 0.083 | rel-q27-head-k2 |
| megakernel, track B (s11 s0020) | 7.257 | 7.248–7.271 | 7.718 | 7.713–7.724 | 1.31× | 1.33× | 136.3 | 1,006.5 | 0.088 | rel-q27-head-k2 |
| SGLang | 9.527 | 9.525–9.531 | 10.233 | 10.230–10.237 | 1.00× | 1.00× | 104.2 | 773.0 | 0.064 | rel-q27-head-f |
| vLLM | 10.201 | 10.159–10.261 | 10.739 | 10.734–10.744 | 0.93× | 0.95× | 95.7 | 739.9 | 0.044 | rel-q27-head-f |
| megakernel, track B, earlier build (s9 s0023) | 7.712 | 7.695–7.721 | 8.180 | 8.161–8.208 | 1.24× | 1.25× | 128.1 | 950.5 | 0.089 | rel-q27-head-f |
| SGLang | 9.530 | 9.529–9.531 | 1.00× | 104.4 | 0.062 | rel-q27-axes-2 | ||||
| vLLM | 10.167 | 10.159–10.176 | 0.94× | 98.1 | 0.043 | rel-q27-axes-2 | ||||
| megakernel, track A, earlier build (s5 s0016) | 8.888 | 8.886–8.889 | 9.843 | 9.841–9.844 | 1.07× | 111.6 | 798.2 | 0.084 | rel-q27-axes-2 | |
| 1 sequence, per token | 8 sequences, per step | × fastest general | tokens/s | TTFT, s | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| System | ms | range | ms | range | 1 seq. | 8 seq. | 1 seq. | 8 seq. | 1 seq. | Job |
| SGLang | 11.426 | 11.425–11.427 | 11.682 | 11.679–11.684 | 1.00× | 1.00× | 87.5 | 683.1 | 0.021 | rel-q32-head-k2 |
| vLLM | 12.252 | 12.250–12.254 | 12.227 | 12.221–12.230 | 0.93× | 0.96× | 81.6 | 652.8 | 0.020 | rel-q32-head-k2 |
| megakernel, track A (s12 s0033) | 10.114 | 10.111–10.119 | 10.435 | 10.431–10.438 | 1.13× | 1.12× | 93.0 | 723.8 | 0.658 | rel-q32-head-k2 |
| megakernel, track B (s12 s0036) | 8.988 | 8.984–8.992 | 9.435 | 9.432–9.437 | 1.27× | 1.24× | 104.7 | 801.2 | 0.582 | rel-q32-head-k2 |
| 1 sequence, per token | 8 sequences, per step | × fastest general | tokens/s | TTFT, s | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| System | ms | range | ms | range | 1 seq. | 8 seq. | 1 seq. | 8 seq. | 1 seq. | Job |
| SGLang | 2.216 | 2.215–2.217 | 3.328 | 3.236–3.379 | 1.00× | 1.00× | 448.6 | 2,388.0 | 0.016 | rel-oss120-head-k2 |
| vLLM | 2.522 | 3.772 | 3.763–3.777 | 0.88× | 0.88× | 393.8 | 2,157.4 | 0.020 | rel-oss120-head-k2 | |
| megakernel, track A (s12 s0049) | 1.776 | 1.775–1.776 | 3.073 | 3.071–3.076 | 1.25× | 1.08× | 521.3 | 2,435.6 | 0.148 | rel-oss120-head-k2 |
| megakernel, track B (s12 s0051) | 1.688 | 1.685–1.689 | 3.026 | 3.024–3.027 | 1.31× | 1.10× | 546.6 | 2,471.9 | 0.147 | rel-oss120-head-k2 |
| 1 sequence, per token | 8 sequences, per step | × fastest general | tokens/s | TTFT, s | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| System | ms | range | ms | range | 1 seq. | 8 seq. | 1 seq. | 8 seq. | 1 seq. | Job |
| SGLang | 4.737 | 4.734–4.743 | 6.553 | 6.534–6.587 | 1.00× | 1.00× | 210.3 | 1,214.8 | 0.023 | rel-ink-head-2 |
| megakernel, track A (s7 s0034) | 3.303 | 5.391 | 1.43× | 1.22× | 281.4 | 1,417.9 | 0.259 | rel-ink-head-2 | ||
| megakernel, track B (s7 s0036) | 3.289 | 3.288–3.289 | 5.377 | 5.376–5.378 | 1.44× | 1.22× | 282.7 | 1,421.9 | 0.257 | rel-ink-head-2 |
| SGLang | 4.716 | 4.709–4.723 | 6.571 | 6.545–6.597 | 1.00× | 1.00× | 211.0 | 1,205.1 | 0.023 | rel-inkon2-cells |
| megakernel, serving, track A (s8 s0043) | 11.442 | 11.433–11.451 | 14.125 | 14.117–14.133 | 0.41× | 0.47× | 87.3 | 564.3 | 0.019 | rel-inkon2-cells |
| SGLang | 4.745 | 4.736–4.754 | 1.00× | 209.9 | 0.025 | rel-ink-axes-2 | ||||
| megakernel, track A, earlier build (s7 s0020) | 3.268 | 3.267–3.268 | 5.386 | 5.385–5.389 | 1.45× | 284.5 | 1,419.0 | 0.256 | rel-ink-axes-2 | |
64-token prompt, 1,024 tokens each, greedy. Time per step exists where all requests decode at once; total tokens per second includes the wait for the first token. A steady-decode engine holds 8 sequences and serves more requests in turns.
| ms per step at batch | total tokens per second at batch | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| System | 1 | 2 | 4 | 8 | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
| SGLang | 6.086 | 6.715 | 6.694 | 6.853 | 164 | 297 | 597 | 1,164 | 2,251 | 4,107 | 6,869 |
| vLLM | 6.753 | 6.692 | 6.718 | 6.947 | 148 | 299 | 595 | 1,149 | 2,233 | 4,135 | 7,034 |
| megakernel, track A (s11 s0016) | 5.133 | 5.201 | 5.318 | 5.478 | 182 | 359 | 704 | 1,381 | 1,381 | 1,381 | |
| megakernel, track B (s11 s0020) | 3.827 | 3.939 | 4.110 | 4.254 | 243 | 473 | 908 | 1,777 | 1,778 | 1,778 | |
| megakernel, track A, earlier build (s10 s0003) | 5.377 | 5.474 | 5.481 | 5.636 | 174 | 342 | 683 | 1,343 | 1,342 | 1,343 | |
| megakernel, track B, earlier build (s10 s0002) | 4.158 | 4.229 | 4.323 | 4.469 | 224 | 442 | 865 | 1,694 | 1,695 | 1,695 | |
| megakernel, serving, track A (s3 s0038) | 6.096 | 6.101 | 6.106 | 6.289 | 164 | 328 | 654 | 1,269 | 2,385 | 4,245 | 6,908 |
| megakernel, serving, track B (s3 s0039) | 5.624 | 5.661 | 5.694 | 5.834 | 178 | 353 | 701 | 1,369 | 2,549 | 4,429 | 7,064 |
| ms per step at batch | total tokens per second at batch | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| System | 1 | 2 | 4 | 8 | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
| vLLM | 5.057 | 5.936 | 8.011 | 9.175 | 197 | 336 | 498 | 869 | 1,378 | 2,228 | |
| ms per step at batch | total tokens per second at batch | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| System | 1 | 2 | 4 | 8 | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
| SGLang | 3.504 | 3.970 | 4.421 | 5.217 | 285 | 496 | 893 | 1,520 | 2,762 | 4,837 | |
| vLLM | 3.312 | 3.640 | 4.084 | 4.929 | 301 | 547 | 977 | 1,627 | 3,017 | 5,228 | |
| megakernel, track A (s12 s0047) | 1.874 | 2.252 | 2.862 | 3.388 | 497 | 831 | 1,334 | 2,271 | 2,267 | 2,264 | |
| megakernel, track B (s12 s0050) | 1.615 | 2.092 | 2.656 | 3.388 | 571 | 891 | 1,415 | 2,268 | 2,272 | 2,265 | |
| ms per step at batch | total tokens per second at batch | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| System | 1 | 2 | 4 | 8 | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
| vLLM | 1.989 | 2.258 | 2.785 | 3.785 | 486 | 842 | 1,378 | 2,050 | 3,053 | 4,454 | |
| megakernel, track A (s11 s0026) | 1.392 | 1.800 | 2.345 | 3.354 | 660 | 1,030 | 1,601 | 2,291 | 2,289 | 2,283 | |
| megakernel, track B (s11 s0027) | 1.356 | 1.763 | 2.341 | 3.349 | 676 | 1,050 | 1,604 | 2,295 | 2,293 | 2,287 | |
| megakernel, track A, earlier build (s4 s0077) | 1.478 | 1.894 | 2.421 | 3.398 | 622 | 980 | 1,551 | 2,261 | 2,259 | 2,254 | |
| megakernel, track B, earlier build (s4 s0055) | 1.477 | 1.910 | 2.719 | 3.953 | 623 | 971 | 1,380 | 1,935 | 1,939 | 1,932 | |
| megakernel, track A, earlier build (s4 s0054) | 1.525 | 1.946 | 2.677 | 3.881 | 604 | 954 | 1,406 | 1,975 | 1,974 | 1,969 | |
| ms per step at batch | total tokens per second at batch | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| System | 1 | 2 | 4 | 8 | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
| SGLang | 9.531 | 9.719 | 9.847 | 10.300 | 104 | 205 | 402 | 770 | 1,441 | 2,566 | |
| vLLM | 10.194 | 10.344 | 10.465 | 10.739 | 98 | 192 | 379 | 739 | 1,387 | 2,518 | |
| megakernel, track A (s9 s0022) | 8.450 | 8.653 | 8.642 | 8.886 | 117 | 229 | 456 | 882 | 883 | 882 | |
| megakernel, track B (s11 s0020) | 7.257 | 7.550 | 7.539 | 7.718 | 136 | 261 | 520 | 1,007 | 1,008 | 1,008 | |
| megakernel, track B, earlier build (s9 s0023) | 7.712 | 8.018 | 8.022 | 8.180 | 128 | 246 | 489 | 951 | 953 | 953 | |
| megakernel, track A, earlier build (s5 s0016) | 8.888 | 9.050 | 9.167 | 9.843 | 112 | 219 | 430 | 798 | 798 | 798 | |
| ms per step at batch | total tokens per second at batch | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| System | 1 | 2 | 4 | 8 | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
| SGLang | 11.426 | 11.606 | 11.609 | 11.682 | 87 | 172 | 343 | 683 | 1,338 | 2,438 | |
| vLLM | 12.252 | 12.358 | 12.413 | 12.227 | 82 | 162 | 322 | 653 | 1,284 | 2,426 | |
| megakernel, track A (s12 s0033) | 10.114 | 10.232 | 10.470 | 10.435 | 93 | 184 | 360 | 724 | 724 | 724 | |
| megakernel, track B (s12 s0036) | 8.988 | 9.150 | 9.352 | 9.435 | 105 | 206 | 403 | 801 | 801 | 801 | |
| ms per step at batch | total tokens per second at batch | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| System | 1 | 2 | 4 | 8 | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
| SGLang | 2.216 | 2.572 | 2.945 | 3.328 | 449 | 773 | 1,339 | 2,388 | 3,870 | 6,316 | |
| vLLM | 2.522 | 2.872 | 3.187 | 3.772 | 394 | 684 | 1,237 | 2,157 | 3,432 | 5,658 | |
| megakernel, track A (s12 s0049) | 1.776 | 2.287 | 2.614 | 3.073 | 521 | 819 | 1,461 | 2,436 | 2,444 | 2,467 | |
| megakernel, track B (s12 s0051) | 1.688 | 2.207 | 2.553 | 3.026 | 547 | 846 | 1,496 | 2,472 | 2,481 | 2,506 | |
| ms per step at batch | total tokens per second at batch | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| System | 1 | 2 | 4 | 8 | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
| SGLang | 4.737 | 5.114 | 5.875 | 6.553 | 210 | 389 | 678 | 1,215 | 2,116 | 3,366 | 5,409 |
| megakernel, track A (s7 s0034) | 3.303 | 3.755 | 4.330 | 5.391 | 281 | 497 | 869 | 1,418 | 1,422 | 1,426 | |
| megakernel, track B (s7 s0036) | 3.289 | 3.739 | 4.316 | 5.377 | 283 | 499 | 872 | 1,422 | 1,426 | 1,430 | |
| megakernel, serving, track A (s8 s0043) | 11.442 | 11.829 | 12.351 | 14.125 | 87 | 169 | 323 | 564 | 1,043 | 1,817 | 3,154 |
| megakernel, track A, earlier build (s7 s0020) | 3.268 | 3.751 | 4.333 | 5.386 | 285 | 498 | 868 | 1,419 | 1,423 | 1,427 | |
One sequence unless noted, greedy. Generation: median time per token over a generation of 4,096 or 16,384 tokens (64-token prompt). Prompt: time to the first token for a prompt of the given length (16 tokens generated). The last columns are eight requests with 4,096-token prompts and 256 tokens each. fails marks a request the engine refused or could not serve (hover for the reason); "> context" marks a request beyond the context the engine is checked to, which is 2,048 tokens for the steady-decode engines.
| generation, ms per token | TTFT (s) by prompt tokens | 8 × (4,096 + 256) | ||||||
|---|---|---|---|---|---|---|---|---|
| System | 4,096 | 16,384 | 1,024 | 4,096 | 16,384 | 32,000 | TTFT, s | tokens/s |
| SGLang | 6.597 | 0.029 | 0.100 | 0.490 | 1.237 | 0.408 | 726 | |
| vLLM | 7.115 | 0.029 | 0.102 | 0.492 | 1.235 | 0.499 | 720 | |
| megakernel, serving, track A (s3 s0038) | 7.207 | 0.032 | 0.130 | 0.669 | 1.779 | 0.554 | 664 | |
| megakernel, serving, track B (s3 s0039) | 6.879 | 0.029 | 0.118 | 0.605 | 1.634 | 0.514 | 701 | |
| generation, ms per token | TTFT (s) by prompt tokens | 8 × (4,096 + 256) | ||||||
|---|---|---|---|---|---|---|---|---|
| System | 4,096 | 16,384 | 1,024 | 4,096 | 16,384 | 32,000 | TTFT, s | tokens/s |
| vLLM | 4.837 | 4.979 | 0.047 | 0.084 | 0.355 | |||
| generation, ms per token | TTFT (s) by prompt tokens | 8 × (4,096 + 256) | ||||||
|---|---|---|---|---|---|---|---|---|
| System | 4,096 | 16,384 | 1,024 | 4,096 | 16,384 | 32,000 | TTFT, s | tokens/s |
| vLLM | 1.989 | 2.014 | 0.064 | 0.065 | 0.126 | 0.223 | 1,661 | |
| megakernel, track A, earlier build (s4 s0054) | > context | > context | fails | > context | > context | > context | > context | |
| generation, ms per token | TTFT (s) by prompt tokens | 8 × (4,096 + 256) | ||||||
|---|---|---|---|---|---|---|---|---|
| System | 4,096 | 16,384 | 1,024 | 4,096 | 16,384 | 32,000 | TTFT, s | tokens/s |
| SGLang | 9.949 | 11.414 | 0.064 | 0.195 | 0.914 | 1.001 | 471 | |
| vLLM | 10.187 | 10.273 | 0.075 | 0.194 | 0.715 | 1.198 | 510 | |
| megakernel, track A, earlier build (s5 s0016) | > context | > context | 0.434 | > context | > context | > context | > context | |
| generation, ms per token | TTFT (s) by prompt tokens | 8 × (4,096 + 256) | ||||||
|---|---|---|---|---|---|---|---|---|
| System | 4,096 | 16,384 | 1,024 | 4,096 | 16,384 | 32,000 | TTFT, s | tokens/s |
| SGLang | 4.718 | 4.730 | 0.036 | 0.071 | 0.210 | 0.408 | 0.292 | 1,020 |
| megakernel, serving, track A (s8 s0043) | 11.687 | 0.035 | 0.100 | 0.356 | 0.704 | 0.352 | 484 | |
| megakernel, track A, earlier build (s7 s0020) | > context | > context | 3.391 | > context | > context | > context | > context | |
Milliseconds per token with 1 sequence and per step with 8; 64-token prompts, 1,024 new tokens, two rounds. Random tokens are the benchmark's prompts; natural text is prose and code. Sampling is temperature 0.7, top-p 0.8, top-k 20. Rows named mkserve are our engines behind the repository's server, which does the sampling; the rows without it are the same engines behind the evaluation's own front end (greedy only).
| random tokens, greedy | natural text, greedy | natural text, sampled | ||||
|---|---|---|---|---|---|---|
| System | 1 seq. | 8 seq. | 1 seq. | 8 seq. | 1 seq. | 8 seq. |
| SGLang | 6.035 | 6.834 | 6.024 | 6.838 | 6.100 | 6.948 |
| vLLM | 6.785 | 6.883 | 6.598 | 6.882 | 6.729 | 7.031 |
| megakernel, track A, mkserve | 5.380 | 5.641 | 5.385 | 5.642 | 5.524 | 5.794 |
| megakernel, track B, mkserve | 4.166 | 4.469 | 4.165 | 4.469 | 4.308 | 4.625 |
| random tokens, greedy | natural text, greedy | natural text, sampled | ||||
|---|---|---|---|---|---|---|
| System | 1 seq. | 8 seq. | 1 seq. | 8 seq. | 1 seq. | 8 seq. |
| vLLM | 4.890 | 9.273 | 4.897 | 10.167 | 5.082 | 10.367 |
| megakernel, track A, mkserve | 2.586 | 7.025 | 2.582 | 7.989 | 2.734 | 7.967 |
| megakernel, track B, mkserve | 2.271 | 6.280 | 2.276 | 6.989 | 2.415 | 6.898 |
| megakernel, track A (s0022) | 2.536 | 6.955 | 2.536 | 7.924 | ||
| megakernel, track B (s0021) | 2.153 | 6.135 | 2.153 | 6.852 | ||
| random tokens, greedy | natural text, greedy | natural text, sampled | ||||
|---|---|---|---|---|---|---|
| System | 1 seq. | 8 seq. | 1 seq. | 8 seq. | 1 seq. | 8 seq. |
| vLLM | 1.970 | 3.786 | 1.971 | 3.890 | 2.089 | 4.020 |
| megakernel, track A, mkserve | 1.554 | 3.487 | 1.554 | 3.618 | 1.682 | 3.763 |
| megakernel, track B, mkserve | 1.550 | 4.078 | 1.550 | 4.183 | 1.676 | 4.318 |
| random tokens, greedy | natural text, greedy | natural text, sampled | ||||
|---|---|---|---|---|---|---|
| System | 1 seq. | 8 seq. | 1 seq. | 8 seq. | 1 seq. | 8 seq. |
| SGLang | 9.524 | 10.232 | 9.525 | 10.229 | 9.612 | 10.333 |
| vLLM | 10.160 | 10.733 | 10.160 | 10.734 | 10.324 | 10.915 |
| megakernel, track A, mkserve | 8.548 | 9.001 | 8.543 | 9.000 | 8.644 | 9.100 |
| megakernel, track B, mkserve | 7.771 | 8.233 | 7.770 | 8.232 | 7.881 | 8.366 |
| random tokens, greedy | natural text, greedy | natural text, sampled | ||||
|---|---|---|---|---|---|---|
| System | 1 seq. | 8 seq. | 1 seq. | 8 seq. | 1 seq. | 8 seq. |
| SGLang | 4.735 | 6.543 | 4.739 | 6.777 | 4.774 | 6.878 |
| megakernel, track A, mkserve | 3.484 | 5.563 | 3.480 | 5.895 | 3.621 | 6.044 |
| megakernel, track B, mkserve | 3.467 | 5.559 | 3.468 | 5.895 | 3.613 | 6.055 |
The first 300 seconds of the Mooncake tool-and-agent trace, thinned to the nominal rate, every system behind HTTP on the same GPU or node; rounds after one warm-up round. Rate is the nominal rate and req/s what the GPU was offered. In SLO: first token within 5 s and a mean time between tokens of at most 0.1 s. TBT is the time between a request's tokens; stall is the largest gap any decoding request saw; tokens/s is the output of all requests together.
| TTFT, s | TBT, ms | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Rate | req/s | System | Requests | In SLO, % | p50 | p99 | p50 | p99 | Stall, ms | Tokens/s | Rounds | Job |
| 1 | 0.79 | SGLang | 238 | 68.5 | 1.41 | 5.89 | 19.1 | 1,852 | 6,159 | 167 | 2 | rel-q8-replay-a2 |
| 1 | 0.79 | vLLM | 238 | 70.4 | 1.37 | 5.96 | 21.6 | 275 | 504 | 167 | 2 | rel-q8-replay-a2 |
| 1 | 0.79 | megakernel, serving, track A | 238 | 95.4 | 0.23 | 7.81 | 19.5 | 72 | 109 | 166 | 2 | rel-q8-replay-a2 |
| 1 | 0.79 | megakernel, serving, track B | 238 | 96.6 | 0.21 | 7.43 | 20.3 | 68 | 88 | 167 | 2 | rel-q8-replay-a2 |
| 2 | 1.61 | SGLang | 483 | 71.9 | 1.29 | 5.55 | 25.1 | 2,064 | 6,925 | 325 | 2 | rel-q8-replay-a2 |
| 2 | 1.61 | vLLM | 483 | 75.2 | 1.42 | 5.92 | 23.4 | 302 | 577 | 325 | 2 | rel-q8-replay-a2 |
| 2 | 1.61 | megakernel, serving, track A | 483 | 95.7 | 0.23 | 7.96 | 22.4 | 76 | 88 | 323 | 2 | rel-q8-replay-a2 |
| 2 | 1.61 | megakernel, serving, track B | 483 | 96.9 | 0.21 | 7.19 | 24.0 | 75 | 87 | 325 | 2 | rel-q8-replay-a2 |
| 3 | 2.39 | SGLang | 716 | 59.2 | 2.31 | 12.10 | 42.8 | 1,563 | 7,328 | 467 | 2 | rel-q8-replay-b |
| 3 | 2.39 | vLLM | 716 | 67.9 | 2.07 | 8.85 | 43.9 | 298 | 631 | 469 | 2 | rel-q8-replay-b |
| 3 | 2.39 | megakernel, serving, track A | 716 | 89.6 | 0.42 | 15.40 | 60.6 | 79 | 91 | 460 | 2 | rel-q8-replay-b |
| 3 | 2.39 | megakernel, serving, track B | 716 | 93.2 | 0.37 | 12.09 | 57.8 | 77 | 92 | 466 | 2 | rel-q8-replay-b |
| 4 | 3.49 | SGLang | 1,046 | 13.2 | 25.31 | 74.03 | 52.1 | 435 | 9,151 | 528 | 2 | rel-q8-replay-b |
| 4 | 3.49 | vLLM | 1,046 | 16.7 | 20.18 | 64.52 | 54.5 | 264 | 1,170 | 542 | 2 | rel-q8-replay-b |
| 4 | 3.49 | megakernel, serving, track A | 1,046 | 64.0 | 2.75 | 105.84 | 64.0 | 80 | 102 | 524 | 2 | rel-q8-replay-b |
| 4 | 3.49 | megakernel, serving, track B | 1,046 | 67.6 | 2.59 | 78.68 | 61.9 | 79 | 135 | 537 | 2 | rel-q8-replay-b |
| TTFT, s | TBT, ms | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Rate | req/s | System | Requests | In SLO, % | p50 | p99 | p50 | p99 | Stall, ms | Tokens/s | Rounds | Job |
| 1 | 0.79 | SGLang | 238 | 83.6 | 0.56 | 2.21 | 9.2 | 637 | 2,476 | 169 | 2 | rel-inkon2-replay-r1 |
| 1 | 0.79 | megakernel, serving, track A | 238 | 100.0 | 0.19 | 3.98 | 27.9 | 69 | 75 | 166 | 2 | rel-inkon2-replay-r1 |
| 2 | 1.61 | SGLang | 483 | 84.6 | 0.59 | 2.18 | 9.4 | 732 | 2,350 | 332 | 2 | rel-inkon2-replay-r2 |
| 2 | 1.61 | megakernel, serving, track A | 483 | 100.0 | 0.17 | 3.41 | 29.3 | 69 | 80 | 324 | 2 | rel-inkon2-replay-r2 |
| 4 | 3.49 | SGLang | 1,046 | 83.1 | 0.61 | 2.16 | 10.1 | 755 | 2,602 | 666 | 2 | rel-inkon2-replay-r4 |
| 4 | 3.49 | megakernel, serving, track A | 1,046 | 100.0 | 0.18 | 3.43 | 45.4 | 73 | 87 | 656 | 2 | rel-inkon2-replay-r4 |
| 8 | 7.07 | SGLang | 2,120 | 81.7 | 0.62 | 2.01 | 18.0 | 789 | 4,448 | 1,281 | 2 | rel-inkon2-replay-r8 |
| 8 | 7.07 | megakernel, serving, track A | 2,120 | 99.2 | 0.20 | 4.63 | 60.2 | 80 | 148 | 1,250 | 2 | rel-inkon2-replay-r8 |
Every evaluated submission of every run, with the run's own evaluator's score: the headline is time per token with one sequence (µs; in process for the first run, over HTTP afterwards) or, for a serving run, the share of requests within the service levels at the rate in the metric's name. The same-GPU tables above are the re-measured figures. Source, hypothesis and result of each submission are in the repository under runs/.
| Run | Model | GPU | Agent | Agent-hours (budget) | Submissions | Valid | Purpose |
|---|---|---|---|---|---|---|---|
s2-q8b-a1 | Qwen3-8B | H100 | Claude Opus 5.5 | 40 | 66 | 66 | steady decoding, tracks A and B |
s3-q8b-online-a1 | Qwen3-8B | H100 | Claude Opus 5.5 | 40 | 39 | 36 | serving |
s4-north-h100-a1 | North Mini Code 1.0 | H100 | Claude Opus 5.5 | 24 | 23 | 23 | steady decoding, tracks A and B |
s4-flint-b200-a2 | flint | B200 | Claude Fable 5.2 | 36 | 77 | 76 | steady decoding, tracks A and B |
s4-qwen38-27b-b200-a1 | Qwen3.8-27B | B200 | Claude Opus 5.5 | 24 | 50 | 48 | steady decoding, track A; multi-token decoding |
s5-qwen38-27b-b200-b1 | Qwen3.8-27B | B200 | Claude Opus 5.5 | 6 | 17 | 14 | track B, continuing the run above |
s6-inkling2s-b200x4-a1 | Inkling-2-Small | 4 × B200 | Claude Opus 5.5 | 24 | 14 | 11 | steady decoding, track A |
s7-inkling2s-b200x4-fa1 | Inkling-2-Small | 4 × B200 | Claude Fable 5.2 | 12 | 20 | 19 | steady decoding, track A, continuing the run above |
s7-inkling2s-b200x4-fb1 | Inkling-2-Small | 4 × B200 | Claude Fable 5.2 | 15 | 36 | 35 | steady decoding, track B |
s8-inkling2s-b200x4-on1 | Inkling-2-Small | 4 × B200 | Claude Fable 5.2 | 32 | 43 | 42 | serving |
s9-qwen38-27b-b200-f1 | Qwen3.8-27B | B200 | Claude Fable 5.2 | 8 | 23 | 23 | steady decoding, tracks A and B, continuing the two runs above |
s10-q8b-h100-f1 | Qwen3-8B | H100 | Claude Fable 5.2 | 3 | 4 | 4 | correction of track A at long positions |
s11-q8b-h100-b1 | Qwen3-8B | H100 | Claude Opus 5.5 | 20 | 25 | 24 | track B, continuing the two runs above |
s11-north-h100-b1 | North Mini Code 1.0 | H100 | Claude Opus 5.5 | 20 | 26 | 26 | track B, continuing the run above |
s11-flint-b200-b1 | flint | B200 | Claude Opus 5.5 | 20 | 27 | 27 | track B, continuing the run above |
s11-q27-b200-b1 | Qwen3.8-27B | B200 | Claude Opus 5.5 | 30 | 29 | 29 | track B, continuing the three runs above |
s12-oss20-h100-a1 | gpt-oss-20b | H100 | Claude Opus 5.5 | 32 | 50 | 49 | steady decoding, tracks A and B, from the starter |
s12-q32-b200-a1 | Qwen3-32B | B200 | Claude Opus 5.5 | 36 | 47 | 47 | steady decoding, tracks A and B, from the starter |
s12-oss120-b200-a1 | gpt-oss-120b | B200 | Claude Opus 5.5 | 36 | 59 | 57 | steady decoding, tracks A and B, from the starter |
An agent-hour is an hour of the agent's wall-clock time with a GPU pod ready. The serving run on Inkling-2-Small and the last run on Qwen3.8-27B stopped at 31.2 and 7.2 hours.
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.1 | valid | step_us_b1 | 21,051 | starter: Hugging Face reference | ||
| s0002 | 0.4 | valid | step_us_b1 | 7,366 | own CUDA operators: MMA matvec block team + attention, one launch per operator | ||
| s0003 | 0.4 | valid | step_us_b1 | 6,465 | whole decode step in one kernel launch, grid barriers between phases | ||
| s0004 | 1.0 | valid | step_us_b1 | 6,385 | down-projection entry: copy only live batch columns, no zero fill | ||
| s0005 | 1.1 | valid | step_us_b1 | 6,139 | lean phases: tile tables, fixed-trip unrolled loaders, coordinator warp, no divisions | ||
| s0006 | 1.2 | valid | step_us_b1 | 5,951 | fuse RMSNorm into the previous epilogue: publish bf16(w*x), scale matvec outputs by 1/rms | ||
| s0007 | 1.2 | valid | step_us_b1 | 5,886 | generate(): greedy loop on the device (argmax + token feedback in the kernel, one launch) | ||
| s0008 | 2.4 | valid | step_us_b1 | 5,810 | SMs idle during attention stream their first O tile into shared memory | ||
| s0009 | 2.7 | valid | step_us_b1 | 5,859 | coordinator requests residual rows and norm weights before the tile barrier | ||
| s0010 | 3.3 | valid | step_us_b1 | 5,771 | kernel = call-free interpreter of phase descriptors; loader and coordinator code split | ||
| s0011 | 3.4 | valid | step_us_b1 | 5,756 | 1/rms of a norm phase computed by warps that have no input-copy loads | ||
| s0012 | 3.5 | valid | step_us_b1 | 5,694 | SMs idle during attention first pull the layer's K/V history into L2 | ||
| s0013 | 4.0 | valid | step_us_b1 | 5,659 | attention q/k/v prep: one warp per vector, norm and RoPE by warp shuffles, no block barrier | ||
| s0014 | 4.4 | valid | step_us_b1 | 5,651 | input copy: idle threads leave at once, loads requested in pairs before the stores | ||
| s0015 | 4.7 | valid | step_us_b1 | 5,595 | weight loads use the last-use cache operator (ld.lu) | ||
| s0016 | 6.4 | valid | step_us_b1 | 5,528 | loaders request 2 loads + the extra kstep of the next tile before the tile barrier | ||
| s0017 | 6.6 | valid | step_us_b1 | 5,515 | O phase: request the second tile's first loads while the stash tile is computed | ||
| s0018 | 7.6 | valid | step_us_b1 | 5,455 | at the end of a phase, prefetch the next phase's first 2 ksteps per warp into L2 | ||
| s0019 | 7.9 | valid | step_us_b1 | 5,422 | L2 prefetch in the attention and embedding windows (second O tile, attention blocks' O tile, first QKV tile) | ||
| s0020 | 8.3 | valid | step_us_b1 | 5,398 | while a block waits at the grid barrier, its coordinator prefetches more of the next phase | ||
| s0021 | 8.6 | valid | step_us_b1 | 5,382 | compile with fast-math: approximate division and rsqrt in the scalar epilogue code | ||
| s0022 | 9.0 | valid | step_us_b1 | 5,340 | 24 warps per block (80 registers per thread) with a 3-load tile pipeline | ||
| s0023 | 9.6 | valid | step_us_b1 | 5,336 | source cleanup; lm_head prefetches layer 0 for the next launch | ||
| s0024 | 9.9 | valid | step_us_b1 | 5,347 | 5+ sequences: input fragments stored in the shared-memory layout, input copy becomes a block copy | ||
| s0025 | 11.1 | valid | step_us_b1 | 4,522 | weights stored losslessly in 12 bits, decoded in registers before the MMA | ||
| s0026 | 11.8 | valid | step_us_b1 | 4,464 | exact per-column power-of-two weight scales: no exception-heavy tiles | ||
| s0027 | 12.1 | valid | step_us_b1 | 4,366 | up to 4 sequences: narrow input fragments, block shared memory 223 -> 154 KB | ||
| s0028 | 12.3 | valid | step_us_b1 | 4,270 | 12-bit stream laid out for a mask-and-OR decode (11 instead of 31 ops per MMA) | ||
| s0029 | 13.4 | valid | step_us_b1 | 4,198 | loaders stash the start of the next phase's first tile while the grid barrier runs | ||
| s0030 | 14.2 | valid | step_us_b1 | 4,184 | 3+ sequences: loaders read the input fragments from L2, no input copy | ||
| s0031 | 18.7 | B | valid | step_us_b1 | 4,186 | attention to 9216+ positions: 16 B/position scratch, split over all SMs for 1-2 sequences | |
| s0032 | 19.2 | A | valid | step_us_b1 | 5,386 | track A: BF16 weights as stored, BF16 activations, RMSNorm before the matvec | |
| s0033 | 20.4 | B | valid | step_us_b1 | 4,161 | attention: prefetch the K/V append lines; idle blocks stash after the K loads (track B) | |
| s0034 | 20.4 | A | valid | step_us_b1 | 5,343 | track A with the attention scheduling of s0033 | |
| s0035 | 21.5 | B | valid | step_us_b1 | 4,154 | grid-barrier increment directly behind the fence (track B) | |
| s0036 | 21.6 | A | valid | step_us_b1 | 5,332 | track A with the early grid-barrier increment of s0035 | |
| s0037 | 22.4 | B | valid | step_us_b1 | 4,105 | attention: the idle blocks' stash starts one load at a time (track B) | |
| s0038 | 22.5 | A | valid | step_us_b1 | 5,299 | track A with the gentle attention-phase stash of s0037 | |
| s0039 | 23.0 | B | valid | step_us_b1 | 4,088 | o projection: attention blocks take two tiles; second tile prefetched at phase start (track B) | |
| s0040 | 23.1 | A | valid | step_us_b1 | 5,298 | track A with the o tile table of s0039 (attention blocks take two tiles) | |
| s0041 | 23.7 | B | valid | step_us_b1 | 4,093 | 6+ sequences: stash blocks prefetch all of their second o tile (track B) | |
| s0042 | 23.7 | A | valid | step_us_b1 | 5,304 | track A with the second-tile prefetch rule of s0041 | |
| s0043 | 25.7 | B | valid | step_us_b1 | 4,085 | softmax in two passes less at long context; split path copies scores in one round trip (track B) | |
| s0044 | 25.7 | A | valid | step_us_b1 | 5,286 | track A with the long-context softmax changes of s0043 | |
| s0045 | 27.3 | B | valid | step_us_b1 | 4,095 | long context, 3+ sequences: idle blocks pull the first K tiles into L2 (track B) | |
| s0046 | 27.3 | A | valid | step_us_b1 | 5,288 | track A with the long-context K touch of s0045 | |
| s0047 | 29.1 | B | valid | step_us_b1 | 4,086 | long context, 1 sequence: idle blocks pull V into L2 during score copy + softmax | |
| s0048 | 29.1 | A | valid | step_us_b1 | 5,292 | track A with the split-path V touch of s0047 | |
| s0049 | 30.1 | A | valid | step_us_b1 | 5,278 | track A, 3+ sequences: loaders apply RMSNorm to the input; no norm phase | |
| s0050 | 30.4 | A | valid | step_us_b1 | 5,284 | track A: normalised input kept in shared memory after a block's first tile | |
| s0051 | 30.9 | B | valid | step_us_b1 | 4,094 | long context, 3+ sequences: 6 V loads in flight per warp instead of 4 | |
| s0052 | 31.4 | B | valid | step_us_b1 | 4,083 | attention: no fence behind the K/V append | |
| s0053 | 31.4 | A | valid | step_us_b1 | 5,230 | track A with the fence removal of s0052 and the V batching of s0051 | |
| s0054 | 33.5 | B | valid | step_us_b1 | 4,061 | 1-2 sequences: loader warps fetch their own input slice; no block copy/barrier | |
| s0055 | 33.6 | A | valid | step_us_b1 | 5,233 | track A with the per-warp input slices of s0054 (phases without a norm) | |
| s0056 | 33.9 | B | valid | step_us_b1 | 4,057 | 3+ sequences: no block barrier at the start of each phase | |
| s0057 | 34.4 | B | valid | step_us_b1 | 4,043 | q/k/v, gate+up, down on 128 of the 132 SMs: every block gets the same tile count | |
| s0058 | 34.6 | A | valid | step_us_b1 | 5,235 | track A: no entry barrier with 3+ sequences, matvec tiles on 128 SMs with 3+ | |
| s0059 | 35.7 | B | valid | step_us_b1 | 4,047 | only the attention blocks wait at the q/k/v barrier; the others go on | |
| s0060 | 35.9 | A | valid | step_us_b1 | 5,234 | track A: only attention blocks wait at the q/k/v barrier (3+ sequences) | |
| s0061 | 37.8 | B | valid | step_us_b1 | 4,026 | re-evaluation of the final track B source (s0059 unchanged) | |
| s0062 | 37.9 | A | valid | step_us_b1 | 5,237 | re-evaluation of the final track A source (s0060 unchanged) | |
| s0063 | 38.2 | B | valid | step_us_b1 | 4,020 | attention blocks: L2 prefetch of 3 (not 6) loads per warp of their cold o tile | |
| s0064 | 38.4 | A | valid | step_us_b1 | 5,235 | track A with the o-tile prefetch of s0063 (3 loads per warp) | |
| s0065 | 39.3 | B | valid | step_us_b1 | 4,047 | final track B source (s0063 kernel; host-side diagnostic export only) | |
| s0066 | 39.4 | A | valid | step_us_b1 | 5,236 | final track A source (s0064 kernel; host-side diagnostic export only) |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.0 | incorrect | inherited engine, no online interface | ||||
| s0002 | 1.1 | incorrect | pass engine: decode rows + prefill chunks through one GEMM pass, FIFO | ||||
| s0003 | 1.4 | incorrect | scheduler: deadline-ordered prefill, 60 ms pass budget, prefix page cache | ||||
| s0004 | 2.3 | valid | slo_share_r2 | 83.1% | attention rewrite (block-staged K/V, 2-level fold) + size-class scheduler | ||
| s0005 | 3.0 | valid | slo_share_r2 | 90.5% | GEMM on the Hopper path: wgmma + bulk async copy + mbarriers | ||
| s0006 | 3.2 | valid | slo_share_r2 | 93.7% | attention: K/V staged by the copy engine, no rescale when max unchanged | ||
| s0007 | 3.7 | valid | slo_share_r2 | 94.2% | attention on the warpgroup MMA, pipelined over segments | ||
| s0008 | 4.6 | valid | slo_share_r2 | 94.2% | no host scheduler: the serving loop runs on the device as one graph | ||
| s0009 | 5.5 | valid | slo_share_r2 | 94.2% | 64-row GEMM blocks with 8 stages for passes of up to 64 columns | ||
| s0010 | 6.1 | valid | slo_share_r2 | 95.2% | pass overheads: batched residual loads, 8-byte q/K stores, wgmma stage in flight | ||
| s0011 | 7.5 | valid | slo_share_r2 | 95.2% | GEMM block shape per matrix and pass size; generate() loops on the device | ||
| s0012 | 7.6 | valid | slo_share_r2 | 91.5% | pass target 80 -> 27 ms: median time between tokens under 25 ms | ||
| s0013 | 10.6 | valid | slo_share_r2 | 92.1% | step() as one graph launch; decode rows stop re-clearing their V group | ||
| s0014 | 10.7 | valid | slo_share_r2 | 92.1% | GEMM: four producer threads for 64-row blocks; prefetch of the next kernel's first stages | ||
| s0015 | 11.0 | valid | slo_share_r2 | 92.1% | decode-row attention: operand roles swapped, tiles on three warpgroups | ||
| s0016 | 11.1 | valid | slo_share_r2 | 92.1% | 1/rms computed inside the next GEMM up to 64 columns (two kernels per layer less) | ||
| s0017 | 12.6 | valid | slo_share_r2 | 92.6% | deadline planner: prompt share from the state; 45 ms passes while a prompt is short of slack | ||
| s0018 | 13.5 | valid | slo_share_r2 | 92.6% | q, K, V prepared in the q/k/v GEMM epilogue up to 16 columns (no prep kernel) | ||
| s0019 | 14.4 | valid | slo_share_r2 | 92.6% | deadline order: shortest first among prompts that arrive together | ||
| s0020 | 14.7 | valid | slo_share_r2 | 93.1% | prefill attention: the two warpgroups of a block alternate tensor and scalar phases | ||
| s0021 | 15.4 | valid | slo_share_r2 | 93.1% | decode-row attention in its own kernel (no wgmma serialized for lack of registers) | ||
| s0022 | 15.7 | valid | slo_share_r2 | 93.1% | attention fold: threads without a position of the unit skip | ||
| s0023 | 17.2 | B | valid | slo_share_r2 | 94.7% | tail guard: large passes while a prompt heads for a first token later than 8 s | |
| s0024 | 18.0 | A | valid | slo_share_r2 | 94.7% | track A build (engine/TRACK=A): the engines' formats and data flow | |
| s0025 | 22.4 | B | valid | slo_share_r2 | 94.7% | decode step: PDL + early weights, counter-free decode attention (5.67->5.58 ms) | |
| s0026 | 24.5 | B | valid | slo_share_r2 | 94.7% | light decode passes in the serving loop (SWITCH node), decode pass -0.8 ms | |
| s0027 | 25.6 | B | valid | slo_share_r2 | 94.7% | early weight stages before the dependency wait (offline 5.57 -> 5.46 ms) | |
| s0028 | 25.9 | B | valid | slo_share_r2 | 94.7% | PDL: attention set-up before its wait, non-GEMM kernels trigger at start | |
| s0029 | 27.6 | B | valid | slo_share_r2 | 94.7% | per-shape GEMMs for prompt passes (bucket branches); idle blocks leave early | |
| s0030 | 29.2 | B | valid | slo_share_r2 | 94.7% | GEMM epilogue: unit's 1/rms staged in shared memory | |
| s0031 | 29.8 | B | valid | slo_share_r2 | 95.2% | near-miss trigger on (MK_NEAR_S 0.75): big passes while a prompt is just past 5 s | |
| s0032 | 30.2 | B | valid | slo_share_r2 | 95.8% | tail guard 5 s with the near-miss window | |
| s0033 | 31.6 | B | valid | slo_share_r2 | 95.2% | pass target 25 ms with near-miss window; age cap 10 s | |
| s0034 | 33.7 | A | valid | slo_share_r2 | 94.2% | track A: k_rnorm fills whole lines above 64 columns; near-miss window off | |
| s0035 | 36.0 | B | valid | slo_share_r2 | 95.2% | final track B source (s0033 engine; track-A-only changes since) | |
| s0036 | 36.8 | A | valid | slo_share_r2 | 94.2% | track A: one-column steps normalize inside QKV/GU (no norm kernels at B=1) | |
| s0037 | 37.5 | B | valid | slo_share_r2 | 95.2% | ADD epilogue: last unit's residual staged in smem by cp.async (one round trip) | |
| s0038 | 38.4 | A | valid | slo_share_r2 | 94.7% | track A: residual staged in smem by cp.async in the ADD epilogue (as s0037) | |
| s0039 | 39.4 | B | valid | slo_share_r2 | 95.2% | tail guard 8 -> 7 s (shorter first-token tail; same share and tbt) |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.5 | A | valid | step_us_b1 | 3,497 | v1 megakernel: one persistent kernel, BF16 weights in mma fragment order | |
| s0002 | 0.8 | A | valid | step_us_b1 | 3,050 | v2: static per-block schedule, norm and residual combined in shared memory | |
| s0003 | 1.1 | valid | http_step_us_b1 | 3,603 | 8,995 | OPERATOR: s0001 again, unchanged (evaluated again under the HTTP headline, harness 418e7ad) | |
| s0004 | 1.2 | valid | http_step_us_b1 | 3,169 | 7,927 | OPERATOR: s0002 again, unchanged (evaluated again under the HTTP headline, harness 418e7ad) | |
| s0005 | 2.0 | A | valid | http_step_us_b1 | 3,196 | 7,615 | v4: balanced quantum schedule, shallow HBM queue (2 loads in flight), leader-first routing |
| s0006 | 3.3 | A | valid | http_step_us_b1 | 2,933 | 7,257 | v5: attention in 8 position chunks over 8 blocks; device loop behind submit/poll |
| s0007 | 3.8 | B | valid | http_step_us_b1 | 2,628 | 6,938 | track B v6: lossless 12-bit weights (row exponents, neuron shifts, exact escapes) |
| s0008 | 4.8 | B | valid | http_step_us_b1 | 2,598 | 6,698 | track B: layer 0 from token tables (q|k|v and MLP output per token id) |
| s0009 | 5.6 | B | valid | http_step_us_b1 | 2,572 | 6,699 | track B: per-layer skeleton trimmed (residual add on 4 warps, top-8 by warp reduce, norm totals per column) |
| s0010 | 5.7 | B | valid | http_step_us_b1 | 2,474 | 6,616 | track B: certified int8 sketch head on the greedy path (integer tensor core, exact survivors) |
| s0011 | 7.0 | B | valid | http_step_us_b1 | 2,412 | 6,421 | track B: stage B start balance (early union, leader first, delay-aware gate|up shares) |
| s0012 | 7.2 | B | valid | http_step_us_b1 | 2,366 | 6,382 | track B: 12-bit format re-laid out for a 3-instruction decode (rotate + LOP3) |
| s0013 | 7.8 | B | valid | http_step_us_b1 | 2,380 | 6,310 | track B: o_proj in stage B at batch > 1 (attention-done stamps; shares include it) |
| s0014 | 8.5 | B | valid | http_step_us_b1 | 2,373 | 6,240 | track B: RMSNorm rounding with cvt.rn.bf16x2 (bit-identical, fewer instructions) |
| s0015 | 8.9 | B | valid | http_step_us_b1 | 2,365 | 6,207 | track B: stage C starts without dependent round trips (late escape gather, local B=1 routing) |
| s0016 | 13.2 | A | valid | http_step_us_b1 | 2,785 | 7,157 | track A: s0006 + the scheduling work since (shares by start delay, o_proj in B at B>1, decode-free paths) |
| s0017 | 15.4 | B | valid | http_step_us_b1 | 2,252 | 6,049 | track B: weight loads skip L1; zero operand from shared memory (stage C x stays in L1) |
| s0018 | 18.6 | B | valid | http_step_us_b1 | 2,187 | 6,061 | track B: o_proj in stage B at batch 1 too (now that stage C's x stays in L1) |
| s0019 | 19.7 | A | valid | http_step_us_b1 | 2,573 | 6,904 | track A: BF16 engine + L1 no-allocate weight loads, o_proj in stage B (kernel organisation only) |
| s0020 | 20.9 | B | valid | http_step_us_b1 | 2,173 | 6,039 | track B: gate|up results and residual rows as exchanges instead of store + fence |
| s0021 | 21.7 | B | valid | http_step_us_b1 | 2,146 | 6,028 | track B: union load issued during the last stage-A quantum (batch-1 kernel) |
| s0022 | 22.0 | A | valid | http_step_us_b1 | 2,533 | 6,893 | track A: s0019 + exchanges instead of fences, early union load at batch 1 |
| s0023 | 22.6 | B | valid | http_step_us_b1 | 2,150 | 5,989 | track B: share model weights attention's per-K-tile cost more (long contexts) |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.8 | A | valid | step_us_b1 | 2,679 | First megakernel: one launch per step, tensor-core tile GEMV (track A) | |
| s0002 | 0.9 | valid | http_step_us_b1 | 3,003 | 7,816 | OPERATOR: s0001 again, unchanged (evaluated again under the HTTP headline, harness 418e7ad) | |
| s0003 | 1.6 | A | incorrect | 512-thread blocks, tiles staged in registers, L1-bypass loads, submit/poll serving | |||
| s0004 | 1.8 | B | valid | http_step_us_b1 | 2,194 | 7,294 | Track B precision mode: float32 residual stream, hi+lo router input (same kernel as s0003) |
| s0005 | 2.6 | B | valid | http_step_us_b1 | 1,968 | 6,273 | Attention mix by dim-chunk warps, parallel routing finals, K/V staging levels, head on 152 SMs |
| s0006 | 2.7 | A | valid | http_step_us_b1 | 1,901 | 6,081 | Track A build of the s0005 kernel (pure BF16 arithmetic, 2 accumulators per chain) |
| s0007 | 3.1 | B | valid | http_step_us_b1 | 1,782 | 5,245 | Shared-memory reservation 224 -> 128 KB: L1 left for global loads |
| s0008 | 3.2 | A | valid | http_step_us_b1 | 1,685 | 4,796 | Track A build of the s0007 kernel (128 KB reservation, 16-tile expert batches) |
| s0009 | 5.1 | B | valid | http_step_us_b1 | 1,772 | 4,158 | Race-free hand-overs (atomic writes), batch-8 routing and attention placement |
| s0010 | 5.2 | A | valid | http_step_us_b1 | 1,780 | 4,235 | Track A build of the s0009 kernel (race-free hand-overs, batch-8 changes) |
| s0011 | 5.4 | B | valid | http_step_us_b1 | 1,745 | 4,152 | Two kernel instances (batch <=2, >2), down-phase loop of s0008 restored |
| s0012 | 5.5 | A | valid | http_step_us_b1 | 1,734 | 4,079 | Track A build of the s0011 kernel (two instances, s0008 down-phase loop) |
| s0013 | 6.2 | B | valid | http_step_us_b1 | 1,746 | 4,101 | Batch >2 kernel: split tile requests in the down phase, norm scratch, deferred writes |
| s0014 | 6.3 | A | valid | http_step_us_b1 | 1,734 | 3,993 | Track A build of the s0013 kernel (batch >2: split tile requests, norm scratch) |
| s0015 | 6.8 | B | valid | http_step_us_b1 | 1,733 | 4,100 | Head in 16-tile batches (staging registers) |
| s0016 | 6.8 | A | valid | http_step_us_b1 | 1,724 | 3,989 | Track A build of the s0015 kernel (head in 16-tile batches) |
| s0017 | 9.0 | B | valid | http_step_us_b1 | 1,786 | 4,298 | Strict hand-overs: sentinel words, signal at phase start, readers poll the data |
| s0018 | 9.6 | B | valid | http_step_us_b1 | 1,733 | 4,268 | Sentinels issued a phase ahead, confirmed by read-back; second stream buffer |
| s0019 | 9.6 | A | valid | http_step_us_b1 | 1,783 | 4,179 | Track A build of the s0018 kernel (strict sentinel hand-overs) |
| s0020 | 10.4 | B | valid | http_step_us_b1 | 1,733 | 4,233 | Attention multiply-adds written out: bitwise batch invariance at batch 7-8 |
| s0021 | 10.8 | B | valid | http_step_us_b1 | 1,715 | 4,231 | Router top-6 by ranks instead of six rounds of warp maxima (batch 1-2) |
| s0022 | 10.8 | B | valid | http_step_us_b1 | 1,701 | 4,224 | Expert phase, batch 1-2: idle warps fold and write the first slot's hidden units |
| s0023 | 10.8 | B | valid | http_step_us_b1 | 1,699 | 4,221 | Shared-expert epilogue and K/V cache stores moved off the critical path |
| s0024 | 10.9 | A | valid | http_step_us_b1 | 1,732 | 4,111 | Track A build of the s0023 kernel (ranks, epilogues off the critical path) |
| s0025 | 11.2 | B | valid | http_step_us_b1 | 1,663 | 4,222 | Attention at long contexts: K/V cache tiles 8 per request, not one per trip |
| s0026 | 11.3 | A | valid | http_step_us_b1 | 1,679 | 4,116 | Track A build of the s0025 kernel (K/V cache tiles 8 per request) |
| s0027 | 11.6 | B | valid | http_step_us_b1 | 1,646 | 4,227 | Attention: first extra batch of V tiles requested right after the scores |
| s0028 | 11.7 | A | valid | http_step_us_b1 | 1,669 | 4,118 | Track A build of the s0027 kernel (V tiles requested after the scores) |
| s0029 | 12.2 | B | valid | http_step_us_b1 | 1,646 | 4,188 | Batch > 2: the 12 routing candidates of a sequence merged by ranks |
| s0030 | 12.3 | A | valid | http_step_us_b1 | 1,671 | 4,056 | Track A build of the s0029 kernel (batch > 2 routing merge by ranks) |
| s0031 | 12.4 | B | valid | http_step_us_b1 | 1,646 | 4,123 | Batch > 2: router blocks preload router tiles, stage K/V after the router |
| s0032 | 12.4 | A | valid | http_step_us_b1 | 1,671 | 4,085 | Track A build of the s0031 kernel (router tiles preloaded at batch > 2) |
| s0033 | 12.9 | B | valid | http_step_us_b1 | 1,652 | 4,120 | Attention: the k/v-row loads are issued together with the q-row loads |
| s0034 | 13.0 | A | valid | http_step_us_b1 | 1,671 | 4,097 | Track A build of the s0033 kernel (attention load order) |
| s0035 | 13.1 | B | valid | http_step_us_b1 | 1,650 | 4,117 | Host sees the kernel's watchdog; contexts above 2048 positions refused |
| s0036 | 13.2 | A | valid | http_step_us_b1 | 1,672 | 4,099 | Track A build of the s0035 engine (watchdog visible, context cap) |
| s0037 | 13.7 | B | valid | http_step_us_b1 | 1,606 | 4,116 | generate/serve at batch 1-2: the 152 blocks' best tokens gathered in one trip |
| s0038 | 13.8 | A | valid | http_step_us_b1 | 1,606 | 4,094 | Track A build of the s0037 kernel (token gather in one trip) |
| s0039 | 14.0 | B | valid | http_step_us_b1 | 1,606 | 4,088 | generate/serve at batch 3-8: the blocks' best tokens gathered in 16 strides |
| s0040 | 14.0 | A | valid | http_step_us_b1 | 1,608 | 4,028 | Track A build of the s0039 kernel (batch 3-8 token gather in 16 strides) |
| s0041 | 14.8 | B | valid | http_step_us_b1 | 1,560 | 4,055 | Softmax weights: one reciprocal and an exact FMA correction, not 4224 divisions |
| s0042 | 14.8 | A | valid | http_step_us_b1 | 1,548 | 3,928 | Track A build of the s0041 kernel (softmax weights by reciprocal + correction) |
| s0043 | 15.6 | B | valid | http_step_us_b1 | 1,558 | 4,039 | Batch 7-8: main blocks with an attention pair poll phase A's counter early |
| s0044 | 15.6 | A | valid | http_step_us_b1 | 1,546 | 3,920 | Track A build of the s0043 kernel (early poll of phase A's counter, batch 7-8) |
| s0045 | 16.6 | B | valid | http_step_us_b1 | 1,559 | 4,021 | Batch > 2: the norm's scale of each sequence computed once per warp, not by every thread |
| s0046 | 16.6 | A | valid | http_step_us_b1 | 1,546 | 3,912 | Track A build of the s0045 kernel (norm scales once per warp at batch > 2) |
| s0047 | 18.5 | B | valid | http_step_us_b1 | 1,566 | 3,979 | Sentinel reads: the abort word is loaded on every 16th retry, not before each reload |
| s0048 | 18.6 | A | valid | http_step_us_b1 | 1,527 | 3,914 | Track A build of the s0047 kernel (abort word loaded on every 16th retry) |
| s0049 | 19.3 | B | valid | http_step_us_b1 | 1,560 | 3,936 | Hand-over counters placed by a start-up timing: fastest of 16 pages per kernel |
| s0050 | 19.4 | A | valid | http_step_us_b1 | 1,527 | 3,870 | Track A build of the s0049 engine (counters placed by a start-up timing) |
| s0051 | 19.9 | B | valid | http_step_us_b1 | 1,555 | 3,961 | Start-up placement extended: 3 more buffers at batch 1, 64 counter pages at batch 8 |
| s0052 | 19.9 | A | valid | http_step_us_b1 | 1,516 | 3,898 | Track A build of the s0051 engine (extended start-up placement) |
| s0053 | 20.2 | B | valid | http_step_us_b1 | 1,557 | 3,950 | Placement: batch 8 back to the 16 counter pages of s0049; batch 1 buffers on own slabs |
| s0054 | 20.2 | A | valid | http_step_us_b1 | 1,516 | 3,878 | Track A build of the s0053 engine (batch-8 counters as s0050, batch-1 buffers on own slabs) |
| s0055 | 21.3 | B | valid | http_step_us_b1 | 1,473 | 3,963 | Track B: kernel body wrapped in an inlined function: better register allocation |
| s0056 | 21.4 | A | valid | http_step_us_b1 | 1,516 | 3,874 | Track A build of the s0055 source (wrapper off on this track: build unchanged) |
| s0057 | 23.6 | B | valid | http_step_us_b1 | 1,475 | 3,950 | Hand-over reads as strong loads (ld.relaxed.gpu) instead of ld.global.cg |
| s0058 | 23.6 | A | valid | http_step_us_b1 | 1,520 | 3,877 | Track A build of the s0057 source (hand-over reads as strong loads) |
| s0059 | 24.0 | A | valid | http_step_us_b1 | 1,522 | 3,898 | Audit defects: over-long served requests and batch > max_batch now raise |
| s0060 | 24.8 | A | valid | http_step_us_b1 | 1,531 | 3,842 | Fragment getter without a conditional load (batch > 2 kernel) |
| s0061 | 26.1 | A | valid | http_step_us_b1 | 1,514 | 3,829 | Hand-over counters polled from warp 6, not from the leader warp 0 |
| s0062 | 26.8 | A | valid | http_step_us_b1 | 1,494 | 3,784 | Attention K/V batches pinned with volatile asm; used in the batch > 2 kernel |
| s0063 | 27.7 | A | valid | http_step_us_b1 | 1,497 | 3,735 | Copy of the attention output with 4 loads in flight (batch > 2 kernel) |
| s0064 | 28.0 | A | valid | http_step_us_b1 | 1,496 | 3,577 | Down phase: one fragment copy up front (compact slots), rounds stream tiles only |
| s0065 | 29.1 | A | valid | http_step_us_b1 | 1,492 | 3,571 | Same engine as s0064; the later experiments behind switches, all off |
| s0066 | 29.3 | A | valid | http_step_us_b1 | 1,493 | 3,527 | Down-phase copy without its pointer array (fewer spills in the batch > 2 kernel) |
| s0067 | 29.7 | A | valid | http_step_us_b1 | 1,493 | 3,499 | Routing merge: the lane's candidate from shared memory, not from a stack array |
| s0068 | 29.9 | A | valid | http_step_us_b1 | 1,493 | 3,460 | Batch 7-8: output projection of the 8 pair blocks computed by 8 other main blocks |
| s0069 | 30.8 | A | valid | http_step_us_b1 | 1,493 | 3,448 | Hosted output projection: hosts moved from blocks 8..15 to blocks 24..31 |
| s0070 | 31.1 | A | valid | http_step_us_b1 | 1,493 | 3,426 | Attention-output copy: all pending words of a batch re-requested per attempt |
| s0071 | 32.6 | A | valid | http_step_us_b1 | 1,493 | 3,426 | Same engine as s0070; cycle-counter study build and two off switches added |
| s0072 | 33.6 | A | valid | http_step_us_b1 | 1,491 | 3,412 | Routing union: two barriers fewer (table zeroed in stage 1, written by stage 2) |
| s0073 | 34.1 | A | valid | http_step_us_b1 | 1,490 | 3,405 | Routing candidates by rank counts in shared memory, no shuffle rounds (batch>2) |
| s0074 | 34.4 | A | valid | http_step_us_b1 | 1,484 | 3,408 | Batch 1-2: routing merge ranks at most 27 pruned candidates, one per lane |
| s0075 | 35.3 | A | valid | http_step_us_b1 | 1,478 | 3,397 | Routing union: last pass (unit tables) on all 16 warps, 4 sequences per thread |
| s0076 | 35.6 | A | valid | http_step_us_b1 | 1,483 | 3,394 | Routing merge: table bytes from the ranking lanes; weights on idle warps (B>2) |
| s0077 | 35.9 | A | valid | http_step_us_b1 | 1,479 | 3,397 | Routing union, last pass: counts of the earlier warps without a loop (B>2) |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 2.0 | A | valid | http_step_us_b1 | 16,990 | 17,606 | Megakernel v0: one launch per step, weights in MMA fragment order (unoptimized) |
| s0002 | 2.2 | A | valid | http_step_us_b1 | 10,834 | 14,059 | Kernel v3: no per-thread stack traffic, activations via L1, one stream loop per warp |
| s0003 | 2.5 | B | error | Exact multi-token passes: MTP-head drafts verified P positions at a time | |||
| s0004 | 2.7 | B | incorrect | Exact draft-and-verify: MTP head + context copy, up to 16 positions per pass | |||
| s0005 | 3.0 | B | valid | http_step_us_b1 | 1,145 | 7,372 | Draft-and-verify engine resubmitted with release/acquire grid synchronisation |
| s0006 | 3.5 | B | valid | http_step_us_b1 | 1,100 | 6,502 | Coalesced RMSNorm prologue and 32-column loop order (faster multi-column passes) |
| s0007 | 3.9 | B | valid | http_step_us_b1 | 824 | 6,460 | 32-position verify passes, L2 prefetch of the next matrix, barrier-free recurrence |
| s0008 | 4.2 | B | valid | http_step_us_b1 | 843 | 5,738 | Per-sequence verify depth in a batch: copyable sequences get more positions |
| s0009 | 4.5 | A | valid | http_step_us_b1 | 10,337 | 12,028 | Track-A engine: everything since s0002 that keeps the checkpoint's formats and data flow |
| s0010 | 6.0 | B | valid | http_step_us_b1 | 622 | 4,496 | Attention per column group, recurrence split over 2 SMs per head, B=8 columns to loops |
| s0011 | 6.5 | A | valid | http_step_us_b1 | 10,267 | 11,795 | Track-A build of the s0010 kernel plus faster 32-column tails |
| s0012 | 6.8 | B | valid | http_step_us_b1 | 582 | 4,145 | Shorter GEMM tails at 32 columns: batched finalize loads, per-kind epilogues, free warp |
| s0013 | 6.9 | A | valid | http_step_us_b1 | 10,184 | 11,724 | Track-A build of the s0012 kernel |
| s0014 | 7.6 | B | valid | http_step_us_b1 | 512 | 3,847 | No replay after full acceptance; spill-free split recurrence; 4-token copy match |
| s0015 | 7.7 | A | valid | http_step_us_b1 | 10,175 | 11,718 | Track-A build of the s0014 kernel |
| s0016 | 9.0 | B | valid | http_step_us_b1 | 506 | 3,762 | Non-blocking GEMM prologues: cp.async residual rows, barrier-free RMSNorm scalar |
| s0017 | 9.1 | A | valid | http_step_us_b1 | 10,195 | 11,473 | Track-A build of the s0016 kernel |
| s0018 | 10.7 | B | valid | http_step_us_b1 | 488 | 3,695 | Attention output folded into value sum; recurrent-op inputs 4 channels per thread |
| s0019 | 10.8 | A | valid | http_step_us_b1 | 10,150 | 11,435 | Track-A build of the s0018 kernel |
| s0020 | 11.2 | B | valid | http_step_us_b1 | 479 | 3,690 | First 8 verify passes one at a time: copied drafts take effect one pass earlier |
| s0021 | 11.7 | B | valid | http_step_us_b1 | 468 | 3,626 | Verify passes skip the MTP head while copied drafts keep being accepted |
| s0022 | 11.9 | B | valid | http_step_us_b1 | 458 | 3,625 | B=1 ramp: pass length chosen at launch from the previous pass's copy result |
| s0023 | 12.4 | A | valid | http_step_us_b1 | 10,154 | 10,988 | Track A: two recurrent units per block at one position (B=8 step -4 %) |
| s0024 | 13.0 | B | valid | http_step_us_b1 | 457 | 3,604 | B=8 recurrent units balanced across blocks; replay deltas via shared memory |
| s0025 | 15.5 | B | valid | http_step_us_b1 | 458 | 3,604 | Serving fixes for 2-7 sequences (token buffers, unit dispatch); same speed |
| s0026 | 15.7 | B | valid | http_step_us_b1 | 457 | 3,626 | Softmax with four warps per row |
| s0027 | 15.8 | A | valid | http_step_us_b1 | 10,139 | 10,925 | Track-A build of the s0026 kernel |
| s0028 | 16.3 | B | valid | http_step_us_b1 | 457 | 3,624 | Prompt prefill in chunks (first token 0.68 s -> 0.09 s); same decode |
| s0029 | 16.4 | A | valid | http_step_us_b1 | 10,146 | 10,923 | Track-A build with chunked prompt prefill |
| s0030 | 17.0 | B | valid | http_step_us_b1 | 450 | 3,573 | Cut tiles handed over with sentinels (one memory trip instead of three) |
| s0031 | 17.1 | A | valid | http_step_us_b1 | 9,981 | 10,713 | Track-A build of the s0030 kernel (sentinel hand-over of cut tiles) |
| s0032 | 17.4 | B | valid | http_step_us_b1 | 449 | 3,571 | Split recurrent op: norm sums exchanged through sentinel slots |
| s0033 | 17.5 | A | valid | http_step_us_b1 | 9,968 | 10,728 | Track-A build of the s0032 kernel |
| s0034 | 18.2 | A | valid | http_step_us_b1 | 9,874 | 10,622 | Track A: no far L2 prefetch before small ops (only 128 KB before each GEMM) |
| s0035 | 18.8 | A | valid | http_step_us_b1 | 9,502 | 10,291 | Track A: GEMM hand-over without 64-bit divisions after the stream |
| s0036 | 18.9 | C | valid | http_step_us_b1 | 432 | 3,437 | Track C build of the s0035 kernel (division-free hand-over, no far prefetch) |
| s0037 | 19.2 | A | valid | http_step_us_b1 | 9,380 | 10,199 | Track A: attention units spread over blocks; recurrent op without a start load |
| s0038 | 19.9 | A | valid | http_step_us_b1 | 9,281 | 10,065 | Track A: GEMM finalize roles at each warp's stream end; leaner prologue |
| s0039 | 20.0 | C | valid | http_step_us_b1 | 426 | 3,404 | Track C build of the s0038 kernel |
| s0040 | 20.4 | A | valid | http_step_us_b1 | 9,258 | 10,065 | Track A: softmax in registers, early gates, split recurrence at B=1 |
| s0041 | 20.6 | C | valid | http_step_us_b1 | 426 | 3,393 | Track C build of the s0040 kernel |
| s0042 | 21.5 | A | valid | http_step_us_b1 | 9,006 | 9,900 | Track A: no local-memory reloads after op calls in the main loop |
| s0043 | 21.6 | C | valid | http_step_us_b1 | 414 | 3,275 | Track C build of the s0042 kernel |
| s0044 | 22.6 | A | valid | http_step_us_b1 | 8,965 | 9,833 | Track A: debug timers compiled out of the default build |
| s0045 | 22.7 | C | valid | http_step_us_b1 | 415 | 3,300 | Track C build of the s0044 kernel (debug timers compiled out) |
| s0046 | 23.0 | A | valid | http_step_us_b1 | 8,947 | 9,819 | Track A: remaining debug stamp checks compiled out |
| s0047 | 23.0 | C | valid | http_step_us_b1 | 417 | 3,310 | Track C build of the s0046 kernel (final) |
| s0048 | 23.3 | A | valid | http_step_us_b1 | 8,938 | 9,800 | Track A: GEMM debug-flag checks compiled out |
| s0049 | 23.4 | C | valid | http_step_us_b1 | 415 | 3,296 | Track C build of the s0048 kernel (final) |
| s0050 | 23.6 | B | valid | http_step_us_b1 | 10,687 | 11,270 | Track B: lossless 12-bit weights (W12) on the s0048 track-A engine |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.1 | A | valid | http_step_us_b1 | 8,940 | 9,808 | Inherited engine s4/s0048 re-measured as this run's baseline |
| s0002 | 0.6 | B | incorrect | W12: exact 12-bit weights, per-row exponent windows, escapes as sparse correction | |||
| s0003 | 0.9 | B | incorrect | Exact W12: 12-bit weights with per-row exponent windows, bit-identical to BF16 | |||
| s0004 | 1.1 | B | valid | http_step_us_b1 | 8,670 | 9,193 | Exact W12 (bit-identical to BF16), now also for multi-column prefill passes |
| s0005 | 1.5 | B | error | Exact W12, W12-only build: unused BF16 and old W12 stream calls compiled out | |||
| s0006 | 1.5 | B | valid | http_step_us_b1 | 8,557 | 9,136 | Exact W12, W12-only build (resubmission of s0005, lost to pod failures) |
| s0007 | 1.6 | A | valid | http_step_us_b1 | 8,892 | 9,812 | Track A: activation fragments loaded only for real columns (zeros for the rest) |
| s0008 | 2.0 | B | valid | http_step_us_b1 | 8,409 | 9,131 | Exact W12 + compact one-column activation layout at B=1 (8x smaller L1 footprint) |
| s0009 | 2.4 | B | valid | http_step_us_b1 | 8,263 | 9,009 | Exact W12 with 16-exponent row windows: half the escapes, less patching |
| s0010 | 2.8 | B | valid | http_step_us_b1 | 8,227 | 8,975 | Exact W12: escape positions counted down, loop counters back in registers |
| s0011 | 4.0 | B | valid | http_step_us_b1 | 8,194 | 8,938 | Exact W12, byte-permute layout: cheaper decode (5 ops per register) |
| s0012 | 4.2 | B | valid | http_step_us_b1 | 8,169 | 8,908 | Exact W12: activation loads issued before the per-quad escape test |
| s0013 | 4.6 | B | valid | http_step_us_b1 | 8,162 | 8,897 | Exact W12: compact-layout test hoisted out of the activation producers' loops |
| s0014 | 5.0 | B | valid | http_step_us_b1 | 8,120 | 8,914 | Exact W12 + BF16 out_proj/o_proj for one-column passes (storage chosen per op) |
| s0015 | 5.2 | B | valid | http_step_us_b1 | 8,104 | 8,860 | Exact W12: 96 KB per block of the next matrix prefetched into L2 during small ops |
| s0016 | 5.5 | A | valid | http_step_us_b1 | 8,880 | 9,827 | Track A: grid-barrier spin back-off 200 -> 25 ns |
| s0017 | 5.6 | B | valid | http_step_us_b1 | 8,095 | 8,862 | Exact W12 with the 25 ns grid-barrier back-off |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 4.2 | A | error | Persistent per-card kernel, TP4 attention + intra-expert TP4, B=1 redundant mode | |||
| s0002 | 4.7 | A | valid | http_step_us_b1 | 6,564 | 10,330 | Fix deadlock (header by value); relaxed signals; smem-staged B frags; phases pre-wait |
| s0003 | 5.5 | A | valid | http_step_us_b1 | 5,324 | 9,310 | 768-thread blocks, block-contiguous GEMV, per-strip GEMVs, vectorised token phases |
| s0004 | 6.2 | A | incorrect | LL tagged-data cross-card exchange, router per-strip, attention and mix fixes | |||
| s0005 | 6.4 | A | valid | http_step_us_b1 | 4,778 | 8,845 | Slot-order invariant expert scaling (explicit FMA) on top of s0004 |
| s0006 | 6.8 | A | valid | http_step_us_b1 | 4,581 | 8,242 | HW E2M1 rounding, 64-position attention chunks at B=1, w2 prefetch only at B=1 |
| s0007 | 8.5 | A | valid | http_step_us_b1 | 4,541 | 7,731 | B=8 expert GEMVs: w2 weights+fragments in flight across experts, quant2 warp tasks; split-K router |
| s0008 | 9.3 | A | valid | http_step_us_b1 | 4,311 | 7,477 | Cross-card polls with all loads in flight before spinning; redux top-8; prefetch by last warp |
| s0009 | 10.0 | A | valid | http_step_us_b1 | 4,260 | 7,415 | Load-depth tuning per mode (FP4/MXFP8 unroll), router bias before the poll, async conv loads at B>=2 |
| s0010 | 10.5 | A | valid | http_step_us_b1 | 4,078 | 7,214 | Attention starts on a counter of the qkv strips it reads; acq_rel hand-off atomics |
| s0011 | 11.2 | A | valid | http_step_us_b1 | 4,049 | 7,260 | Fix B>=2 combine/gate-row race (barrier back at B>=2); paired o_proj strips |
| s0012 | 12.2 | A | valid | http_step_us_b1 | 3,959 | 7,194 | Register relief: 32-bit GEMV indices, launch state in shared memory; two-chain B=1 quant2 |
| s0013 | 12.7 | A | valid | http_step_us_b1 | 3,624 | 6,955 | Cross-card partial sums as bf16 pairs (half the polled bytes); adaptive B>=2 chunks |
| s0014 | 13.6 | A | build_failed | B>=2: whole-region K/V prefetch (barrier no longer waits for prefetch issue); routing-masked w2 inputs |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.1 | A | valid | http_step_us_b1 | 4,807 | 8,861 | Baseline: run s6's s0005 source unchanged (LL exchange, slot-invariant FMA) |
| s0002 | 1.0 | A | valid | http_step_us_b1 | 4,163 | 7,125 | Team vector phases: a token's mix/norm/quantize on 6 blocks, 1 element per thread |
| s0003 | 1.8 | A | valid | http_step_us_b1 | 3,984 | 6,537 | GEMV phases: two strips per block at once, w2 quant per routed pair, no staging for qkv/router |
| s0004 | 2.8 | A | valid | http_step_us_b1 | 3,891 | 6,335 | Attention scores as BF16 MMAs; tag-checked hand-offs without fences in the w13 path |
| s0005 | 3.1 | A | valid | http_step_us_b1 | 3,646 | 5,870 | Shared memory per block 222 -> 121 KB: the SM's L1 gets the difference |
| s0006 | 3.5 | A | valid | http_step_us_b1 | 3,661 | 5,742 | Expert GEMV unroll 5 (NVFP4) / 4 (MXFP8); activation function moved to the w2 quantization block |
| s0007 | 3.8 | A | valid | http_step_us_b1 | 3,440 | 5,773 | w13 GEMV for few experts: whole strips per block, four at a time (B=1) |
| s0008 | 4.0 | A | valid | http_step_us_b1 | 3,401 | 5,758 | Attention: 32 or 64 positions per chunk block while blocks are free |
| s0009 | 4.2 | A | valid | http_step_us_b1 | 3,310 | 5,716 | Output-gate projection computed beside the attention phase by idle blocks |
| s0010 | 4.6 | A | valid | http_step_us_b1 | 3,322 | 5,660 | Gate split with 24 free blocks (B=8 up to context 1024); own strip piece from registers |
| s0011 | 4.9 | A | valid | http_step_us_b1 | 3,314 | 5,599 | Flat GEMV strip hand-off without a counter: the first-piece warp finishes the strip |
| s0012 | 5.1 | A | valid | http_step_us_b1 | 3,299 | 5,558 | Attention: the block of the last chunk combines; no chunk counter, no fence on its path |
| s0013 | 5.8 | A | valid | http_step_us_b1 | 3,296 | 5,484 | Attention, routing and w2-quantization phases compiled out of line (register cap) |
| s0014 | 7.8 | A | error | NVFP4 w13 at batch >= 4: tcgen05 block-scaled MMAs from a second layout | |||
| s0015 | 8.7 | A | valid | http_step_us_b1 | 3,300 | 5,552 | s0014 again: tcgen05 w13 at batch >= 4, race fix and a stall report |
| s0016 | 9.0 | A | valid | http_step_us_b1 | 3,300 | 5,413 | Tensor-core kernel: its calls moved out of the kernel body (routing phase) |
| s0017 | 10.4 | A | valid | http_step_us_b1 | 3,298 | 5,420 | Progress pointer out of the launch header: plain kernel back at STACK:96 |
| s0018 | 10.9 | A | valid | http_step_us_b1 | 3,298 | 5,440 | Tensor-core w13: sub-items of 12 units instead of 9 |
| s0019 | 11.3 | A | valid | http_step_us_b1 | 3,307 | 5,390 | Tensor-core w13: its scale-factor stores spread over three warp groups |
| s0020 | 11.5 | A | valid | http_step_us_b1 | 3,307 | 5,378 | Tensor-core kernel: w13 quantization emit inlined in its team phase |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.1 | A | incorrect | Inherited engine unchanged (s6 s0003): baseline on this run's node | |||
| s0002 | 0.2 | A | valid | http_step_us_b1 | 5,317 | 9,340 | Inherited engine + explicit FMAs in expert scaling (slot-order invariant) |
| s0003 | 0.4 | A | valid | http_step_us_b1 | 4,819 | 8,895 | Kernel of run s6's s0005: tagged-data cross-card exchange, router per strip, attention fixes |
| s0004 | 1.1 | A | valid | http_step_us_b1 | 4,794 | 8,486 | Parallel mailbox polls; B=8 w2 prefetch issued at once; B=1 token phases as own functions |
| s0005 | 1.3 | A | valid | http_step_us_b1 | 4,797 | 8,031 | B>=2: w2 input as compact per-pair fragments staged in shared memory |
| s0006 | 1.7 | A | valid | http_step_us_b1 | 4,682 | 7,614 | w13 and unembedding on block-contiguous strips with shared-memory hand-off; 4 FP4 units in flight at B>=2 |
| s0007 | 2.4 | A | valid | http_step_us_b1 | 4,688 | 7,606 | Loader only: runtime buffers allocated before the unembedding; unembedding storage chosen at load |
| s0008 | 2.6 | B | valid | http_step_us_b1 | 4,679 | 7,596 | Track B: unembedding stored in an exact 12-bit code (bit-identical values, 25% fewer bytes) |
| s0009 | 3.6 | A | valid | http_step_us_b1 | 4,568 | 7,263 | Prefetch issued by idle warps; one cross-card store per destination per strip; batched re-polls at B>=2 |
| s0010 | 3.8 | A | valid | http_step_us_b1 | 4,246 | 7,117 | NVFP4 input quantization with the hardware E2M1 conversion; w2 inputs one 16-block per thread |
| s0011 | 4.8 | A | valid | http_step_us_b1 | 4,239 | 7,035 | B>=2: routing per token block, conv work on idle blocks; top-8 with a shorter dependency chain |
| s0012 | 5.1 | A | valid | http_step_us_b1 | 4,235 | 7,044 | Kernel prepared for load-time token tables (switch off): reference for the track-B submissions |
| s0013 | 5.1 | B | valid | http_step_us_b1 | 4,222 | 7,037 | Track B: load-time token tables for layer 0 (normed embedding, layer-0 projections) |
| s0014 | 5.6 | B | valid | http_step_us_b1 | 4,195 | 7,004 | Track B: 12-bit unembedding and layer-0 token tables together |
| s0015 | 5.9 | A | valid | http_step_us_b1 | 4,215 | 6,899 | B>=2: MXFP8 expert GEMVs load the next unit before computing the current one |
| s0016 | 6.1 | B | valid | http_step_us_b1 | 4,188 | 6,884 | Track B on the s0015 kernel: 12-bit unembedding and layer-0 token tables |
| s0017 | 6.6 | A | valid | http_step_us_b1 | 4,174 | 6,901 | B=1: attention scores on the tensor cores; value sum with 8 dimensions per thread |
| s0018 | 6.8 | B | valid | http_step_us_b1 | 4,151 | 6,882 | Track B on the s0017 kernel: 12-bit unembedding and layer-0 token tables |
| s0019 | 7.3 | A | valid | http_step_us_b1 | 4,176 | 6,803 | B>=2: NVFP4 units loaded per group: 6 for w13, 5 for w2 (was 4) |
| s0020 | 8.0 | A | valid | http_step_us_b1 | 4,074 | 6,540 | Prefetch policy re-measured per kernel; reader-side fences removed; tensor-core scores at B>=2 |
| s0021 | 8.4 | B | valid | http_step_us_b1 | 4,051 | 6,513 | Track B on the s0020 kernel: 12-bit unembedding and layer-0 token tables |
| s0022 | 9.0 | A | valid | http_step_us_b1 | 4,014 | 6,425 | Block-per-strip GEMVs run a block's two strips at once on the two halves of its warps |
| s0023 | 9.1 | B | valid | http_step_us_b1 | 3,991 | 6,406 | Track B on the s0022 kernel: 12-bit unembedding and layer-0 token tables |
| s0024 | 11.1 | A | valid | http_step_us_b1 | 3,997 | 6,459 | B=1: attention items of 64 positions (twice as many blocks busy, each with half the work) |
| s0025 | 11.2 | B | valid | http_step_us_b1 | 3,975 | 6,436 | Track B on the s0024 kernel: 12-bit unembedding and layer-0 token tables |
| s0026 | 11.3 | B | valid | http_step_us_b1 | 3,975 | 6,436 | Track B, final: s0025 with the MK_NO_RED debugging switch made harmless (host side only) |
| s0027 | 11.5 | A | valid | http_step_us_b1 | 3,997 | 6,455 | B>=2 kernel builds its pair map for every batch size (a single sequence no longer hangs it) |
| s0028 | 11.6 | B | valid | http_step_us_b1 | 3,974 | 6,433 | Track B, final: 12-bit unembedding and layer-0 token tables on the s0027 kernel |
| s0029 | 11.8 | A | valid | http_step_us_b1 | 3,308 | 5,394 | Sibling run fa1's final kernel (its s0020 source, unchanged) as this run's track-A baseline |
| s0030 | 12.5 | B | valid | http_step_us_b1 | 3,301 | 5,376 | Track B on the sibling's final kernel: 12-bit unembedding + layer-0 token tables; context limit enforced |
| s0031 | 12.8 | B | valid | http_step_us_b1 | 3,291 | 5,376 | Track B: unembedding decoder chosen at build time (s0030's binary carried both) |
| s0032 | 13.0 | A | valid | http_step_us_b1 | 3,308 | 5,395 | Track-A twin of s0031: the s0020 kernel with the context guard (same source, TRACK = A) |
| s0033 | 13.7 | B | valid | http_step_us_b1 | 3,290 | 5,378 | Track B: s0031 plus a token-id range check in submit() (same kernel source) |
| s0034 | 14.2 | A | valid | http_step_us_b1 | 3,307 | 5,394 | Track A from a binary that serves both tracks (unembedding decoder chosen at load, compiled per kernel) |
| s0035 | 14.2 | B | valid | http_step_us_b1 | 3,295 | 5,383 | Track B from the same binary as s0034: 12-bit unembedding + layer-0 token tables |
| s0036 | 14.4 | B | valid | http_step_us_b1 | 3,291 | 5,378 | Track B: s0033 with the 12-bit round trip asserted at every load (same kernel source) |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.1 | A | incorrect | inherited engine | |||
| s0002 | 2.0 | A | valid | slo_share_r2 | 8.5% | pass engine: rows of all requests through one tcgen05 pass, paged KV, prefix cache | |
| s0003 | 2.4 | A | valid | slo_share_r2 | 86.8% | scheduler: 80 ms passes, shortest-remaining-first within the TTFT limit; wider GEMM tiles | |
| s0004 | 3.1 | A | valid | slo_share_r2 | 88.9% | pass length from the decoding requests' gap credit; cheaper decode rows and small passes | |
| s0005 | 3.5 | A | valid | slo_share_r2 | 92.1% | experts move between cards with the traffic; expert outputs summed on one card | |
| s0006 | 3.9 | A | valid | slo_share_r2 | 92.6% | tcgen05 chains: one issuing thread per row group; pass length follows used-up credit | |
| s0007 | 4.3 | A | valid | slo_share_r2 | 93.7% | dense GEMMs: four row groups per block on large passes (one round of block-units) | |
| s0008 | 5.1 | A | valid | slo_share_r2 | 92.6% | no host scheduler: admission, page store, prefix cache and pass composition in a kernel | |
| s0009 | 5.3 | A | valid | slo_share_r2 | 93.1% | expert placement on the device too: swaps decided by the scheduler kernel | |
| s0010 | 6.3 | A | valid | slo_share_r2 | 95.8% | attention on tensor cores: scores, softmax and value product in one block | |
| s0011 | 6.8 | A | valid | slo_share_r2 | 96.3% | GEMM chains: fewer, larger operand copies per stage (dense 4 K steps, experts 2) | |
| s0012 | 8.1 | A | valid | slo_share_r2 | 97.9% | a home card per row: residual, mixes, o_proj and router once, not on every card | |
| s0013 | 8.5 | A | valid | slo_share_r2 | 98.4% | passes of up to 2560 rows (they no longer step up above 2048) | |
| s0014 | 9.2 | A | valid | slo_share_r2 | 98.4% | attention: far keys without bias or mask, fold with 16-byte loads | |
| s0015 | 9.3 | A | valid | slo_share_r2 | 98.4% | submit() and poll() make no call into the cards: prompts are copied in by the scheduler kernel | |
| s0016 | 9.9 | A | valid | slo_share_r2 | 98.4% | small passes: three expert row groups per block when one per block needs more rounds | |
| s0017 | 10.1 | A | valid | slo_share_r2 | 99.5% | GEMM read-out by 16 warps instead of 4 | |
| s0018 | 10.6 | A | valid | slo_share_r2 | 99.5% | assignment kernel: ids read 16 bytes at a time, four rows ahead; unused experts skipped | |
| s0019 | 10.8 | A | valid | slo_share_r2 | 100.0% | largest passes (3584 rows) only while a waiting prompt would miss its first-token time | |
| s0020 | 12.5 | A | valid | slo_share_r2 | 100.0% | recomputations of evicted blocks counted on the device; source comments brought up to date | |
| s0021 | 13.0 | A | valid | slo_share_r2 | 100.0% | softmax: first-pass scores kept in tensor memory (tcgen05.st), zeros through exp | |
| s0022 | 13.2 | A | valid | slo_share_r2 | 100.0% | pressure rule with 0.3 s of slack: the last request of a burst no longer ends at the limit | |
| s0023 | 13.8 | A | valid | slo_share_r2 | 100.0% | dense GEMMs sum their K range as two halves: on two SMs in small passes | |
| s0024 | 14.6 | A | valid | slo_share_r2 | 100.0% | scheduler: key table overrun for prompts > 33,279 tokens fixed; submit guards | |
| s0025 | 15.5 | A | valid | slo_share_r2 | 100.0% | guards: token ring cannot lap a slow host; token ids checked at submit() | |
| s0026 | 16.9 | A | valid | slo_share_r2 | 100.0% | scheduler kernel: its serial per-pass work by all threads; eviction without scan | |
| s0027 | 17.1 | A | valid | slo_share_r2 | 100.0% | scheduler: cache table keeps empty slots over hours; a waiting prompt hashed once | |
| s0028 | 17.9 | A | valid | slo_share_r2 | 100.0% | unembedding in small passes: both K halves in one block, 128-row units | |
| s0029 | 19.2 | A | valid | slo_share_r2 | 100.0% | small passes: end of a layer fused with the next mix; fetch kernels loop over rows | |
| s0030 | 19.7 | A | valid | slo_share_r2 | 100.0% | step()/generate() after submit()/poll() on one engine: loop stopped and restarted | |
| s0031 | 20.1 | A | valid | slo_share_r2 | 100.0% | routing fetch back to one block per row: the row loop cost large passes 1 ms | |
| s0032 | 20.6 | A | valid | slo_share_r2 | 100.0% | routing fetch: a row's scales, ids, weights, outer scale as one contiguous piece | |
| s0033 | 21.4 | A | valid | slo_share_r2 | 100.0% | convolution-state fetch: 576 threads per row, two remote loads in series, not six | |
| s0034 | 22.7 | A | valid | slo_share_r2 | 100.0% | attention-output gather: one launch, 768 threads, one remote load per thread | |
| s0035 | 23.2 | A | valid | slo_share_r2 | 100.0% | routing fetch: the packed piece read by threads that carry no codes | |
| s0036 | 24.3 | A | valid | slo_share_r8 | 83.6% | audit fixes: slot buffers for 64 slots at any max_batch; cache matches verified | |
| s0037 | 24.7 | A | valid | slo_share_r8 | 99.0% | 128 sequences in flight instead of 64: at 8 req/s the slots were the capacity | |
| s0038 | 25.1 | A | valid | slo_share_r8 | 99.0% | admission by priority: on-time and cheapest first, late requests when there is room | |
| s0039 | 26.3 | A | valid | slo_share_r8 | 99.0% | largest passes for a prompt nobody waits behind: 32,000 tokens 808 -> 752 ms | |
| s0040 | 26.9 | A | valid | slo_share_r8 | 99.0% | audit doubts: failed page allocation surfaces as an error; open request ids unique | |
| s0041 | 27.4 | A | valid | slo_share_r8 | 99.4% | page store from the memory really free: 2149 -> 2439 pages | |
| s0042 | 28.9 | A | valid | slo_share_r8 | 99.7% | pressure rule: the largest passes 2 s earlier while 32+ sequences decode | |
| s0043 | 30.5 | A | valid | slo_share_r8 | 99.7% | prompts: promotion ahead of shortest-first when it saves one and costs nobody |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.2 | A | valid | http_step_us_b1 | 8,885 | 9,836 | Inherited engine (s5 s0016) with the stats() track label fixed |
| s0002 | 1.0 | A | valid | http_step_us_b1 | 8,701 | 9,453 | Weight-stream loads use the cache-streaming operator (ld.global.cs) |
| s0003 | 1.4 | A | valid | http_step_us_b1 | 8,694 | 9,222 | B=8 recurrent op in one pass per block (three units, inputs loaded once) |
| s0004 | 1.7 | A | valid | http_step_us_b1 | 8,641 | 9,145 | Barrier-time prefetch re-tuned for cs loads: near 160 KB, far 64 KB |
| s0005 | 1.9 | A | valid | http_step_us_b1 | 8,628 | 9,134 | Served decode: tokens written by the kernel into host-mapped memory |
| s0006 | 2.2 | A | valid | http_step_us_b1 | 8,587 | 9,114 | Prefetch during small ops: consecutive pieces, idle blocks take their whole share |
| s0007 | 2.4 | B | valid | http_step_us_b1 | 7,847 | 8,360 | Track B build of the s0006 kernel: exact 12-bit weight storage |
| s0008 | 3.3 | A | valid | http_step_us_b1 | 8,492 | 9,077 | Attention layer as one op: a block per 256-key chunk, sentinel hand-overs |
| s0009 | 3.4 | B | valid | http_step_us_b1 | 7,706 | 8,253 | Track B build of the s0008 kernel: exact 12-bit weight storage |
| s0010 | 4.0 | A | valid | http_step_us_b1 | 8,494 | 8,982 | B=8: RMSNorm partials by the four prefetched warps, r finished before the stream ends |
| s0011 | 4.0 | B | valid | http_step_us_b1 | 7,709 | 8,184 | Track B build of the s0010 kernel: exact 12-bit weight storage |
| s0012 | 4.3 | A | valid | http_step_us_b1 | 8,492 | 8,956 | B=8 recurrent op: the next state column's loads fly while the current one is computed |
| s0013 | 4.4 | B | valid | http_step_us_b1 | 7,708 | 8,166 | Track B build of the s0012 kernel: exact 12-bit weight storage |
| s0014 | 4.8 | A | valid | http_step_us_b1 | 8,492 | 8,948 | B=8: conv-history rows prefetched during in_proj; far prefetch piece 80 KB |
| s0015 | 5.0 | B | valid | http_step_us_b1 | 7,708 | 8,193 | Track B build of the s0014 kernel: exact 12-bit weight storage |
| s0016 | 5.2 | A | valid | http_step_us_b1 | 8,494 | 8,905 | Attention blocks ask for their K chunk at the barrier before the op (B >= 4) |
| s0017 | 5.3 | B | valid | http_step_us_b1 | 7,710 | 8,131 | Track B build of the s0016 kernel: exact 12-bit weight storage |
| s0018 | 6.1 | A | valid | http_step_us_b1 | 8,466 | 8,899 | Grid barrier: a block's prefetch requests go out after its counter increment |
| s0019 | 6.2 | B | valid | http_step_us_b1 | 7,689 | 8,116 | Track B build of the s0018 kernel: exact 12-bit weight storage |
| s0020 | 6.4 | A | valid | http_step_us_b1 | 8,455 | 8,880 | Grid barrier: one prefetch piece per poll, none after the release |
| s0021 | 6.6 | B | valid | http_step_us_b1 | 7,665 | 8,102 | Track B build of the s0020 kernel: exact 12-bit weight storage |
| s0022 | 7.1 | A | valid | http_step_us_b1 | 8,445 | 8,878 | Far prefetch piece 128 KB for the BF16 engine (host constant) |
| s0023 | 7.2 | B | valid | http_step_us_b1 | 7,668 | 8,114 | Track B build of the s0022 source (unchanged from s0021: repeat measurement) |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.3 | A | valid | http_step_us_b1 | 5,376 | 5,634 | Track A: attention scores stay float32 (no BF16 rounding of q.k) |
| s0002 | 0.7 | B | valid | http_step_us_b1 | 4,136 | 4,461 | Track B build of the s0001 source (12-bit weight code, float32 intermediates) |
| s0003 | 1.5 | A | valid | http_step_us_b1 | 5,373 | 5,634 | Track A: final source (test-only switches, trace-build fix), same device code as s0001 |
| s0004 | 2.3 | A | valid | http_step_us_b1 | 5,365 | 5,622 | Track A: idle blocks pull K of all units into L2 first (1-2 sequences) |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.1 | B | valid | http_step_us_b1 | 4,162 | 4,457 | Inherited engine, track B (s10 s0002 device code), re-measured on this pod |
| s0002 | 0.8 | A | valid | http_step_us_b1 | 5,363 | 5,684 | L2 staging of gate+up tiles by idle blocks during attention (track A) |
| s0003 | 0.8 | B | valid | http_step_us_b1 | 4,095 | 4,476 | L2 staging of gate+up tiles by idle blocks during attention (track B) |
| s0004 | 0.9 | A | valid | http_step_us_b1 | 5,227 | 5,663 | L2 staging of the next phase's middle tile at each matvec phase end (track A) |
| s0005 | 1.0 | B | valid | http_step_us_b1 | 4,015 | 4,477 | L2 staging of the next phase's middle tile at each matvec phase end (track B) |
| s0006 | 1.1 | B | valid | http_step_us_b1 | 3,998 | 4,415 | Tail L2 staging also with 3-8 sequences (3 x 16 KB per block) |
| s0007 | 1.2 | B | valid | http_step_us_b1 | 4,008 | 4,369 | Layer-0 q/k/v from a per-token table built by the kernel's own q/k/v phase |
| s0008 | 1.3 | B | valid | http_step_us_b1 | 3,952 | 4,317 | Online submit/poll contract: one device-loop launch per cohort, tokens via host-mapped memory |
| s0009 | 1.4 | A | valid | http_step_us_b1 | 5,160 | 5,588 | Online submit/poll contract (track A) |
| s0010 | 1.4 | A | valid | http_step_us_b1 | 5,162 | 5,486 | Tail L2 staging also with 3-8 sequences (track A) |
| s0011 | 1.5 | B | valid | http_step_us_b1 | 3,895 | 4,248 | Certified int8 sketch of lm_head + exact rescoring on the greedy/online paths |
| s0012 | 1.8 | B | valid | http_step_us_b1 | 3,869 | 4,263 | Token ids outside the vocabulary: no read past the tables, ValueError |
| s0013 | 2.0 | B | valid | http_step_us_b1 | 3,881 | 4,247 | Head sketch: x preparation with 16-byte loads in the raw fragment layout |
| s0014 | 2.7 | B | valid | http_step_us_b1 | 3,865 | 4,255 | No L2 prefetch of the second tile behind a stashed first tile (O2PF 0) |
| s0015 | 3.1 | A | incorrect | Track A re-tuned for the staged engine: 1.5 staged tiles, 4 stash loads in flight, no o prefetch | |||
| s0016 | 3.2 | A | valid | http_step_us_b1 | 5,134 | 5,462 | Track A re-tuned: 1.5 staged tiles, 4 stash loads in flight (fix of s0015) |
| s0017 | 3.6 | B | valid | http_step_us_b1 | 3,837 | 4,240 | Exception MMAs read x from shared memory instead of L2 (from a phase's 2nd tile) |
| s0018 | 3.9 | B | valid | http_step_us_b1 | 3,825 | 4,258 | Exception x from shared memory only in the greedy kernel (fixes s0017's step() spill) |
| s0019 | 6.2 | B | valid | http_step_us_b1 | 3,843 | 4,257 | Tail staging after o targets the block's 3rd gate+up tile instead of the middle one |
| s0020 | 6.8 | B | valid | http_step_us_b1 | 3,813 | 4,253 | Attention window: shorter score-stamp wait, more polite o-stash loads (track B re-tune) |
| s0021 | 9.6 | B | valid | http_step_us_b1 | 3,822 | 4,241 | Head sketch: rescoring of candidate tiles spread over all blocks (3+ sequences) |
| s0022 | 12.4 | B | valid | http_step_us_b1 | 3,813 | 4,231 | 3+ sequences: stage gate+up into L2 after the attention blocks' score pass |
| s0023 | 12.6 | B | valid | http_step_us_b1 | 3,815 | 4,221 | 3+ sequences: idle blocks' o-tile prefetch deferred until the score pass is done |
| s0024 | 12.9 | B | valid | http_step_us_b1 | 3,817 | 4,211 | 3+ sequences, long contexts: idle blocks pull V (only) into L2 during attention |
| s0025 | 13.2 | A | valid | http_step_us_b1 | 5,134 | 5,485 | Track A: idle blocks pull V into L2 during attention beyond the touch limit |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.7 | B | valid | http_step_us_b1 | 2,123 | 5,769 | Track B made bit-identical: W12 escapes patched into mma operands, L0 tables by kernel |
| s0002 | 2.5 | A | valid | http_step_us_b1 | 2,540 | 6,901 | Track A rebuild of the current source (bit-identical partner of track B) |
| s0003 | 2.7 | B | valid | http_step_us_b1 | 2,108 | 5,769 | Batch-1 kernel bulk-prefetches its next-layer stage-A tiles into L2 (W12 only) |
| s0004 | 3.1 | A | valid | http_step_us_b1 | 2,535 | 6,823 | Pad 8-column operand strides (bank conflicts) - track A |
| s0005 | 3.3 | B | valid | http_step_us_b1 | 2,099 | 5,646 | Pad 8-column operand strides (bank conflicts) - track B |
| s0006 | 5.0 | A | valid | http_step_us_b1 | 2,524 | 6,745 | Attention runs two chunks at a time (8 warps each) - track A |
| s0007 | 5.1 | B | valid | http_step_us_b1 | 2,103 | 5,612 | Attention runs two chunks at a time (8 warps each) - track B |
| s0008 | 6.2 | A | valid | http_step_us_b1 | 2,534 | 6,753 | Share model: attention cost per K tile 12 at batch > 1 - track A |
| s0009 | 6.3 | B | valid | http_step_us_b1 | 2,109 | 5,587 | Share model: attention cost per K tile 12 at batch > 1 - track B |
| s0010 | 6.7 | A | valid | http_step_us_b1 | 2,526 | 6,688 | Batch>1 kernels: 8 head loads in flight per thread - track A |
| s0011 | 6.8 | B | valid | http_step_us_b1 | 2,099 | 5,533 | Batch>1 kernels: 8 head-sketch loads in flight per thread - track B |
| s0012 | 7.2 | A | valid | http_step_us_b1 | 2,526 | 6,569 | Batch>1 kernels launch with 16 KB less shared memory (larger L1) - track A |
| s0013 | 7.3 | B | valid | http_step_us_b1 | 2,107 | 5,437 | Batch>1 kernels launch with 16 KB less shared memory (larger L1) - track B |
| s0014 | 7.6 | A | valid | http_step_us_b1 | 2,517 | 6,567 | Gate|up share table cached across steps of a launch - track A |
| s0015 | 7.7 | B | valid | http_step_us_b1 | 2,104 | 5,408 | Gate|up share table cached across steps of a launch - track B |
| s0016 | 7.8 | A | valid | http_step_us_b1 | 2,517 | 6,526 | RMSNorm loads of the residual bypass L1 - track A |
| s0017 | 7.9 | B | valid | http_step_us_b1 | 2,101 | 5,392 | RMSNorm loads of the residual bypass L1 - track B |
| s0018 | 8.4 | B | valid | http_step_us_b1 | 2,076 | 5,384 | Batch-1 L2 prefetch limited to the half tile and the first whole stage-A tile |
| s0019 | 8.7 | B | valid | http_step_us_b1 | 2,051 | 5,385 | Batch-1: down tiles of the first two experts to L2 while waiting at barrier B |
| s0020 | 9.1 | A | valid | http_step_us_b1 | 2,506 | 6,525 | Batch-1: down tiles of the first two experts to L2 at barrier B - track A |
| s0021 | 9.2 | B | valid | http_step_us_b1 | 2,044 | 5,392 | Batch-1: previous token's experts' gate|up tiles to L2 during the union wait |
| s0022 | 10.0 | B | valid | http_step_us_b1 | 2,040 | 5,391 | Barrier-wait L2 prefetches (stage-A tiles, first down tiles) also at batch > 1 |
| s0023 | 10.7 | B | valid | http_step_us_b1 | 2,045 | 5,320 | Greedy head sketch skips the 439 tiles made only of copies of earlier rows |
| s0024 | 11.9 | A | valid | http_step_us_b1 | 2,501 | 6,493 | Track A rebuild of the final source (partner of track B s0023) |
| s0025 | 12.9 | B | valid | http_step_us_b1 | 2,050 | 5,336 | Final source, track B (no functional change from s0023) |
| s0026 | 14.6 | B | valid | http_step_us_b1 | 2,038 | 5,334 | Batch-1: barrier-B poll with the last o_proj quantum, attention delay 30 in the share model |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.9 | A | valid | http_step_us_b1 | 1,488 | 3,426 | Track A re-measure: inherited s0077 kernel, track-B code compiled out |
| s0002 | 0.9 | B | valid | http_step_us_b1 | 1,473 | 3,438 | Track B: certified int8 sketch of lm_head + exact rescoring on the greedy paths |
| s0003 | 1.9 | A | valid | http_step_us_b1 | 1,477 | 3,417 | Track A: attention scores as K x q^T, 8 mma per key group instead of 16 |
| s0004 | 2.3 | B | valid | http_step_us_b1 | 1,471 | 3,437 | Track B on s0003: int8 head sketch on the greedy paths (scores code per track) |
| s0005 | 2.7 | B | valid | http_step_us_b1 | 1,472 | 3,428 | Track B: head sketch in the batch 1-2 kernel only; batch>2 kernel = track A's |
| s0006 | 3.2 | B | valid | http_step_us_b1 | 1,472 | 3,456 | Track B final: s0005 kernel + submit() refuses out-of-range token ids |
| s0007 | 4.6 | A | valid | http_step_us_b1 | 1,463 | 3,422 | Track A: L2 prefetch of next q/k/v and shared expert from an idle warp |
| s0008 | 5.9 | A | valid | http_step_us_b1 | 1,452 | 3,426 | Track A: q/k/v prefetch at the end of C, shared expert at the start of A |
| s0009 | 7.7 | A | valid | http_step_us_b1 | 1,422 | 3,427 | Track A: 512-byte device allocation before the engine's own (buffer layout) |
| s0010 | 8.5 | A | valid | http_step_us_b1 | 1,419 | 3,427 | Track A re-measure of s0009 (identical source): layout fix reproducibility |
| s0011 | 10.0 | A | valid | http_step_us_b1 | 1,430 | 3,443 | Track A trade-off: 4 accumulators in phase D (B=8 -57 us, B=1 +11 us) |
| s0012 | 10.4 | A | valid | http_step_us_b1 | 1,419 | 3,425 | Track A: two K batches in flight for attention pairs on main blocks (batch 8) |
| s0013 | 11.1 | A | valid | http_step_us_b1 | 1,417 | 3,356 | Track A: L2 evict-first hint on the expert and head tile loads |
| s0014 | 11.6 | A | valid | http_step_us_b1 | 1,412 | 3,373 | Track A: evict-first expert/head loads without L1::no_allocate |
| s0015 | 12.0 | A | valid | http_step_us_b1 | 1,408 | 3,368 | Track A: last layer's idle q/k/v prefetch slot fetches the head's first 20 MB |
| s0016 | 12.8 | B | valid | http_step_us_b1 | 1,400 | 3,362 | Track B: int8 head sketch on the s0015 kernel, with the 8-mma scores form |
| s0017 | 14.1 | A | valid | http_step_us_b1 | 1,418 | 3,379 | Track A: dense-layer MLP tiles preloaded before their barriers |
| s0018 | 14.2 | B | valid | http_step_us_b1 | 1,380 | 3,367 | Track B: s0017's dense-layer MLP preloads with the int8 head sketch |
| s0019 | 14.8 | A | valid | http_step_us_b1 | 1,402 | 3,367 | Track A: dense-layer MLP epilogue: plain stores + one fence, down preload after |
| s0020 | 14.8 | B | valid | http_step_us_b1 | 1,374 | 3,350 | Track B: s0019's dense-layer epilogue with the int8 head sketch |
| s0021 | 15.5 | A | valid | http_step_us_b1 | 1,398 | 3,361 | Track A: dense-layer gate/up loop unrolled by 16 (more tiles in flight) |
| s0022 | 15.6 | B | valid | http_step_us_b1 | 1,372 | 3,353 | Track B: s0021's dense-layer gate/up unroll with the int8 head sketch |
| s0023 | 16.0 | B | valid | http_step_us_b1 | 1,368 | 3,352 | Track B: head sketch pass 2 staged through shared memory (one load round) |
| s0024 | 16.9 | A | valid | http_step_us_b1 | 1,400 | 3,355 | Track A: chosen tokens published by atomic exchange, bump before the host word |
| s0025 | 16.9 | B | valid | http_step_us_b1 | 1,361 | 3,332 | Track B: s0024's token publish with the int8 head sketch |
| s0026 | 19.5 | A | valid | http_step_us_b1 | 1,393 | 3,339 | Track A: drop a never-true clause that kept a dead branch in the batch-1 kernel |
| s0027 | 19.6 | B | valid | http_step_us_b1 | 1,353 | 3,333 | Track B: s0026's dead-branch removal with the int8 head sketch |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.7 | B | valid | http_step_us_b1 | 7,557 | 8,003 | W12 rows stored scaled by exact powers of two: one decode base, factor in epilogue |
| s0002 | 0.9 | B | valid | http_step_us_b1 | 7,542 | 7,969 | Layer-0 in_proj output from a per-token table built by the kernel itself |
| s0003 | 1.4 | B | valid | http_step_us_b1 | 7,525 | 7,950 | Greedy passes: certified int8 sketch of lm_head + exact rescoring of candidate tiles |
| s0004 | 1.8 | B | valid | http_step_us_b1 | 7,495 | 7,936 | Head sketch in its own launch after the main kernel; x statistics in the sketch |
| s0005 | 2.5 | B | valid | http_step_us_b1 | 7,337 | 7,763 | W12 stream: next escape position kept in a register, no shuffle per iteration |
| s0006 | 4.9 | B | valid | http_step_us_b1 | 7,327 | 7,763 | W12 build: epilogue norm gains and row factors copied to shared memory at op start |
| s0007 | 5.0 | B | valid | http_step_us_b1 | 7,304 | 7,741 | No BF16 twins of out_proj/o_proj: W12 at every batch size, BF16 loops not compiled |
| s0008 | 5.3 | B | valid | http_step_us_b1 | 7,313 | 7,738 | Head sketch: per-block tile counts balanced by measured block speed at load |
| s0009 | 5.6 | B | valid | http_step_us_b1 | 7,291 | 7,712 | Head sketch in 7 bits per weight (1.11 GB instead of 1.27 GB), still certified exact |
| s0010 | 6.2 | B | valid | http_step_us_b1 | 7,273 | 7,683 | Barrier L2 prefetch of the next GEMM's first piece per block: 144 KB -> 176 KB |
| s0011 | 7.3 | B | valid | http_step_us_b1 | 7,275 | 7,697 | Head kernel: argmax op folded into the finish op (one grid barrier less) |
| s0012 | 7.5 | B | valid | http_step_us_b1 | 7,266 | 7,682 | Head kernel: exact rescoring chain relayed in 40 turns of 8 k-steps (was 64 of 5) |
| s0013 | 8.2 | B | valid | http_step_us_b1 | 7,263 | 7,675 | Head sketch: power-of-two x scale, digits by exact multiply instead of division |
| s0014 | 9.1 | B | valid | http_step_us_b1 | 7,261 | 7,674 | B<=3 attention barrier prefetch: 256 KB of o_proj + the K chunk (12-bit build) |
| s0015 | 9.6 | B | valid | http_step_us_b1 | 7,258 | 7,688 | B<=3: attention barrier asks for 20 K chunks per unit (was 5), 12-bit build |
| s0016 | 10.5 | B | valid | http_step_us_b1 | 7,249 | 7,682 | B<=3 prefetch table retune: no K past chunk 5, whole o_proj share, 40 idle blocks |
| s0017 | 13.3 | B | valid | http_step_us_b1 | 7,244 | 7,684 | Head sketch: 94 % of tiles in 6 bits, frequent candidates stay 7-bit (0.96 GB) |
| s0018 | 13.7 | B | valid | http_step_us_b1 | 7,239 | 7,677 | Head sketch at B>1: all-7-bit copy through the plain kernel instance (B=1 keeps 6/7) |
| s0019 | 14.6 | B | valid | http_step_us_b1 | 7,245 | 7,680 | Mixed head sketch: per-block tile counts calibrated at load again (B=1) |
| s0020 | 15.2 | B | valid | http_step_us_b1 | 7,238 | 7,679 | B<=3 near prefetch per matrix: gate_up 160 KB, down 192 KB (others 176) |
| s0021 | 15.9 | B | valid | http_step_us_b1 | 7,242 | 7,679 | Confirmation: s0020 configuration with the later experiments compiled out (no change) |
| s0022 | 16.7 | B | valid | http_step_us_b1 | 7,259 | 7,708 | B=1 head kernel: last rescoring block finishes the pass (two grid barriers less) |
| s0023 | 16.8 | B | valid | http_step_us_b1 | 7,261 | 7,711 | Final state: s0020 configuration (SK_LAST and other experiments compiled out) |
| s0024 | 17.8 | B | valid | http_step_us_b1 | 7,252 | 7,705 | B=1 head sketch: frequent-candidate tiles in 8 bits instead of 7 (6 % of tiles) |
| s0025 | 18.2 | B | valid | http_step_us_b1 | 7,255 | 7,707 | Head sketch: the 6/8-bit mixed sketch for every batch size (no all-7-bit copy) |
| s0026 | 19.0 | B | valid | http_step_us_b1 | 7,250 | 7,692 | Head sketch prologue: x read as aligned u32 pairs instead of u16 scalars |
| s0027 | 22.0 | B | valid | http_step_us_b1 | 7,221 | 7,698 | Recurrent op re-reads B and spec flag from shared memory (no spill reloads) |
| s0028 | 25.0 | B | valid | http_step_us_b1 | 7,175 | 7,681 | ptxas register-usage level 10 for the track-B build (same source) |
| s0029 | 26.0 | B | valid | http_step_us_b1 | 7,170 | 7,652 | ptxas -disable-optimizer-consts added to the track-B build (same source) |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.6 | A | valid | http_step_us_b1 | 2,314 | 5,745 | Port of the gpt-oss-120b B200 megakernel to 20b/H100 (sm_90 e2m1 decode) |
| s0002 | 1.0 | A | valid | http_step_us_b1 | 2,275 | 4,284 | MoE as whole-unit warp streams + last-arriver down combine + cheaper e2m1 decode |
| s0003 | 1.7 | A | valid | http_step_us_b1 | 2,275 | 3,853 | Faster norms (one warp per row, loads in flight) and down combine (4 slots/sequence) |
| s0004 | 2.0 | A | valid | http_step_us_b1 | 2,254 | 3,850 | Router on 45 blocks: K-slice partials summed by every block, top-4 in smem |
| s0005 | 2.1 | B | valid | http_step_us_b1 | 2,061 | 3,671 | Track B: certified int8 head sketch on the greedy path, exact recompute of candidates |
| s0006 | 2.7 | B | valid | http_step_us_b1 | 2,037 | 3,641 | K/V cache in MMA fragment order: attention reads with 16-byte loads |
| s0007 | 3.2 | A | valid | http_step_us_b1 | 2,237 | 3,819 | Track A: K/V cache in MMA fragment order (16-byte loads in attention) |
| s0008 | 3.3 | B | valid | http_step_us_b1 | 2,026 | 3,599 | Release atomics in barrier/counters, warp-parallel greedy token choice |
| s0009 | 3.4 | B | valid | http_step_us_b1 | 1,992 | 3,457 | Routing block in registers: per-warp logits+top-4, warp-0 union/slots (2 syncs, was 5) |
| s0010 | 3.5 | B | valid | http_step_us_b1 | 1,935 | 3,411 | Attention as in-block flash decoding: online softmax per warp, 2 syncs (was 5) |
| s0011 | 3.7 | A | valid | http_step_us_b1 | 2,072 | 3,509 | Track A catch-up: routing in registers, flash attention, release atomics, sync trims |
| s0012 | 3.9 | B | valid | http_step_us_b1 | 1,896 | 3,345 | Routing tail over 3 warps, 5-stage qkv/o segments, sketch loads in flight 15 |
| s0013 | 4.3 | B | valid | http_step_us_b1 | 1,863 | 3,333 | Track B: 6-bit certified head sketch (434 MB instead of 579) |
| s0014 | 4.4 | A | valid | http_step_us_b1 | 2,078 | 3,501 | Track A: stage-aligned qkv/o segments, routing tail over 3 warps |
| s0015 | 4.9 | B | valid | http_step_us_b1 | 1,813 | 3,328 | Track B: router weights folded with ln2; router partials in o_proj epilogue, no phase D |
| s0016 | 5.4 | B | valid | http_step_us_b1 | 1,802 | 3,343 | Track B: L2 prefetch of each attention block's K/V window at the start of phase A |
| s0017 | 5.6 | B | valid | http_step_us_b1 | 1,777 | 3,353 | Track B: norms take sums of squares from partials of the producing phase (one pass) |
| s0018 | 5.9 | B | valid | http_step_us_b1 | 1,766 | 3,354 | qkv epilogue bias and RoPE inputs loaded between the stream and the block barrier |
| s0019 | 6.3 | A | valid | http_step_us_b1 | 2,063 | 3,520 | Track A catch-up: K/V window L2 prefetch, qkv epilogue inputs before the block barrier |
| s0020 | 6.6 | B | valid | http_step_us_b1 | 1,752 | 3,331 | E2M1 decode by masked integer multiply (same bits, ~30% fewer decode instructions) |
| s0021 | 7.0 | B | valid | http_step_us_b1 | 1,734 | 3,322 | Track B: single-pass norms split over 4 warps per row at B<=2 |
| s0022 | 7.1 | B | valid | http_step_us_b1 | 1,721 | 3,331 | Prefetch of cold epilogue inputs: folded router rows (L1) and down biases (L2) |
| s0023 | 7.2 | B | valid | http_step_us_b1 | 1,713 | 3,315 | L2 prefetch of the next layer's norm weights, biases and sinks during the down phase |
| s0024 | 7.4 | A | valid | http_step_us_b1 | 2,035 | 3,480 | Track A catch-up: masked-multiply decode, L2 prefetch of small vectors and down biases |
| s0025 | 7.5 | A | valid | http_step_us_b1 | 1,952 | 3,487 | Track A: single-pass norms with sums of squares accumulated where rows are produced |
| s0026 | 7.9 | B | valid | http_step_us_b1 | 1,704 | 3,308 | Consume loop counter in a register at RS=2 (no local-memory round trip per stage) |
| s0027 | 8.2 | B | valid | http_step_us_b1 | 1,699 | 3,305 | 4-warp norm loads its norm weights together with the row |
| s0028 | 8.5 | B | valid | http_step_us_b1 | 1,692 | 3,283 | MXFP4 items stored codes+scales contiguous: one bulk copy per ring stage |
| s0029 | 8.6 | A | valid | http_step_us_b1 | 1,924 | 3,444 | Track A catch-up: fused MXFP4 items, register loop counter, 4-warp norm weight loads |
| s0030 | 8.9 | B | valid | http_step_us_b1 | 1,680 | 3,274 | Earlier o_proj ring prefetch; sketch loads with L2 evict_first |
| s0031 | 10.2 | B | valid | http_step_us_b1 | 1,680 | 3,250 | gate_up units dealt round-robin over blocks (B=8 balance) |
| s0032 | 10.4 | B | valid | http_step_us_b1 | 1,678 | 3,255 | Grid barrier: acq_rel atomic add; the last arriving block skips polling |
| s0033 | 10.5 | A | valid | http_step_us_b1 | 1,896 | 3,393 | Track A catch-up: acq_rel grid barrier, round-robin gate_up units |
| s0034 | 11.4 | B | valid | http_step_us_b1 | 1,668 | 3,406 | B=1-specialised kernel build per cohort; down units split in K halves (track B) |
| s0035 | 11.7 | B | valid | http_step_us_b1 | 1,653 | 3,405 | B=1 build: contiguous gate_up unit ranges (round-robin only in the general build) |
| s0036 | 12.0 | A | valid | http_step_us_b1 | 1,877 | 3,394 | Track A: batch-size-1 specialised kernel build per cohort, contiguous gate_up ranges there |
| s0037 | 12.2 | B | valid | http_step_us_b1 | 1,651 | 3,408 | B=1 build: sketch loads 10 k-steps per batch, no down-bias prefetch |
| s0038 | 13.0 | B | valid | http_step_us_b1 | 1,647 | 3,405 | B=1 build: down units half-major (both K halves of a rowtile in different blocks' ranges) |
| s0039 | 13.4 | B | valid | http_step_us_b1 | 1,639 | 3,396 | Head recompute: candidate tile loaded by one 92 KB bulk copy (TMA) instead of a load loop |
| s0040 | 13.7 | B | valid | http_step_us_b1 | 1,639 | 3,386 | Head recompute copy in 6 chunks so the exact chain overlaps it (general build, B=8) |
| s0041 | 13.8 | A | valid | http_step_us_b1 | 1,878 | 3,394 | Track A catch-up: B=1 build with half-major down units and no down-bias prefetch |
| s0042 | 14.3 | B | valid | http_step_us_b1 | 1,634 | 3,405 | Head sketch: 5-bit codes with per-32-column u8 group scales (381 MB instead of 434) |
| s0043 | 14.4 | B | valid | http_step_us_b1 | 1,634 | 3,396 | Keep the 6-bit head sketch for B>1 cohorts; 5-bit group-scaled sketch only at B=1 |
| s0044 | 14.7 | B | valid | http_step_us_b1 | 1,634 | 3,392 | B=1 build: L2 prefetch of each warp's first sketch batch at the end of the last layer |
| s0045 | 15.3 | B | valid | http_step_us_b1 | 1,629 | 3,396 | B=1 build: double-buffered sketch loads in head pass 1 (next batch in flight) |
| s0046 | 15.9 | B | valid | http_step_us_b1 | 1,625 | 3,396 | B=1 build: router top-4 by rank counting (independent shuffles, same selection) |
| s0047 | 16.7 | A | valid | http_step_us_b1 | 1,871 | 3,394 | Track A: router top-4 by rank counting in the batch-size-1 build |
| s0048 | 17.0 | B | build_failed | B=1 build: router bias loaded at the start of the routing warp | |||
| s0049 | 17.1 | B | valid | http_step_us_b1 | 1,617 | 3,399 | B=1 build: router bias loaded at the start of the routing warp (resubmit, fixed build) |
| s0050 | 17.3 | B | valid | http_step_us_b1 | 1,613 | 3,388 | B=1 build: gate_up bias line prefetched into L1 when each unit's task is built |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.5 | A | valid | http_step_us_b1 | 11,488 | 11,763 | Own persistent megakernel v1: split-K GEMM ops, tensor-core GQA attention |
| s0002 | 2.2 | A | valid | http_step_us_b1 | 10,631 | 11,480 | 512-thread blocks (128 regs), split attention, batch-class norm prep |
| s0003 | 3.1 | A | valid | http_step_us_b1 | 10,586 | 11,290 | Attention waits on per-group q/k/v tile counters; o waits on unit counter |
| s0004 | 3.5 | B | valid | http_step_us_b1 | 9,625 | 10,407 | Track B: exact 12-bit weight storage, bit-identical to track A |
| s0005 | 3.7 | A | valid | http_step_us_b1 | 10,489 | 11,175 | Track A: work split in quads of 4 k-steps (same source as s0004) |
| s0006 | 3.8 | A | valid | http_step_us_b1 | 10,522 | 10,994 | Track A: RMSNorm applied per k-step in the stream at B>=3 (no norm prep) |
| s0007 | 3.9 | A | valid | http_step_us_b1 | 10,334 | 10,833 | Cut tiles handed over with a sentinel slot instead of an atomic counter |
| s0008 | 4.0 | B | valid | http_step_us_b1 | 9,467 | 10,277 | Track B build of s0007's source (12-bit storage + sentinel cut tiles) |
| s0009 | 4.5 | A | valid | http_step_us_b1 | 10,323 | 10,768 | B>=7 attention combine through DSMEM (clusters of 2); separate small/large-batch kernels |
| s0010 | 5.2 | B | valid | http_step_us_b1 | 9,444 | 10,356 | Track B build of s0009's source (12-bit storage, DSMEM combine, two kernels) |
| s0011 | 5.6 | B | valid | http_step_us_b1 | 9,453 | 10,265 | Track B: small-batch kernel instance at all batch sizes (B=8 regression of s0010) |
| s0012 | 6.3 | B | valid | http_step_us_b1 | 9,402 | 10,049 | Track B: escape patches through shared memory; last-warp prefetch issue |
| s0013 | 6.7 | A | valid | http_step_us_b1 | 10,351 | 10,773 | Track A: three kernel instances (1-2, 3-6, 7-8 sequences) and per-warp argmax keys |
| s0014 | 7.3 | A | valid | http_step_us_b1 | 10,354 | 10,766 | Track A final source: cohort-tagged served tokens, 20 ms gather window |
| s0015 | 7.4 | B | valid | http_step_us_b1 | 9,402 | 10,071 | Track B final source: cohort-tagged served tokens, 20 ms gather window |
| s0016 | 8.6 | A | valid | http_step_us_b1 | 10,299 | 10,763 | K/V of cached positions prefetched into L2 before the q/k/v wait (B<=2) |
| s0017 | 9.0 | A | valid | http_step_us_b1 | 10,304 | 10,742 | B=8 attention in fewer rounds: one warp per part softmax, 3 K blocks per warp |
| s0018 | 10.3 | A | valid | http_step_us_b1 | 10,344 | 10,715 | Attention combine folds two halves in parallel (cluster blocks, same split globally) |
| s0019 | 10.6 | A | valid | http_step_us_b1 | 10,230 | 10,718 | Split grid barrier: first weight batch requested before the wait (1-2 sequences) |
| s0020 | 10.7 | B | valid | http_step_us_b1 | 9,301 | 9,943 | Track B of s0019's source (attention rounds, two-half combine, split barrier) |
| s0021 | 11.3 | A | valid | http_step_us_b1 | 10,284 | 10,539 | Split grid barrier in every kernel instance (wait state kept in shared memory) |
| s0022 | 11.8 | B | valid | http_step_us_b1 | 9,336 | 9,918 | Track B of s0021's source; 7-8 sequences on the cluster-combine kernel instance |
| s0023 | 12.3 | A | valid | http_step_us_b1 | 10,265 | 10,535 | First weight batch also requested across the RMSNorm prep (1-2 sequences) |
| s0024 | 13.3 | B | valid | http_step_us_b1 | 9,250 | 9,780 | Track B: 3 quads in flight per warp in the norm GEMMs (inputs from smem just in time) |
| s0025 | 13.7 | B | valid | http_step_us_b1 | 9,197 | 9,840 | Track B, B=1: o and down read their input column from smem, 3 quads in flight |
| s0026 | 14.2 | B | valid | http_step_us_b1 | 9,156 | 9,714 | RMSNorm prep with packed bf16 arithmetic (same bits, half the instructions) |
| s0027 | 14.5 | A | valid | http_step_us_b1 | 10,249 | 10,537 | Track A build of s0026's source (packed bf16 norm prep) |
| s0028 | 15.7 | A | valid | http_step_us_b1 | 10,141 | 10,533 | Attention combine split over all blocks of a unit (1-6 sequences) |
| s0029 | 15.8 | B | valid | http_step_us_b1 | 9,063 | 9,703 | Track B of s0028's source (attention combine split over a unit's blocks) |
| s0030 | 16.4 | A | valid | http_step_us_b1 | 10,131 | 10,528 | Attention prep constants (norm weights, cos, sin) requested before the q/k/v wait |
| s0031 | 16.5 | B | valid | http_step_us_b1 | 9,049 | 9,683 | Track B of the attention-prep source (s0030) |
| s0032 | 17.2 | B | valid | http_step_us_b1 | 9,035 | 9,495 | Track B, 7-8 sequences: norm prep split over the cluster pair via DSMEM |
| s0033 | 17.5 | A | valid | http_step_us_b1 | 10,121 | 10,440 | Track A of s0032's source; 7-8 instance's unused norm-prep path instantiated with the cluster prep |
| s0034 | 18.8 | B | valid | http_step_us_b1 | 9,016 | 9,433 | Track B: a quad's escapes patched by their lanes in parallel |
| s0035 | 21.0 | B | valid | http_step_us_b1 | 9,025 | 9,449 | Track B: at most 10 blocks per attention unit at B=1 |
| s0036 | 21.6 | B | valid | http_step_us_b1 | 8,995 | 9,439 | Track B: idle blocks prefetch 128 KB of o during attention (runtime default) |
| s0037 | 24.5 | A | valid | http_step_us_b1 | 10,083 | 10,404 | Track A: next op's first-bytes prefetch issued by the last warp, not warp 0 |
| s0038 | 24.8 | B | valid | http_step_us_b1 | 8,956 | 9,390 | GEMM block ranges and prefetch plans with 32-bit instead of 64-bit divisions |
| s0039 | 25.0 | A | valid | http_step_us_b1 | 10,061 | 10,407 | Track A of s0038's source (32-bit range divisions) |
| s0040 | 25.4 | A | valid | http_step_us_b1 | 10,012 | 10,410 | Attention combine arrival: release + poll a cumulative per-launch count (no atomic/flag) |
| s0041 | 25.5 | B | valid | http_step_us_b1 | 8,948 | 9,404 | Track B of s0040's source (counter-based combine arrival) |
| s0042 | 28.3 | A | valid | http_step_us_b1 | 9,981 | 10,408 | Track A: norm weights staged into shared memory during the barrier wait (1-4 seq) |
| s0043 | 28.4 | B | valid | http_step_us_b1 | 8,937 | 9,383 | Track B of s0042's source (track-B SASS identical to s0041) |
| s0044 | 29.0 | A | valid | http_step_us_b1 | 9,974 | 10,276 | Track A: per-k-step norm weights pre-ordered on the host, staged during barrier wait |
| s0045 | 29.1 | B | valid | http_step_us_b1 | 8,938 | 9,373 | Track B of s0044's source (track-B SASS identical to s0041) |
| s0046 | 29.4 | B | valid | http_step_us_b1 | 8,911 | 9,332 | Track B: row factors copied with cp.async (no L2 stall of warps 0-1 at stream start) |
| s0047 | 29.6 | B | valid | http_step_us_b1 | 8,792 | 9,238 | Track B: escape-list start and first window loaded independently before the wait |
| Submission | Agent-hours | Track | Status | Metric | Headline | 8 seq., µs | Title |
|---|---|---|---|---|---|---|---|
| s0001 | 0.5 | A | valid | http_step_us_b1 | 3,613 | 8,775 | v1 megakernel: one cooperative kernel, grid barrier between 6 phases per layer |
| s0002 | 1.0 | A | build_failed | TMA per-warp weight rings, cross-phase prefetch, fragment-order B operands | |||
| s0003 | 1.0 | A | error | TMA per-warp weight rings, cross-phase prefetch, fragment-order B operands | |||
| s0004 | 1.1 | A | valid | http_step_us_b1 | 2,523 | 6,735 | TMA weight rings + prefetch (s0003 fixed: attention P built from scores, no smem overlap) |
| s0005 | 1.2 | A | valid | http_step_us_b1 | 2,276 | 6,738 | Down phase at B<=2: even (expert, rowtile) units, partials summed by the next phase |
| s0006 | 1.7 | A | valid | http_step_us_b1 | 2,161 | 6,872 | MXFP4 dequant by scaled e2m1 cvt (exact); norm weights bulk-copied into smem early |
| s0007 | 2.0 | A | valid | http_step_us_b1 | 2,079 | 6,640 | Attention split into 64-key chunks, one warp each, last warp merges (flash decoding) |
| s0008 | 2.2 | A | valid | http_step_us_b1 | 2,099 | 4,646 | B>=3: expert phases without reduction rounds (warp per unit), even down units + combine step |
| s0009 | 2.4 | A | valid | http_step_us_b1 | 2,071 | 4,325 | MXFP4 records interleave codes and scales (one bulk copy per stage); batched norm loads |
| s0010 | 2.6 | A | valid | http_step_us_b1 | 2,035 | 4,279 | Router split 4 ways in K over 32 blocks; E sums the partials |
| s0011 | 2.7 | A | valid | http_step_us_b1 | 1,997 | 4,144 | Top-4 by one warp integer max per round; expert union in warp 0 (match/reduce) |
| s0012 | 3.0 | A | valid | http_step_us_b1 | 1,959 | 4,017 | E->F grid barrier replaced by per-expert-slot completion counters |
| s0013 | 3.2 | A | valid | http_step_us_b1 | 1,951 | 3,953 | Two kernel instances (B<=2 / B>=3); wide instance norms one warp per column |
| s0014 | 3.9 | A | valid | http_step_us_b1 | 1,927 | 3,824 | Drop the per-lane B-column selects (absent columns only feed discarded outputs); batched o_proj input copy |
| s0015 | 4.0 | A | valid | http_step_us_b1 | 1,906 | 3,756 | Two mma accumulators per ring stage (even/odd items), fp4 and bf16 |
| s0016 | 4.8 | A | valid | http_step_us_b1 | 1,840 | 3,708 | K/V cache in mma fragment order (16 B loads); attention partials handed over with one acq_rel atomic |
| s0017 | 5.0 | A | valid | http_step_us_b1 | 1,858 | 3,545 | Expert bias rows prefetched when a unit starts (B>=3 epilogues no longer stall the stream) |
| s0018 | 6.7 | B | valid | http_step_us_b1 | 1,799 | 3,500 | Track B: certified int8 head sketch + exact rescoring of candidate rows (B<=2 greedy) |
| s0019 | 7.4 | B | valid | http_step_us_b1 | 1,803 | 3,459 | Track B: head sketch in the B>=3 instance too (inlined); same tokens |
| s0020 | 7.9 | B | valid | http_step_us_b1 | 1,798 | 3,413 | Track B: B>=3 staging by bulk copies (attention output, act rows in fragment order) |
| s0021 | 8.1 | B | valid | http_step_us_b1 | 1,774 | 3,403 | Track B: B<=2 instance stages attention output and act rows by bulk copies too |
| s0022 | 8.3 | A | valid | http_step_us_b1 | 1,831 | 3,391 | Track A: current engine without the head sketch (bulk-copy staging, batched act loads) |
| s0023 | 8.7 | B | valid | http_step_us_b1 | 1,739 | 3,323 | Track B: o_proj K split 16 -> 4 tasks per rowtile (fewer, longer ring tasks) |
| s0024 | 8.9 | B | valid | http_step_us_b1 | 1,739 | 3,219 | Track B: ring tasks on whole stages (o_proj segments 63/63/63/67, router 5x9 ksteps) |
| s0025 | 9.3 | B | valid | http_step_us_b1 | 1,737 | 3,237 | Track B: B<=2 attention loads V by cp.async with K |
| s0026 | 9.4 | A | valid | http_step_us_b1 | 1,810 | 3,228 | Track A: current engine without the head sketch (whole-stage tasks, bulk-copy staging) |
| s0027 | 9.8 | B | valid | http_step_us_b1 | 1,730 | 3,200 | Track B: head candidates rescored with 20 stages in flight; B>=3 down tail units as segments |
| s0028 | 10.2 | B | valid | http_step_us_b1 | 1,731 | 3,174 | Track B: short attention units per block with smem merge; B>=3 gate_up tail units as segments |
| s0029 | 10.4 | A | valid | http_step_us_b1 | 1,799 | 3,174 | Track A: current engine without the head sketch (attention short units per block, tail segments) |
| s0030 | 10.6 | B | valid | http_step_us_b1 | 1,719 | 3,177 | Track B: head candidates rescored in parallel (pooled ring slots), no L2 prefetch of local candidates |
| s0031 | 10.9 | B | valid | http_step_us_b1 | 1,715 | 3,144 | Track B: head sketch in the B>=3 instance via an out-of-line call; warp-parallel token pick |
| s0032 | 11.2 | B | valid | http_step_us_b1 | 1,720 | 3,139 | Track B: sketch's global bound read in parallel per column |
| s0033 | 11.4 | A | valid | http_step_us_b1 | 1,796 | 3,164 | Track A: current engine without the head sketch (since s0029: warp-parallel token pick) |
| s0034 | 12.4 | B | valid | http_step_us_b1 | 1,722 | 3,125 | Track B: wide head rescoring copies the prep's normed rows instead of re-norming |
| s0035 | 13.6 | B | valid | http_step_us_b1 | 1,722 | 3,115 | Track B: down-phase tail units back to whole units (FTAIL 0) |
| s0036 | 13.6 | B | valid | http_step_us_b1 | 1,722 | 3,105 | Track B: B>=3 attention units up to 16 chunks on one block (2 chunks per warp) |
| s0037 | 13.8 | A | valid | http_step_us_b1 | 1,799 | 3,135 | Track A: s0036's engine without the head sketch |
| s0038 | 14.1 | B | valid | http_step_us_b1 | 1,722 | 3,083 | Track B: wide norm loads each weight word one iteration ahead (NORM_PIPE1) |
| s0039 | 14.7 | A | valid | http_step_us_b1 | 1,798 | 3,117 | Track A: B>=3 instance prefetches 3 ring stages across barriers (was 2) |
| s0040 | 15.2 | B | valid | http_step_us_b1 | 1,719 | 3,083 | Track B: B<=2 attention loads V after the scores again (V_ASYNC_FLAT 0) |
| s0041 | 15.6 | B | valid | http_step_us_b1 | 1,703 | 3,066 | Track B: phase A pulls each block's attention K/V chunks into L2 |
| s0042 | 15.8 | B | valid | http_step_us_b1 | 1,700 | 3,064 | Track B: phase A also pulls the layer's biases and sinks into L2 |
| s0043 | 15.9 | A | valid | http_step_us_b1 | 1,787 | 3,114 | Track A: B<=2 attention loads V after the scores (as track B since s0040) |
| s0044 | 17.0 | B | valid | http_step_us_b1 | 1,706 | 3,054 | Track B: gate_up tail units back to whole units (ETAIL 0) |
| s0045 | 17.9 | B | valid | http_step_us_b1 | 1,693 | 3,026 | Track B: wide head re-norms its rows for the rescoring again (HSK_XG 0) |
| s0046 | 18.2 | B | valid | http_step_us_b1 | 1,693 | 3,017 | Track B: wide head rescores candidate tiles with pooled ring slots (FASTRS_W 1) |
| s0047 | 20.1 | B | valid | http_step_us_b1 | 1,691 | 3,021 | Track B: B<=2 qkv on 2 rowtiles per block; extra V rowtiles as spread segments |
| s0048 | 21.6 | A | valid | http_step_us_b1 | 1,775 | 3,101 | Track A: exact head's last rowtiles split over warps (HTAIL) + wide norm pipelining |
| s0049 | 22.0 | A | valid | http_step_us_b1 | 1,774 | 3,074 | Track A: K/V chunks prefetched into L2 in the B>=3 instance (KVPF) |
| s0050 | 22.7 | A | valid | http_step_us_b1 | 1,776 | 3,061 | Track A: head tail rowtiles in 2 K halves instead of 4 quarters (HTS 2) |
| s0051 | 23.6 | B | valid | http_step_us_b1 | 1,687 | 3,029 | Track B: final engine as committed (s0047's kernel; track A code compiled out) |
| s0052 | 25.0 | B | valid | http_step_us_b1 | 1,690 | 3,019 | Track B: grid barrier arrives by red.release and polls with a 40 ns back-off |
| s0053 | 25.1 | A | valid | http_step_us_b1 | 1,771 | 3,075 | Track A: grid barrier arrives by red.release and polls with a 40 ns back-off |
| s0054 | 26.2 | B | valid | http_step_us_b1 | 1,690 | 3,007 | Track B: B>=3 head rescoring splits candidate tiles into 4 K segments over warps |
| s0055 | 26.5 | B | valid | http_step_us_b1 | 1,690 | 3,000 | Track B: wide head copies the published normed rows by bulk copy (HSK_XG 2) |
| s0056 | 26.9 | B | valid | http_step_us_b1 | 1,690 | 2,991 | Track B: wide head sketch prep fuses the norm and digit passes (HSK_PREP2) |
| s0057 | 27.4 | A | valid | http_step_us_b1 | 1,760 | 3,067 | Track A: exact head tail claimed dynamically (HDYN) + re-swept KVPF_F/QXO |
| s0058 | 27.8 | B | valid | http_step_us_b1 | 1,692 | 3,010 | Track B: head sketch tail tiles claimed dynamically by free warps (SDYN) |
| s0059 | 29.9 | A | valid | http_step_us_b1 | 1,738 | 3,076 | Track A: B<=2 instance prefetches the layer's small parameters into L2 (SPF_F) |
As recorded by each job: the GPU, the versions inside each system's environment, and the command each server was started with (model path, host and port removed).
| Record |
|---|
| gpu NVIDIA H100 80GB HBM3, 81559 MiB, 595.71.05, 1980 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours q8A4-s0016 sha256=621e47c041a73f2f |
| ours q8B4-s0020 sha256=b76d9f48be969976 |
| sglang=0.5.18 torch=2.13.0+cu130 cuda=13.0 arch=x86_64 gpu=NVIDIA H100 80GB HBM3 |
| vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=x86_64 gpu=NVIDIA H100 80GB HBM3 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q8A4-s0016 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q8B4-s0020 |
| sglang.launch_server --model-path Qwen/Qwen3-8B --dtype bfloat16 --enable-torch-compile |
| vllm serve Qwen/Qwen3-8B --dtype bfloat16 --gpu-memory-utilization 0.90 |
| Record |
|---|
| gpu NVIDIA H100 80GB HBM3, 81559 MiB, 595.71.05, 1980 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours q8A2-s0003 sha256=5d107a61cec081ff |
| ours q8B2-s0002 sha256=d59994213a77d62e |
| sglang=0.5.18 torch=2.13.0+cu130 cuda=13.0 arch=x86_64 gpu=NVIDIA H100 80GB HBM3 |
| vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=x86_64 gpu=NVIDIA H100 80GB HBM3 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q8A2-s0003 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q8B2-s0002 |
| sglang.launch_server --model-path Qwen/Qwen3-8B --dtype bfloat16 --enable-torch-compile |
| vllm serve Qwen/Qwen3-8B --dtype bfloat16 --gpu-memory-utilization 0.90 |
| Record |
|---|
| gpu NVIDIA H100 80GB HBM3, 81559 MiB, 595.71.05, 1980 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours q8onA-s0038 sha256=a6316f60bec0cc7c |
| ours q8onB-s0039 sha256=40e46bf2c02cc6e8 |
| sglang=0.5.18 torch=2.13.0+cu130 cuda=13.0 arch=x86_64 gpu=NVIDIA H100 80GB HBM3 |
| vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=x86_64 gpu=NVIDIA H100 80GB HBM3 |
| serve_engine.py engine --max-batch 64 --max-context 34816 + frontend; source q8onA-s0038 |
| serve_engine.py engine --max-batch 64 --max-context 34816 + frontend; source q8onB-s0039 |
| sglang.launch_server --model-path Qwen/Qwen3-8B --dtype bfloat16 --enable-torch-compile |
| vllm serve Qwen/Qwen3-8B --dtype bfloat16 --gpu-memory-utilization 0.90 |
| Record |
|---|
| gpu NVIDIA H100 80GB HBM3, 81559 MiB, 595.71.05, 1980 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours northA-s0022 sha256=c79cf7faa931872d |
| vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=x86_64 gpu=NVIDIA H100 80GB HBM3 |
| serve_engine.py engine --max-batch 8 --max-context 18432 + frontend; source northA-s0022 |
| vllm serve CohereLabs/North-Mini-Code-1.0 --dtype bfloat16 --gpu-memory-utilization 0.90 --max-model-len 32768 |
| Record |
|---|
| gpu NVIDIA H100 80GB HBM3, 81559 MiB, 595.71.05, 1980 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours oss20A4-s0047 sha256=dd03ba032f05b691 |
| ours oss20B4-s0050 sha256=47936dfcd785078d |
| vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=x86_64 gpu=NVIDIA H100 80GB HBM3 |
| sglang=0.5.23 torch=2.14.1+cu130 cuda=13.0 arch=x86_64 gpu=NVIDIA H100 80GB HBM3 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source oss20A4-s0047 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source oss20B4-s0050 |
| vllm serve openai/gpt-oss-20b --dtype bfloat16 --gpu-memory-utilization 0.90 --no-enable-prefix-caching --max-cudagraph-capture-size 2048 --max-num-batched-tokens 8 |
| sglang.launch_server --model-path openai/gpt-oss-20b --dtype bfloat16 |
| Record |
|---|
| gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours flintA3-s0026 sha256=8060d34f88c738a7 |
| ours flintB3-s0027 sha256=8ef9677fe7a2a6b2 |
| vllm=0.33.0 torch=2.14.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source flintA3-s0026 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source flintB3-s0027 |
| vllm serve RecursiveSI/flint --dtype bfloat16 --gpu-memory-utilization 0.90 |
| Record |
|---|
| gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours flintA2-s0077 sha256=5c3396ee29ca2c89 |
| ours flintB-s0055 sha256=97ec5b442dd27867 |
| vllm=0.33.0 torch=2.14.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source flintA2-s0077 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source flintB-s0055 |
| vllm serve RecursiveSI/flint --dtype bfloat16 --gpu-memory-utilization 0.90 |
| Record |
|---|
| gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours flintA-s0054 sha256=22a115614e575e72 |
| vllm=0.33.0 torch=2.14.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200 |
| serve_engine.py engine --max-batch 8 --max-context 18432 + frontend; source flintA-s0054 |
| vllm serve RecursiveSI/flint --dtype bfloat16 --gpu-memory-utilization 0.90 |
| Record |
|---|
| gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours q27A2-s0022 sha256=67f78b19aaf2551b |
| ours q27B3-s0020 sha256=860af7b83f4e929e |
| sglang=0.5.23 torch=2.14.1+cu130 cuda=13.0 arch=aarch64 gpu=NVIDIA GB200 |
| vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q27A2-s0022 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q27B3-s0020 |
| sglang.launch_server --model-path Qwen/Qwen3.8-27B --dtype bfloat16 |
| vllm serve Qwen/Qwen3.8-27B --dtype bfloat16 --gpu-memory-utilization 0.90 |
| Record |
|---|
| gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours q27A2-s0022 sha256=67f78b19aaf2551b |
| ours q27B2-s0023 sha256=3941f614d5c3a090 |
| sglang=0.5.23 torch=2.14.1+cu130 cuda=13.0 arch=aarch64 gpu=NVIDIA GB200 |
| vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q27A2-s0022 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q27B2-s0023 |
| sglang.launch_server --model-path Qwen/Qwen3.8-27B --dtype bfloat16 |
| vllm serve Qwen/Qwen3.8-27B --dtype bfloat16 --gpu-memory-utilization 0.90 |
| Record |
|---|
| gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours q27A-s0016 sha256=7c131f50e038b675 |
| sglang=0.5.23 torch=2.14.1+cu130 cuda=13.0 arch=aarch64 gpu=NVIDIA GB200 |
| vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200 |
| serve_engine.py engine --max-batch 8 --max-context 18432 + frontend; source q27A-s0016 |
| sglang.launch_server --model-path Qwen/Qwen3.8-27B --dtype bfloat16 |
| vllm serve Qwen/Qwen3.8-27B --dtype bfloat16 --gpu-memory-utilization 0.90 |
| Record |
|---|
| gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours q32A3-s0033 sha256=1d029c8737bc23dd |
| ours q32B3-s0036 sha256=d5c730744c3b4706 |
| sglang=0.5.23 torch=2.14.1+cu130 cuda=13.0 arch=aarch64 gpu=NVIDIA GB200 |
| vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q32A3-s0033 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q32B3-s0036 |
| sglang.launch_server --model-path Qwen/Qwen3-32B --dtype bfloat16 |
| vllm serve Qwen/Qwen3-32B --dtype bfloat16 --gpu-memory-utilization 0.90 |
| Record |
|---|
| gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours oss120A3-s0049 sha256=26c8833113e22068 |
| ours oss120B3-s0051 sha256=63d4b0a5c2c83fb9 |
| sglang=0.5.23 torch=2.14.1+cu130 cuda=13.0 arch=aarch64 gpu=NVIDIA GB200 |
| vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source oss120A3-s0049 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source oss120B3-s0051 |
| sglang.launch_server --model-path openai/gpt-oss-120b --dtype bfloat16 |
| vllm serve openai/gpt-oss-120b --dtype bfloat16 --gpu-memory-utilization 0.90 --no-enable-prefix-caching --max-cudagraph-capture-size 2048 --max-num-batched-tokens |
| Record |
|---|
| gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours inkA2-s0034 sha256=a144977d2dc5042c |
| ours inkB2-s0036 sha256=02dc9a234e11d4e2 |
| img sglang ref=lmsysorg/sglang@sha256:198af079c599cbf37ad371faca16c988a71cea771b3c51fbe21d018f6e400c9e arch=arm64 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source inkA2-s0034 |
| serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source inkB2-s0036 |
| sglang(image lmsysorg/sglang@sha256:198af079c599cbf37ad371faca16c988a71cea771b3c51fbe21d018f6e400c9e) env SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 launch_server --model-path thinkingmachines/Inkling-2-Small |
| Record |
|---|
| gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours inkOn2-s0043 sha256=65058690258bae89 |
| img sglang ref=lmsysorg/sglang@sha256:198af079c599cbf37ad371faca16c988a71cea771b3c51fbe21d018f6e400c9e arch=arm64 |
| serve_engine.py engine --max-batch 64 --max-context 34816 + frontend; source inkOn2-s0043 |
| sglang(image lmsysorg/sglang@sha256:198af079c599cbf37ad371faca16c988a71cea771b3c51fbe21d018f6e400c9e) env SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 launch_server --model-path thinkingmachines/Inkling-2-Small |
| Record |
|---|
| gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz |
| eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0 |
| ours inkA-s0020 sha256=1c05b00e58dc971c |
| img sglang ref=lmsysorg/sglang@sha256:198af079c599cbf37ad371faca16c988a71cea771b3c51fbe21d018f6e400c9e arch=arm64 |
| serve_engine.py engine --max-batch 8 --max-context 18432 + frontend; source inkA-s0020 |
| sglang(image lmsysorg/sglang@sha256:198af079c599cbf37ad371faca16c988a71cea771b3c51fbe21d018f6e400c9e) env SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 launch_server --model-path thinkingmachines/Inkling-2-Small |