Megakernel data tables

The measurements behind A Megakernel per Model. Code, engines and run records: megakernel-zoo. Generated from job logs on Nov 18, 2026, 20:53 Pacific time; a row names the job it was measured in. General engines are in gray. The same cells are in this page as JSON (<script id="data-tables">).

Engines

The engines of megakernel-zoo, as registered for its server mkserve. Sequences and context are the limits the engine is served with. Extra checks: logits against float64 at every batch size from 1 to 8; bit-identical under a permutation of the slots; a sequence among other neighbors; bit-identical on repetition; bit-identical after a full context and a reset; identical after a process restart (each in two processes). Wrapper checks: scripts/mkserve_check.py against the served engine. Audit: the conclusion of an independent agent that read the source and ran its own jobs.

ModelEngineTrackKindRun and submissionSeq.ContextExtra checksWrapperAudit
Qwen3-8Bonline-b (default)Bonlines3-q8b-online-a1 · s00396434,8166 of 612 of 12no violation found; not bit-identical to track A; same tokens alone and under load (24 of 24 runs); the engine does not enforce its own limits
Qwen3-8Bonline-aAonlines3-q8b-online-a1 · s00386434,8166 of 612 of 12no violation found; chunked prompts, paging and prefix reuse bit-identical to stepping; the engine does not enforce its own limits; under load, tokens can differ from the request alone at near-ties
Qwen3-8Boffline-bBofflines11-q8b-h100-b1 · s002082,048––not audited
Qwen3-8Boffline-aAofflines11-q8b-h100-b1 · s001682,048––not audited
flintoffline-a (default)Aofflines11-flint-b200-b1 · s002682,048––not audited
flintoffline-bBofflines11-flint-b200-b1 · s002782,048––not audited
Qwen3.8-27Boffline-a (default)Aofflines9-qwen38-27b-b200-f1 · s002282,0486 of 612 of 12no violation found; logits bit-identical to the engine audited first at 18,700 positions; RMSNorm scale applied at a different rounding point than Hugging Face; the engine does not check invalid requests
Qwen3.8-27Boffline-bBofflines11-q27-b200-b1 · s002082,048––not audited
North Mini Code 1.0offline-a (default)Aofflines4-north-h100-a1 · s002282,0486 of 612 of 12no violation found
North Mini Code 1.0offline-bBofflines4-north-h100-a1 · s002182,0486 of 612 of 12no violation found; outputs not bit-identical to track A
Inkling-2-Smalloffline-a (default)Aofflines7-inkling2s-b200x4-fb1 · s003482,0486 of 612 of 12no violation found; logits bit-identical to the audited kernel, with the context limit now enforced
Inkling-2-Smalloffline-bBofflines7-inkling2s-b200x4-fb1 · s003682,0486 of 612 of 12no violation found; logits bit-identical to track A at every position checked
Inkling-2-Smallonline-aAonlines8-inkling2s-b200x4-on1 · s00436434,8166 of 612 of 12no violation found; the model's arithmetic is that of the engine audited first (same tokens); the first audit's defects are fixed: no out-of-bounds write at small batch, prefixes compared token by token, requests beyond the limits refused; 130 concurrent requests give the tokens of each request alone
gpt-oss-20boffline-a (default)Aofflines12-oss20-h100-a1 · s004782,048––not audited
gpt-oss-20boffline-bBofflines12-oss20-h100-a1 · s005082,048––not audited
Qwen3-32Boffline-a (default)Aofflines12-q32-b200-a1 · s003382,048––not audited
Qwen3-32Boffline-bBofflines12-q32-b200-a1 · s003682,048––not audited
gpt-oss-120boffline-a (default)Aofflines12-oss120-b200-a1 · s004982,048––not audited
gpt-oss-120boffline-bBofflines12-oss120-b200-a1 · s005182,048––not audited

Steady decoding on the same GPU

64-token prompt, 1,024 generated tokens, greedy, every system behind streaming HTTP and timed by the same client; all systems of a table ran in the job named, one after another, three rounds. Time per token is taken over the window in which all sequences decode; the range is the lowest and highest round. × is the fastest general engine's time over the row's.

Qwen3-8B · 1 × H100

1 sequence, per token8 sequences, per step× fastest generaltokens/sTTFT, s
Systemmsrangemsrange1 seq.8 seq.1 seq.8 seq.1 seq.Job
SGLang6.0866.085–6.0876.8536.850–6.8591.00×1.00×164.21,163.50.014rel-q8-head-k3
vLLM6.7536.718–6.7726.9476.881–7.0760.90×0.99×148.41,149.20.013rel-q8-head-k3
megakernel, track A (s11 s0016)5.1335.132–5.1355.4785.459–5.5041.19×1.25×182.01,380.60.377rel-q8-head-k3
megakernel, track B (s11 s0020)3.8273.820–3.8394.2544.254–4.2551.59×1.61×243.01,777.30.297rel-q8-head-k3
SGLang6.1306.129–6.1316.8396.829–6.8521.00×1.00×163.01,166.10.012rel-q8-head-f
vLLM6.6706.601–6.7866.9576.891–7.0890.92×0.98×149.51,147.70.012rel-q8-head-f
megakernel, track A, earlier build (s10 s0003)5.3775.374–5.3805.6365.634–5.6371.14×1.21×173.91,342.60.386rel-q8-head-f
megakernel, track B, earlier build (s10 s0002)4.1584.139–4.1734.4694.460–4.4741.47×1.53×224.51,694.40.308rel-q8-head-f
SGLang6.1406.136–6.1446.8476.844–6.8491.00×1.00×162.71,165.10.013rel-q8-axes-1
vLLM6.7966.786–6.8057.0937.090–7.0960.90×0.97×147.01,125.60.013rel-q8-axes-1
megakernel, serving, track A (s3 s0038)6.0966.2891.01×1.09×164.01,269.30.009rel-q8-axes-1
megakernel, serving, track B (s3 s0039)5.6245.8341.09×1.17×177.71,368.50.008rel-q8-axes-1

North Mini Code 1.0 · 1 × H100

1 sequence, per token8 sequences, per step× fastest generaltokens/sTTFT, s
Systemmsrangemsrange1 seq.8 seq.1 seq.8 seq.1 seq.Job
Cohere megakernel (vendor's own)3.6197.0661.00×1.00×276.31,132.20.002compare-final-north-2
vLLM5.0479.4590.72×0.75×198.1845.70.022compare-final-north-2
megakernel, track A (s4 s0022)2.5386.8971.43×1.02×394.01,159.90.171compare-final-north-2
megakernel, track B (s4 s0021)2.1506.0511.68×1.17×465.01,322.10.155compare-final-north-2

gpt-oss-20b · 1 × H100

1 sequence, per token8 sequences, per step× fastest generaltokens/sTTFT, s
Systemmsrangemsrange1 seq.8 seq.1 seq.8 seq.1 seq.Job
SGLang3.5043.502–3.5065.2175.214–5.2200.95×0.94×284.61,520.30.013rel-oss20-head-k3
vLLM3.3123.200–3.3794.9294.810–4.9941.00×1.00×300.61,627.40.013rel-oss20-head-k3
megakernel, track A (s12 s0047)1.8741.872–1.8753.3883.387–3.3891.77×1.45×496.92,270.70.144rel-oss20-head-k3
megakernel, track B (s12 s0050)1.6151.615–1.6163.3883.387–3.3892.05×1.46×571.02,267.50.141rel-oss20-head-k3

flint · 1 × B200

1 sequence, per token8 sequences, per step× fastest generaltokens/sTTFT, s
Systemmsrangemsrange1 seq.8 seq.1 seq.8 seq.1 seq.Job
vLLM1.9891.986–1.9913.7853.783–3.7861.00×1.00×486.32,050.20.063rel-flint-head-k2
megakernel, track A (s11 s0026)1.3921.392–1.3933.3543.353–3.3551.43×1.13×659.82,290.90.127rel-flint-head-k2
megakernel, track B (s11 s0027)1.3561.356–1.3573.3493.348–3.3491.47×1.13×676.22,294.50.126rel-flint-head-k2
vLLM1.9893.7913.783–3.8081.00×1.00×485.82,040.20.066rel-flint-head-f
megakernel, track A, earlier build (s4 s0077)1.4781.478–1.4793.3983.397–3.3981.35×1.12×622.02,261.00.133rel-flint-head-f
megakernel, track B, earlier build (s4 s0055)1.4771.476–1.4783.9533.950–3.9551.35×0.96×623.01,935.40.133rel-flint-head-f
vLLM1.9961.973–2.0191.00×485.70.066rel-flint-axes-2
megakernel, track A, earlier build (s4 s0054)1.5251.521–1.5283.8813.877–3.8881.31×604.11,975.10.135rel-flint-axes-2

Qwen3.8-27B · 1 × B200

1 sequence, per token8 sequences, per step× fastest generaltokens/sTTFT, s
Systemmsrangemsrange1 seq.8 seq.1 seq.8 seq.1 seq.Job
SGLang9.5319.529–9.53310.30010.236–10.4291.00×1.00×104.1770.40.060rel-q27-head-k2
vLLM10.19410.166–10.22810.73910.734–10.7430.93×0.96×97.6738.90.043rel-q27-head-k2
megakernel, track A (s9 s0022)8.4508.447–8.4518.8868.883–8.8881.13×1.16×117.3882.30.083rel-q27-head-k2
megakernel, track B (s11 s0020)7.2577.248–7.2717.7187.713–7.7241.31×1.33×136.31,006.50.088rel-q27-head-k2
SGLang9.5279.525–9.53110.23310.230–10.2371.00×1.00×104.2773.00.064rel-q27-head-f
vLLM10.20110.159–10.26110.73910.734–10.7440.93×0.95×95.7739.90.044rel-q27-head-f
megakernel, track B, earlier build (s9 s0023)7.7127.695–7.7218.1808.161–8.2081.24×1.25×128.1950.50.089rel-q27-head-f
SGLang9.5309.529–9.5311.00×104.40.062rel-q27-axes-2
vLLM10.16710.159–10.1760.94×98.10.043rel-q27-axes-2
megakernel, track A, earlier build (s5 s0016)8.8888.886–8.8899.8439.841–9.8441.07×111.6798.20.084rel-q27-axes-2

Qwen3-32B · 1 × B200

1 sequence, per token8 sequences, per step× fastest generaltokens/sTTFT, s
Systemmsrangemsrange1 seq.8 seq.1 seq.8 seq.1 seq.Job
SGLang11.42611.425–11.42711.68211.679–11.6841.00×1.00×87.5683.10.021rel-q32-head-k2
vLLM12.25212.250–12.25412.22712.221–12.2300.93×0.96×81.6652.80.020rel-q32-head-k2
megakernel, track A (s12 s0033)10.11410.111–10.11910.43510.431–10.4381.13×1.12×93.0723.80.658rel-q32-head-k2
megakernel, track B (s12 s0036)8.9888.984–8.9929.4359.432–9.4371.27×1.24×104.7801.20.582rel-q32-head-k2

gpt-oss-120b · 1 × B200

1 sequence, per token8 sequences, per step× fastest generaltokens/sTTFT, s
Systemmsrangemsrange1 seq.8 seq.1 seq.8 seq.1 seq.Job
SGLang2.2162.215–2.2173.3283.236–3.3791.00×1.00×448.62,388.00.016rel-oss120-head-k2
vLLM2.5223.7723.763–3.7770.88×0.88×393.82,157.40.020rel-oss120-head-k2
megakernel, track A (s12 s0049)1.7761.775–1.7763.0733.071–3.0761.25×1.08×521.32,435.60.148rel-oss120-head-k2
megakernel, track B (s12 s0051)1.6881.685–1.6893.0263.024–3.0271.31×1.10×546.62,471.90.147rel-oss120-head-k2

Inkling-2-Small · 4 × B200 (one node)

1 sequence, per token8 sequences, per step× fastest generaltokens/sTTFT, s
Systemmsrangemsrange1 seq.8 seq.1 seq.8 seq.1 seq.Job
SGLang4.7374.734–4.7436.5536.534–6.5871.00×1.00×210.31,214.80.023rel-ink-head-2
megakernel, track A (s7 s0034)3.3035.3911.43×1.22×281.41,417.90.259rel-ink-head-2
megakernel, track B (s7 s0036)3.2893.288–3.2895.3775.376–5.3781.44×1.22×282.71,421.90.257rel-ink-head-2
SGLang4.7164.709–4.7236.5716.545–6.5971.00×1.00×211.01,205.10.023rel-inkon2-cells
megakernel, serving, track A (s8 s0043)11.44211.433–11.45114.12514.117–14.1330.41×0.47×87.3564.30.019rel-inkon2-cells
SGLang4.7454.736–4.7541.00×209.90.025rel-ink-axes-2
megakernel, track A, earlier build (s7 s0020)3.2683.267–3.2685.3865.385–5.3891.45×284.51,419.00.256rel-ink-axes-2

One setting away from the benchmark

Requests started together: 1 to 64

64-token prompt, 1,024 tokens each, greedy. Time per step exists where all requests decode at once; total tokens per second includes the wait for the first token. A steady-decode engine holds 8 sequences and serves more requests in turns.

Qwen3-8B · 1 × H100

ms per step at batchtotal tokens per second at batch
System12481248163264
SGLang6.0866.7156.6946.8531642975971,1642,2514,1076,869
vLLM6.7536.6926.7186.9471482995951,1492,2334,1357,034
megakernel, track A (s11 s0016)5.1335.2015.3185.4781823597041,3811,3811,381
megakernel, track B (s11 s0020)3.8273.9394.1104.2542434739081,7771,7781,778
megakernel, track A, earlier build (s10 s0003)5.3775.4745.4815.6361743426831,3431,3421,343
megakernel, track B, earlier build (s10 s0002)4.1584.2294.3234.4692244428651,6941,6951,695
megakernel, serving, track A (s3 s0038)6.0966.1016.1066.2891643286541,2692,3854,2456,908
megakernel, serving, track B (s3 s0039)5.6245.6615.6945.8341783537011,3692,5494,4297,064

North Mini Code 1.0 · 1 × H100

ms per step at batchtotal tokens per second at batch
System12481248163264
vLLM5.0575.9368.0119.1751973364988691,3782,228

gpt-oss-20b · 1 × H100

ms per step at batchtotal tokens per second at batch
System12481248163264
SGLang3.5043.9704.4215.2172854968931,5202,7624,837
vLLM3.3123.6404.0844.9293015479771,6273,0175,228
megakernel, track A (s12 s0047)1.8742.2522.8623.3884978311,3342,2712,2672,264
megakernel, track B (s12 s0050)1.6152.0922.6563.3885718911,4152,2682,2722,265

flint · 1 × B200

ms per step at batchtotal tokens per second at batch
System12481248163264
vLLM1.9892.2582.7853.7854868421,3782,0503,0534,454
megakernel, track A (s11 s0026)1.3921.8002.3453.3546601,0301,6012,2912,2892,283
megakernel, track B (s11 s0027)1.3561.7632.3413.3496761,0501,6042,2952,2932,287
megakernel, track A, earlier build (s4 s0077)1.4781.8942.4213.3986229801,5512,2612,2592,254
megakernel, track B, earlier build (s4 s0055)1.4771.9102.7193.9536239711,3801,9351,9391,932
megakernel, track A, earlier build (s4 s0054)1.5251.9462.6773.8816049541,4061,9751,9741,969

Qwen3.8-27B · 1 × B200

ms per step at batchtotal tokens per second at batch
System12481248163264
SGLang9.5319.7199.84710.3001042054027701,4412,566
vLLM10.19410.34410.46510.739981923797391,3872,518
megakernel, track A (s9 s0022)8.4508.6538.6428.886117229456882883882
megakernel, track B (s11 s0020)7.2577.5507.5397.7181362615201,0071,0081,008
megakernel, track B, earlier build (s9 s0023)7.7128.0188.0228.180128246489951953953
megakernel, track A, earlier build (s5 s0016)8.8889.0509.1679.843112219430798798798

Qwen3-32B · 1 × B200

ms per step at batchtotal tokens per second at batch
System12481248163264
SGLang11.42611.60611.60911.682871723436831,3382,438
vLLM12.25212.35812.41312.227821623226531,2842,426
megakernel, track A (s12 s0033)10.11410.23210.47010.43593184360724724724
megakernel, track B (s12 s0036)8.9889.1509.3529.435105206403801801801

gpt-oss-120b · 1 × B200

ms per step at batchtotal tokens per second at batch
System12481248163264
SGLang2.2162.5722.9453.3284497731,3392,3883,8706,316
vLLM2.5222.8723.1873.7723946841,2372,1573,4325,658
megakernel, track A (s12 s0049)1.7762.2872.6143.0735218191,4612,4362,4442,467
megakernel, track B (s12 s0051)1.6882.2072.5533.0265478461,4962,4722,4812,506

Inkling-2-Small · 4 × B200 (one node)

ms per step at batchtotal tokens per second at batch
System12481248163264
SGLang4.7375.1145.8756.5532103896781,2152,1163,3665,409
megakernel, track A (s7 s0034)3.3033.7554.3305.3912814978691,4181,4221,426
megakernel, track B (s7 s0036)3.2893.7394.3165.3772834998721,4221,4261,430
megakernel, serving, track A (s8 s0043)11.44211.82912.35114.125871693235641,0431,8173,154
megakernel, track A, earlier build (s7 s0020)3.2683.7514.3335.3862854988681,4191,4231,427

Long generations and long prompts

One sequence unless noted, greedy. Generation: median time per token over a generation of 4,096 or 16,384 tokens (64-token prompt). Prompt: time to the first token for a prompt of the given length (16 tokens generated). The last columns are eight requests with 4,096-token prompts and 256 tokens each. fails marks a request the engine refused or could not serve (hover for the reason); "> context" marks a request beyond the context the engine is checked to, which is 2,048 tokens for the steady-decode engines.

Qwen3-8B · 1 × H100

generation, ms per tokenTTFT (s) by prompt tokens8 × (4,096 + 256)
System4,09616,3841,0244,09616,38432,000TTFT, stokens/s
SGLang6.5970.0290.1000.4901.2370.408726
vLLM7.1150.0290.1020.4921.2350.499720
megakernel, serving, track A (s3 s0038)7.2070.0320.1300.6691.7790.554664
megakernel, serving, track B (s3 s0039)6.8790.0290.1180.6051.6340.514701

North Mini Code 1.0 · 1 × H100

generation, ms per tokenTTFT (s) by prompt tokens8 × (4,096 + 256)
System4,09616,3841,0244,09616,38432,000TTFT, stokens/s
vLLM4.8374.9790.0470.0840.355

flint · 1 × B200

generation, ms per tokenTTFT (s) by prompt tokens8 × (4,096 + 256)
System4,09616,3841,0244,09616,38432,000TTFT, stokens/s
vLLM1.9892.0140.0640.0650.1260.2231,661
megakernel, track A, earlier build (s4 s0054)> context> contextfails> context> context> context> context

Qwen3.8-27B · 1 × B200

generation, ms per tokenTTFT (s) by prompt tokens8 × (4,096 + 256)
System4,09616,3841,0244,09616,38432,000TTFT, stokens/s
SGLang9.94911.4140.0640.1950.9141.001471
vLLM10.18710.2730.0750.1940.7151.198510
megakernel, track A, earlier build (s5 s0016)> context> context0.434> context> context> context> context

Inkling-2-Small · 4 × B200 (one node)

generation, ms per tokenTTFT (s) by prompt tokens8 × (4,096 + 256)
System4,09616,3841,0244,09616,38432,000TTFT, stokens/s
SGLang4.7184.7300.0360.0710.2100.4080.2921,020
megakernel, serving, track A (s8 s0043)11.6870.0350.1000.3560.7040.352484
megakernel, track A, earlier build (s7 s0020)> context> context3.391> context> context> context> context

Natural-text prompts and sampling

Milliseconds per token with 1 sequence and per step with 8; 64-token prompts, 1,024 new tokens, two rounds. Random tokens are the benchmark's prompts; natural text is prose and code. Sampling is temperature 0.7, top-p 0.8, top-k 20. Rows named mkserve are our engines behind the repository's server, which does the sampling; the rows without it are the same engines behind the evaluation's own front end (greedy only).

Qwen3-8B · 1 × H100 rel-q8-nat-f

random tokens, greedynatural text, greedynatural text, sampled
System1 seq.8 seq.1 seq.8 seq.1 seq.8 seq.
SGLang6.0356.8346.0246.8386.1006.948
vLLM6.7856.8836.5986.8826.7297.031
megakernel, track A, mkserve5.3805.6415.3855.6425.5245.794
megakernel, track B, mkserve4.1664.4694.1654.4694.3084.625

North Mini Code 1.0 · 1 × H100 rel-north-nat-1

random tokens, greedynatural text, greedynatural text, sampled
System1 seq.8 seq.1 seq.8 seq.1 seq.8 seq.
vLLM4.8909.2734.89710.1675.08210.367
megakernel, track A, mkserve2.5867.0252.5827.9892.7347.967
megakernel, track B, mkserve2.2716.2802.2766.9892.4156.898
megakernel, track A (s0022)2.5366.9552.5367.924
megakernel, track B (s0021)2.1536.1352.1536.852

flint · 1 × B200 rel-flint-nat-f

random tokens, greedynatural text, greedynatural text, sampled
System1 seq.8 seq.1 seq.8 seq.1 seq.8 seq.
vLLM1.9703.7861.9713.8902.0894.020
megakernel, track A, mkserve1.5543.4871.5543.6181.6823.763
megakernel, track B, mkserve1.5504.0781.5504.1831.6764.318

Qwen3.8-27B · 1 × B200 rel-q27-nat-f

random tokens, greedynatural text, greedynatural text, sampled
System1 seq.8 seq.1 seq.8 seq.1 seq.8 seq.
SGLang9.52410.2329.52510.2299.61210.333
vLLM10.16010.73310.16010.73410.32410.915
megakernel, track A, mkserve8.5489.0018.5439.0008.6449.100
megakernel, track B, mkserve7.7718.2337.7708.2327.8818.366

Inkling-2-Small · 4 × B200 (one node) rel-ink-nat-2

random tokens, greedynatural text, greedynatural text, sampled
System1 seq.8 seq.1 seq.8 seq.1 seq.8 seq.
SGLang4.7356.5434.7396.7774.7746.878
megakernel, track A, mkserve3.4845.5633.4805.8953.6216.044
megakernel, track B, mkserve3.4675.5593.4685.8953.6136.055

Replay of a production request trace

The first 300 seconds of the Mooncake tool-and-agent trace, thinned to the nominal rate, every system behind HTTP on the same GPU or node; rounds after one warm-up round. Rate is the nominal rate and req/s what the GPU was offered. In SLO: first token within 5 s and a mean time between tokens of at most 0.1 s. TBT is the time between a request's tokens; stall is the largest gap any decoding request saw; tokens/s is the output of all requests together.

Qwen3-8B · 1 × H100

TTFT, sTBT, ms
Ratereq/sSystemRequestsIn SLO, %p50p99p50p99Stall, msTokens/sRoundsJob
10.79SGLang23868.51.415.8919.11,8526,1591672rel-q8-replay-a2
10.79vLLM23870.41.375.9621.62755041672rel-q8-replay-a2
10.79megakernel, serving, track A23895.40.237.8119.5721091662rel-q8-replay-a2
10.79megakernel, serving, track B23896.60.217.4320.368881672rel-q8-replay-a2
21.61SGLang48371.91.295.5525.12,0646,9253252rel-q8-replay-a2
21.61vLLM48375.21.425.9223.43025773252rel-q8-replay-a2
21.61megakernel, serving, track A48395.70.237.9622.476883232rel-q8-replay-a2
21.61megakernel, serving, track B48396.90.217.1924.075873252rel-q8-replay-a2
32.39SGLang71659.22.3112.1042.81,5637,3284672rel-q8-replay-b
32.39vLLM71667.92.078.8543.92986314692rel-q8-replay-b
32.39megakernel, serving, track A71689.60.4215.4060.679914602rel-q8-replay-b
32.39megakernel, serving, track B71693.20.3712.0957.877924662rel-q8-replay-b
43.49SGLang1,04613.225.3174.0352.14359,1515282rel-q8-replay-b
43.49vLLM1,04616.720.1864.5254.52641,1705422rel-q8-replay-b
43.49megakernel, serving, track A1,04664.02.75105.8464.0801025242rel-q8-replay-b
43.49megakernel, serving, track B1,04667.62.5978.6861.9791355372rel-q8-replay-b

Inkling-2-Small · 4 × B200 (one node)

TTFT, sTBT, ms
Ratereq/sSystemRequestsIn SLO, %p50p99p50p99Stall, msTokens/sRoundsJob
10.79SGLang23883.60.562.219.26372,4761692rel-inkon2-replay-r1
10.79megakernel, serving, track A238100.00.193.9827.969751662rel-inkon2-replay-r1
21.61SGLang48384.60.592.189.47322,3503322rel-inkon2-replay-r2
21.61megakernel, serving, track A483100.00.173.4129.369803242rel-inkon2-replay-r2
43.49SGLang1,04683.10.612.1610.17552,6026662rel-inkon2-replay-r4
43.49megakernel, serving, track A1,046100.00.183.4345.473876562rel-inkon2-replay-r4
87.07SGLang2,12081.70.622.0118.07894,4481,2812rel-inkon2-replay-r8
87.07megakernel, serving, track A2,12099.20.204.6360.2801481,2502rel-inkon2-replay-r8

Runs and submissions

Every evaluated submission of every run, with the run's own evaluator's score: the headline is time per token with one sequence (µs; in process for the first run, over HTTP afterwards) or, for a serving run, the share of requests within the service levels at the rate in the metric's name. The same-GPU tables above are the re-measured figures. Source, hypothesis and result of each submission are in the repository under runs/.

RunModelGPUAgentAgent-hours (budget)SubmissionsValidPurpose
s2-q8b-a1Qwen3-8BH100Claude Opus 5.5406666steady decoding, tracks A and B
s3-q8b-online-a1Qwen3-8BH100Claude Opus 5.5403936serving
s4-north-h100-a1North Mini Code 1.0H100Claude Opus 5.5242323steady decoding, tracks A and B
s4-flint-b200-a2flintB200Claude Fable 5.2367776steady decoding, tracks A and B
s4-qwen38-27b-b200-a1Qwen3.8-27BB200Claude Opus 5.5245048steady decoding, track A; multi-token decoding
s5-qwen38-27b-b200-b1Qwen3.8-27BB200Claude Opus 5.561714track B, continuing the run above
s6-inkling2s-b200x4-a1Inkling-2-Small4 × B200Claude Opus 5.5241411steady decoding, track A
s7-inkling2s-b200x4-fa1Inkling-2-Small4 × B200Claude Fable 5.2122019steady decoding, track A, continuing the run above
s7-inkling2s-b200x4-fb1Inkling-2-Small4 × B200Claude Fable 5.2153635steady decoding, track B
s8-inkling2s-b200x4-on1Inkling-2-Small4 × B200Claude Fable 5.2324342serving
s9-qwen38-27b-b200-f1Qwen3.8-27BB200Claude Fable 5.282323steady decoding, tracks A and B, continuing the two runs above
s10-q8b-h100-f1Qwen3-8BH100Claude Fable 5.2344correction of track A at long positions
s11-q8b-h100-b1Qwen3-8BH100Claude Opus 5.5202524track B, continuing the two runs above
s11-north-h100-b1North Mini Code 1.0H100Claude Opus 5.5202626track B, continuing the run above
s11-flint-b200-b1flintB200Claude Opus 5.5202727track B, continuing the run above
s11-q27-b200-b1Qwen3.8-27BB200Claude Opus 5.5302929track B, continuing the three runs above
s12-oss20-h100-a1gpt-oss-20bH100Claude Opus 5.5325049steady decoding, tracks A and B, from the starter
s12-q32-b200-a1Qwen3-32BB200Claude Opus 5.5364747steady decoding, tracks A and B, from the starter
s12-oss120-b200-a1gpt-oss-120bB200Claude Opus 5.5365957steady decoding, tracks A and B, from the starter

An agent-hour is an hour of the agent's wall-clock time with a GPU pod ready. The serving run on Inkling-2-Small and the last run on Qwen3.8-27B stopped at 31.2 and 7.2 hours.

s2-q8b-a1 · Qwen3-8B, H100 · 66 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.1validstep_us_b121,051starter: Hugging Face reference
s00020.4validstep_us_b17,366own CUDA operators: MMA matvec block team + attention, one launch per operator
s00030.4validstep_us_b16,465whole decode step in one kernel launch, grid barriers between phases
s00041.0validstep_us_b16,385down-projection entry: copy only live batch columns, no zero fill
s00051.1validstep_us_b16,139lean phases: tile tables, fixed-trip unrolled loaders, coordinator warp, no divisions
s00061.2validstep_us_b15,951fuse RMSNorm into the previous epilogue: publish bf16(w*x), scale matvec outputs by 1/rms
s00071.2validstep_us_b15,886generate(): greedy loop on the device (argmax + token feedback in the kernel, one launch)
s00082.4validstep_us_b15,810SMs idle during attention stream their first O tile into shared memory
s00092.7validstep_us_b15,859coordinator requests residual rows and norm weights before the tile barrier
s00103.3validstep_us_b15,771kernel = call-free interpreter of phase descriptors; loader and coordinator code split
s00113.4validstep_us_b15,7561/rms of a norm phase computed by warps that have no input-copy loads
s00123.5validstep_us_b15,694SMs idle during attention first pull the layer's K/V history into L2
s00134.0validstep_us_b15,659attention q/k/v prep: one warp per vector, norm and RoPE by warp shuffles, no block barrier
s00144.4validstep_us_b15,651input copy: idle threads leave at once, loads requested in pairs before the stores
s00154.7validstep_us_b15,595weight loads use the last-use cache operator (ld.lu)
s00166.4validstep_us_b15,528loaders request 2 loads + the extra kstep of the next tile before the tile barrier
s00176.6validstep_us_b15,515O phase: request the second tile's first loads while the stash tile is computed
s00187.6validstep_us_b15,455at the end of a phase, prefetch the next phase's first 2 ksteps per warp into L2
s00197.9validstep_us_b15,422L2 prefetch in the attention and embedding windows (second O tile, attention blocks' O tile, first QKV tile)
s00208.3validstep_us_b15,398while a block waits at the grid barrier, its coordinator prefetches more of the next phase
s00218.6validstep_us_b15,382compile with fast-math: approximate division and rsqrt in the scalar epilogue code
s00229.0validstep_us_b15,34024 warps per block (80 registers per thread) with a 3-load tile pipeline
s00239.6validstep_us_b15,336source cleanup; lm_head prefetches layer 0 for the next launch
s00249.9validstep_us_b15,3475+ sequences: input fragments stored in the shared-memory layout, input copy becomes a block copy
s002511.1validstep_us_b14,522weights stored losslessly in 12 bits, decoded in registers before the MMA
s002611.8validstep_us_b14,464exact per-column power-of-two weight scales: no exception-heavy tiles
s002712.1validstep_us_b14,366up to 4 sequences: narrow input fragments, block shared memory 223 -> 154 KB
s002812.3validstep_us_b14,27012-bit stream laid out for a mask-and-OR decode (11 instead of 31 ops per MMA)
s002913.4validstep_us_b14,198loaders stash the start of the next phase's first tile while the grid barrier runs
s003014.2validstep_us_b14,1843+ sequences: loaders read the input fragments from L2, no input copy
s003118.7Bvalidstep_us_b14,186attention to 9216+ positions: 16 B/position scratch, split over all SMs for 1-2 sequences
s003219.2Avalidstep_us_b15,386track A: BF16 weights as stored, BF16 activations, RMSNorm before the matvec
s003320.4Bvalidstep_us_b14,161attention: prefetch the K/V append lines; idle blocks stash after the K loads (track B)
s003420.4Avalidstep_us_b15,343track A with the attention scheduling of s0033
s003521.5Bvalidstep_us_b14,154grid-barrier increment directly behind the fence (track B)
s003621.6Avalidstep_us_b15,332track A with the early grid-barrier increment of s0035
s003722.4Bvalidstep_us_b14,105attention: the idle blocks' stash starts one load at a time (track B)
s003822.5Avalidstep_us_b15,299track A with the gentle attention-phase stash of s0037
s003923.0Bvalidstep_us_b14,088o projection: attention blocks take two tiles; second tile prefetched at phase start (track B)
s004023.1Avalidstep_us_b15,298track A with the o tile table of s0039 (attention blocks take two tiles)
s004123.7Bvalidstep_us_b14,0936+ sequences: stash blocks prefetch all of their second o tile (track B)
s004223.7Avalidstep_us_b15,304track A with the second-tile prefetch rule of s0041
s004325.7Bvalidstep_us_b14,085softmax in two passes less at long context; split path copies scores in one round trip (track B)
s004425.7Avalidstep_us_b15,286track A with the long-context softmax changes of s0043
s004527.3Bvalidstep_us_b14,095long context, 3+ sequences: idle blocks pull the first K tiles into L2 (track B)
s004627.3Avalidstep_us_b15,288track A with the long-context K touch of s0045
s004729.1Bvalidstep_us_b14,086long context, 1 sequence: idle blocks pull V into L2 during score copy + softmax
s004829.1Avalidstep_us_b15,292track A with the split-path V touch of s0047
s004930.1Avalidstep_us_b15,278track A, 3+ sequences: loaders apply RMSNorm to the input; no norm phase
s005030.4Avalidstep_us_b15,284track A: normalised input kept in shared memory after a block's first tile
s005130.9Bvalidstep_us_b14,094long context, 3+ sequences: 6 V loads in flight per warp instead of 4
s005231.4Bvalidstep_us_b14,083attention: no fence behind the K/V append
s005331.4Avalidstep_us_b15,230track A with the fence removal of s0052 and the V batching of s0051
s005433.5Bvalidstep_us_b14,0611-2 sequences: loader warps fetch their own input slice; no block copy/barrier
s005533.6Avalidstep_us_b15,233track A with the per-warp input slices of s0054 (phases without a norm)
s005633.9Bvalidstep_us_b14,0573+ sequences: no block barrier at the start of each phase
s005734.4Bvalidstep_us_b14,043q/k/v, gate+up, down on 128 of the 132 SMs: every block gets the same tile count
s005834.6Avalidstep_us_b15,235track A: no entry barrier with 3+ sequences, matvec tiles on 128 SMs with 3+
s005935.7Bvalidstep_us_b14,047only the attention blocks wait at the q/k/v barrier; the others go on
s006035.9Avalidstep_us_b15,234track A: only attention blocks wait at the q/k/v barrier (3+ sequences)
s006137.8Bvalidstep_us_b14,026re-evaluation of the final track B source (s0059 unchanged)
s006237.9Avalidstep_us_b15,237re-evaluation of the final track A source (s0060 unchanged)
s006338.2Bvalidstep_us_b14,020attention blocks: L2 prefetch of 3 (not 6) loads per warp of their cold o tile
s006438.4Avalidstep_us_b15,235track A with the o-tile prefetch of s0063 (3 loads per warp)
s006539.3Bvalidstep_us_b14,047final track B source (s0063 kernel; host-side diagnostic export only)
s006639.4Avalidstep_us_b15,236final track A source (s0064 kernel; host-side diagnostic export only)
s3-q8b-online-a1 · Qwen3-8B, H100 · 39 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.0incorrectinherited engine, no online interface
s00021.1incorrectpass engine: decode rows + prefill chunks through one GEMM pass, FIFO
s00031.4incorrectscheduler: deadline-ordered prefill, 60 ms pass budget, prefix page cache
s00042.3validslo_share_r283.1%attention rewrite (block-staged K/V, 2-level fold) + size-class scheduler
s00053.0validslo_share_r290.5%GEMM on the Hopper path: wgmma + bulk async copy + mbarriers
s00063.2validslo_share_r293.7%attention: K/V staged by the copy engine, no rescale when max unchanged
s00073.7validslo_share_r294.2%attention on the warpgroup MMA, pipelined over segments
s00084.6validslo_share_r294.2%no host scheduler: the serving loop runs on the device as one graph
s00095.5validslo_share_r294.2%64-row GEMM blocks with 8 stages for passes of up to 64 columns
s00106.1validslo_share_r295.2%pass overheads: batched residual loads, 8-byte q/K stores, wgmma stage in flight
s00117.5validslo_share_r295.2%GEMM block shape per matrix and pass size; generate() loops on the device
s00127.6validslo_share_r291.5%pass target 80 -> 27 ms: median time between tokens under 25 ms
s001310.6validslo_share_r292.1%step() as one graph launch; decode rows stop re-clearing their V group
s001410.7validslo_share_r292.1%GEMM: four producer threads for 64-row blocks; prefetch of the next kernel's first stages
s001511.0validslo_share_r292.1%decode-row attention: operand roles swapped, tiles on three warpgroups
s001611.1validslo_share_r292.1%1/rms computed inside the next GEMM up to 64 columns (two kernels per layer less)
s001712.6validslo_share_r292.6%deadline planner: prompt share from the state; 45 ms passes while a prompt is short of slack
s001813.5validslo_share_r292.6%q, K, V prepared in the q/k/v GEMM epilogue up to 16 columns (no prep kernel)
s001914.4validslo_share_r292.6%deadline order: shortest first among prompts that arrive together
s002014.7validslo_share_r293.1%prefill attention: the two warpgroups of a block alternate tensor and scalar phases
s002115.4validslo_share_r293.1%decode-row attention in its own kernel (no wgmma serialized for lack of registers)
s002215.7validslo_share_r293.1%attention fold: threads without a position of the unit skip
s002317.2Bvalidslo_share_r294.7%tail guard: large passes while a prompt heads for a first token later than 8 s
s002418.0Avalidslo_share_r294.7%track A build (engine/TRACK=A): the engines' formats and data flow
s002522.4Bvalidslo_share_r294.7%decode step: PDL + early weights, counter-free decode attention (5.67->5.58 ms)
s002624.5Bvalidslo_share_r294.7%light decode passes in the serving loop (SWITCH node), decode pass -0.8 ms
s002725.6Bvalidslo_share_r294.7%early weight stages before the dependency wait (offline 5.57 -> 5.46 ms)
s002825.9Bvalidslo_share_r294.7%PDL: attention set-up before its wait, non-GEMM kernels trigger at start
s002927.6Bvalidslo_share_r294.7%per-shape GEMMs for prompt passes (bucket branches); idle blocks leave early
s003029.2Bvalidslo_share_r294.7%GEMM epilogue: unit's 1/rms staged in shared memory
s003129.8Bvalidslo_share_r295.2%near-miss trigger on (MK_NEAR_S 0.75): big passes while a prompt is just past 5 s
s003230.2Bvalidslo_share_r295.8%tail guard 5 s with the near-miss window
s003331.6Bvalidslo_share_r295.2%pass target 25 ms with near-miss window; age cap 10 s
s003433.7Avalidslo_share_r294.2%track A: k_rnorm fills whole lines above 64 columns; near-miss window off
s003536.0Bvalidslo_share_r295.2%final track B source (s0033 engine; track-A-only changes since)
s003636.8Avalidslo_share_r294.2%track A: one-column steps normalize inside QKV/GU (no norm kernels at B=1)
s003737.5Bvalidslo_share_r295.2%ADD epilogue: last unit's residual staged in smem by cp.async (one round trip)
s003838.4Avalidslo_share_r294.7%track A: residual staged in smem by cp.async in the ADD epilogue (as s0037)
s003939.4Bvalidslo_share_r295.2%tail guard 8 -> 7 s (shorter first-token tail; same share and tbt)
s4-north-h100-a1 · North Mini Code 1.0, H100 · 23 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.5Avalidstep_us_b13,497v1 megakernel: one persistent kernel, BF16 weights in mma fragment order
s00020.8Avalidstep_us_b13,050v2: static per-block schedule, norm and residual combined in shared memory
s00031.1validhttp_step_us_b13,6038,995OPERATOR: s0001 again, unchanged (evaluated again under the HTTP headline, harness 418e7ad)
s00041.2validhttp_step_us_b13,1697,927OPERATOR: s0002 again, unchanged (evaluated again under the HTTP headline, harness 418e7ad)
s00052.0Avalidhttp_step_us_b13,1967,615v4: balanced quantum schedule, shallow HBM queue (2 loads in flight), leader-first routing
s00063.3Avalidhttp_step_us_b12,9337,257v5: attention in 8 position chunks over 8 blocks; device loop behind submit/poll
s00073.8Bvalidhttp_step_us_b12,6286,938track B v6: lossless 12-bit weights (row exponents, neuron shifts, exact escapes)
s00084.8Bvalidhttp_step_us_b12,5986,698track B: layer 0 from token tables (q|k|v and MLP output per token id)
s00095.6Bvalidhttp_step_us_b12,5726,699track B: per-layer skeleton trimmed (residual add on 4 warps, top-8 by warp reduce, norm totals per column)
s00105.7Bvalidhttp_step_us_b12,4746,616track B: certified int8 sketch head on the greedy path (integer tensor core, exact survivors)
s00117.0Bvalidhttp_step_us_b12,4126,421track B: stage B start balance (early union, leader first, delay-aware gate|up shares)
s00127.2Bvalidhttp_step_us_b12,3666,382track B: 12-bit format re-laid out for a 3-instruction decode (rotate + LOP3)
s00137.8Bvalidhttp_step_us_b12,3806,310track B: o_proj in stage B at batch > 1 (attention-done stamps; shares include it)
s00148.5Bvalidhttp_step_us_b12,3736,240track B: RMSNorm rounding with cvt.rn.bf16x2 (bit-identical, fewer instructions)
s00158.9Bvalidhttp_step_us_b12,3656,207track B: stage C starts without dependent round trips (late escape gather, local B=1 routing)
s001613.2Avalidhttp_step_us_b12,7857,157track A: s0006 + the scheduling work since (shares by start delay, o_proj in B at B>1, decode-free paths)
s001715.4Bvalidhttp_step_us_b12,2526,049track B: weight loads skip L1; zero operand from shared memory (stage C x stays in L1)
s001818.6Bvalidhttp_step_us_b12,1876,061track B: o_proj in stage B at batch 1 too (now that stage C's x stays in L1)
s001919.7Avalidhttp_step_us_b12,5736,904track A: BF16 engine + L1 no-allocate weight loads, o_proj in stage B (kernel organisation only)
s002020.9Bvalidhttp_step_us_b12,1736,039track B: gate|up results and residual rows as exchanges instead of store + fence
s002121.7Bvalidhttp_step_us_b12,1466,028track B: union load issued during the last stage-A quantum (batch-1 kernel)
s002222.0Avalidhttp_step_us_b12,5336,893track A: s0019 + exchanges instead of fences, early union load at batch 1
s002322.6Bvalidhttp_step_us_b12,1505,989track B: share model weights attention's per-K-tile cost more (long contexts)
s4-flint-b200-a2 · flint, B200 · 77 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.8Avalidstep_us_b12,679First megakernel: one launch per step, tensor-core tile GEMV (track A)
s00020.9validhttp_step_us_b13,0037,816OPERATOR: s0001 again, unchanged (evaluated again under the HTTP headline, harness 418e7ad)
s00031.6Aincorrect512-thread blocks, tiles staged in registers, L1-bypass loads, submit/poll serving
s00041.8Bvalidhttp_step_us_b12,1947,294Track B precision mode: float32 residual stream, hi+lo router input (same kernel as s0003)
s00052.6Bvalidhttp_step_us_b11,9686,273Attention mix by dim-chunk warps, parallel routing finals, K/V staging levels, head on 152 SMs
s00062.7Avalidhttp_step_us_b11,9016,081Track A build of the s0005 kernel (pure BF16 arithmetic, 2 accumulators per chain)
s00073.1Bvalidhttp_step_us_b11,7825,245Shared-memory reservation 224 -> 128 KB: L1 left for global loads
s00083.2Avalidhttp_step_us_b11,6854,796Track A build of the s0007 kernel (128 KB reservation, 16-tile expert batches)
s00095.1Bvalidhttp_step_us_b11,7724,158Race-free hand-overs (atomic writes), batch-8 routing and attention placement
s00105.2Avalidhttp_step_us_b11,7804,235Track A build of the s0009 kernel (race-free hand-overs, batch-8 changes)
s00115.4Bvalidhttp_step_us_b11,7454,152Two kernel instances (batch <=2, >2), down-phase loop of s0008 restored
s00125.5Avalidhttp_step_us_b11,7344,079Track A build of the s0011 kernel (two instances, s0008 down-phase loop)
s00136.2Bvalidhttp_step_us_b11,7464,101Batch >2 kernel: split tile requests in the down phase, norm scratch, deferred writes
s00146.3Avalidhttp_step_us_b11,7343,993Track A build of the s0013 kernel (batch >2: split tile requests, norm scratch)
s00156.8Bvalidhttp_step_us_b11,7334,100Head in 16-tile batches (staging registers)
s00166.8Avalidhttp_step_us_b11,7243,989Track A build of the s0015 kernel (head in 16-tile batches)
s00179.0Bvalidhttp_step_us_b11,7864,298Strict hand-overs: sentinel words, signal at phase start, readers poll the data
s00189.6Bvalidhttp_step_us_b11,7334,268Sentinels issued a phase ahead, confirmed by read-back; second stream buffer
s00199.6Avalidhttp_step_us_b11,7834,179Track A build of the s0018 kernel (strict sentinel hand-overs)
s002010.4Bvalidhttp_step_us_b11,7334,233Attention multiply-adds written out: bitwise batch invariance at batch 7-8
s002110.8Bvalidhttp_step_us_b11,7154,231Router top-6 by ranks instead of six rounds of warp maxima (batch 1-2)
s002210.8Bvalidhttp_step_us_b11,7014,224Expert phase, batch 1-2: idle warps fold and write the first slot's hidden units
s002310.8Bvalidhttp_step_us_b11,6994,221Shared-expert epilogue and K/V cache stores moved off the critical path
s002410.9Avalidhttp_step_us_b11,7324,111Track A build of the s0023 kernel (ranks, epilogues off the critical path)
s002511.2Bvalidhttp_step_us_b11,6634,222Attention at long contexts: K/V cache tiles 8 per request, not one per trip
s002611.3Avalidhttp_step_us_b11,6794,116Track A build of the s0025 kernel (K/V cache tiles 8 per request)
s002711.6Bvalidhttp_step_us_b11,6464,227Attention: first extra batch of V tiles requested right after the scores
s002811.7Avalidhttp_step_us_b11,6694,118Track A build of the s0027 kernel (V tiles requested after the scores)
s002912.2Bvalidhttp_step_us_b11,6464,188Batch > 2: the 12 routing candidates of a sequence merged by ranks
s003012.3Avalidhttp_step_us_b11,6714,056Track A build of the s0029 kernel (batch > 2 routing merge by ranks)
s003112.4Bvalidhttp_step_us_b11,6464,123Batch > 2: router blocks preload router tiles, stage K/V after the router
s003212.4Avalidhttp_step_us_b11,6714,085Track A build of the s0031 kernel (router tiles preloaded at batch > 2)
s003312.9Bvalidhttp_step_us_b11,6524,120Attention: the k/v-row loads are issued together with the q-row loads
s003413.0Avalidhttp_step_us_b11,6714,097Track A build of the s0033 kernel (attention load order)
s003513.1Bvalidhttp_step_us_b11,6504,117Host sees the kernel's watchdog; contexts above 2048 positions refused
s003613.2Avalidhttp_step_us_b11,6724,099Track A build of the s0035 engine (watchdog visible, context cap)
s003713.7Bvalidhttp_step_us_b11,6064,116generate/serve at batch 1-2: the 152 blocks' best tokens gathered in one trip
s003813.8Avalidhttp_step_us_b11,6064,094Track A build of the s0037 kernel (token gather in one trip)
s003914.0Bvalidhttp_step_us_b11,6064,088generate/serve at batch 3-8: the blocks' best tokens gathered in 16 strides
s004014.0Avalidhttp_step_us_b11,6084,028Track A build of the s0039 kernel (batch 3-8 token gather in 16 strides)
s004114.8Bvalidhttp_step_us_b11,5604,055Softmax weights: one reciprocal and an exact FMA correction, not 4224 divisions
s004214.8Avalidhttp_step_us_b11,5483,928Track A build of the s0041 kernel (softmax weights by reciprocal + correction)
s004315.6Bvalidhttp_step_us_b11,5584,039Batch 7-8: main blocks with an attention pair poll phase A's counter early
s004415.6Avalidhttp_step_us_b11,5463,920Track A build of the s0043 kernel (early poll of phase A's counter, batch 7-8)
s004516.6Bvalidhttp_step_us_b11,5594,021Batch > 2: the norm's scale of each sequence computed once per warp, not by every thread
s004616.6Avalidhttp_step_us_b11,5463,912Track A build of the s0045 kernel (norm scales once per warp at batch > 2)
s004718.5Bvalidhttp_step_us_b11,5663,979Sentinel reads: the abort word is loaded on every 16th retry, not before each reload
s004818.6Avalidhttp_step_us_b11,5273,914Track A build of the s0047 kernel (abort word loaded on every 16th retry)
s004919.3Bvalidhttp_step_us_b11,5603,936Hand-over counters placed by a start-up timing: fastest of 16 pages per kernel
s005019.4Avalidhttp_step_us_b11,5273,870Track A build of the s0049 engine (counters placed by a start-up timing)
s005119.9Bvalidhttp_step_us_b11,5553,961Start-up placement extended: 3 more buffers at batch 1, 64 counter pages at batch 8
s005219.9Avalidhttp_step_us_b11,5163,898Track A build of the s0051 engine (extended start-up placement)
s005320.2Bvalidhttp_step_us_b11,5573,950Placement: batch 8 back to the 16 counter pages of s0049; batch 1 buffers on own slabs
s005420.2Avalidhttp_step_us_b11,5163,878Track A build of the s0053 engine (batch-8 counters as s0050, batch-1 buffers on own slabs)
s005521.3Bvalidhttp_step_us_b11,4733,963Track B: kernel body wrapped in an inlined function: better register allocation
s005621.4Avalidhttp_step_us_b11,5163,874Track A build of the s0055 source (wrapper off on this track: build unchanged)
s005723.6Bvalidhttp_step_us_b11,4753,950Hand-over reads as strong loads (ld.relaxed.gpu) instead of ld.global.cg
s005823.6Avalidhttp_step_us_b11,5203,877Track A build of the s0057 source (hand-over reads as strong loads)
s005924.0Avalidhttp_step_us_b11,5223,898Audit defects: over-long served requests and batch > max_batch now raise
s006024.8Avalidhttp_step_us_b11,5313,842Fragment getter without a conditional load (batch > 2 kernel)
s006126.1Avalidhttp_step_us_b11,5143,829Hand-over counters polled from warp 6, not from the leader warp 0
s006226.8Avalidhttp_step_us_b11,4943,784Attention K/V batches pinned with volatile asm; used in the batch > 2 kernel
s006327.7Avalidhttp_step_us_b11,4973,735Copy of the attention output with 4 loads in flight (batch > 2 kernel)
s006428.0Avalidhttp_step_us_b11,4963,577Down phase: one fragment copy up front (compact slots), rounds stream tiles only
s006529.1Avalidhttp_step_us_b11,4923,571Same engine as s0064; the later experiments behind switches, all off
s006629.3Avalidhttp_step_us_b11,4933,527Down-phase copy without its pointer array (fewer spills in the batch > 2 kernel)
s006729.7Avalidhttp_step_us_b11,4933,499Routing merge: the lane's candidate from shared memory, not from a stack array
s006829.9Avalidhttp_step_us_b11,4933,460Batch 7-8: output projection of the 8 pair blocks computed by 8 other main blocks
s006930.8Avalidhttp_step_us_b11,4933,448Hosted output projection: hosts moved from blocks 8..15 to blocks 24..31
s007031.1Avalidhttp_step_us_b11,4933,426Attention-output copy: all pending words of a batch re-requested per attempt
s007132.6Avalidhttp_step_us_b11,4933,426Same engine as s0070; cycle-counter study build and two off switches added
s007233.6Avalidhttp_step_us_b11,4913,412Routing union: two barriers fewer (table zeroed in stage 1, written by stage 2)
s007334.1Avalidhttp_step_us_b11,4903,405Routing candidates by rank counts in shared memory, no shuffle rounds (batch>2)
s007434.4Avalidhttp_step_us_b11,4843,408Batch 1-2: routing merge ranks at most 27 pruned candidates, one per lane
s007535.3Avalidhttp_step_us_b11,4783,397Routing union: last pass (unit tables) on all 16 warps, 4 sequences per thread
s007635.6Avalidhttp_step_us_b11,4833,394Routing merge: table bytes from the ranking lanes; weights on idle warps (B>2)
s007735.9Avalidhttp_step_us_b11,4793,397Routing union, last pass: counts of the earlier warps without a loop (B>2)
s4-qwen38-27b-b200-a1 · Qwen3.8-27B, B200 · 50 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00012.0Avalidhttp_step_us_b116,99017,606Megakernel v0: one launch per step, weights in MMA fragment order (unoptimized)
s00022.2Avalidhttp_step_us_b110,83414,059Kernel v3: no per-thread stack traffic, activations via L1, one stream loop per warp
s00032.5BerrorExact multi-token passes: MTP-head drafts verified P positions at a time
s00042.7BincorrectExact draft-and-verify: MTP head + context copy, up to 16 positions per pass
s00053.0Bvalidhttp_step_us_b11,1457,372Draft-and-verify engine resubmitted with release/acquire grid synchronisation
s00063.5Bvalidhttp_step_us_b11,1006,502Coalesced RMSNorm prologue and 32-column loop order (faster multi-column passes)
s00073.9Bvalidhttp_step_us_b18246,46032-position verify passes, L2 prefetch of the next matrix, barrier-free recurrence
s00084.2Bvalidhttp_step_us_b18435,738Per-sequence verify depth in a batch: copyable sequences get more positions
s00094.5Avalidhttp_step_us_b110,33712,028Track-A engine: everything since s0002 that keeps the checkpoint's formats and data flow
s00106.0Bvalidhttp_step_us_b16224,496Attention per column group, recurrence split over 2 SMs per head, B=8 columns to loops
s00116.5Avalidhttp_step_us_b110,26711,795Track-A build of the s0010 kernel plus faster 32-column tails
s00126.8Bvalidhttp_step_us_b15824,145Shorter GEMM tails at 32 columns: batched finalize loads, per-kind epilogues, free warp
s00136.9Avalidhttp_step_us_b110,18411,724Track-A build of the s0012 kernel
s00147.6Bvalidhttp_step_us_b15123,847No replay after full acceptance; spill-free split recurrence; 4-token copy match
s00157.7Avalidhttp_step_us_b110,17511,718Track-A build of the s0014 kernel
s00169.0Bvalidhttp_step_us_b15063,762Non-blocking GEMM prologues: cp.async residual rows, barrier-free RMSNorm scalar
s00179.1Avalidhttp_step_us_b110,19511,473Track-A build of the s0016 kernel
s001810.7Bvalidhttp_step_us_b14883,695Attention output folded into value sum; recurrent-op inputs 4 channels per thread
s001910.8Avalidhttp_step_us_b110,15011,435Track-A build of the s0018 kernel
s002011.2Bvalidhttp_step_us_b14793,690First 8 verify passes one at a time: copied drafts take effect one pass earlier
s002111.7Bvalidhttp_step_us_b14683,626Verify passes skip the MTP head while copied drafts keep being accepted
s002211.9Bvalidhttp_step_us_b14583,625B=1 ramp: pass length chosen at launch from the previous pass's copy result
s002312.4Avalidhttp_step_us_b110,15410,988Track A: two recurrent units per block at one position (B=8 step -4 %)
s002413.0Bvalidhttp_step_us_b14573,604B=8 recurrent units balanced across blocks; replay deltas via shared memory
s002515.5Bvalidhttp_step_us_b14583,604Serving fixes for 2-7 sequences (token buffers, unit dispatch); same speed
s002615.7Bvalidhttp_step_us_b14573,626Softmax with four warps per row
s002715.8Avalidhttp_step_us_b110,13910,925Track-A build of the s0026 kernel
s002816.3Bvalidhttp_step_us_b14573,624Prompt prefill in chunks (first token 0.68 s -> 0.09 s); same decode
s002916.4Avalidhttp_step_us_b110,14610,923Track-A build with chunked prompt prefill
s003017.0Bvalidhttp_step_us_b14503,573Cut tiles handed over with sentinels (one memory trip instead of three)
s003117.1Avalidhttp_step_us_b19,98110,713Track-A build of the s0030 kernel (sentinel hand-over of cut tiles)
s003217.4Bvalidhttp_step_us_b14493,571Split recurrent op: norm sums exchanged through sentinel slots
s003317.5Avalidhttp_step_us_b19,96810,728Track-A build of the s0032 kernel
s003418.2Avalidhttp_step_us_b19,87410,622Track A: no far L2 prefetch before small ops (only 128 KB before each GEMM)
s003518.8Avalidhttp_step_us_b19,50210,291Track A: GEMM hand-over without 64-bit divisions after the stream
s003618.9Cvalidhttp_step_us_b14323,437Track C build of the s0035 kernel (division-free hand-over, no far prefetch)
s003719.2Avalidhttp_step_us_b19,38010,199Track A: attention units spread over blocks; recurrent op without a start load
s003819.9Avalidhttp_step_us_b19,28110,065Track A: GEMM finalize roles at each warp's stream end; leaner prologue
s003920.0Cvalidhttp_step_us_b14263,404Track C build of the s0038 kernel
s004020.4Avalidhttp_step_us_b19,25810,065Track A: softmax in registers, early gates, split recurrence at B=1
s004120.6Cvalidhttp_step_us_b14263,393Track C build of the s0040 kernel
s004221.5Avalidhttp_step_us_b19,0069,900Track A: no local-memory reloads after op calls in the main loop
s004321.6Cvalidhttp_step_us_b14143,275Track C build of the s0042 kernel
s004422.6Avalidhttp_step_us_b18,9659,833Track A: debug timers compiled out of the default build
s004522.7Cvalidhttp_step_us_b14153,300Track C build of the s0044 kernel (debug timers compiled out)
s004623.0Avalidhttp_step_us_b18,9479,819Track A: remaining debug stamp checks compiled out
s004723.0Cvalidhttp_step_us_b14173,310Track C build of the s0046 kernel (final)
s004823.3Avalidhttp_step_us_b18,9389,800Track A: GEMM debug-flag checks compiled out
s004923.4Cvalidhttp_step_us_b14153,296Track C build of the s0048 kernel (final)
s005023.6Bvalidhttp_step_us_b110,68711,270Track B: lossless 12-bit weights (W12) on the s0048 track-A engine
s5-qwen38-27b-b200-b1 · Qwen3.8-27B, B200 · 17 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.1Avalidhttp_step_us_b18,9409,808Inherited engine s4/s0048 re-measured as this run's baseline
s00020.6BincorrectW12: exact 12-bit weights, per-row exponent windows, escapes as sparse correction
s00030.9BincorrectExact W12: 12-bit weights with per-row exponent windows, bit-identical to BF16
s00041.1Bvalidhttp_step_us_b18,6709,193Exact W12 (bit-identical to BF16), now also for multi-column prefill passes
s00051.5BerrorExact W12, W12-only build: unused BF16 and old W12 stream calls compiled out
s00061.5Bvalidhttp_step_us_b18,5579,136Exact W12, W12-only build (resubmission of s0005, lost to pod failures)
s00071.6Avalidhttp_step_us_b18,8929,812Track A: activation fragments loaded only for real columns (zeros for the rest)
s00082.0Bvalidhttp_step_us_b18,4099,131Exact W12 + compact one-column activation layout at B=1 (8x smaller L1 footprint)
s00092.4Bvalidhttp_step_us_b18,2639,009Exact W12 with 16-exponent row windows: half the escapes, less patching
s00102.8Bvalidhttp_step_us_b18,2278,975Exact W12: escape positions counted down, loop counters back in registers
s00114.0Bvalidhttp_step_us_b18,1948,938Exact W12, byte-permute layout: cheaper decode (5 ops per register)
s00124.2Bvalidhttp_step_us_b18,1698,908Exact W12: activation loads issued before the per-quad escape test
s00134.6Bvalidhttp_step_us_b18,1628,897Exact W12: compact-layout test hoisted out of the activation producers' loops
s00145.0Bvalidhttp_step_us_b18,1208,914Exact W12 + BF16 out_proj/o_proj for one-column passes (storage chosen per op)
s00155.2Bvalidhttp_step_us_b18,1048,860Exact W12: 96 KB per block of the next matrix prefetched into L2 during small ops
s00165.5Avalidhttp_step_us_b18,8809,827Track A: grid-barrier spin back-off 200 -> 25 ns
s00175.6Bvalidhttp_step_us_b18,0958,862Exact W12 with the 25 ns grid-barrier back-off
s6-inkling2s-b200x4-a1 · Inkling-2-Small, 4 × B200 · 14 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00014.2AerrorPersistent per-card kernel, TP4 attention + intra-expert TP4, B=1 redundant mode
s00024.7Avalidhttp_step_us_b16,56410,330Fix deadlock (header by value); relaxed signals; smem-staged B frags; phases pre-wait
s00035.5Avalidhttp_step_us_b15,3249,310768-thread blocks, block-contiguous GEMV, per-strip GEMVs, vectorised token phases
s00046.2AincorrectLL tagged-data cross-card exchange, router per-strip, attention and mix fixes
s00056.4Avalidhttp_step_us_b14,7788,845Slot-order invariant expert scaling (explicit FMA) on top of s0004
s00066.8Avalidhttp_step_us_b14,5818,242HW E2M1 rounding, 64-position attention chunks at B=1, w2 prefetch only at B=1
s00078.5Avalidhttp_step_us_b14,5417,731B=8 expert GEMVs: w2 weights+fragments in flight across experts, quant2 warp tasks; split-K router
s00089.3Avalidhttp_step_us_b14,3117,477Cross-card polls with all loads in flight before spinning; redux top-8; prefetch by last warp
s000910.0Avalidhttp_step_us_b14,2607,415Load-depth tuning per mode (FP4/MXFP8 unroll), router bias before the poll, async conv loads at B>=2
s001010.5Avalidhttp_step_us_b14,0787,214Attention starts on a counter of the qkv strips it reads; acq_rel hand-off atomics
s001111.2Avalidhttp_step_us_b14,0497,260Fix B>=2 combine/gate-row race (barrier back at B>=2); paired o_proj strips
s001212.2Avalidhttp_step_us_b13,9597,194Register relief: 32-bit GEMV indices, launch state in shared memory; two-chain B=1 quant2
s001312.7Avalidhttp_step_us_b13,6246,955Cross-card partial sums as bf16 pairs (half the polled bytes); adaptive B>=2 chunks
s001413.6Abuild_failedB>=2: whole-region K/V prefetch (barrier no longer waits for prefetch issue); routing-masked w2 inputs
s7-inkling2s-b200x4-fa1 · Inkling-2-Small, 4 × B200 · 20 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.1Avalidhttp_step_us_b14,8078,861Baseline: run s6's s0005 source unchanged (LL exchange, slot-invariant FMA)
s00021.0Avalidhttp_step_us_b14,1637,125Team vector phases: a token's mix/norm/quantize on 6 blocks, 1 element per thread
s00031.8Avalidhttp_step_us_b13,9846,537GEMV phases: two strips per block at once, w2 quant per routed pair, no staging for qkv/router
s00042.8Avalidhttp_step_us_b13,8916,335Attention scores as BF16 MMAs; tag-checked hand-offs without fences in the w13 path
s00053.1Avalidhttp_step_us_b13,6465,870Shared memory per block 222 -> 121 KB: the SM's L1 gets the difference
s00063.5Avalidhttp_step_us_b13,6615,742Expert GEMV unroll 5 (NVFP4) / 4 (MXFP8); activation function moved to the w2 quantization block
s00073.8Avalidhttp_step_us_b13,4405,773w13 GEMV for few experts: whole strips per block, four at a time (B=1)
s00084.0Avalidhttp_step_us_b13,4015,758Attention: 32 or 64 positions per chunk block while blocks are free
s00094.2Avalidhttp_step_us_b13,3105,716Output-gate projection computed beside the attention phase by idle blocks
s00104.6Avalidhttp_step_us_b13,3225,660Gate split with 24 free blocks (B=8 up to context 1024); own strip piece from registers
s00114.9Avalidhttp_step_us_b13,3145,599Flat GEMV strip hand-off without a counter: the first-piece warp finishes the strip
s00125.1Avalidhttp_step_us_b13,2995,558Attention: the block of the last chunk combines; no chunk counter, no fence on its path
s00135.8Avalidhttp_step_us_b13,2965,484Attention, routing and w2-quantization phases compiled out of line (register cap)
s00147.8AerrorNVFP4 w13 at batch >= 4: tcgen05 block-scaled MMAs from a second layout
s00158.7Avalidhttp_step_us_b13,3005,552s0014 again: tcgen05 w13 at batch >= 4, race fix and a stall report
s00169.0Avalidhttp_step_us_b13,3005,413Tensor-core kernel: its calls moved out of the kernel body (routing phase)
s001710.4Avalidhttp_step_us_b13,2985,420Progress pointer out of the launch header: plain kernel back at STACK:96
s001810.9Avalidhttp_step_us_b13,2985,440Tensor-core w13: sub-items of 12 units instead of 9
s001911.3Avalidhttp_step_us_b13,3075,390Tensor-core w13: its scale-factor stores spread over three warp groups
s002011.5Avalidhttp_step_us_b13,3075,378Tensor-core kernel: w13 quantization emit inlined in its team phase
s7-inkling2s-b200x4-fb1 · Inkling-2-Small, 4 × B200 · 36 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.1AincorrectInherited engine unchanged (s6 s0003): baseline on this run's node
s00020.2Avalidhttp_step_us_b15,3179,340Inherited engine + explicit FMAs in expert scaling (slot-order invariant)
s00030.4Avalidhttp_step_us_b14,8198,895Kernel of run s6's s0005: tagged-data cross-card exchange, router per strip, attention fixes
s00041.1Avalidhttp_step_us_b14,7948,486Parallel mailbox polls; B=8 w2 prefetch issued at once; B=1 token phases as own functions
s00051.3Avalidhttp_step_us_b14,7978,031B>=2: w2 input as compact per-pair fragments staged in shared memory
s00061.7Avalidhttp_step_us_b14,6827,614w13 and unembedding on block-contiguous strips with shared-memory hand-off; 4 FP4 units in flight at B>=2
s00072.4Avalidhttp_step_us_b14,6887,606Loader only: runtime buffers allocated before the unembedding; unembedding storage chosen at load
s00082.6Bvalidhttp_step_us_b14,6797,596Track B: unembedding stored in an exact 12-bit code (bit-identical values, 25% fewer bytes)
s00093.6Avalidhttp_step_us_b14,5687,263Prefetch issued by idle warps; one cross-card store per destination per strip; batched re-polls at B>=2
s00103.8Avalidhttp_step_us_b14,2467,117NVFP4 input quantization with the hardware E2M1 conversion; w2 inputs one 16-block per thread
s00114.8Avalidhttp_step_us_b14,2397,035B>=2: routing per token block, conv work on idle blocks; top-8 with a shorter dependency chain
s00125.1Avalidhttp_step_us_b14,2357,044Kernel prepared for load-time token tables (switch off): reference for the track-B submissions
s00135.1Bvalidhttp_step_us_b14,2227,037Track B: load-time token tables for layer 0 (normed embedding, layer-0 projections)
s00145.6Bvalidhttp_step_us_b14,1957,004Track B: 12-bit unembedding and layer-0 token tables together
s00155.9Avalidhttp_step_us_b14,2156,899B>=2: MXFP8 expert GEMVs load the next unit before computing the current one
s00166.1Bvalidhttp_step_us_b14,1886,884Track B on the s0015 kernel: 12-bit unembedding and layer-0 token tables
s00176.6Avalidhttp_step_us_b14,1746,901B=1: attention scores on the tensor cores; value sum with 8 dimensions per thread
s00186.8Bvalidhttp_step_us_b14,1516,882Track B on the s0017 kernel: 12-bit unembedding and layer-0 token tables
s00197.3Avalidhttp_step_us_b14,1766,803B>=2: NVFP4 units loaded per group: 6 for w13, 5 for w2 (was 4)
s00208.0Avalidhttp_step_us_b14,0746,540Prefetch policy re-measured per kernel; reader-side fences removed; tensor-core scores at B>=2
s00218.4Bvalidhttp_step_us_b14,0516,513Track B on the s0020 kernel: 12-bit unembedding and layer-0 token tables
s00229.0Avalidhttp_step_us_b14,0146,425Block-per-strip GEMVs run a block's two strips at once on the two halves of its warps
s00239.1Bvalidhttp_step_us_b13,9916,406Track B on the s0022 kernel: 12-bit unembedding and layer-0 token tables
s002411.1Avalidhttp_step_us_b13,9976,459B=1: attention items of 64 positions (twice as many blocks busy, each with half the work)
s002511.2Bvalidhttp_step_us_b13,9756,436Track B on the s0024 kernel: 12-bit unembedding and layer-0 token tables
s002611.3Bvalidhttp_step_us_b13,9756,436Track B, final: s0025 with the MK_NO_RED debugging switch made harmless (host side only)
s002711.5Avalidhttp_step_us_b13,9976,455B>=2 kernel builds its pair map for every batch size (a single sequence no longer hangs it)
s002811.6Bvalidhttp_step_us_b13,9746,433Track B, final: 12-bit unembedding and layer-0 token tables on the s0027 kernel
s002911.8Avalidhttp_step_us_b13,3085,394Sibling run fa1's final kernel (its s0020 source, unchanged) as this run's track-A baseline
s003012.5Bvalidhttp_step_us_b13,3015,376Track B on the sibling's final kernel: 12-bit unembedding + layer-0 token tables; context limit enforced
s003112.8Bvalidhttp_step_us_b13,2915,376Track B: unembedding decoder chosen at build time (s0030's binary carried both)
s003213.0Avalidhttp_step_us_b13,3085,395Track-A twin of s0031: the s0020 kernel with the context guard (same source, TRACK = A)
s003313.7Bvalidhttp_step_us_b13,2905,378Track B: s0031 plus a token-id range check in submit() (same kernel source)
s003414.2Avalidhttp_step_us_b13,3075,394Track A from a binary that serves both tracks (unembedding decoder chosen at load, compiled per kernel)
s003514.2Bvalidhttp_step_us_b13,2955,383Track B from the same binary as s0034: 12-bit unembedding + layer-0 token tables
s003614.4Bvalidhttp_step_us_b13,2915,378Track B: s0033 with the 12-bit round trip asserted at every load (same kernel source)
s8-inkling2s-b200x4-on1 · Inkling-2-Small, 4 × B200 · 43 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.1Aincorrectinherited engine
s00022.0Avalidslo_share_r28.5%pass engine: rows of all requests through one tcgen05 pass, paged KV, prefix cache
s00032.4Avalidslo_share_r286.8%scheduler: 80 ms passes, shortest-remaining-first within the TTFT limit; wider GEMM tiles
s00043.1Avalidslo_share_r288.9%pass length from the decoding requests' gap credit; cheaper decode rows and small passes
s00053.5Avalidslo_share_r292.1%experts move between cards with the traffic; expert outputs summed on one card
s00063.9Avalidslo_share_r292.6%tcgen05 chains: one issuing thread per row group; pass length follows used-up credit
s00074.3Avalidslo_share_r293.7%dense GEMMs: four row groups per block on large passes (one round of block-units)
s00085.1Avalidslo_share_r292.6%no host scheduler: admission, page store, prefix cache and pass composition in a kernel
s00095.3Avalidslo_share_r293.1%expert placement on the device too: swaps decided by the scheduler kernel
s00106.3Avalidslo_share_r295.8%attention on tensor cores: scores, softmax and value product in one block
s00116.8Avalidslo_share_r296.3%GEMM chains: fewer, larger operand copies per stage (dense 4 K steps, experts 2)
s00128.1Avalidslo_share_r297.9%a home card per row: residual, mixes, o_proj and router once, not on every card
s00138.5Avalidslo_share_r298.4%passes of up to 2560 rows (they no longer step up above 2048)
s00149.2Avalidslo_share_r298.4%attention: far keys without bias or mask, fold with 16-byte loads
s00159.3Avalidslo_share_r298.4%submit() and poll() make no call into the cards: prompts are copied in by the scheduler kernel
s00169.9Avalidslo_share_r298.4%small passes: three expert row groups per block when one per block needs more rounds
s001710.1Avalidslo_share_r299.5%GEMM read-out by 16 warps instead of 4
s001810.6Avalidslo_share_r299.5%assignment kernel: ids read 16 bytes at a time, four rows ahead; unused experts skipped
s001910.8Avalidslo_share_r2100.0%largest passes (3584 rows) only while a waiting prompt would miss its first-token time
s002012.5Avalidslo_share_r2100.0%recomputations of evicted blocks counted on the device; source comments brought up to date
s002113.0Avalidslo_share_r2100.0%softmax: first-pass scores kept in tensor memory (tcgen05.st), zeros through exp
s002213.2Avalidslo_share_r2100.0%pressure rule with 0.3 s of slack: the last request of a burst no longer ends at the limit
s002313.8Avalidslo_share_r2100.0%dense GEMMs sum their K range as two halves: on two SMs in small passes
s002414.6Avalidslo_share_r2100.0%scheduler: key table overrun for prompts > 33,279 tokens fixed; submit guards
s002515.5Avalidslo_share_r2100.0%guards: token ring cannot lap a slow host; token ids checked at submit()
s002616.9Avalidslo_share_r2100.0%scheduler kernel: its serial per-pass work by all threads; eviction without scan
s002717.1Avalidslo_share_r2100.0%scheduler: cache table keeps empty slots over hours; a waiting prompt hashed once
s002817.9Avalidslo_share_r2100.0%unembedding in small passes: both K halves in one block, 128-row units
s002919.2Avalidslo_share_r2100.0%small passes: end of a layer fused with the next mix; fetch kernels loop over rows
s003019.7Avalidslo_share_r2100.0%step()/generate() after submit()/poll() on one engine: loop stopped and restarted
s003120.1Avalidslo_share_r2100.0%routing fetch back to one block per row: the row loop cost large passes 1 ms
s003220.6Avalidslo_share_r2100.0%routing fetch: a row's scales, ids, weights, outer scale as one contiguous piece
s003321.4Avalidslo_share_r2100.0%convolution-state fetch: 576 threads per row, two remote loads in series, not six
s003422.7Avalidslo_share_r2100.0%attention-output gather: one launch, 768 threads, one remote load per thread
s003523.2Avalidslo_share_r2100.0%routing fetch: the packed piece read by threads that carry no codes
s003624.3Avalidslo_share_r883.6%audit fixes: slot buffers for 64 slots at any max_batch; cache matches verified
s003724.7Avalidslo_share_r899.0%128 sequences in flight instead of 64: at 8 req/s the slots were the capacity
s003825.1Avalidslo_share_r899.0%admission by priority: on-time and cheapest first, late requests when there is room
s003926.3Avalidslo_share_r899.0%largest passes for a prompt nobody waits behind: 32,000 tokens 808 -> 752 ms
s004026.9Avalidslo_share_r899.0%audit doubts: failed page allocation surfaces as an error; open request ids unique
s004127.4Avalidslo_share_r899.4%page store from the memory really free: 2149 -> 2439 pages
s004228.9Avalidslo_share_r899.7%pressure rule: the largest passes 2 s earlier while 32+ sequences decode
s004330.5Avalidslo_share_r899.7%prompts: promotion ahead of shortest-first when it saves one and costs nobody
s9-qwen38-27b-b200-f1 · Qwen3.8-27B, B200 · 23 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.2Avalidhttp_step_us_b18,8859,836Inherited engine (s5 s0016) with the stats() track label fixed
s00021.0Avalidhttp_step_us_b18,7019,453Weight-stream loads use the cache-streaming operator (ld.global.cs)
s00031.4Avalidhttp_step_us_b18,6949,222B=8 recurrent op in one pass per block (three units, inputs loaded once)
s00041.7Avalidhttp_step_us_b18,6419,145Barrier-time prefetch re-tuned for cs loads: near 160 KB, far 64 KB
s00051.9Avalidhttp_step_us_b18,6289,134Served decode: tokens written by the kernel into host-mapped memory
s00062.2Avalidhttp_step_us_b18,5879,114Prefetch during small ops: consecutive pieces, idle blocks take their whole share
s00072.4Bvalidhttp_step_us_b17,8478,360Track B build of the s0006 kernel: exact 12-bit weight storage
s00083.3Avalidhttp_step_us_b18,4929,077Attention layer as one op: a block per 256-key chunk, sentinel hand-overs
s00093.4Bvalidhttp_step_us_b17,7068,253Track B build of the s0008 kernel: exact 12-bit weight storage
s00104.0Avalidhttp_step_us_b18,4948,982B=8: RMSNorm partials by the four prefetched warps, r finished before the stream ends
s00114.0Bvalidhttp_step_us_b17,7098,184Track B build of the s0010 kernel: exact 12-bit weight storage
s00124.3Avalidhttp_step_us_b18,4928,956B=8 recurrent op: the next state column's loads fly while the current one is computed
s00134.4Bvalidhttp_step_us_b17,7088,166Track B build of the s0012 kernel: exact 12-bit weight storage
s00144.8Avalidhttp_step_us_b18,4928,948B=8: conv-history rows prefetched during in_proj; far prefetch piece 80 KB
s00155.0Bvalidhttp_step_us_b17,7088,193Track B build of the s0014 kernel: exact 12-bit weight storage
s00165.2Avalidhttp_step_us_b18,4948,905Attention blocks ask for their K chunk at the barrier before the op (B >= 4)
s00175.3Bvalidhttp_step_us_b17,7108,131Track B build of the s0016 kernel: exact 12-bit weight storage
s00186.1Avalidhttp_step_us_b18,4668,899Grid barrier: a block's prefetch requests go out after its counter increment
s00196.2Bvalidhttp_step_us_b17,6898,116Track B build of the s0018 kernel: exact 12-bit weight storage
s00206.4Avalidhttp_step_us_b18,4558,880Grid barrier: one prefetch piece per poll, none after the release
s00216.6Bvalidhttp_step_us_b17,6658,102Track B build of the s0020 kernel: exact 12-bit weight storage
s00227.1Avalidhttp_step_us_b18,4458,878Far prefetch piece 128 KB for the BF16 engine (host constant)
s00237.2Bvalidhttp_step_us_b17,6688,114Track B build of the s0022 source (unchanged from s0021: repeat measurement)
s10-q8b-h100-f1 · Qwen3-8B, H100 · 4 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.3Avalidhttp_step_us_b15,3765,634Track A: attention scores stay float32 (no BF16 rounding of q.k)
s00020.7Bvalidhttp_step_us_b14,1364,461Track B build of the s0001 source (12-bit weight code, float32 intermediates)
s00031.5Avalidhttp_step_us_b15,3735,634Track A: final source (test-only switches, trace-build fix), same device code as s0001
s00042.3Avalidhttp_step_us_b15,3655,622Track A: idle blocks pull K of all units into L2 first (1-2 sequences)
s11-q8b-h100-b1 · Qwen3-8B, H100 · 25 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.1Bvalidhttp_step_us_b14,1624,457Inherited engine, track B (s10 s0002 device code), re-measured on this pod
s00020.8Avalidhttp_step_us_b15,3635,684L2 staging of gate+up tiles by idle blocks during attention (track A)
s00030.8Bvalidhttp_step_us_b14,0954,476L2 staging of gate+up tiles by idle blocks during attention (track B)
s00040.9Avalidhttp_step_us_b15,2275,663L2 staging of the next phase's middle tile at each matvec phase end (track A)
s00051.0Bvalidhttp_step_us_b14,0154,477L2 staging of the next phase's middle tile at each matvec phase end (track B)
s00061.1Bvalidhttp_step_us_b13,9984,415Tail L2 staging also with 3-8 sequences (3 x 16 KB per block)
s00071.2Bvalidhttp_step_us_b14,0084,369Layer-0 q/k/v from a per-token table built by the kernel's own q/k/v phase
s00081.3Bvalidhttp_step_us_b13,9524,317Online submit/poll contract: one device-loop launch per cohort, tokens via host-mapped memory
s00091.4Avalidhttp_step_us_b15,1605,588Online submit/poll contract (track A)
s00101.4Avalidhttp_step_us_b15,1625,486Tail L2 staging also with 3-8 sequences (track A)
s00111.5Bvalidhttp_step_us_b13,8954,248Certified int8 sketch of lm_head + exact rescoring on the greedy/online paths
s00121.8Bvalidhttp_step_us_b13,8694,263Token ids outside the vocabulary: no read past the tables, ValueError
s00132.0Bvalidhttp_step_us_b13,8814,247Head sketch: x preparation with 16-byte loads in the raw fragment layout
s00142.7Bvalidhttp_step_us_b13,8654,255No L2 prefetch of the second tile behind a stashed first tile (O2PF 0)
s00153.1AincorrectTrack A re-tuned for the staged engine: 1.5 staged tiles, 4 stash loads in flight, no o prefetch
s00163.2Avalidhttp_step_us_b15,1345,462Track A re-tuned: 1.5 staged tiles, 4 stash loads in flight (fix of s0015)
s00173.6Bvalidhttp_step_us_b13,8374,240Exception MMAs read x from shared memory instead of L2 (from a phase's 2nd tile)
s00183.9Bvalidhttp_step_us_b13,8254,258Exception x from shared memory only in the greedy kernel (fixes s0017's step() spill)
s00196.2Bvalidhttp_step_us_b13,8434,257Tail staging after o targets the block's 3rd gate+up tile instead of the middle one
s00206.8Bvalidhttp_step_us_b13,8134,253Attention window: shorter score-stamp wait, more polite o-stash loads (track B re-tune)
s00219.6Bvalidhttp_step_us_b13,8224,241Head sketch: rescoring of candidate tiles spread over all blocks (3+ sequences)
s002212.4Bvalidhttp_step_us_b13,8134,2313+ sequences: stage gate+up into L2 after the attention blocks' score pass
s002312.6Bvalidhttp_step_us_b13,8154,2213+ sequences: idle blocks' o-tile prefetch deferred until the score pass is done
s002412.9Bvalidhttp_step_us_b13,8174,2113+ sequences, long contexts: idle blocks pull V (only) into L2 during attention
s002513.2Avalidhttp_step_us_b15,1345,485Track A: idle blocks pull V into L2 during attention beyond the touch limit
s11-north-h100-b1 · North Mini Code 1.0, H100 · 26 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.7Bvalidhttp_step_us_b12,1235,769Track B made bit-identical: W12 escapes patched into mma operands, L0 tables by kernel
s00022.5Avalidhttp_step_us_b12,5406,901Track A rebuild of the current source (bit-identical partner of track B)
s00032.7Bvalidhttp_step_us_b12,1085,769Batch-1 kernel bulk-prefetches its next-layer stage-A tiles into L2 (W12 only)
s00043.1Avalidhttp_step_us_b12,5356,823Pad 8-column operand strides (bank conflicts) - track A
s00053.3Bvalidhttp_step_us_b12,0995,646Pad 8-column operand strides (bank conflicts) - track B
s00065.0Avalidhttp_step_us_b12,5246,745Attention runs two chunks at a time (8 warps each) - track A
s00075.1Bvalidhttp_step_us_b12,1035,612Attention runs two chunks at a time (8 warps each) - track B
s00086.2Avalidhttp_step_us_b12,5346,753Share model: attention cost per K tile 12 at batch > 1 - track A
s00096.3Bvalidhttp_step_us_b12,1095,587Share model: attention cost per K tile 12 at batch > 1 - track B
s00106.7Avalidhttp_step_us_b12,5266,688Batch>1 kernels: 8 head loads in flight per thread - track A
s00116.8Bvalidhttp_step_us_b12,0995,533Batch>1 kernels: 8 head-sketch loads in flight per thread - track B
s00127.2Avalidhttp_step_us_b12,5266,569Batch>1 kernels launch with 16 KB less shared memory (larger L1) - track A
s00137.3Bvalidhttp_step_us_b12,1075,437Batch>1 kernels launch with 16 KB less shared memory (larger L1) - track B
s00147.6Avalidhttp_step_us_b12,5176,567Gate|up share table cached across steps of a launch - track A
s00157.7Bvalidhttp_step_us_b12,1045,408Gate|up share table cached across steps of a launch - track B
s00167.8Avalidhttp_step_us_b12,5176,526RMSNorm loads of the residual bypass L1 - track A
s00177.9Bvalidhttp_step_us_b12,1015,392RMSNorm loads of the residual bypass L1 - track B
s00188.4Bvalidhttp_step_us_b12,0765,384Batch-1 L2 prefetch limited to the half tile and the first whole stage-A tile
s00198.7Bvalidhttp_step_us_b12,0515,385Batch-1: down tiles of the first two experts to L2 while waiting at barrier B
s00209.1Avalidhttp_step_us_b12,5066,525Batch-1: down tiles of the first two experts to L2 at barrier B - track A
s00219.2Bvalidhttp_step_us_b12,0445,392Batch-1: previous token's experts' gate|up tiles to L2 during the union wait
s002210.0Bvalidhttp_step_us_b12,0405,391Barrier-wait L2 prefetches (stage-A tiles, first down tiles) also at batch > 1
s002310.7Bvalidhttp_step_us_b12,0455,320Greedy head sketch skips the 439 tiles made only of copies of earlier rows
s002411.9Avalidhttp_step_us_b12,5016,493Track A rebuild of the final source (partner of track B s0023)
s002512.9Bvalidhttp_step_us_b12,0505,336Final source, track B (no functional change from s0023)
s002614.6Bvalidhttp_step_us_b12,0385,334Batch-1: barrier-B poll with the last o_proj quantum, attention delay 30 in the share model
s11-flint-b200-b1 · flint, B200 · 27 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.9Avalidhttp_step_us_b11,4883,426Track A re-measure: inherited s0077 kernel, track-B code compiled out
s00020.9Bvalidhttp_step_us_b11,4733,438Track B: certified int8 sketch of lm_head + exact rescoring on the greedy paths
s00031.9Avalidhttp_step_us_b11,4773,417Track A: attention scores as K x q^T, 8 mma per key group instead of 16
s00042.3Bvalidhttp_step_us_b11,4713,437Track B on s0003: int8 head sketch on the greedy paths (scores code per track)
s00052.7Bvalidhttp_step_us_b11,4723,428Track B: head sketch in the batch 1-2 kernel only; batch>2 kernel = track A's
s00063.2Bvalidhttp_step_us_b11,4723,456Track B final: s0005 kernel + submit() refuses out-of-range token ids
s00074.6Avalidhttp_step_us_b11,4633,422Track A: L2 prefetch of next q/k/v and shared expert from an idle warp
s00085.9Avalidhttp_step_us_b11,4523,426Track A: q/k/v prefetch at the end of C, shared expert at the start of A
s00097.7Avalidhttp_step_us_b11,4223,427Track A: 512-byte device allocation before the engine's own (buffer layout)
s00108.5Avalidhttp_step_us_b11,4193,427Track A re-measure of s0009 (identical source): layout fix reproducibility
s001110.0Avalidhttp_step_us_b11,4303,443Track A trade-off: 4 accumulators in phase D (B=8 -57 us, B=1 +11 us)
s001210.4Avalidhttp_step_us_b11,4193,425Track A: two K batches in flight for attention pairs on main blocks (batch 8)
s001311.1Avalidhttp_step_us_b11,4173,356Track A: L2 evict-first hint on the expert and head tile loads
s001411.6Avalidhttp_step_us_b11,4123,373Track A: evict-first expert/head loads without L1::no_allocate
s001512.0Avalidhttp_step_us_b11,4083,368Track A: last layer's idle q/k/v prefetch slot fetches the head's first 20 MB
s001612.8Bvalidhttp_step_us_b11,4003,362Track B: int8 head sketch on the s0015 kernel, with the 8-mma scores form
s001714.1Avalidhttp_step_us_b11,4183,379Track A: dense-layer MLP tiles preloaded before their barriers
s001814.2Bvalidhttp_step_us_b11,3803,367Track B: s0017's dense-layer MLP preloads with the int8 head sketch
s001914.8Avalidhttp_step_us_b11,4023,367Track A: dense-layer MLP epilogue: plain stores + one fence, down preload after
s002014.8Bvalidhttp_step_us_b11,3743,350Track B: s0019's dense-layer epilogue with the int8 head sketch
s002115.5Avalidhttp_step_us_b11,3983,361Track A: dense-layer gate/up loop unrolled by 16 (more tiles in flight)
s002215.6Bvalidhttp_step_us_b11,3723,353Track B: s0021's dense-layer gate/up unroll with the int8 head sketch
s002316.0Bvalidhttp_step_us_b11,3683,352Track B: head sketch pass 2 staged through shared memory (one load round)
s002416.9Avalidhttp_step_us_b11,4003,355Track A: chosen tokens published by atomic exchange, bump before the host word
s002516.9Bvalidhttp_step_us_b11,3613,332Track B: s0024's token publish with the int8 head sketch
s002619.5Avalidhttp_step_us_b11,3933,339Track A: drop a never-true clause that kept a dead branch in the batch-1 kernel
s002719.6Bvalidhttp_step_us_b11,3533,333Track B: s0026's dead-branch removal with the int8 head sketch
s11-q27-b200-b1 · Qwen3.8-27B, B200 · 29 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.7Bvalidhttp_step_us_b17,5578,003W12 rows stored scaled by exact powers of two: one decode base, factor in epilogue
s00020.9Bvalidhttp_step_us_b17,5427,969Layer-0 in_proj output from a per-token table built by the kernel itself
s00031.4Bvalidhttp_step_us_b17,5257,950Greedy passes: certified int8 sketch of lm_head + exact rescoring of candidate tiles
s00041.8Bvalidhttp_step_us_b17,4957,936Head sketch in its own launch after the main kernel; x statistics in the sketch
s00052.5Bvalidhttp_step_us_b17,3377,763W12 stream: next escape position kept in a register, no shuffle per iteration
s00064.9Bvalidhttp_step_us_b17,3277,763W12 build: epilogue norm gains and row factors copied to shared memory at op start
s00075.0Bvalidhttp_step_us_b17,3047,741No BF16 twins of out_proj/o_proj: W12 at every batch size, BF16 loops not compiled
s00085.3Bvalidhttp_step_us_b17,3137,738Head sketch: per-block tile counts balanced by measured block speed at load
s00095.6Bvalidhttp_step_us_b17,2917,712Head sketch in 7 bits per weight (1.11 GB instead of 1.27 GB), still certified exact
s00106.2Bvalidhttp_step_us_b17,2737,683Barrier L2 prefetch of the next GEMM's first piece per block: 144 KB -> 176 KB
s00117.3Bvalidhttp_step_us_b17,2757,697Head kernel: argmax op folded into the finish op (one grid barrier less)
s00127.5Bvalidhttp_step_us_b17,2667,682Head kernel: exact rescoring chain relayed in 40 turns of 8 k-steps (was 64 of 5)
s00138.2Bvalidhttp_step_us_b17,2637,675Head sketch: power-of-two x scale, digits by exact multiply instead of division
s00149.1Bvalidhttp_step_us_b17,2617,674B<=3 attention barrier prefetch: 256 KB of o_proj + the K chunk (12-bit build)
s00159.6Bvalidhttp_step_us_b17,2587,688B<=3: attention barrier asks for 20 K chunks per unit (was 5), 12-bit build
s001610.5Bvalidhttp_step_us_b17,2497,682B<=3 prefetch table retune: no K past chunk 5, whole o_proj share, 40 idle blocks
s001713.3Bvalidhttp_step_us_b17,2447,684Head sketch: 94 % of tiles in 6 bits, frequent candidates stay 7-bit (0.96 GB)
s001813.7Bvalidhttp_step_us_b17,2397,677Head sketch at B>1: all-7-bit copy through the plain kernel instance (B=1 keeps 6/7)
s001914.6Bvalidhttp_step_us_b17,2457,680Mixed head sketch: per-block tile counts calibrated at load again (B=1)
s002015.2Bvalidhttp_step_us_b17,2387,679B<=3 near prefetch per matrix: gate_up 160 KB, down 192 KB (others 176)
s002115.9Bvalidhttp_step_us_b17,2427,679Confirmation: s0020 configuration with the later experiments compiled out (no change)
s002216.7Bvalidhttp_step_us_b17,2597,708B=1 head kernel: last rescoring block finishes the pass (two grid barriers less)
s002316.8Bvalidhttp_step_us_b17,2617,711Final state: s0020 configuration (SK_LAST and other experiments compiled out)
s002417.8Bvalidhttp_step_us_b17,2527,705B=1 head sketch: frequent-candidate tiles in 8 bits instead of 7 (6 % of tiles)
s002518.2Bvalidhttp_step_us_b17,2557,707Head sketch: the 6/8-bit mixed sketch for every batch size (no all-7-bit copy)
s002619.0Bvalidhttp_step_us_b17,2507,692Head sketch prologue: x read as aligned u32 pairs instead of u16 scalars
s002722.0Bvalidhttp_step_us_b17,2217,698Recurrent op re-reads B and spec flag from shared memory (no spill reloads)
s002825.0Bvalidhttp_step_us_b17,1757,681ptxas register-usage level 10 for the track-B build (same source)
s002926.0Bvalidhttp_step_us_b17,1707,652ptxas -disable-optimizer-consts added to the track-B build (same source)
s12-oss20-h100-a1 · gpt-oss-20b, H100 · 50 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.6Avalidhttp_step_us_b12,3145,745Port of the gpt-oss-120b B200 megakernel to 20b/H100 (sm_90 e2m1 decode)
s00021.0Avalidhttp_step_us_b12,2754,284MoE as whole-unit warp streams + last-arriver down combine + cheaper e2m1 decode
s00031.7Avalidhttp_step_us_b12,2753,853Faster norms (one warp per row, loads in flight) and down combine (4 slots/sequence)
s00042.0Avalidhttp_step_us_b12,2543,850Router on 45 blocks: K-slice partials summed by every block, top-4 in smem
s00052.1Bvalidhttp_step_us_b12,0613,671Track B: certified int8 head sketch on the greedy path, exact recompute of candidates
s00062.7Bvalidhttp_step_us_b12,0373,641K/V cache in MMA fragment order: attention reads with 16-byte loads
s00073.2Avalidhttp_step_us_b12,2373,819Track A: K/V cache in MMA fragment order (16-byte loads in attention)
s00083.3Bvalidhttp_step_us_b12,0263,599Release atomics in barrier/counters, warp-parallel greedy token choice
s00093.4Bvalidhttp_step_us_b11,9923,457Routing block in registers: per-warp logits+top-4, warp-0 union/slots (2 syncs, was 5)
s00103.5Bvalidhttp_step_us_b11,9353,411Attention as in-block flash decoding: online softmax per warp, 2 syncs (was 5)
s00113.7Avalidhttp_step_us_b12,0723,509Track A catch-up: routing in registers, flash attention, release atomics, sync trims
s00123.9Bvalidhttp_step_us_b11,8963,345Routing tail over 3 warps, 5-stage qkv/o segments, sketch loads in flight 15
s00134.3Bvalidhttp_step_us_b11,8633,333Track B: 6-bit certified head sketch (434 MB instead of 579)
s00144.4Avalidhttp_step_us_b12,0783,501Track A: stage-aligned qkv/o segments, routing tail over 3 warps
s00154.9Bvalidhttp_step_us_b11,8133,328Track B: router weights folded with ln2; router partials in o_proj epilogue, no phase D
s00165.4Bvalidhttp_step_us_b11,8023,343Track B: L2 prefetch of each attention block's K/V window at the start of phase A
s00175.6Bvalidhttp_step_us_b11,7773,353Track B: norms take sums of squares from partials of the producing phase (one pass)
s00185.9Bvalidhttp_step_us_b11,7663,354qkv epilogue bias and RoPE inputs loaded between the stream and the block barrier
s00196.3Avalidhttp_step_us_b12,0633,520Track A catch-up: K/V window L2 prefetch, qkv epilogue inputs before the block barrier
s00206.6Bvalidhttp_step_us_b11,7523,331E2M1 decode by masked integer multiply (same bits, ~30% fewer decode instructions)
s00217.0Bvalidhttp_step_us_b11,7343,322Track B: single-pass norms split over 4 warps per row at B<=2
s00227.1Bvalidhttp_step_us_b11,7213,331Prefetch of cold epilogue inputs: folded router rows (L1) and down biases (L2)
s00237.2Bvalidhttp_step_us_b11,7133,315L2 prefetch of the next layer's norm weights, biases and sinks during the down phase
s00247.4Avalidhttp_step_us_b12,0353,480Track A catch-up: masked-multiply decode, L2 prefetch of small vectors and down biases
s00257.5Avalidhttp_step_us_b11,9523,487Track A: single-pass norms with sums of squares accumulated where rows are produced
s00267.9Bvalidhttp_step_us_b11,7043,308Consume loop counter in a register at RS=2 (no local-memory round trip per stage)
s00278.2Bvalidhttp_step_us_b11,6993,3054-warp norm loads its norm weights together with the row
s00288.5Bvalidhttp_step_us_b11,6923,283MXFP4 items stored codes+scales contiguous: one bulk copy per ring stage
s00298.6Avalidhttp_step_us_b11,9243,444Track A catch-up: fused MXFP4 items, register loop counter, 4-warp norm weight loads
s00308.9Bvalidhttp_step_us_b11,6803,274Earlier o_proj ring prefetch; sketch loads with L2 evict_first
s003110.2Bvalidhttp_step_us_b11,6803,250gate_up units dealt round-robin over blocks (B=8 balance)
s003210.4Bvalidhttp_step_us_b11,6783,255Grid barrier: acq_rel atomic add; the last arriving block skips polling
s003310.5Avalidhttp_step_us_b11,8963,393Track A catch-up: acq_rel grid barrier, round-robin gate_up units
s003411.4Bvalidhttp_step_us_b11,6683,406B=1-specialised kernel build per cohort; down units split in K halves (track B)
s003511.7Bvalidhttp_step_us_b11,6533,405B=1 build: contiguous gate_up unit ranges (round-robin only in the general build)
s003612.0Avalidhttp_step_us_b11,8773,394Track A: batch-size-1 specialised kernel build per cohort, contiguous gate_up ranges there
s003712.2Bvalidhttp_step_us_b11,6513,408B=1 build: sketch loads 10 k-steps per batch, no down-bias prefetch
s003813.0Bvalidhttp_step_us_b11,6473,405B=1 build: down units half-major (both K halves of a rowtile in different blocks' ranges)
s003913.4Bvalidhttp_step_us_b11,6393,396Head recompute: candidate tile loaded by one 92 KB bulk copy (TMA) instead of a load loop
s004013.7Bvalidhttp_step_us_b11,6393,386Head recompute copy in 6 chunks so the exact chain overlaps it (general build, B=8)
s004113.8Avalidhttp_step_us_b11,8783,394Track A catch-up: B=1 build with half-major down units and no down-bias prefetch
s004214.3Bvalidhttp_step_us_b11,6343,405Head sketch: 5-bit codes with per-32-column u8 group scales (381 MB instead of 434)
s004314.4Bvalidhttp_step_us_b11,6343,396Keep the 6-bit head sketch for B>1 cohorts; 5-bit group-scaled sketch only at B=1
s004414.7Bvalidhttp_step_us_b11,6343,392B=1 build: L2 prefetch of each warp's first sketch batch at the end of the last layer
s004515.3Bvalidhttp_step_us_b11,6293,396B=1 build: double-buffered sketch loads in head pass 1 (next batch in flight)
s004615.9Bvalidhttp_step_us_b11,6253,396B=1 build: router top-4 by rank counting (independent shuffles, same selection)
s004716.7Avalidhttp_step_us_b11,8713,394Track A: router top-4 by rank counting in the batch-size-1 build
s004817.0Bbuild_failedB=1 build: router bias loaded at the start of the routing warp
s004917.1Bvalidhttp_step_us_b11,6173,399B=1 build: router bias loaded at the start of the routing warp (resubmit, fixed build)
s005017.3Bvalidhttp_step_us_b11,6133,388B=1 build: gate_up bias line prefetched into L1 when each unit's task is built
s12-q32-b200-a1 · Qwen3-32B, B200 · 47 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.5Avalidhttp_step_us_b111,48811,763Own persistent megakernel v1: split-K GEMM ops, tensor-core GQA attention
s00022.2Avalidhttp_step_us_b110,63111,480512-thread blocks (128 regs), split attention, batch-class norm prep
s00033.1Avalidhttp_step_us_b110,58611,290Attention waits on per-group q/k/v tile counters; o waits on unit counter
s00043.5Bvalidhttp_step_us_b19,62510,407Track B: exact 12-bit weight storage, bit-identical to track A
s00053.7Avalidhttp_step_us_b110,48911,175Track A: work split in quads of 4 k-steps (same source as s0004)
s00063.8Avalidhttp_step_us_b110,52210,994Track A: RMSNorm applied per k-step in the stream at B>=3 (no norm prep)
s00073.9Avalidhttp_step_us_b110,33410,833Cut tiles handed over with a sentinel slot instead of an atomic counter
s00084.0Bvalidhttp_step_us_b19,46710,277Track B build of s0007's source (12-bit storage + sentinel cut tiles)
s00094.5Avalidhttp_step_us_b110,32310,768B>=7 attention combine through DSMEM (clusters of 2); separate small/large-batch kernels
s00105.2Bvalidhttp_step_us_b19,44410,356Track B build of s0009's source (12-bit storage, DSMEM combine, two kernels)
s00115.6Bvalidhttp_step_us_b19,45310,265Track B: small-batch kernel instance at all batch sizes (B=8 regression of s0010)
s00126.3Bvalidhttp_step_us_b19,40210,049Track B: escape patches through shared memory; last-warp prefetch issue
s00136.7Avalidhttp_step_us_b110,35110,773Track A: three kernel instances (1-2, 3-6, 7-8 sequences) and per-warp argmax keys
s00147.3Avalidhttp_step_us_b110,35410,766Track A final source: cohort-tagged served tokens, 20 ms gather window
s00157.4Bvalidhttp_step_us_b19,40210,071Track B final source: cohort-tagged served tokens, 20 ms gather window
s00168.6Avalidhttp_step_us_b110,29910,763K/V of cached positions prefetched into L2 before the q/k/v wait (B<=2)
s00179.0Avalidhttp_step_us_b110,30410,742B=8 attention in fewer rounds: one warp per part softmax, 3 K blocks per warp
s001810.3Avalidhttp_step_us_b110,34410,715Attention combine folds two halves in parallel (cluster blocks, same split globally)
s001910.6Avalidhttp_step_us_b110,23010,718Split grid barrier: first weight batch requested before the wait (1-2 sequences)
s002010.7Bvalidhttp_step_us_b19,3019,943Track B of s0019's source (attention rounds, two-half combine, split barrier)
s002111.3Avalidhttp_step_us_b110,28410,539Split grid barrier in every kernel instance (wait state kept in shared memory)
s002211.8Bvalidhttp_step_us_b19,3369,918Track B of s0021's source; 7-8 sequences on the cluster-combine kernel instance
s002312.3Avalidhttp_step_us_b110,26510,535First weight batch also requested across the RMSNorm prep (1-2 sequences)
s002413.3Bvalidhttp_step_us_b19,2509,780Track B: 3 quads in flight per warp in the norm GEMMs (inputs from smem just in time)
s002513.7Bvalidhttp_step_us_b19,1979,840Track B, B=1: o and down read their input column from smem, 3 quads in flight
s002614.2Bvalidhttp_step_us_b19,1569,714RMSNorm prep with packed bf16 arithmetic (same bits, half the instructions)
s002714.5Avalidhttp_step_us_b110,24910,537Track A build of s0026's source (packed bf16 norm prep)
s002815.7Avalidhttp_step_us_b110,14110,533Attention combine split over all blocks of a unit (1-6 sequences)
s002915.8Bvalidhttp_step_us_b19,0639,703Track B of s0028's source (attention combine split over a unit's blocks)
s003016.4Avalidhttp_step_us_b110,13110,528Attention prep constants (norm weights, cos, sin) requested before the q/k/v wait
s003116.5Bvalidhttp_step_us_b19,0499,683Track B of the attention-prep source (s0030)
s003217.2Bvalidhttp_step_us_b19,0359,495Track B, 7-8 sequences: norm prep split over the cluster pair via DSMEM
s003317.5Avalidhttp_step_us_b110,12110,440Track A of s0032's source; 7-8 instance's unused norm-prep path instantiated with the cluster prep
s003418.8Bvalidhttp_step_us_b19,0169,433Track B: a quad's escapes patched by their lanes in parallel
s003521.0Bvalidhttp_step_us_b19,0259,449Track B: at most 10 blocks per attention unit at B=1
s003621.6Bvalidhttp_step_us_b18,9959,439Track B: idle blocks prefetch 128 KB of o during attention (runtime default)
s003724.5Avalidhttp_step_us_b110,08310,404Track A: next op's first-bytes prefetch issued by the last warp, not warp 0
s003824.8Bvalidhttp_step_us_b18,9569,390GEMM block ranges and prefetch plans with 32-bit instead of 64-bit divisions
s003925.0Avalidhttp_step_us_b110,06110,407Track A of s0038's source (32-bit range divisions)
s004025.4Avalidhttp_step_us_b110,01210,410Attention combine arrival: release + poll a cumulative per-launch count (no atomic/flag)
s004125.5Bvalidhttp_step_us_b18,9489,404Track B of s0040's source (counter-based combine arrival)
s004228.3Avalidhttp_step_us_b19,98110,408Track A: norm weights staged into shared memory during the barrier wait (1-4 seq)
s004328.4Bvalidhttp_step_us_b18,9379,383Track B of s0042's source (track-B SASS identical to s0041)
s004429.0Avalidhttp_step_us_b19,97410,276Track A: per-k-step norm weights pre-ordered on the host, staged during barrier wait
s004529.1Bvalidhttp_step_us_b18,9389,373Track B of s0044's source (track-B SASS identical to s0041)
s004629.4Bvalidhttp_step_us_b18,9119,332Track B: row factors copied with cp.async (no L2 stall of warps 0-1 at stream start)
s004729.6Bvalidhttp_step_us_b18,7929,238Track B: escape-list start and first window loaded independently before the wait
s12-oss120-b200-a1 · gpt-oss-120b, B200 · 59 submissions
SubmissionAgent-hoursTrackStatusMetricHeadline8 seq., µsTitle
s00010.5Avalidhttp_step_us_b13,6138,775v1 megakernel: one cooperative kernel, grid barrier between 6 phases per layer
s00021.0Abuild_failedTMA per-warp weight rings, cross-phase prefetch, fragment-order B operands
s00031.0AerrorTMA per-warp weight rings, cross-phase prefetch, fragment-order B operands
s00041.1Avalidhttp_step_us_b12,5236,735TMA weight rings + prefetch (s0003 fixed: attention P built from scores, no smem overlap)
s00051.2Avalidhttp_step_us_b12,2766,738Down phase at B<=2: even (expert, rowtile) units, partials summed by the next phase
s00061.7Avalidhttp_step_us_b12,1616,872MXFP4 dequant by scaled e2m1 cvt (exact); norm weights bulk-copied into smem early
s00072.0Avalidhttp_step_us_b12,0796,640Attention split into 64-key chunks, one warp each, last warp merges (flash decoding)
s00082.2Avalidhttp_step_us_b12,0994,646B>=3: expert phases without reduction rounds (warp per unit), even down units + combine step
s00092.4Avalidhttp_step_us_b12,0714,325MXFP4 records interleave codes and scales (one bulk copy per stage); batched norm loads
s00102.6Avalidhttp_step_us_b12,0354,279Router split 4 ways in K over 32 blocks; E sums the partials
s00112.7Avalidhttp_step_us_b11,9974,144Top-4 by one warp integer max per round; expert union in warp 0 (match/reduce)
s00123.0Avalidhttp_step_us_b11,9594,017E->F grid barrier replaced by per-expert-slot completion counters
s00133.2Avalidhttp_step_us_b11,9513,953Two kernel instances (B<=2 / B>=3); wide instance norms one warp per column
s00143.9Avalidhttp_step_us_b11,9273,824Drop the per-lane B-column selects (absent columns only feed discarded outputs); batched o_proj input copy
s00154.0Avalidhttp_step_us_b11,9063,756Two mma accumulators per ring stage (even/odd items), fp4 and bf16
s00164.8Avalidhttp_step_us_b11,8403,708K/V cache in mma fragment order (16 B loads); attention partials handed over with one acq_rel atomic
s00175.0Avalidhttp_step_us_b11,8583,545Expert bias rows prefetched when a unit starts (B>=3 epilogues no longer stall the stream)
s00186.7Bvalidhttp_step_us_b11,7993,500Track B: certified int8 head sketch + exact rescoring of candidate rows (B<=2 greedy)
s00197.4Bvalidhttp_step_us_b11,8033,459Track B: head sketch in the B>=3 instance too (inlined); same tokens
s00207.9Bvalidhttp_step_us_b11,7983,413Track B: B>=3 staging by bulk copies (attention output, act rows in fragment order)
s00218.1Bvalidhttp_step_us_b11,7743,403Track B: B<=2 instance stages attention output and act rows by bulk copies too
s00228.3Avalidhttp_step_us_b11,8313,391Track A: current engine without the head sketch (bulk-copy staging, batched act loads)
s00238.7Bvalidhttp_step_us_b11,7393,323Track B: o_proj K split 16 -> 4 tasks per rowtile (fewer, longer ring tasks)
s00248.9Bvalidhttp_step_us_b11,7393,219Track B: ring tasks on whole stages (o_proj segments 63/63/63/67, router 5x9 ksteps)
s00259.3Bvalidhttp_step_us_b11,7373,237Track B: B<=2 attention loads V by cp.async with K
s00269.4Avalidhttp_step_us_b11,8103,228Track A: current engine without the head sketch (whole-stage tasks, bulk-copy staging)
s00279.8Bvalidhttp_step_us_b11,7303,200Track B: head candidates rescored with 20 stages in flight; B>=3 down tail units as segments
s002810.2Bvalidhttp_step_us_b11,7313,174Track B: short attention units per block with smem merge; B>=3 gate_up tail units as segments
s002910.4Avalidhttp_step_us_b11,7993,174Track A: current engine without the head sketch (attention short units per block, tail segments)
s003010.6Bvalidhttp_step_us_b11,7193,177Track B: head candidates rescored in parallel (pooled ring slots), no L2 prefetch of local candidates
s003110.9Bvalidhttp_step_us_b11,7153,144Track B: head sketch in the B>=3 instance via an out-of-line call; warp-parallel token pick
s003211.2Bvalidhttp_step_us_b11,7203,139Track B: sketch's global bound read in parallel per column
s003311.4Avalidhttp_step_us_b11,7963,164Track A: current engine without the head sketch (since s0029: warp-parallel token pick)
s003412.4Bvalidhttp_step_us_b11,7223,125Track B: wide head rescoring copies the prep's normed rows instead of re-norming
s003513.6Bvalidhttp_step_us_b11,7223,115Track B: down-phase tail units back to whole units (FTAIL 0)
s003613.6Bvalidhttp_step_us_b11,7223,105Track B: B>=3 attention units up to 16 chunks on one block (2 chunks per warp)
s003713.8Avalidhttp_step_us_b11,7993,135Track A: s0036's engine without the head sketch
s003814.1Bvalidhttp_step_us_b11,7223,083Track B: wide norm loads each weight word one iteration ahead (NORM_PIPE1)
s003914.7Avalidhttp_step_us_b11,7983,117Track A: B>=3 instance prefetches 3 ring stages across barriers (was 2)
s004015.2Bvalidhttp_step_us_b11,7193,083Track B: B<=2 attention loads V after the scores again (V_ASYNC_FLAT 0)
s004115.6Bvalidhttp_step_us_b11,7033,066Track B: phase A pulls each block's attention K/V chunks into L2
s004215.8Bvalidhttp_step_us_b11,7003,064Track B: phase A also pulls the layer's biases and sinks into L2
s004315.9Avalidhttp_step_us_b11,7873,114Track A: B<=2 attention loads V after the scores (as track B since s0040)
s004417.0Bvalidhttp_step_us_b11,7063,054Track B: gate_up tail units back to whole units (ETAIL 0)
s004517.9Bvalidhttp_step_us_b11,6933,026Track B: wide head re-norms its rows for the rescoring again (HSK_XG 0)
s004618.2Bvalidhttp_step_us_b11,6933,017Track B: wide head rescores candidate tiles with pooled ring slots (FASTRS_W 1)
s004720.1Bvalidhttp_step_us_b11,6913,021Track B: B<=2 qkv on 2 rowtiles per block; extra V rowtiles as spread segments
s004821.6Avalidhttp_step_us_b11,7753,101Track A: exact head's last rowtiles split over warps (HTAIL) + wide norm pipelining
s004922.0Avalidhttp_step_us_b11,7743,074Track A: K/V chunks prefetched into L2 in the B>=3 instance (KVPF)
s005022.7Avalidhttp_step_us_b11,7763,061Track A: head tail rowtiles in 2 K halves instead of 4 quarters (HTS 2)
s005123.6Bvalidhttp_step_us_b11,6873,029Track B: final engine as committed (s0047's kernel; track A code compiled out)
s005225.0Bvalidhttp_step_us_b11,6903,019Track B: grid barrier arrives by red.release and polls with a 40 ns back-off
s005325.1Avalidhttp_step_us_b11,7713,075Track A: grid barrier arrives by red.release and polls with a 40 ns back-off
s005426.2Bvalidhttp_step_us_b11,6903,007Track B: B>=3 head rescoring splits candidate tiles into 4 K segments over warps
s005526.5Bvalidhttp_step_us_b11,6903,000Track B: wide head copies the published normed rows by bulk copy (HSK_XG 2)
s005626.9Bvalidhttp_step_us_b11,6902,991Track B: wide head sketch prep fuses the norm and digit passes (HSK_PREP2)
s005727.4Avalidhttp_step_us_b11,7603,067Track A: exact head tail claimed dynamically (HDYN) + re-swept KVPF_F/QXO
s005827.8Bvalidhttp_step_us_b11,6923,010Track B: head sketch tail tiles claimed dynamically by free warps (SDYN)
s005929.9Avalidhttp_step_us_b11,7383,076Track A: B<=2 instance prefetches the layer's small parameters into L2 (SPF_F)

Versions and commands of the same-GPU jobs

As recorded by each job: the GPU, the versions inside each system's environment, and the command each server was started with (model path, host and port removed).

rel-q8-head-k3
Record
gpu NVIDIA H100 80GB HBM3, 81559 MiB, 595.71.05, 1980 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours q8A4-s0016 sha256=621e47c041a73f2f
ours q8B4-s0020 sha256=b76d9f48be969976
sglang=0.5.18 torch=2.13.0+cu130 cuda=13.0 arch=x86_64 gpu=NVIDIA H100 80GB HBM3
vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=x86_64 gpu=NVIDIA H100 80GB HBM3
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q8A4-s0016
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q8B4-s0020
sglang.launch_server --model-path Qwen/Qwen3-8B --dtype bfloat16 --enable-torch-compile
vllm serve Qwen/Qwen3-8B --dtype bfloat16 --gpu-memory-utilization 0.90
rel-q8-head-f
Record
gpu NVIDIA H100 80GB HBM3, 81559 MiB, 595.71.05, 1980 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours q8A2-s0003 sha256=5d107a61cec081ff
ours q8B2-s0002 sha256=d59994213a77d62e
sglang=0.5.18 torch=2.13.0+cu130 cuda=13.0 arch=x86_64 gpu=NVIDIA H100 80GB HBM3
vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=x86_64 gpu=NVIDIA H100 80GB HBM3
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q8A2-s0003
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q8B2-s0002
sglang.launch_server --model-path Qwen/Qwen3-8B --dtype bfloat16 --enable-torch-compile
vllm serve Qwen/Qwen3-8B --dtype bfloat16 --gpu-memory-utilization 0.90
rel-q8-axes-1
Record
gpu NVIDIA H100 80GB HBM3, 81559 MiB, 595.71.05, 1980 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours q8onA-s0038 sha256=a6316f60bec0cc7c
ours q8onB-s0039 sha256=40e46bf2c02cc6e8
sglang=0.5.18 torch=2.13.0+cu130 cuda=13.0 arch=x86_64 gpu=NVIDIA H100 80GB HBM3
vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=x86_64 gpu=NVIDIA H100 80GB HBM3
serve_engine.py engine --max-batch 64 --max-context 34816 + frontend; source q8onA-s0038
serve_engine.py engine --max-batch 64 --max-context 34816 + frontend; source q8onB-s0039
sglang.launch_server --model-path Qwen/Qwen3-8B --dtype bfloat16 --enable-torch-compile
vllm serve Qwen/Qwen3-8B --dtype bfloat16 --gpu-memory-utilization 0.90
rel-north-axes-1
Record
gpu NVIDIA H100 80GB HBM3, 81559 MiB, 595.71.05, 1980 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours northA-s0022 sha256=c79cf7faa931872d
vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=x86_64 gpu=NVIDIA H100 80GB HBM3
serve_engine.py engine --max-batch 8 --max-context 18432 + frontend; source northA-s0022
vllm serve CohereLabs/North-Mini-Code-1.0 --dtype bfloat16 --gpu-memory-utilization 0.90 --max-model-len 32768
rel-oss20-head-k3
Record
gpu NVIDIA H100 80GB HBM3, 81559 MiB, 595.71.05, 1980 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours oss20A4-s0047 sha256=dd03ba032f05b691
ours oss20B4-s0050 sha256=47936dfcd785078d
vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=x86_64 gpu=NVIDIA H100 80GB HBM3
sglang=0.5.23 torch=2.14.1+cu130 cuda=13.0 arch=x86_64 gpu=NVIDIA H100 80GB HBM3
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source oss20A4-s0047
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source oss20B4-s0050
vllm serve openai/gpt-oss-20b --dtype bfloat16 --gpu-memory-utilization 0.90 --no-enable-prefix-caching --max-cudagraph-capture-size 2048 --max-num-batched-tokens 8
sglang.launch_server --model-path openai/gpt-oss-20b --dtype bfloat16
rel-flint-head-k2
Record
gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours flintA3-s0026 sha256=8060d34f88c738a7
ours flintB3-s0027 sha256=8ef9677fe7a2a6b2
vllm=0.33.0 torch=2.14.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source flintA3-s0026
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source flintB3-s0027
vllm serve RecursiveSI/flint --dtype bfloat16 --gpu-memory-utilization 0.90
rel-flint-head-f
Record
gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours flintA2-s0077 sha256=5c3396ee29ca2c89
ours flintB-s0055 sha256=97ec5b442dd27867
vllm=0.33.0 torch=2.14.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source flintA2-s0077
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source flintB-s0055
vllm serve RecursiveSI/flint --dtype bfloat16 --gpu-memory-utilization 0.90
rel-flint-axes-2
Record
gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours flintA-s0054 sha256=22a115614e575e72
vllm=0.33.0 torch=2.14.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200
serve_engine.py engine --max-batch 8 --max-context 18432 + frontend; source flintA-s0054
vllm serve RecursiveSI/flint --dtype bfloat16 --gpu-memory-utilization 0.90
rel-q27-head-k2
Record
gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours q27A2-s0022 sha256=67f78b19aaf2551b
ours q27B3-s0020 sha256=860af7b83f4e929e
sglang=0.5.23 torch=2.14.1+cu130 cuda=13.0 arch=aarch64 gpu=NVIDIA GB200
vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q27A2-s0022
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q27B3-s0020
sglang.launch_server --model-path Qwen/Qwen3.8-27B --dtype bfloat16
vllm serve Qwen/Qwen3.8-27B --dtype bfloat16 --gpu-memory-utilization 0.90
rel-q27-head-f
Record
gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours q27A2-s0022 sha256=67f78b19aaf2551b
ours q27B2-s0023 sha256=3941f614d5c3a090
sglang=0.5.23 torch=2.14.1+cu130 cuda=13.0 arch=aarch64 gpu=NVIDIA GB200
vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q27A2-s0022
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q27B2-s0023
sglang.launch_server --model-path Qwen/Qwen3.8-27B --dtype bfloat16
vllm serve Qwen/Qwen3.8-27B --dtype bfloat16 --gpu-memory-utilization 0.90
rel-q27-axes-2
Record
gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours q27A-s0016 sha256=7c131f50e038b675
sglang=0.5.23 torch=2.14.1+cu130 cuda=13.0 arch=aarch64 gpu=NVIDIA GB200
vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200
serve_engine.py engine --max-batch 8 --max-context 18432 + frontend; source q27A-s0016
sglang.launch_server --model-path Qwen/Qwen3.8-27B --dtype bfloat16
vllm serve Qwen/Qwen3.8-27B --dtype bfloat16 --gpu-memory-utilization 0.90
rel-q32-head-k2
Record
gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours q32A3-s0033 sha256=1d029c8737bc23dd
ours q32B3-s0036 sha256=d5c730744c3b4706
sglang=0.5.23 torch=2.14.1+cu130 cuda=13.0 arch=aarch64 gpu=NVIDIA GB200
vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q32A3-s0033
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source q32B3-s0036
sglang.launch_server --model-path Qwen/Qwen3-32B --dtype bfloat16
vllm serve Qwen/Qwen3-32B --dtype bfloat16 --gpu-memory-utilization 0.90
rel-oss120-head-k2
Record
gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours oss120A3-s0049 sha256=26c8833113e22068
ours oss120B3-s0051 sha256=63d4b0a5c2c83fb9
sglang=0.5.23 torch=2.14.1+cu130 cuda=13.0 arch=aarch64 gpu=NVIDIA GB200
vllm=0.32.0 torch=2.13.0+cu132 cuda=13.2 arch=aarch64 gpu=NVIDIA GB200
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source oss120A3-s0049
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source oss120B3-s0051
sglang.launch_server --model-path openai/gpt-oss-120b --dtype bfloat16
vllm serve openai/gpt-oss-120b --dtype bfloat16 --gpu-memory-utilization 0.90 --no-enable-prefix-caching --max-cudagraph-capture-size 2048 --max-num-batched-tokens
rel-ink-head-2
Record
gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours inkA2-s0034 sha256=a144977d2dc5042c
ours inkB2-s0036 sha256=02dc9a234e11d4e2
img sglang ref=lmsysorg/sglang@sha256:198af079c599cbf37ad371faca16c988a71cea771b3c51fbe21d018f6e400c9e arch=arm64
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source inkA2-s0034
serve_engine.py engine --max-batch 8 --max-context 2048 + frontend; source inkB2-s0036
sglang(image lmsysorg/sglang@sha256:198af079c599cbf37ad371faca16c988a71cea771b3c51fbe21d018f6e400c9e) env SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 launch_server --model-path thinkingmachines/Inkling-2-Small
rel-inkon2-cells
Record
gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours inkOn2-s0043 sha256=65058690258bae89
img sglang ref=lmsysorg/sglang@sha256:198af079c599cbf37ad371faca16c988a71cea771b3c51fbe21d018f6e400c9e arch=arm64
serve_engine.py engine --max-batch 64 --max-context 34816 + frontend; source inkOn2-s0043
sglang(image lmsysorg/sglang@sha256:198af079c599cbf37ad371faca16c988a71cea771b3c51fbe21d018f6e400c9e) env SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 launch_server --model-path thinkingmachines/Inkling-2-Small
rel-ink-axes-2
Record
gpu NVIDIA GB200, 189471 MiB, 595.71.05, 2062 MHz
eval torch=2.15.0+cu132 cuda=13.2 transformers=5.23.0
ours inkA-s0020 sha256=1c05b00e58dc971c
img sglang ref=lmsysorg/sglang@sha256:198af079c599cbf37ad371faca16c988a71cea771b3c51fbe21d018f6e400c9e arch=arm64
serve_engine.py engine --max-batch 8 --max-context 18432 + frontend; source inkA-s0020
sglang(image lmsysorg/sglang@sha256:198af079c599cbf37ad371faca16c988a71cea771b3c51fbe21d018f6e400c9e) env SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 launch_server --model-path thinkingmachines/Inkling-2-Small