KernelAscent
KernelAscent asks whether models get better at writing GPU kernels. A model writes a kernel. We grade it for correctness against an fp32 reference and for speed against a baseline. Task 1, Capability measures raw skill in one shot. Task 2, Weight RSI measures recursive self improvement through weight training, for open weight models. Task 3, Procedure RSI lets open and closed models improve their own executable research procedure, weights fixed. Open and closed models both compete on Tasks 1 and 3. Task 2 needs weight access.
All three tasks use GPU kernels and the same grader. A kernel is correct when its output matches the fp32 reference. Speed is scored against an eager baseline today. A unified torch.compile baseline is being added and reported separately. Task 1 is a single solve. Task 2 improves the weights across rounds. Task 3 improves the executable procedure across rounds with the weights fixed.
Can the model write a correct and fast kernel in one shot. Correct means it matches the fp32 gold. Fast means it beats the eager baseline today, with a compiled baseline being added. Correctness ranks the low and mid band. Real speedups separate the frontier.
Does self improvement compound. Each round the model writes kernels for a train split, trains on its own correct kernels, and is re-scored on a held out split. The weights change, so the improver itself gets better. A frozen or API model cannot cross this line.
A closed model cannot change its weights, but it can improve its own executable research procedure. The editable state is a strategy library, a solver prompt, and an archive of verified kernels. Each round the model solves tasks, then rewrites its own procedure from the evidence. The improved procedure drives the next round. Weights stay fixed. This is how API models join the RSI axis.
Fixed tasks, k = 5. Correct matches fp32 gold. Speed is scored against an eager baseline today, with a compiled baseline being added. Open and closed models both compete. Click a header to sort.
| Model | Kind | correct | fast | mean C |
|---|
Open models only, five rounds on the difficulty standardized hard bank. Three arms run in parallel. Self trains on its own kernels. Fresh-frozen trains on kernels freshly sampled from the unchanged base each round. Round0 retrains on the first round's data. The key column is self vs fresh-frozen. A positive value means an improving producer causes better descendants than a frozen producer given the same fresh data. That is the causal recursion signal, stronger than beating a static replay. Correctness and compiled speedup are shown separately so a gain in reliably writing a correct kernel is not confused with a gain in writing a faster one.
| Model | Family | rounds | C0 | final | correct | compiled x | delta vs frozen | self vs fresh-frozen | verdict |
|---|
Each model runs on a bank difficulty-filtered against a frozen anchor of its own scale (small 1.5B, mid 7B, large 14B), scored on speed vs torch.compile where every size has headroom. This removes the benchmark confound so results are pure capability.
| Model | Tier | rounds | C0 (speed) | C final | self vs fresh-frozen | verdict |
|---|
A closed model (Fable-5.1, fixed weights) rewrites the training harness each round; the harness trains an open model. The key column is improved vs frozen harness on the open trainee. Positive means the closed model's procedure edits causally produced a better open model.
| Open trainee | Researcher | rounds | C0 | C improved | C frozen | improved vs frozen | verdict |
|---|
Every gain is checked against non-recursive methods at the same generation budget (best-of-k sampling, in-context self-refine, archive retrieval), all frozen-weight and scored with the identical grader. recursion gain = weight-RSI final C minus the best non-recursive baseline. Positive means training on your own kernels beats simply sampling more.
| Model | weight-RSI C | best-of-k | self-refine | retrieval | recursion gain |
|---|
A model with fixed weights rewrites its own executable research procedure each round (strategy library, solver prompt, verified-kernel archive). Q is held-out quality of the current procedure; the key column is Q gain vs the frozen procedure (self-modify arm). Open and closed models both compete.
| Model | Mode | rounds | Q0 | Q final | Q gain vs frozen | verdict |
|---|
Fixed-task self-training can only deliver a one-time upgrade (our recursion-interruption control shows continued-self ≈ checkpoint-frozen once the bank is fit). Task 5 removes that ceiling: each round a proposer (Fable-5.1) mutates the kernels the model just solved into strictly harder variants, GPU-validates them, and grows the frontier — so difficulty co-evolves with the solver. Holding or rising capability on a growing frontier is genuine compounding, not fitting a fixed set.
Twelve rounds of weight-RSI, logging the state of each model's own training signal every round. The story is in the curves: models that keep self-data diversity high and retain their round-0 solutions keep gaining, while models whose self-generated kernels collapse in diversity and forget prior solutions drive self-minus-fresh negative (overfit). Each line is a model.
Every task ships a public split for development and a disjoint held out split that is never released. The board is scored only on held out, so scores cannot be overfit.
| Task | Dataset | Tiers | Public | Held out |
|---|---|---|---|---|
| Capability | GPU kernel bank | L1 memory, L2 matmul epilogue, L3 attention and norm | HF, dataset/kernel_bank/ | disjoint kernel set, unreleased |
| RSI | Standardized hard bank, Fable curated and GPU validated | L1, L2, L3, filtered to hard but learnable | dataset/kernel_bank/rsi_bank_hard.json | held out kernels re-scored each round |
from datasets import load_dataset
ds = load_dataset("muahmed7338/kernelascent-tasks") # public split, by tierEverything runs behind one entrypoint. The board is scored on a private held out split.
docker build -t ka -f docker/Dockerfile .
# Task 1, Capability, open or closed
docker run --rm -e AWS_SHARED_CREDENTIALS_FILE=/creds -e AWS_PROFILE=bedrock \
-v $PWD/creds:/creds:ro -v $PWD/out:/out ka \
--track capability --api-model <model> --tier medium
# Task 2, RSI, open weight, GPU
docker run --rm --gpus all -v $PWD/out:/out ka \
--track rsi --model <hf-id> --rounds 5 --k 3
# each writes out/scorecard.jsonOpen a model submission issue with your scorecard.json. Maintainers re-run on the held out split and add your row. Automatic submission through GitHub is coming soon.