KernelAscent
Models write GPU kernels. Each kernel is graded for correctness against a reference and for speed against a baseline. The benchmark asks two things: how well a model does in one shot, and whether it gets better when it trains on, or edits, its own work over several rounds.
| Task | What it measures | Models |
|---|---|---|
| 1. Capability | Can the model write a correct, fast kernel in one shot? | open and closed |
| 2. Weight self-improvement | The model trains on its own correct kernels each round. Does it get better than a matched control? | open weights |
| 3. Procedure self-improvement | Weights stay fixed; the model rewrites its own strategy and solver prompt each round. | open and closed |
| 4. Closed → open | A closed model rewrites the training setup of an open model. | closed + open |
| 5. Self-play | The model writes harder tasks for itself, then solves them. | open and closed |
15 fixed tasks, 5 tries each. correct: output matches the reference. fast: correct and faster than PyTorch. Click a header to sort.
| Model | Kind | correct | fast | mean C |
|---|
Five rounds. self vs fresh-frozen: the model trained on its own kernels minus a model trained on kernels from the unchanged base. Positive means self-training helped.
| Model | Family | rounds | C0 | final | correct | compiled x | delta vs frozen | self vs fresh-frozen | verdict |
|---|
Each size runs on tasks filtered for its own difficulty, scored on speed against torch.compile.
| Model | Tier | rounds | C0 (speed) | C final | self vs fresh-frozen | verdict |
|---|
Self-training compared with methods that don't train, at the same number of generations. recursion gain: self-training minus the best of these.
| Model | weight-RSI C | best-of-k | self-refine | retrieval | recursion gain |
|---|
Q is the held-out quality of the model's current procedure. Q gain vs frozen: improvement over the procedure it started with.
| Model | Mode | rounds | Q0 | Q final | Q gain vs frozen | verdict |
|---|
improved − frozen: the open model trained with the closed model's setup minus one trained with the original setup. peak: best round.
| Researcher | Open trainee | rounds | C0 | C improved | C frozen | improved − frozen | peak | verdict |
|---|
Three versions run side by side: fixed tasks (static), harder tasks written by the original model (frozen-author), and harder tasks written by the improving model (live). L − F: live minus frozen-author. authored: model-written tasks that passed the validity check.
| Model | Tier | rounds | C live | C frozen-auth | C static | L − F | F − S | authored | verdict |
|---|
| Model | Tier | rounds | C live | C frozen-auth | C static | L − F | F − S | authored | verdict |
|---|
Each task bank has a public split and a held-out split.
| Bank | Contents | Where |
|---|---|---|
| Kernel bank | 29 tasks in three tiers: memory-bound, matmul epilogue, attention and norm | Hugging Face, dataset/kernel_bank/ |
| Hard bank | filtered to hard but learnable tasks, used for self-improvement | dataset/kernel_bank/rsi_bank_hard.json |
| Generated bank | 456 tasks over 16 operation families | dataset/kernel_bank/kernel_tasks_dsl_validated.json |
from datasets import load_dataset
ds = load_dataset("muahmed7338/kernelascent-tasks")Open a pull request adding one JSON file under submissions/<track>/. It is checked automatically and appears below as self-reported until a maintainer re-runs it.
python3 scripts/validate_submission.py submissions/capability/your-model.jsonGuide: submissions/README.md
| Track | Model / recipe | Kind | headline | status | by |
|---|