An experiment in LLMs for embedded code generation. Control loops, security, and real peripherals implemented on Cortex-M or RISC-V micro. Each LLM model gets a task and requirements, the vendor's documents, a debugger, and it goes for it. A judge checks what it built and measures what the hardware (or emulator) does.
Part built, or just a plan. Nothing from these is on the site yet.
Same data.
Every run a model has had on a rig, and what to know before you believe its hand-back.
All model pages →Models down the side, tasks across the top, one square per judged run. It's dense, but it's where you go to check a claim.
The results grid →Every demo runs in C. Some run again in Dulse, a language I'm working on, whose compiler checks firmware against the chip's manual and its deadlines. So I have a horse in that race, which is why the switch on each demo just shows the runs and the argument stays on its own page.
The comparison →While a wave runs, a dashboard shows who has the board, what each agent is up to, and the clips as they get cut.
dulse-benchdash; not public yetFour groups and no overall score, as one big number would hide which of these a model is actually good at. Each result says if it came from a rig or a simulator, and the simulated pendulum stays out of the rig's tables and the results grid.
The rules get written first. Each demo has a pre-registration file committed before its first run, and any later change or voided run is an amendment in that file, so I can't quietly move the goalposts. On the servo, a written review of each tree is committed before any verdict gets read.
The board decides. Levels and passes come from the judge's own trials: a camera following the rod, lost-tick counters, the encoder. The hand-back can say whatever it likes, it never decides a result.
One run is one run. Most cells hold one to three runs, and each gets its own square so a 1 of 1 never looks like a 3 of 3. A single run says what happened, not how good a model is.
Not all dollars are the same. Claude runs made directly are what the CLI reports at API list prices, on a subscription token, so not a bill. OpenRouter runs are what was charged. Codex runs are tokens at list prices. No run set an effort level.
Harness faults get fixed and rerun, model behaviour stands. A run that died on a proxy bug or a dead endpoint is voided and listed with its cost. A model that blames the board keeps its score.
Every number names its file. Hover a cell or read the line under a block for the results file or the run record it came from. If a number has no file beside it, that's a bug in the page, so please tell me.