The 1/20-cost model scored less than half the coding points — so I made the 60-second proof public

The 1/20-cost model scored less than half the coding points — so I made the 60-second proof public

I've been running a small eval harness that pits a budget model against a frontier one on the same fixed set of coding tasks. The result kept surprising me: the cheap model runs at roughly 1/20 the per-token cost and lands well under half the coding score of the big one. Numbers on a spreadsheet don't convince anyone, so I recorded the whole run as a short walkthrough: https://www.youtube.com/shorts/Kk0DbwAPBTU

Here's the concrete detail of how it works. The harness feeds each task to both models with identical prompts and temperature 0, then scores pass@1 against a held-out test set — no retries, no cherry-picking. Cost is computed from the provider's published per-token rates times actual completion tokens, so a verbose model pays for its own verbosity. That last part is what makes the comparison honest: the cheap model isn't winning because it answers less, it's winning on raw task completion per dollar.

The thing it costs me as the operator is real money and real time. Every full benchmark sweep burns through a few dollars in API calls and about twenty minutes of wall-clock time waiting on the frontier model's slower completions. That's the trade I accept — I'd rather spend that than publish a chart nobody believes. If you've ever run your own model comparisons, you know the gap between "I measured this" and "watch me measure this" is the whole ballgame.