Karpathy nanochat: How GPT-2-Class Training Fell From $43,000 to $48 in 7 Years (2026 H100 Spot Math)

OpenAI’s 2019 GPT-2 training run cost $43,000. Karpathy’s nanochat, which shipped this year, prints a CORE score that beats it for $48. That delta looked wrong to me, so I opened the repo and priced a run against current spot rates to check.

I’ve been working through nanochat’s 8,159 lines one directory at a time for a book I’m writing. That process led me to a fairly modest realization about how the “GPT-2 got cheap” arc actually plays out in the record: there was never a single dramatic compression event, only a slow pileup of dated incremental wins. This post walks through the receipts.

One bias to name up front: I wrote a 200-page reading guide to nanochat, so I’m already inclined to find it interesting. The $48 number itself didn’t surprise me, given that cheap-LLM headlines have become their own genre. My actual reaction was to the format of Karpathy’s dev/LEADERBOARD.md, which reads more like a public accounting ledger than a marketing artifact. Every row carries a date, a wall-clock time, and a diff URL, which almost never happens in this space.

The 2019 receipt

The $43,000 traces back to dev/LEADERBOARD.md, row zero: OpenAI’s original GPT-2 at 1.5B parameters, trained on 32 TPU v3 chips for 168 hours of wall-clock at roughly $8/chip/hour in 2019 rates. The multiplication comes out to $43,008 exactly. Rounded, that matches the figure OpenAI’s blog gestured toward without ever putting on the price tag.

A couple of caveats matter here. The $43K already assumed on-prem cloud discounts rather than sticker rate; a naïve at-the-time rental of 32 TPU v3s for a week would have run considerably higher. And when I say “GPT-2” I specifically mean the CORE score of 0.256525 as measured on DCLM’s benchmark suite, which is the target the leaderboard exists to hit.

The 2026 receipt

The file that made nanochat famous, runs/speedrun.sh, weighs in at 79 lines. It launches a d24 (24-layer) model on 8xH100 hardware for around two wall-clock hours. As of March 14, 2026, row 6 of the leaderboard puts the current time at 1.65 hours. According to the README’s cost note, that translates to $48 on a Lambda 8xH100 node at on-demand rates, or roughly $15 if you’re running on spot.

The $48 figure served as my starting point. Cited in the repo and clearly real, it already comes in at 900x cheaper than 2019. However, the quote referenced list rates from the 8xH100 tier as of publication, and the H100 market has drifted considerably through 2026.

Re-pricing at July 2026 spot rates

So I pulled current prices from four providers I’ve actually used, then reran the numbers against the same 1.65-hour run:

Provider	H100 tier	Rate (per GPU-hour)	8-GPU run × 1.65h
Lambda Labs on-demand	H100 SXM	~$2.49	~$32.87
RunPod on-demand	H100 SXM	~$2.69	~$35.51
RunPod spot (Community Cloud)	H100 SXM	~$1.19	~$15.71
Spheron spot	H100 SXM5	~$1.03	~$13.60
Vast.ai peer-to-peer	H100 varies	~$1.60	~$21.12

At spot pricing today, the floor sits around $13-16 for a full GPT-2 speedrun. Lambda’s listed $48 captured a snapshot of a specific day and tier; the market has drifted downward from there. Even the on-demand tier now sits closer to $33 than $48.

A few caveats these numbers exclude. Spot interruption risk means your 1.65-hour run may restart from a checkpoint, which pushes the actual cost above the naïve multiplication. Egress fees, snapshot storage, and the WandB run written out by speedrun.sh add up to real but modest overhead. Assume something like a 10-20% margin over the spot floor if you’re new to this, more if you don’t already have uv and cached datasets on hand.

Where the 100x-plus savings actually came from

The dev/LEADERBOARD.md table answers the “where did the money go?” question directly. No single mechanism carries the whole story; instead you get a sequence of dated, attributed pull requests, each shaving minutes off the wall clock:

Row	Date	Time	CORE	What changed
0	2019	168h	0.2565	OpenAI GPT-2 original
1	2026-01-29	3.04h	0.2585	d24 baseline, slightly overtrained
2	2026-02-02	2.91h	0.2578	d26 slightly undertrained + fp8
3	2026-02-05	2.76h	0.2602	Batch size raised to 1M tokens
4	2026-03-04	2.02h	0.2571	Switched dataset to NVIDIA ClimbMix
5	2026-03-09	1.80h	0.2690	Autoresearch round 1
6	2026-03-14	1.65h	0.2626	Autoresearch round 2

Going from 168 to 1.65 hours works out to roughly a 100x compression, though no individual row accounts for more than about a third of the total. FP8 kernels via torch._scaled_mm accounted for one slice. Batch size retuning contributed further reductions in wall-clock time. Swapping in NVIDIA’s ClimbMix dataset added yet more. The remainder came from two rounds of what Karpathy calls “autoresearch,” where he lets an LLM propose training-loop tweaks and A/B-tests them.

Silicon obviously factors in as well. TPU v3 to H100 delivers roughly a 15-20x throughput improvement per chip on bf16 workloads, and H100 fp8 doubles that again on the layers where it stays stable. Yet the leaderboard’s own history shows algorithmic wins pulling more of the weight than hardware within the 2026-to-2026 window, since the hardware itself didn’t change inside that year.

The compute-optimal knob that quietly matters

Row 2 flips a fairly subtle switch. The --target-param-data-ratio argument in scripts/base_train.py defaults to 12, while Chinchilla and the broader compute-optimal literature suggest roughly 20 tokens per parameter. Inside runs/speedrun.sh, that value gets set to 8.

That’s deliberate undertraining. The comment in the shell script owns it: “slightly undertrained to beat GPT-2.” The objective here isn’t producing the best model your compute budget allows; it’s reaching a specific CORE score as quickly as possible. Anyone chasing “cheapest GPT-2” ends up walking away from compute-optimal on purpose because compute-optimal targets a different metric entirely.

I bumped into this reading through my own book draft, and it shifted how I think about the “smaller and faster is always worse” reflex. Smaller and faster is worse when your metric is “best model.” Once the metric becomes “clear a specific bar cheaply,” undertraining turns into a valid lever. Add this to the pile of things I would have sworn were obvious and turned out not to be.

What this changes for a working engineer

If you’ve been putting off touching pretraining because it felt like a $50,000 hobby, there are two shifts worth registering.

One is that pretraining now costs about the same as a Saturday car rental. Given a $50 budget and eight hours, you can produce a model that beats what OpenAI announced with a blog post in 2019, at least on DCLM CORE. That doesn’t make you competitive with 2026 frontier models (a d24 is nowhere near GPT-5), but the “I have never actually trained a real LLM” barrier now runs roughly the price of a decent lunch.

The other is that the pipeline you clone from nanochat is legibly the same pipeline frontier labs run. Their version uses bigger models, better data, and more compute, though the overall shape stays consistent: tokenizer training, pretraining with fp8 and Flash Attention 3, SFT, evaluation, inference with KV cache. Reading those 8,159 lines is the fastest crash course I’ve come across for what “training an LLM” actually looks like at the function-call level.

For anyone who’s been shipping RAG apps and prompt harnesses while wondering when the magic-feeling wears off, this is when. Becoming a pretraining specialist isn’t required. Running bash runs/speedrun.sh once and watching the wandb curves is enough on its own. Most of the mystique evaporates within an afternoon.

The uncomfortable line

One sentence kept nagging at me as I re-read dev/LEADERBOARD.md for the tenth time. The 2019 GPT-2 model that OpenAI famously withheld on safety grounds now costs less than a mid-range mechanical keyboard.

My reaction there is fairly flat rather than heated. What I have is a mild sense that the discourse around “who should be allowed to train large models” was, in retrospect, aimed at a moving target that nobody modeled correctly, plus an even milder sense that today’s discourse is probably aiming at another target on the move. Cheap GPT-2 counts as a data point on that pattern.

The 79 lines of speedrun.sh will most likely keep shrinking, and the dev leaderboard will keep getting rewritten alongside them. dev/LEADERBOARD.md is worth bookmarking specifically because it operates as a living document. By 2027 nobody will care whether GPT-2 got cheap (that answer is already settled); the real question will be which capability tier crossed the $100 line this quarter.

Comparing official AI agents from OpenAI and Anthropic — same-shape argument for the coding-agent tier
Natural-language agent harnesses I learned from arXiv — the algorithmic-tweak vocabulary that autoresearch uses

Sources: karpathy/nanochat repo, nanochat Discussion #481 — Beating GPT-2 for <<$100, H100 rental price comparisons IntuitionLabs, Spheron GPU pricing 2026.