Karpathy Nanochat: $43K GPT-2 Training in 8,000 Lines (2026)
In 2019, OpenAI paid roughly $43,000 to train GPT-2. Thirty-two TPU v3 chips, 168 hours, $8/hour on the price sheet of the day. The number is right there in dev/LEADERBOARD.md in Karpathy’s nanochat repo, along with the receipts.
Seven years later, I ran wc -l on the same repository. 8,159 lines of Python and shell reproduce a model that beats GPT-2’s DCLM CORE score (0.2565), on a single 8×H100 node, in about 2 hours, for $48. Spot instances drop that to roughly $15.
That is not a rounding error. It is a 900× reduction in the price of “having your own GPT-2.”
And there are two things I did not expect about how small the number gets.
$43K, then $48, then $0
The runs/speedrun.sh script that produces the $48 run is 79 lines. I counted. It exports two environment variables, installs uv, syncs dependencies, and then walks the whole training pipeline top to bottom: tokenizer training, pretrain, SFT eval, chat SFT, chat eval. If you have $48 and eight H100s handy, that file is the whole recipe.
But that is not the interesting file. The interesting file is one directory over: runs/runcpu.sh.
runcpu.sh is the same pipeline, aggressively shrunk, targeted at a machine with no GPU at all. It exists so that someone with only a laptop can walk the full tokenizer → pretrain → SFT → chat loop end to end on their own hardware. The tuning notes in the script are candid: this is not going to produce a smart model. It is going to produce a model that (the comment says) might know that the capital of France is Paris, might know that the sky is blue, and might respond a little better if you say “hi” first.
That is fine. That is the point.
Karpathy’s benchmark on his own MacBook Pro M3 Max, taken straight from the script comments:
- Tokenizer training: ~34 seconds (8 shards, ~2B characters)
- Pretrain: ~30 minutes (
--depth=6,--max-seq-len=512, 5,000 iterations) - SFT: ~10 minutes (1,500 iterations)
About 40 minutes, no GPU, no cloud bill, no coordination email to the finance team. You end up with a small model that can hold a broken conversation about the sky being blue, and you end up with the pipeline living in your muscle memory.
I have run vLLM benchmarks that cost more than that in electricity.
8,000 lines split across six directories
The 8,159-line number sounds abstract until you look at where the lines actually live. I ran wc -l on the top-level directories:
| Directory | Lines | Role |
|---|---|---|
nanochat/ | 3,414 | The GPT model, tokenizer, optimizer, inference engine |
scripts/ | 2,592 | Entry points: one script advances the pipeline by one stage |
tests/ | 1,060 | Unit tests for the pieces above |
tasks/ | 581 | GSM8K, MMLU, and other eval task loaders |
runs/ | 384 | speedrun.sh, runcpu.sh, miniseries.sh and friends |
dev/ | 128 | Reference implementations, plus a 1,089-line LOG.md |
The interesting split is between nanochat/ (things reused across stages) and scripts/ (thin entry points that stitch pieces together). Almost every line of code either models something or measures something. Nothing feels defensive.
The dev/LOG.md file is not counted in the 8,159 above — it is Markdown, not code. But it is worth reading. It is Karpathy’s real training diary: what he tried, what broke, why he undid it. If you have ever wondered what “productively arguing with a training loop” looks like written down, that file is 1,089 lines of it.
Where the price actually fell
The 900× cost reduction is not one clever trick. Reading through the leaderboard entries in dev/LEADERBOARD.md, the wall time on 8×H100 comes down from 3.04 hours to 1.65 hours across six documented iterations, and each step is a different vector of improvement:
| # | Time | CORE | What changed |
|---|---|---|---|
| 0 | 168 h | 0.2565 | Original OpenAI GPT-2 (2019, 32 TPU v3) |
| 1 | 3.04 h | 0.2585 | --depth=24 baseline on 8×H100 |
| 2 | 2.91 h | 0.2578 | --depth=26 + fp8 |
| 3 | 2.76 h | 0.2602 | Total batch size bumped to 1M tokens |
| 4 | 2.02 h | 0.2571 | Swapped dataset to NVIDIA ClimbMix |
| 5 | 1.80 h | 0.2690 | Autoresearch round 1 |
| 6 | 1.65 h | 0.2626 | Autoresearch round 2 |
fp8, batch tuning, dataset swaps, hyperparameter search. The improvements are spread across the stack, and each one moves the number by a few percent. Nobody is going to write a Medium post called “the one weird trick that cut GPT-2 training by 100×,” because that is not how it happened. Death by a hundred incremental gains is a real category, and this is one of them.
The larger jump, the one from $43K in 2019 to $48 in 2026, is mostly the GPU price/perf curve. An H100 in 2026 is not 900× faster than a TPU v3 in 2019, but it is a lot faster, and $/hour on rental GPU markets has come down at the same time. Give the credit to the semiconductor industry, not to any one lab.
The number I keep thinking about
$48 is impressive. $15 on spot is more impressive. The number I keep thinking about, though, is 40 minutes on a MacBook.
Because $48 is still an expense you need to justify. You need to spin up a GPU, you need cloud credits, you need a Wandb account, you need someone to notice if the run OOMs at 3 AM. It is a Saturday project, but it is a Saturday project.
40 minutes on a laptop you already own is a lunch-break project. You do not need to justify it to anyone. You do not need to remember to shut anything off. When it finishes, the model is on your disk and you can python -m scripts.chat_cli and talk to the thing you just made.
That is the shift that snuck up on me. The interesting move Karpathy pulled with nanochat is not the $48 headline. It is the runcpu.sh file, the one that reduces LLM training from “expensive engineering project” to “here, try it, it will finish before your coffee gets cold.” The distance between the two is measured in dollars but felt in willingness.
If you have been vaguely meaning to understand what a training loop actually does, the kind of vague meaning-to that has survived several New Year’s resolutions, this is the on-ramp.
What I read next
Two adjacent posts from earlier in this year, if you want to keep pulling on the thread:
- ChatGPT Codex vs Claude Code: Official Coding Agents from OpenAI and Anthropic, for the version of this story where you buy the agent instead of building it.
- Natural Language Agent Harnesses on arXiv, where the training-your-own-harness-on-top-of-a-base-model idea gets its own literature.
And if you want the Japanese-language version of this post, with a longer breakdown of the leaderboard math and the depth-24 sizing table, I wrote it up here about a week before this one.
The 8,159 lines are on GitHub. The 79-line speedrun.sh is in the runs/ folder. The 40-minute laptop run is one bash runs/runcpu.sh away. That is a shorter distance than most tutorials I have written.
Was this article helpful?