The Parallelism Tax
We split our test suite into hundreds of parallel pieces to make CI faster. It worked: a run that used to take eleven minutes took eight. Then the bill arrived. In one month our continuous-integration compute went up roughly a hundredfold — enough that, one morning, our CI provider simply refused to start any more jobs. We had optimized the one number we were watching and blown up the one we weren’t.
What we did, and why it looked smart
Our codebase is a single repository holding many subsystems — a database core, the node that wraps it, a schema service, some serverless functions. A full test run compiles and exercises the lot. To make that run wall-clock faster, we did the obvious thing: we cut the suite into shards and ran them in parallel. Two shards, then ten, then — chasing the last minute off the clock — two hundred and sixty-seven. Each shard runs a small slice of the tests at the same time as all the others, so the whole thing finishes when the slowest single slice finishes, not when the sum of them would. On the stopwatch, it was a win. The stopwatch was the problem.
The catch: every shard pays the toll again
A parallel job is not free the instant it starts. Before it runs a single test it has to check out the code, install the toolchain, restore its cache, and compile — five to seven minutes of billed machine time, every time, on every shard. Split one job into six and you have not divided the work into six. You have divided the tests into six and multiplied the setup by six. The wall-clock falls because the slices run side by side. The bill rises because you are now paying for that fixed startup two hundred and sixty-seven times instead of a handful.
This is the whole trap in one sentence: you are billed for the total area, but you were only looking at the width. Parallelism is a lever that trades money for time, and we had been pulling it as if it were free. Two hundred and sixty-seven shards, each re-compiling, added up to hundreds of thousands of machine-minutes a month — a bill that dwarfed the eleven-to-eight-minute prize we’d bought with it.
The fix: run only what the change can touch
The deeper waste was subtler than the setup toll. Every change ran every shard — a one-line edit in the schema service still recompiled and retested the database core, the node, the functions, everything. But most changes only touch one subsystem. So we taught CI to look at what actually changed and run only the parts that could possibly be affected by it.
The subsystems form a dependency graph: some are foundations that others build on, some are leaves that nothing else depends on. When a change lands in a leaf, nothing downstream can break, so nothing downstream needs to run — just that leaf. When a change lands in a foundation, everything built on it is in play, so everything runs. CI now follows that graph instead of a single on/off switch.
The rule is deliberately paranoid in one direction: when in doubt, run more, not less. Anything that could affect the whole build — a lockfile, a toolchain bump, the CI config itself — still triggers the full suite, and a catch-all shard sweeps up any new code that hasn’t been assigned a home yet. Cheaper must never mean untested; it only means not-pointlessly-retested. With that in place, plus a sober look at how many shards we actually needed, a typical change now runs a fraction of what it used to, and a full run costs a fraction of what it did — with the same tests covering the same code.
How we stop it happening again
Fixing the bill was the easy part. The interesting question is why it happened at all — and the answer is not “someone was careless.” The person who added the shards was optimizing the number they could see. The number they couldn’t see — the compute bill — had nothing pushing back on it. A system that only shows you wall-clock will get optimized straight into a compute explosion by someone doing their honest best.
So the durable fix isn’t a smarter config. It’s making the invisible cost impossible to ignore, at three different moments:
- At merge time — a check that caps how far the test matrix is allowed to fan out. Push past the cap and the pull request is blocked, with a message explaining that raising it requires measuring the compute cost first. This is the wall you hit the instant you try to re-spiral.
- Once a day — a watchdog that reads the actual billing numbers and raises a flag if spend, run volume, or per-run cost drifts up. It catches the things a merge-time check can’t see: a slow creep, a runaway retry loop, cost quietly migrating somewhere new.
- Always — a written rule, kept where the people (and the automated agents) doing this work will read it before they optimize: parallelism buys wall-clock with money; never trade unbounded compute for speed, and never raise a limit without measuring what it costs.
None of the three is sufficient alone. The merge check only sees shard counts; the watchdog is a day late; the rule is only words. But a spiral has to slip past all three at once, and each covers the others’ blind spot. That is the actual lesson, and it is older than CI: a metric you optimize without its cost beside it will get optimized until the cost is a crisis. The fix is never just to undo the damage — it’s to put the missing number back in the frame, permanently, so the next well-meaning optimization can see what we couldn’t.
More on how we work in the open: why a green pull request isn’t proof, and a week of speedups we didn’t write.