Anatomy of a Sync Outage
Cloud sync stopped. Uploads failed for everyone, and the error in the app pointed at the wrong thing entirely. What looked like one bug turned out to be three — stacked, each hiding the one behind it. We fixed the first two before realizing neither was what users were hitting. This is the whole descent, told straight, and the rules we walked away with.
We build LastDB on LastDB — our own tools run on it, and they sync to the cloud like anyone else’s. So when sync broke, it broke for us too, which is the only reason this story has a happy ending: we were the affected user, and we could poke it directly.
The error that lied
The app reported a sync failure that read, in effect: “auth error: HTTP 500.” So the first instinct was to go look at authentication. That was a trap. The string was assembled by a generic wrapper that turns any server-side failure on the sync path into an “auth”-flavored message. Authentication was fine. The 500 was coming from somewhere else, wearing auth’s coat.
This is the shape of every hard outage: the first message you read is a label, not a location. The path a sync upload travels is short — device, gateway, the sync service, the object store — and a 500 with the body “Internal Server Error” is the gateway’s way of saying it gave up waiting for the service behind it. Not an auth rejection. A timeout.
The first wrong turn
Before we found the timeout, we found something that looked like a cause and wasn’t. The sync service logs were full of failures talking to one of our object-storage backends, and reproducing the call by hand returned an error rendered as the bare words “service error.” Unmasked, the real message was concrete and alarming: the storage account had blown through its daily transaction cap. Every call was getting rejected.
That sent us down a satisfying path: raise the cap, ship the obvious storage fix, declare the incident closed. It was real — the cap really was exhausted — but it was a symptom, not the disease. The sync data lives in a different store, one that was perfectly healthy. The transaction cap was being eaten by the same runaway behavior that was breaking sync, but lifting it wouldn’t bring sync back. We’d found a problem. It wasn’t the problem.
Lesson: a masked error is a lie of omission
The SDK printed every failure — a permissions denial, a quota wall, a missing record — as the same two words: “service error.” That uniform vagueness cost us hours, because two completely different failures looked identical in the logs. An error that hides its cause is worse than no error: it’s a confident pointer in the wrong direction.
Layer one: a scan where there should have been a question
The actual timeout lived in the upload path. Before handing back an upload URL, the sync service did a quota check. A quota check should be a single cheap lookup: how much is this account using? — one read of a number we already keep.
A recent change had quietly turned that question into a survey. To keep the usage number fresh, it reconciled the figure by listing every object the account owned — on the upload path, on every upload. For a small account, invisible. For a large one, ruinous: the scan took longer than the service was allowed to run, so it never finished. And it got worse. The reconcile only recorded “done” on success, so a scan that timed out left no mark — which meant the next upload tried the same doomed scan, and the next. A permanent loop of timeouts, each one a 500, each one a failed sync.
We took the scan off the hot path. The quota check went back to reading the number it already had — the survey moved out of the request entirely. Presigns dropped from ten seconds (and failing) to a hundred milliseconds. We shipped it, watched the error logs go quiet, and called it fixed.
Declaring victory too early
It was not fixed. Two things had fooled us.
The first was metric lag. The dashboards that should have confirmed the fix were averaging over a window that still contained the broken minutes. The graph looked like it was healing because old data was aging out, not because new data was healthy. We read a falling line as success when it was just arithmetic.
The second was worse, and it’s the most useful thing in this whole post.
Layer three: the fix that never ran
Our deploy had updated the service’s code, but production doesn’t serve the latest code. It serves a published, frozen version, reached through a stable pointer — an alias — with warm capacity held against it. We’d updated the editable draft. The alias still pointed at the old frozen version. The fix was sitting in production, deployed, and completely unreachable by a single real request.
This is why the first two fixes appeared to do nothing: they did nothing, because nothing ran them. The moment we published the new version and moved the alias to point at it, the change took effect — and the presign timeouts vanished for real.
Layer two: the list that couldn’t finish
With presigns fast and the alias pointed correctly, sync started flowing — partially. Uploads landed. But the app still showed errors, and about half of all requests still timed out. There was a second, independent scan we hadn’t touched.
To sync, a device first lists what’s already in the cloud, then uploads what’s missing. That list paginates the account’s entire log. And here we hit the real, underlying rot: this account’s log had grown to roughly 35,000 entries — with exactly one snapshot. The log was never being compacted. Listing 35,000 objects took about twenty seconds, which — you’ve learned the rhythm by now — is longer than the service was allowed to run.
A healthy log is small: a recent snapshot, plus the handful of entries written since. Compaction is what keeps it that way — it folds old entries into a snapshot and truncates them. Without it, the log grows forever, and one ordinary day it grows past the point where you can even list it.
The honest immediate fix was a stopgap: we raised the service’s time limit so a twenty-second list could complete. That bought sync back. But it is explicitly a band-aid — the log keeps growing, twenty seconds creeps toward the ceiling, and the only durable fix is to make compaction keep the log small. We wrote that down as the next thing to do, in plain view, so the band-aid wouldn’t be mistaken for a cure.
How we actually knew it was fixed
Three times we’d believed a graph and been wrong. So for the last fix we stopped trusting metrics and went looking for ground truth. Because we run our own tools on LastDB, there was a direct way to get it: make a real write. One small change in one of our apps produces a real sync — the exact request that had been failing all day.
Verify against reality, not the dashboard
We made the write, watched a single real sync travel the whole path — list completes, upload lands, object appears in the store — and only then believed it. The final confirmation wasn’t a chart. It was the sync indicator in the app going green. A metric tells you about a population; a real request tells you about the thing in front of you. When you’re unsure, reach for the second one.
What we changed
- The scan came off the upload path. The quota check reads the number we already keep; it never lists a bucket to answer a routine question.
- The fix was published and the alias moved — so the running version is actually the fixed one — then folded into our normal deploy so it can’t drift back.
- The time limit was raised as a safety net, not a solution, with a note saying exactly that, pointing at the real fix.
- Compaction — keeping the log small so the list is fast — was filed as the priority follow-up, because everything above is downstream of it.
How we keep it from happening again
Every line of this outage was a known anti-pattern we’d let slip onto a hot path. The fixes are specific; the rules are general.
- Keep scans off hot paths. A request that runs on every action must do bounded, predictable work. Expensive reconciliation belongs in the background — event-driven or scheduled — feeding a number the hot path can read in one shot. “While we’re here, let’s also recompute the truth” is how a fast path dies.
- Bound every pagination. Any loop that walks “all of something” will, given enough time, walk too much. Cap it, or hand back a cursor and let the caller paginate. Two separate timeouts in this incident were the same unbounded loop wearing different hats.
- Compaction is not optional. Append-only logs are wonderful until they’re thirty-five thousand entries long. If a structure grows with use, something must shrink it on a schedule — and you should alert when it stops, long before a list can’t finish.
- Unmask your errors. “Service error” is not an error message; it’s a shrug. Surface the real cause — the status, the code, the resource — at the point of failure. The hours you save are your own.
- Deploy where traffic actually goes. If production serves a pinned version behind an alias, then updating the draft is theater. Make the deploy and the verification target the alias, so “it’s deployed” and “it’s running” are the same sentence.
- Alarm on the wall before you hit it. A quota, a cap, a time limit — each should page you on the approach, not on the crash. The transaction cap blew silently; the first we knew of it was the failures it caused.
- A timeout is a safety net, not a fix. Raising a limit so slow work can finish buys time and hides the slowness. Fine, as long as you say so out loud and point at the real repair. A band-aid mistaken for a cure is just a slower outage.
- Tell “a real problem” from “the user’s problem.” We fixed two genuine bugs that weren’t what anyone was hitting. Finding something broken is not the same as finding the break. Keep asking, against real traffic, until the symptom is actually gone.
None of these are clever. That’s the point: outages are rarely a failure of cleverness. They’re ordinary shortcuts — a scan here, a missing bound there, a deploy aimed at the wrong target — that wait quietly until an account gets big enough to find them all at once. The work isn’t avoiding the brilliant mistake. It’s refusing the ordinary one, on the path that runs a thousand times a minute.
More on how we build in the open: our autonomous build loop, and a week of speedups we didn’t write.