[← Blog]

The Thrash Was in the Order

2026-07-15

We re-enabled cloud sync after a memory incident. Within minutes the machine was swapping hard again. Status said download was idle (zero new entries) and the local pending queue was thousands deep. We still never saw an upload start. The thrash was not the queue. It was the order of work inside one sync tick.

This is a write-up of what we got wrong, what the logs actually meant, and what we changed so we do not re-learn it under pressure. It is not a product pitch. The point is the failure mode.

What we thought was wrong

Diagnosis 1: download catch-up. The first incident looked like unbounded concurrent fetches on a cold cursor. That path is real. We put hard per-cycle caps on download (entry count, byte budget, per-object size) and shipped them. Re-enabling sync was supposed to be the end of it.

Re-enable failed again. The download line in the log was explicit: 0 new entries, cursor already at head. Pending stayed large. Swap climbed. The process never logged “beginning upload.” So we wrote a second story.

Diagnosis 2: the pending outbox. If download is idle and pending is fat, the engine must be materializing thousands of multi-megabyte batches into RAM before seal. We bounded that path too: queue depth, entries per tick, bytes per tick. We stopped force-admitting an oversize head “so catch-up can progress” — that rule alone is a thrash vector. Those caps were necessary. They were still not the mainline cause of this re-enable failure.

What was actually wrong

When we finally logged the target list at the start of a cycle, the picture flipped. One personal log. Then roughly two dozen shared (org) logs after it — many of them stale dogfood registrations that shared a human-readable label but not an identity. Personal download finished in milliseconds. The first shared download alone drove resident set size into multi-gigabyte territory. Upload ran only after every target finished downloading. A few kilobytes of personal work never got a turn.

What we changed

A multi-device account is not one pipe. There is personal history, and there may be shared logs. Shared pull matters. Letting shared pull monopolize a tick does not.

The corrected tick:

1. Download personal — including the key check that has to pass before we append personal ciphertext.
2. Upload a bounded batch — entry and byte caps; never force-admit an object larger than the budget.
3. At most one scoped download after that. If the active shared-target count is absurdly high (we used >4 as a leak signal), skip scoped work for the tick and clean the registry offline.

We also put the process memory guard back to the ceiling we actually meant to ship (6 GiB on this machine). Mid-incident someone had raised it so restarts would stop. That does not fix thrash; it only delays noticing it. Verification was a multi-sample status window under that guard: last-success advancing, RSS under limit, no guard restart, swap not ballooning into the tens of gigabytes.

Stale registrations as input, not atmosphere

Having ~20 shared-log registrations under the same product label was bad hygiene. Leaving them active made every tick a fan-out. Deactivating them was correct ops.

It also exposed a missing invariant: the sync engine assumed a small target list. Unit tests use personal-only or one shared prefix. A long-lived node accumulates probes, abandoned workspaces, and labels that collide while the underlying ids do not. If work per tick is linear in active targets, the registry is part of the performance envelope — same as outbox depth or object size.

What not to do next time

• Do not only cap the phase the status summary names. Status said download idle + pending fat. The hang was between personal download and upload, on target N of many.
• Do not schedule all downloads before any upload when personal catch-up is what keeps the machine usable. Shared catch-up can round-robin across ticks.
• Do not force-admit oversize heads to “make progress.” Progress that thrash-kills the process is not progress.
• Do not raise the memory guard to clear restarts and call the incident closed. Prove under the limit you intend to keep.
• Do not treat registry cardinality as configuration trivia. Log it. Cap per-tick work. Deactivate leaks.

Earlier sync postmortem: Anatomy of a Sync Outage · related mis-diagnosis pattern: The Fix Was Subtraction

[← Blog]