[← Blog]

The Backup That Wouldn’t Commit

2026-08-04

One job. Make the laptop’s database restorable from the cloud. We re-enabled backup, watched every sealed piece finish uploading, and still woke up to durability: degraded — last good commit days old. The files were in the warehouse. Nobody had stamped which set was the official tip.

This is that story in order: what backup actually means here, what we saw, the one server bug that caused it, the deploy mistake that almost hid the fix, and the short checklist we are keeping.

What “backed up” means here

Our cloud backup for a LastDB home is not a stream of every keystroke. It freezes the sealed store into content-addressed chunks, uploads them, then updates a single cloud pointer — the official tip — that says “this generation is what you restore.” That tip write is a compare-and-swap so two machines cannot clobber each other.

Until the tip moves, you have paid for boxes in a warehouse with no shipping label. Status durability ages the tip, not the last successful chunk put. So a full warehouse and a stale tip can coexist.

CLOUD BACKUP HAS TWO LAYERS SEALED CHUNKS payload in object storage 13,567 / 13,567 PRESENT BYTES IN THE WAREHOUSE PIN LATEST official tip pointer MANIFEST COUNTER STUCK FULL WAREHOUSE · MISSING LABEL = NOT A RESTORABLE BACKUP
Fig. 1 — Chunks in the warehouse; official tip still missing

What we saw

After we turned cloud backup back on, the node did the hard part. Chunks present hit chunks total — 13,567 / 13,567. Cycles reported already-present, zero new bytes. And every fifteen minutes or so the tip step failed the same way: the cloud API refused to flip latest, with a calm message about personal backups only.

The machine was not offline. Sync looked fine. The only honest line on the dashboard was durability getting older. We were not failing to upload. We were failing to commit.

Why the tip refused

We had moved cloud object roots from the account principal to the identity of the database home — the right long-term shape. Clients started sending that new scope on every backup call, including the tip write.

On the server, three revisions of the storage function lined up like this:

  1. Before — tip only for the old personal shape.
  2. Middle (the landmine) — chunk paths accept the new database root; the tip path still treats that root as forbidden and returns a validation error.
  3. After (two hours later in git) — tip resolves the new root the same way chunks do.

Production was still on the middle revision. The fix had merged the same night and never shipped. Clients were already on the new world. Chunks uploaded under the new root. Tip updates were rejected. Half a migration: write path yes, commit path no.

ONE MIGRATION, THREE REVISIONS REV A old roots EARLIER REV B new root for chunks PIN REJECTS NEW ROOT DEPLOYED TO PROD and left there REV C pin accepts root MERGED +2H not deployed SELF-INCONSISTENT WINDOW · CLIENT ALREADY ON C
Fig. 2 — The middle revision shipped; the fix stayed in main

The dangerous deploy

Not always the one that breaks uploads. The one that half-adopts a new identity: objects land under the new root, the official tip still enforces the old rules. Busy looks healthy until you ask whether restore has a label.

Status made it quieter than it should have been. A successful chunk-only cycle could clear the failure streak from the tip refusal. The line that should print FAILING with a cause was gated off once the cut looked complete. Durability only said how old the tip was, not why it would not move. Correct numbers, wrong surface — a pattern we keep relearning.

We shipped the fix — traffic did not

So we built the storage service from current main, published a new version, and pointed the live alias at it. The tip still failed.

The alias said function version equals new. The canary weight said something else: an extra weight of 1.0 still sent one hundred percent of traffic to the old build. We had published. We had not served.

ALIAS VS TRAFFIC ALIAS “LIVE” function version = NEW looks promoted CANARY WEIGHT OLD = 1.0 100% still on old NODE OLD CODE rejects the pin NEW CODE 0% traffic PUBLISHED ≠ SERVING
Fig. 3 — Alias looks new; canary still routes everything to old

Clearing the weight (and restoring the gateway permission a clumsy alias recreate had dropped) put all traffic on the fixed build. The next forced snapshot returned ok. The tip counter advanced. Durability went from days-stale to minutes-fresh. Same warehouse — now with a label.

What we are keeping

Three inequalities, one product rule:

  • Uploaded ≠ committed. Celebrate when the official tip moves, not when the chunk bar hits 100%.
  • Merged ≠ deployed. A two-hour fix sitting only in git is how production freezes on the bad half of a migration.
  • Deployed ≠ serving. Alias version and canary weights are different knobs; promotion is not done until a real tip probe succeeds.

Same rule for status: a health counter is not delivered until the surface that should show FAILING can actually show it for the case you care about.

Anywhere you backup in two steps

If you upload objects and then flip a tip, instrument the tip. If you migrate write roots, ship the tip path in the same cutover. If you promote with an alias and a weight map, check both — then poke the real API.

We run LastDB on LastDB. Notes, the work board, and the rest of the stack share one home. When that home has no cloud tip, the laptop is still the only full copy. Getting the tip green was not a status-line vanity project. It was the difference between “the warehouse is full” and “we can restore.”

Related: The Thrash Was in the Order · Last Store · Blog index

[← Blog]