Checking Every Shelf

2026-07-29

Our own board app and knowledge notes — the apps we dogfood on LastDB every day — got slow. Listing a column could take many seconds. Opening one card could take longer than a coffee pour. The first guess was fashionable: maybe vector search left the database, and now every lookup was doing something expensive and semantic. That guess was wrong. The bug was simpler, meaner, and already sitting in the layout we thought we had fixed.

This is a write-up of what the telemetry said, what one instrumented read proved, what we shipped, and the numbers after it landed on a real primary node. It is a process story about access patterns — not a dump of internal key formats or how to reimplement the engine.

What status looked like

LastDB can report request ops: who called, which kind of work (query vs mutation vs search), how long it took, and a rough cost signal for cold storage work. On a busy window the board app’s query path sat around four seconds average, with a p95 near ten seconds and a max past a minute. Knowledge reads were better on average but still multi-second at the tail. Cold shard loads per call on the main card shape were on the order of a thousand-plus — which, on a 1,024-group layout, is not a keyed read. It is a tour of the building.

Semantic search showed up in the tables as its own verb when we used it. It was not the top offender. Ordinary list and show were.

If the average is seconds and loads-per-call ≈ group count, you are sweeping

Warm caches can mask this until thrash fills the budget. Then every call evicts the last call’s groups, and the tour becomes the steady state. Relayout alone does not help a walk that never qualifies for pruning.

What one cheap read actually did

We boot a throwaway node on a copy of real data (never the live home as the first place a candidate fails). We turn on a gated sweep tracer that only prints when a prefix walk falls back to “visit every group.” Then we ask for the cheapest possible board operation that still returns one card.

The tracer did not whisper. It shouted: hundreds of full-collection sweeps for a single show, across several collections the logical store still has to consider. The fallback reason was always the same idea: the key shape carrying the walk had no partition separator. Under a partition-prefix layout, pruning only kicks in when the prefix already names a shelf. Otherwise the engine is correct to check every bucket — and slow for the same reason.

Fig. 1 — One app asks for one row. Without a shelf label on the scan prefix, the store visits the whole hash-group grid.

Three hot shapes kept showing up. Molecule-wide record walks. Append-log rebuilds that scanned an order prefix even when the count said zero. And a “does this field have unresolved conflicts?” annotation that prefix-scanned on every result field. The third one is easy to miss: it is metadata, not the user’s row, and it still paid warehouse rent.

What we changed

We did not invent a new storage engine. We stopped asking unprunable questions on the hot path.

Paged one-dimensional reads — where records already share a partition, scan the anchored prefix. Where they do not (hash-only cards), serve pages from a partition-pinned derived index and point-read authoritative rows for that page only.
Order logs — if a count key already knows how many entries exist, address them by sequence. Do not prefix-scan an empty log just to learn it is empty.
Conflict flags on query results — keep a small per-molecule list of unresolved conflict ids. Empty means skip. Non-empty means point-get those ids. A first-touch scan can self-heal the list so the second call is cheap.
Rebuild discipline — a page-index rebuild is a pure function of the molecule’s keys. Rebuilding at an unchanged header is waste; repairing a stale window gets a budget of one per header so one bad marker cannot turn every read into another full walk forever.

Fig. 2 — Same ask, with a partition-anchored path or index: only a small neighborhood of groups, then point-get the rows you need.

Before and after

Numbers from three places, same day: live request ops before the fix, CoW microbenches against a real-data copy, and the safe-upgrade latency bar (candidate binary vs then-live binary on identical copies). After cutover, live ops again.

Fig. 3 — Wall-clock shrinkage on the paths we timed. The walk got shorter; we did not invent a new database.

Rough table, honest about what each row measures:

Board-shaped list (scan path) — safe-upgrade bar: about 11s → 1.1s. Live list after cutover: under a second when warm.
Board query average / p95 — live thrash window: about 4s / 10s. Live after: tens of milliseconds / sub-second p95.
Knowledge query average / p95 — from about a second and multi-second tails to ~16ms / ~19ms on a busy healthy window after cutover.
Rich single-card read (CoW, same key) — ~8s and ~40 full-collection sweeps every call on the old binary; after the fix, one self-heal pass then ~30ms and essentially no sweep thrash. Live warm after cutover: about 0.1s.

Cold starts still pay hydration. That is not a regression of the fix; it is physics. The regression was paying full-grid walks on every warm call forever.

What we almost learned wrong

Do not blame the feature that is not in the ops table. Search was not the top row. Query was.
Do not treat “layout is correct” as “reads use it.” Partition-prefix placement was already live. The hot prefixes never contained the separator that pruning requires.
Do not only fix the first named offender. An earlier pass fixed part of the page-index story. Molecule-wide record prefixes, empty order scans, and conflict annotation still walked the warehouse. Done on the board is not done in the store.
Measure with the cost counters that cannot lie about visits. Some key-index shortcuts answer a listing without counting as a cold load. Disable them when proving a walk is gone.

The short version

A database that spreads data across many groups is only as fast as the keys you scan with. If the scan key has no shelf label, you check every shelf. We stopped scanning that way on the paths our own apps hit every minute. The board got snappy again. Vector search was a red herring.

[← Blog]