← Blog

We Deleted the Molecules

Until the end of October, LastDB stored one record as about ten keys: an atom for each field value, a counter for each atom, and a tip for each field. On October 31 we made a record one key with one value, and within a day the old machinery was removed. On the same generated workload the database came out about 40% smaller at the switch, and p95 latency on cold batch reads and batch writes fell by 70% or more in our local comparison. This post is the before and after, with the CI runs behind every number and a list of what does not match.

The old shape

LastDB’s first storage model was built for history. A field’s value was an atom: immutable, content-addressed, never edited in place. A molecule was the index of one field. Its entries, the tips, said which atom was current for which key. A row was never stored as a row. It was assembled at read time from the current atom of each of its fields.

That model is what the July posts describe. In Thin Tips and Honest History we wrote about the per-field tip and its version chain. Last Store counted what a node has to hold: tips, atoms, indexes, sync planes. Those posts are accurate for the formats they describe, formats 4 to 6. The current code no longer contains that layout.

What it cost

We counted keys on a generated benchmark home: 10,000 records, four fields each. The node’s own inventory reported 100,015 keys in the main store: 30,001 atom values, 40,000 per-key molecule entries (the tips), and 30,001 atom counters. That is three atoms, four tips, and three counters for each record. The node’s schema report on a format 7 home of the same records says 10,000 records and no atoms.

BEFORE · FORMATS 4–6 AFTER · FORMAT 7 4 TIPS one per field 3 ATOMS immutable values 3 COUNTERS one per atom 10 KEYS PER RECORD RECORD every field in one value 1 KEY PER RECORD
Fig. 1 — The keys behind one four-field record, counted from the node's inventory on a generated home. The count is for this benchmark record; a record with other fields has its own count.

Ten keys is not just ten writes. Counters decided when an atom, a molecule or a file could leave the store, so there was a drain that removed dead molecules and a planner for the atom counters. A read had to find the tip, then the atom, for each field, then assemble the row. The cost showed up in the places we kept tuning. Checking Every Shelf was mostly about making molecule-wide walks cheaper. This change makes the thing being walked smaller.

The change

A record is one key and one value. All of its fields are in the value, under one key per primary key. A write replaces the value. A read fetches it. At the switch, peer sync merged a record field by field, so two devices that changed different fields of the same record could still agree.

The in-memory graph from Memory First, Disk Later also changed shape. Its unit used to be an atom per field; now it is the record, and a warm entry means one record key whatever the fields.

The work landed as a numbered series of pull requests, from the new key space to the removal of the old code. The record key space and the new home format merged on October 31 (#911). The same day, every new home became format 7 (#929). Over October 31 and November 1 the molecule drain, the counters, the atom readers and the atom and molecule types were removed, finishing with a sweep of leftovers (#989). There is no migration. A new home is format 7, and an older home moves by one copy of its records through the program that wrote it.

What it did to size

Our CI has a growth lane that seeds 512 records, runs three rounds of updates, then inserts up to 2,048 records, and reports the compacted database size per live record. A single CI run, run 41254200450, measured format 6 against format 7 on the same lane. The baseline is 6d5362f3, the last commit that still created format 6 homes; the candidate is the pull request that made the benchmark accept a format 7 home.

  • Read-only seed, 512 records: 1,132 bytes per record → 673 (−40.5%).
  • After three update rounds: 1,374 → 913 (−33.6%).
  • Insert, 2,048 records: 1,188 → 793 (−33.3%).

We repeated the comparison locally on a probe that writes the same generated records in batches of 128, updates all of them, and measures the database after a clean flush. At 10,000 records the database went from 11.19 MiB to 6.68 MiB (−40.3%). At 50,000 records it went from 55.73 MiB to 32.17 MiB (−42.3%). That is 1,174 bytes per record down to 701, and 1,169 down to 675. The two sources agree on the range, about a third to two fifths.

Where the size ended up

The format number has moved a long way since format 7. The benchmark homes are format 15 today. On the same growth lane, the latest CI run on main (run 44196800207) reports 612, 620 and 608 bytes per live record for the same three cases, against format 6’s 1,132, 1,374 and 1,188. The nightly run (run 44056084661) reports 608 bytes per live record at 18,432 live records. Against format 6, that is 46% to 55% fewer bytes per record.

We do not credit all of that to the October 31 switch. The switch itself took about a third to two fifths off. The rest comes from later format changes and other work, and we have not separated them.

What it did to latency

For latency we ran the project’s own benchmark harness on a format 6 build and a format 7 build, five alternating pairs, 10,000 records, 256 samples per case. These are p95 medians across the five pairs, in milliseconds, format 6 to format 7:

P95 LATENCY · FORMAT 6 (HATCH) VS FORMAT 7 (OUTLINE) COLD-READ/32 17.99 ms 2.44 ms INSERT/128 98.99 ms 29.51 ms UPDATE/128 112.34 ms 31.50 ms CONCURRENT-READ/32 · FLAT 13.42 → 13.23 ms
Fig. 2 — p95 latency, drawn to scale within each row. Reads that are already warm and concurrent did not move.
  • Cold batch read (32 records): 17.99 → 2.44. Throughput on that case went from 64.7 to 455.2 requests per second.
  • Batch insert (128 records): 98.99 → 29.51.
  • Batch update (128 records): 112.34 → 31.50.
  • Single insert and single update: 6.59 → 4.14 and 7.28 → 4.40.
  • Sixteen concurrent writers: 67.89 → 23.17.
  • Concurrent reads: about flat. Sixteen readers went from 7.18 to 6.81 and thirty-two from 13.42 to 13.23.

The CI run above saw the same direction with a wider and noisier gap: cold batch read 14.52 → 1.60, batch insert 193.53 → 90.28, and batch update 283.75 → 67.98, from two pairs on a hosted runner. Warm concurrent reads did not get faster, and we are not going to say they did. The win is in writes and in the first read of a record, which is where assembling a row from atoms used to cost.

Where it ended up

The same harness runs on every push to main. On three runs from November 17 (44196800207, 44183519720 and 44161789187), p95 sits at:

  • Cold batch read (32): about 1.5 ms (1.47 in the latest run, 1.47 to 1.69 across the three).
  • Batch insert (128): about 12 to 17 ms (14.73 in the latest run).
  • Batch update (128): about 16 ms in the latest run, with one of the three runs at 22.25.

As with size, this is not the switch alone. It is the switch plus what built on it.1

What we gave up

  • A smaller value limit. An atom could carry up to 256 KiB by default. A record is capped at 64 KiB. An over-cap write is refused with a typed error, not truncated.
  • Aggregates. We removed the aggregate write side with the molecules rather than port it.
  • In-place upgrade. There is no migration of an older home. Moving one is a copy.

How we measured, and what does not match

  • Dev-release builds. Every build in this post, ours and CI’s, uses the dev-release profile. These are not tuned release binaries.
  • Two machines. The local comparison ran on a shared eight-vCPU box, five alternating pairs, medians. CI runs on hosted runners with two pairs. Use the ratios. The absolute times are not portable between the two, and we have not compared them.
  • The before is a commit, not a flag. The format 6 baseline 6d5362f3 already contains the first record code, but still creates format 6 homes. The record build is the first commit where every new home is format 7.
  • The workload moved after October 31. The benchmark has since added a spread seed and cold-read cases over it, writes whole records, and reads each stored record back. A read change that shares opened run readers landed on November 16. So today’s cold-read time is not attributable to the record switch alone, and we compare today’s figures to format 6 only for size, with that caveat.
  • Synthetic data. The workload is generated records with a 512-byte hex body. It is not production data. We have no production numbers for this change and are not claiming any.
  • Noisy cells are left out. Concurrent writes at 32 clients flipped between runs, including one where the new build looked slower, so we do not quote that case in either direction. The concurrent-write numbers above are for 16 writers and fewer, from the five-pair run.

1 The “where it ended up” figures are from CI runs on main on 2026-11-17, before lastdb #1757 merged at 4:00 PM Pacific the same day. That change moved the benchmark scripts onto /v1/read and /v1/write, and its own description says measurements before and after it are not comparable. The figures in this post are all from before it. The format 6 and format 7 comparison, which is the heart of the post, is unaffected, because both sides ran the old scripts.

Related: Thin Tips and Honest History · Last Store · Checking Every Shelf · Memory First, Disk Later · Blog index

← Blog