The problem: memory tracks documents, not traffic
Our production cluster holds about 6.5 million documents, mostly UK Companies House records and sales leads. The second-region replica, region-b, serves almost no traffic, but it was holding 25 GiB. Almost all of that was heap, not cache the kernel could take back.
That told us something useful: queries weren't the cost. Every document lives in RAM for the life of the process, so idle memory was about 4.3 KB per document, and it would keep growing with the data.
Measure first
We took a 1-in-1000 sample of prod documents (8,317 of them) and loaded them into a fresh index under a test allocator that counts live bytes. An average document has 370 bytes of text, 341 bytes of JSON metadata and about 45 distinct search terms. Then we asked where the bytes went:
| Part | Bytes per doc | Notes |
|---|---|---|
| Vector | 772 | Already int8 (768 dims plus a scale). 4× smaller than f32. |
| Document object | ~1,800–2,400 | Text, a parsed JSON metadata tree and owned strings. The biggest item. |
| Keyword index (BM25) | ~800–1,200 | 16-byte postings, about 45 per doc, in lists that grow by doubling. |
| Vector graph (HNSW) | ~200–250 + slack | Per-node neighbour lists, which also grow by doubling. |
| Key maps | ~200 | Copies of the bucket name and id for every doc. |
Change 1: store documents compactly (−647 B per doc)
Metadata was held as a serde_json::Value tree, with a map node and two heap allocations per field. That's several times the size of the 341-byte JSON text it came from. Now each document keeps its metadata as that compact JSON text and parses it only when something needs the value: the top-k hits of a search, a document fetch or an export.
Two smaller fixes went with it. Every document in a bucket now shares one copy of the bucket name instead of carrying its own "companies_house". And immutable strings dropped their spare capacity.
The public API didn't change, and neither did the snapshot file format. The snapshot writer now copies the stored JSON straight to disk instead of re-serialising every document. A byte-for-byte test makes sure old snapshot files still load.
Result: 3,418 → 2,771 bytes per doc (−19%). PR #166
Change 2: one key index instead of two (−100 B per doc)
Lookups by (bucket, id) went through a flat map keyed by two owned strings, and a second map kept a set of ids per bucket. We merged them into one nested map: bucket → id → internal id. Each bucket name is now stored once, the extra set is gone, and lookups no longer allocate a key on every call.
A side benefit: about a dozen hand-written "keep the key map in sync" lines are gone. One function now owns the key index, so it can't drift away from the documents.
Result: 2,771 → 2,671 bytes per doc. PR #167
Change 3: give back the slack (−478 B per doc)
Rust's Vec doubles its capacity when it fills, and it never shrinks on its own. The vector graph has one small neighbour list per node, and the keyword index has one posting list per term. That's millions of small lists, and on a long-running node many of them are half empty: a node that briefly held 33 links keeps room for 64.
The new trim pass shrinks those lists to fit and drops postings that lazy deletion left behind. It works in batches of 20,000 and releases the lock between batches, so a request waits at most a few milliseconds and the trim can run on a node that's serving traffic. Search results are bit-for-bit identical, which the tests check.
A measurement trap we nearly fell into. Our first trim also shrank the big flat arrays, and the counter said it saved 1,178 B per doc. But a byte counter can't tell resident memory from address space. The unused tail of a large allocation is never written, so the OS never backs it with memory. Shrinking it saves nothing real and copies gigabytes while holding a lock. We dropped that part and measured again. The honest number is 478 B per doc.
Result: 2,754 → 2,276 bytes per doc on a node that had seen churn. PR #168
One flag: NEBULA_MEMORY_PROFILE
| Profile | Allocator | Trim |
|---|---|---|
small (default) | jemalloc returns freed pages to the OS after about 1 s. A background thread purges, so an idle node shrinks too. | After boot recovery, then every 15 minutes |
fast | Freed pages kept about 10 s for reuse, so fewer page faults during heavy ingest | Off |
For fine-tuning, NEBULA_MEMORY_TRIM_SECS sets the trim interval (0 turns it off) and NEBULA_MEMORY_TRIM_BATCH sets how much work happens per lock hold. The allocator settings are applied at runtime through jemalloc's mallctl interface, so nothing has to be set before the process starts.
server:
env:
NEBULA_MEMORY_PROFILE: "small" # or "fast" for a bulk-ingest node
What production shows
Only the first change is in production so far. After we deployed it and re-loaded region-b from the leader's snapshot:
| Pod | Before | After |
|---|---|---|
| region-b | 25.0 GiB · 6.22M docs · 4.3 KB/doc | 21.0 GiB · 6.59M docs · 3.4 KB/doc (−21%) |
| leader | 47.7–50.4 GiB · 6.39M docs | 35.8 GiB · 6.75M docs |
Region-b is the clean comparison: an idle node, the same measure before and after, and a 21% drop per document. That matches the 19% the test predicted. The leader's drop is bigger, but don't read it as all from this change. It was also restarted (clearing four days of allocator fragmentation), and it's still running a background re-embedding job.
Why it's still hungrier than Couchbase
Couchbase gives each bucket a RAM quota. Hot data stays in memory, colder data is evicted to disk, and a full quota means slower reads, not a crash. Most search engines also keep their indexes on disk and let the OS cache them. NebulaDB has no disk tier yet. Every document, every posting and every vector is held in RAM, and the pod limit is the only ceiling.
Only about 1 KB per document (the vector and its graph links) really needs to be in RAM for fast search. The rest is a design choice we can revisit. Next, biggest saving first:
- Documents on disk: about −800 B per doc. Keep the document body in an embedded store with a hot cache, since search only needs it for the final hits (design 0014).
- Compressed postings: about −500 B per doc. Delta-encoded ids instead of 16-byte postings.
- Flat graph storage: one array instead of millions of small lists.
- A memory quota: slow down instead of being OOM-killed.
Together these should land around 1.2 KB per document, about 8 GB for today's corpus against the 25 GB we started from. We'll measure each one on real data, the same way, before we claim it.
What we'd tell another team
- Find the idle baseline. A replica that serves no traffic shows you what the data costs on its own.
- Measure on a real sample, not synthetic data. Templated text and metadata trees behave nothing like random strings.
- Know what your counter counts. Requested bytes aren't resident bytes. We nearly shipped a 2.5× overstatement.
- Keep formats stable. Every change here kept the snapshot format, with a test to prove it, so rollback was always a plain redeploy.
Full numbers, the runbook and the open work are in the memory technote.