CameoDB v0.3.5: Commits on a Clock and 370,000 Docs a Second

Commits on a Clock

A commit is what makes a write visible to search. In 0.3.5 the search index commits on a steady clock: once the oldest uncommitted write has waited commit_interval_ms while writes keep arriving, and once more when they settle. The shard writer answers its callers before it commits, so no write ever waits on the indexer, and every write is searchable after the next commit.

The effect on ingest is the headline of this release. Measured on the release build with four shards and wal_sync on, 500-document batches at concurrency 16 went from 19,700 to 130,000 documents a second, and the median request from 420 ms to 53 ms. 5,000-document batches at concurrency 32 went from 184,000 to 328,000. Across every shape we measured, bulk ingest is 1.8 to 6.6 times faster.

[search]
commit_interval_ms = 2000   # the clock; on by default

370,000 Documents a Second, Durably

At saturation a single node sustains 320,000 to 370,000 documents a second, and the best run reached 370,932 with fsync on every commit. With fsync off it measured 365,382, the same within run-to-run noise. On bulk ingest, durability is free.

The node reaches that plateau using about a quarter of its cores: 3.9 of 15 on an Apple M5 Pro. Ingest is shaped by documents in flight rather than processor time, which makes capacity planning refreshingly simple: send fewer, larger batches. Eight connections sending 50,000-document batches reach the plateau with a third of the latency of 32 connections sending 20,000.

A cluster speeds up as well. Each node now serves its peers' share of a bulk write, and its half of a peer's search, on a lane of their own beside the worker pool. Bulk writes sent through every node at once run 2.0 to 2.9 times faster, with p99 down to 40 to 70 ms.

Searches That Scale With Every Core

search_threads now follows the cores the process is given, and in a container it follows the CPU quota. Searches are CPU-bound and nothing else in the read path serializes them, so they scale with every thread you give them: on 15 cores, 2, 6 and 12 threads served 1,900, 6,250 and 11,600 searches a second.

Reads got leaner too. Hits are read straight through redb's own page cache with no extra layer on top, and a scanning read is 35% faster for it.

Room for Many Teams on One Node

A key can now carry a tenant. Every index that key creates belongs to the tenant, and [security.tenants] sets how many indexes and how many bytes the tenant may hold. Several keys can share one tenant and its quota, and ownership is stamped at creation, so it stays put.

index_overrides gives a key a narrower role on named indexes: a writer that should only read the audit index keeps one key instead of two. The rule applies on the HTTP API and over MCP alike.

[[security.api_keys]]
key_hash_file = "/etc/cameodb/keys/acme"
role = "writer"
label = "acme-ingest"
tenant = "acme"
index_overrides = { audit = "reader" }

[security.tenants.acme]
max_indexes = 20
max_bytes = 53687091200          # 50 GiB

[security.limits]
write_documents_per_minute = 5000
write_burst = 20000

Writes are metered in documents rather than requests, so a fair share means the same thing for a single write and a bulk load of thousands. Callers without a key each get their own bucket, per IPv4 address or IPv6 /64. And max_open_indexes keeps the set of open indexes bounded: the least recently used one is committed and closed, and reopens on its next request, so a node can host far more indexes than it keeps warm.

Queries That Stay Quick

Some queries cost far more than they look. 0.3.5 keeps them quick and tells you exactly what it did.

Prefixes have a floor. min_prefix_length defaults to 2. On a 10-million-document shard, a two-character prefix on a hash field runs in about 11 ms, and three characters in under a millisecond.

A bare word has a reach. A term with no field searches up to max_default_fields fields, 64 by default, and an index can name its own default_fields. When the cap narrows a search, the answer lists the fields it reached in _narrowed_default_fields.

A bare prefix just works. qui* with no field now searches the default fields, one prefix per field, ORed together.

Every wildcard is accounted for. A form the grammar cannot use, such as a leading wildcard, is set aside and named in _discarded_clauses with the form that works, and the rest of the query runs.

Dates arrive in more shapes, too: 03/15/2024, 15.03.2024, Mar 15, 2024, 15 March 2024 and RFC 2822, beside ISO, epoch and the rest. data load reads each date column in the order it is written.

Steady Under Load

Under heavy load a node answers the requests it can serve and turns the rest away at the door, with a 503 and a Retry-After the client can act on, before the request has cost anything. Admission predicts the wait ahead of each arrival from a measured spread rather than a mean, as depth × mean + 3σ × √depth, because the spread of a sum grows with the square root of its depth. Every sample is recorded with a single atomic add.

Pushed to twice its capacity in bulk writes, a node keeps a steady 57 requests a second flowing with no timeouts. Inside its capacity it sheds nothing and answers searches in 5.71 ms at p99. /_cluster/health reports queue_depth, predicted_wait_ms and mailbox_service_p90_ms, so you can see at a glance whether a node is busy or slow, and cameodb check-config checks your rate and overload settings before you start.

Get It

Everything in this post ships in CameoDB 0.3.7, the current release, along with a Windows service, disk reclaim after deletes and searches that fetch only the page they return. One binary, nothing else to install.