CameoDB v0.3.10: Compressed Documents, CJK Search and Space Back After Deletes

CameoDB v0.3.10 stores documents compressed, searches Chinese, Japanese and Korean by word, gives deleted space back on its own and reports min and max over every match.

01 · ReleaseIn this post

Documents Stored Compressed

A document row in a shard is now LZ4-compressed against a small dictionary built from the index's own first documents, which fills as they are written and is then fixed. Each row records how much of the dictionary it was compressed against, so every row stays readable while the dictionary fills and nothing is ever re-encoded. There is nothing to configure.

On text indexes the rows take 29% and 61% of their former bytes, and a 3.0 GB shard of mostly synthetic benchmark documents rewrote to 310 MB in 9 seconds. Bulk ingestion on the benchmark rose from 32–33k to 38–39k documents a second, and single writes and search latency were unchanged.

Existing rows are read as they are and compressed when next written. cameodb compact --rewrite compresses every row of a stopped node's files in one pass, and --rewrite --dry-run estimates the result first.

When size matters more than speed, [storage] document_compression = "zstd" compresses against the same dictionary with zstd. On one shard's rows, prose of 2.7 KB a document is stored at 45.0% of its bytes with zstd against 61.0% with LZ4, and short text records at 25.2% against 29.6%. The setting decides writes only: a node reads both methods, an index may hold both, and changing it re-encodes nothing until you run a rewrite.

02 · ReleaseIn this post

Space Back After Deletes

A delete marks a document dead in its segment, and ordinary merging never rewrites a segment because of deletes alone. After a large delete the disk stayed taken until you called the merge endpoint. [search] reclaim_deleted_ratio, a number between 0 and 1, changes that: every segment whose deleted share passes it is rewritten on its own, one segment at a time, after the commit that made the deletes count. Each rewrite is logged as Reclaiming deleted documents. It is off by default, so a node that does not set it merges as before.

03 · ReleaseIn this post

Chinese, Japanese and Korean by Word

Every analyzer until now made one term of a whole clause between two punctuation marks, so only a query for exactly that clause matched. On a sample of classical Chinese books, 道 found 11% of the chunks holding it and 孔子 4%.

The new cjk tokenizer indexes each Han, kana or Hangul character alone and paired with the next. A query word becomes a phrase over them: a character matches wherever it stands, a word where its characters stand together. Latin words and numbers in the same text index as they do under default.

Three folding variants let the old and new spellings of a word find each other:

  • cjk_fold folds traditional Chinese to simplified first, so either script finds both. It is for Chinese only.
  • cjk_ja_fold folds old-style kanji to the post-war forms, so 淸吉 finds 清吉 and 讀書 finds 読書.
  • cjk_ko_fold reads each Hanja as its Hangul syllable, so 女子 finds 여자.
04 · ReleaseIn this post

A Whole Value in Any Case

raw keeps a value whole and keeps its case, and every analyzer that lowercases also splits into words. A field of names, codes or labels whose case varies had no exact match a reader could hit: Vlada Republike Hrvatske beside PREDSJEDNICA REPUBLIKE HRVATSKE, or RJEŠENJE where a reader types rjesenje. raw_fold keeps the value as one term, lowercased and with diacritics removed. Values, quoted values, prefixes, range bounds and IN sets are folded the same way, and hits return the value as written.

05 · ReleaseIn this post

Min and Max on Every Search

Add "metrics": {"min": [...], "max": [...]} to a search and the lowest and highest value of any i64, u64, f64 or date field over every document the query matches, not only the page, comes back as metrics beside total_hits. With "limit": 0 you get the count and the range alone.

Integers are exact at any size, and a date comes back as RFC 3339. A field no matching document holds is null, not 0. Results merge across shards and nodes, MCP search_index takes metrics, and the client has --min FIELD and --max FIELD.

06 · ReleaseIn this post

Agents Read Fields Better

MCP describe_index now describes each field by what a query can do with it: searchable, sortable and returned, plus a tokenizer_hints list that says once per tokenizer whether case counts, whether a word finds its other forms and whether an unaccented word finds an accented one. The string hint says plainly that the whole value is one term, case included. When a search matches nothing, the warning names each value that found nothing on its own and offers the spellings to retry in one IN set.

07 · ReleaseIn this post

Steadier Under Load

A flood of searches no longer shuts writes out at the door, nor a flood of writes searches: each may hold three quarters of max_concurrent_requests. With 16 permits, 64 search connections and 4 writers, 40,222 writes were refused in 15 seconds before and none after. A delete by query on an index being written to no longer looks every match up in the document store, and under a bulk load the commit interval decides the commits again, which measured 61–63k documents a second against 57–59k. On a cluster, a search an index cannot answer is refused instead of returned as a partial page.

08 · ReleaseIn this post

Upgrading

A node starts on your existing config, and every new setting is off until you set it. Four things deserve a read first:

  • There is no downgrade once 0.3.10 has written. 0.3.9 cannot read a compressed row, so keep a copy of the data directory if you need a way back.
  • An index that declares raw_fold, cjk, cjk_fold, cjk_ja_fold or cjk_ko_fold needs 0.3.10 on every node. Upgrade the whole cluster before declaring one.
  • describe_index no longer returns indexed, stored, fast or index_record_option. GET /api/{index}/_config still reports them.
  • A cluster that holds an index refuses a node it has never known. Grow a cluster before its first index, or move a node's disks to the new machine and it rejoins as itself.
09 · ReleaseIn this post

Get It

CameoDB 0.3.10 is the current release: one binary, nothing else to install, signed and checksummed for macOS, Linux and Windows, with deb, rpm and msi packages. The v0.3.9 post covers the OAuth sign-in, ghost fields and delete by query it builds on.