Search Everything: Full-Text Indexing with Tantivy Under the Hood

The Inverted Index Advantage

Traditional databases struggle with text search. A LIKE query with wildcards scans every row, O(n) complexity that destroys performance as data grows. Full-text search engines use inverted indexes, mapping terms to document IDs for O(log n) lookups regardless of dataset size.

CameoDB integrates Tantivy, a Rust-based search library inspired by Lucene. Tantivy provides inverted indexes, term dictionaries, positional data for phrase queries, and BM25 relevance scoring. Combined with redb's ACID KV storage, CameoDB delivers hybrid search, fast point lookups via redb, rich full-text queries via Tantivy.

Index-Only Storage Strategy

Most search engines duplicate data, storing complete documents in the index for retrieval. This bloats index size and slows writes. CameoDB takes a different approach, index-only storage.

// Tantivy stores only indexed fields (no STORED flag)
schema = SchemaBuilder::new()
 .add_text_field("title", TEXT | STORED)
 .add_text_field("content", TEXT) // No STORED
 .build()

// redb stores complete JSON documents
redb.insert("doc:123", full_json_document)

Tantivy stores only the id field and indexed fields without the STORED flag. Complete JSON documents live exclusively in redb. When you search, Tantivy returns matching document IDs and scores, then we batch-fetch the full documents from redb. This split-storage strategy means smaller indices, faster search performance, and zero data duplication.

Field-Level Control and Schema Evolution

CameoDB provides fine-grained control over field indexing through schema configuration. Each field can be configured with:

INDEXED

Enables range queries, filtering, and term lookups on the field.

FAST

Enables sorting and aggregations on the field value.

STORED

Stores the field value in Tantivy for direct retrieval (rarely used).

TEXT

Tokenizes and indexes for full-text search with stemming.

Schema evolution is automatic. When new fields appear in incoming documents, CameoDB infers their types and adds them to the schema. Field fingerprints track schema changes, enabling cache invalidation and distributed synchronization without manual schema migrations.

Query Capabilities: Beyond Simple Search

Tantivy supports sophisticated query patterns that go beyond keyword matching:

// Phrase queries with proximity
query = "database systems"~2 // Within 2 words

// Range queries on numeric fields
query = price:[100 TO 500]

// Boolean combinations
query = (title:rust OR title:go) AND year:[2020 TO 2024]

// Boosting for relevance tuning
query = title:rust^3 OR body:rust

Phrase queries use positional indexes to find terms in specific order with configurable proximity slop. Range queries work on numeric and date fields. Boolean operators enable complex logic with AND, OR, NOT. Field boosting allows you to weight certain fields higher in relevance scoring.

BM25 Relevance Scoring

Tantivy uses BM25, the industry-standard relevance scoring algorithm. BM25 considers term frequency (how often terms appear in a document), inverse document frequency (how rare terms are across the corpus), and document length normalization. This means documents matching rare terms score higher, and longer documents aren't unfairly penalized.

CameoDB returns scores alongside search results, enabling your application to implement custom ranking logic, result re-sorting, or confidence thresholds for filtering low-quality matches.

Segment Merging and Performance

Tantivy stores indexes in immutable segments. As documents are added, new segments are created. Periodic merging combines smaller segments into larger ones, reducing the number of segments that must be searched during queries. This improves read performance while maintaining write throughput.

CameoDB's Supervised Smart Commits control when segments are committed to disk. High-volume workloads trigger smart commits at adaptive thresholds (500-8000 operations), while low-volume patterns use supervised commits after 5 seconds of inactivity. This balances performance with durability guarantees.

The Takeaway

Tantivy provides the full-text search engine that powers CameoDB's hybrid architecture. The index-only storage strategy keeps indices lean while redb provides fast document retrieval. Schema evolution, phrase queries, BM25 scoring, and segment merging deliver production-grade search capabilities without the complexity of managing a separate search service. The result is rich search integrated with ACID-compliant storage.