Redb + Tantivy: The Atomic Dual-Engine at CameoDB's Core

The Hybrid Storage Philosophy

Most databases choose one storage model, relational tables for ACID guarantees, or search engines for rich query capabilities. CameoDB combines both in a single atomic transaction, delivering ACID-compliant key-value storage alongside full-text search without sacrificing consistency or performance.

The hybrid architecture combines two complementary storage engines, redb for durable KV storage with ACID guarantees, and Tantivy for full-text inverted indexes. Every write operation touches both engines atomically, ensuring data consistency across storage and search.

Atomic Write Sequence

Each write operation follows a strict seven-step sequence to guarantee atomicity across both storage engines:

// 1. Generate sequence ID (AtomicU64, lock-free)
seq_id = counter.fetch_add(1)

// 2. Begin redb transaction (ACID isolation)
txn = kv.begin_write()

// 3. Write to WAL (durability before application)
wal.insert(seq_id, serialized_op)

// 4. Write to data table (complete JSON document)
data.insert(id, json_blob)

// 5. Update Tantivy index (in-memory buffer)
writer.add_document(id_only_doc)

// 6. Commit redb transaction (fsync if configured)
txn.commit()

// 7. Signal supervisor for smart commit
supervisor.reset_timer()

The sequence ID ensures monotonic ordering. The WAL (write-ahead log) guarantees durability before the operation is applied. The data table stores the complete JSON document. Tantivy receives only the indexed fields. The redb transaction commits with optional fsync for maximum durability. Finally, the supervisor is signaled for smart commit management.

ACID Guarantees Across Engines

CameoDB provides full ACID guarantees across the hybrid storage system:

Atomicity

All-or-nothing writes. If any step fails, the entire operation rolls back with no partial state.

Consistency

Both stores always reflect the same logical state. Invariants are maintained across KV and search.

Isolation

Concurrent operations don't interfere. redb's MVCC ensures safe parallel reads and writes.

Durability

WAL ensures operations survive crashes. Configurable fsync provides maximum durability guarantees.

Multi-Tenant Architecture

CameoDB supports multiple isolated indices within a single storage instance. Each index operates as an independent tenant with its own data tables, search indices, and sequence counters. This enables multi-tenancy without resource contention between indices.

Data isolation is complete, per-index data_{index} and wal_{index} tables in redb, per-index indices/{index_name}/ directories for Tantivy. Sequence counters are independent per index, enabling parallel write scaling without lock contention.

Recovery and Consistency

On startup, CameoDB performs consistent hybrid recovery. The system reads uncommitted records from the redb WAL and replays them into the Tantivy search index. This ensures that any operations committed to redb but not yet flushed to Tantivy are recovered, maintaining consistency between the two storage engines.

The recovery process is automatic and requires no manual intervention. It handles crash scenarios, power failures, and unclean shutdowns, guaranteeing that the search index eventually reflects all committed writes.

Performance Characteristics

The hybrid architecture delivers balanced performance across operations:

Point Queries

~0.1ms (redb B-tree)

Fast KV lookup by document ID

Range Queries

1-10ms (depends on range size)

Efficient B-tree traversal

Search Queries

10-100ms (Tantivy)

Depends on index size and complexity

Batch Writes

0.05-0.5ms per operation

10-60x faster than single ops

The Takeaway

CameoDB's hybrid storage architecture combines the strengths of two specialized engines. redb provides ACID-compliant KV storage with fast point lookups and range queries. Tantivy provides rich full-text search with inverted indexes and relevance scoring. The atomic write sequence guarantees consistency across both engines. The result is a database that delivers both transactional integrity and search capabilities in a single system, without the complexity of managing separate storage and search infrastructure.