Leaderless Mesh: How CameoDB Scales Without Consensus

The Consensus Problem

Most distributed databases rely on consensus algorithms like Raft or Paxos. These protocols ensure consistency by electing a leader, replicating writes, and requiring majority agreement before commits. The tradeoff, complexity and latency. Every write must travel through the leader, wait for quorum acknowledgment, and handle leader elections during failures. As clusters grow, consensus overhead increases, and scaling becomes painful.

CameoDB takes a different approach, a leaderless mesh. No single point of failure, no leader elections, no quorum waits. Instead, data distribution and routing are handled through consistent hashing and a Kademlia DHT. Each node knows where data belongs, and operations route directly to the owner. The result, linear scalability with minimal coordination overhead.

Consistent Hashing: Data Distribution Without Coordination

At the heart of CameoDB's distribution is a consistent hash ring. Each node is assigned virtual node tokens, and each shard maps to a specific position on the ring. When you write a document with a routing key, the system hashes the key using XXH3, finds the position on the ring, and determines which node owns that shard.

// Routing decision algorithm
hash = xxh3_64(routing_key)
shard_id = ConsistentRing.get_owner(hash)
node_id = shard_assignments[shard_id].owner
decision = if node_id == local_node
 RoutingDecision::Local
 else
 RoutingDecision::Remote { node_id, peer_addr }

This means deterministic routing without coordination. The same routing key always maps to the same shard and node, regardless of which node receives the request. Add a new node to the cluster, and the ring rebalances automatically. Remove a node, and data redistributes to neighbors. No central coordinator required.

Kademlia DHT: Peer Discovery Without Configuration

How do nodes find each other? CameoDB uses a Kademlia DHT built on libp2p. Each node maintains a routing table of known peers, organized by XOR distance from its own node ID. When a node joins, it contacts bootstrap nodes and discovers the network through iterative lookups.

The DHT provides three critical capabilities:

Peer Discovery

Automatically find and connect to cluster nodes without manual configuration.

Routing Metadata

Distribute shard ownership information across the cluster for decentralized routing.

Self-Healing

Detect node failures and automatically rebalance data without human intervention.

Actor-Based Remote Execution

CameoDB uses the Kameo actor framework for distributed execution. Each node runs a NodeOrchestrator actor that manages local shards, and these actors can communicate remotely over libp2p. When the router decides an operation belongs on a remote node, it uses Kameo's remote messaging to forward the request.

// Remote call path
orchestrator_name = format!("orchestrator-{}", target_node_id)
remote_ref = RemoteActorRef::lookup(orchestrator_name).await
result = remote_ref.ask(&ClientOp::Search { ... }).await

Remote actors are registered with stable names like orchestrator-{node_id} and shard-{shard_id}, making them discoverable across the cluster. The same ClientOp message type is used locally and remotely, so operation semantics are consistent regardless of where they execute.

Scatter-Gather: Fan-Out Without Bottlenecks

For queries without a routing key, CameoDB uses scatter-gather broadcast. The router asks the cluster coordinator for known peers, selects up to a fanout limit, and executes the query in parallel across local and remote nodes.

// Broadcast search algorithm
peers = coordinator.get_known_peers()
selected = peers.take(broadcast_fanout_limit)

local_result = handle_client_op(op) // Local search
remote_results = parallel(selected, |peer| {
 remote.ask(peer, op, timeout)
}) // Fan-out to remotes

merged = top_k_merge(local_result, remote_results, limit)

Results are merged using score-aware top-K aggregation, allowing higher-scoring remote hits to displace weaker local results. Bounded concurrency ensures the system doesn't overwhelm itself with parallel requests, and per-call timeouts prevent stragglers from blocking the response.

Event-Driven State Management

Cluster metadata is persisted without polling or background tasks. All state transitions occur on membership events, PeerDiscovered when a node joins, PeerLost when a node leaves. The cluster coordinator maintains a simple state machine with three states, Active (all nodes present), Degraded (some nodes missing), Failed (below quorum).

Metadata is written to metadata.redb inline with state changes, providing crash-safe persistence without separate background processes. On boot, nodes load the persisted snapshot and reconcile with actual cluster state, logging discrepancies for operational visibility.

The Tradeoffs

Leaderless architecture isn't magic. It makes different tradeoffs than consensus-based systems:

Advantages

• No leader election latency
• No single point of failure
• Linear scalability
• Simpler operational model
• Better write throughput

Considerations

• Eventual consistency for some operations
• Requires careful key design
• Network partitions handled via timeout
• Read-your-writes requires routing keys

The Takeaway

CameoDB's leaderless mesh trades consensus complexity for deterministic routing and automatic distribution. Consistent hashing ensures data lands in predictable places, Kademlia DHT handles peer discovery without configuration, and actor-based remote execution provides clean distributed semantics. The result is a database that scales linearly, handles failures gracefully, and keeps operational complexity low. No Raft, no Paxos, just math and networking doing what they do best.