>_ FalkorDB-rs Dev Log

Part 8 of 8

Read your writes: opt-in replica routing via Sentinel

By Barak · · 3 min read

We made it to the finale. This one is short, but it's the post where a one-word default decision is actually a correctness decision in disguise — and getting it wrong is the kind of bug that doesn't show up until production, under load, intermittently. My favorite kind. (It is not my favorite kind.)

The setup

Behind Redis Sentinel, a FalkorDB deployment has a primary and one or more replicas. Reads can be served from replicas to spread load. The catch: a replica applies a write only after the primary has committed it. So a read from a replica can be slightly stale — it might not yet reflect a write you just made.

sequenceDiagram
    autonumber
    participant App
    participant C as falkordb client
    participant Sentinel
    participant P as primary
    participant Rep as replica
    C->>Sentinel: discover topology
    Sentinel-->>C: primary + replica addresses
    App->>C: query("CREATE ...")  (write)
    C->>P: always routed to the primary
    App->>C: ro_query(...).prefer_replica()
    C->>Rep: read served from a replica (may be slightly stale)
    App->>C: ro_query(...).primary_only()
    C->>P: read-your-writes — forced onto the primary
    Note over C,Rep: PreferReplica falls back to the primary<br/>when no replica is available
Writes to the primary; reads where you ask — replica or primary.

The decision: stale-by-accident is not allowed

Here's the part I want to defend, because it was a breaking change and I made people opt back in.

The client used to route read-only queries to replicas automatically when they were available. That's great for throughput and quietly terrible for correctness: the classic failure is "create a thing, then immediately read it back, and it's not there" — because the read went to a replica that hadn't caught up. That's a heisenbug. It reproduces under load, on a good day, once.

So the default flipped: reads go to the primary unless you explicitly opt in. You don't get replication lag unless you ask for it. Throughput is a knob you turn on purpose, with your eyes open — not a default that silently trades correctness for a benchmark number. Routing is now controllable at two levels:

  • Per client — with_read_preference(ReadPreference::PreferReplica) makes every read prefer a replica.
  • Per query — ro_query(..).prefer_replica() opts a single query in, and .primary_only() forces a single query back onto the primary. That last one is the read-your-writes escape hatch: even on a replica-preferring client, the one read that has to be fresh can demand the primary.

Here's the real examples/readonly_replica.rs:

/*
 * Copyright FalkorDB Ltd. 2023 - present
 * Licensed under the MIT License.
 */

//! Demonstrates opt-in routing of read-only queries to replica nodes.
//!
//! Routing reads to replicas is **opt-in**, because a FalkorDB replica applies writes only after
//! the primary has committed them — a read served from a replica can be slightly **stale**. By
//! default every read goes to the primary, so you never observe replication lag unless you ask for
//! it. Opt in either:
//!
//! * **per client** — `FalkorClientBuilder::with_read_preference(ReadPreference::PreferReplica)`
//!   makes every read-only query prefer a replica, or
//! * **per query** — `ro_query(..).prefer_replica()` opts a single query in (and
//!   `.primary_only()` forces a single query back onto the primary for read-your-writes paths).
//!
//! `PreferReplica` transparently falls back to the primary when no replica is available (for
//! example a single-node deployment), so the same code runs everywhere.
//!
//! Set `FALKORDB_CONNECTION` to a Sentinel endpoint (for example
//! `falkor://127.0.0.1:26379`) to see reads routed to replicas; otherwise it
//! defaults to a local single node.

use falkordb::{FalkorClientBuilder, FalkorResult, ReadPreference};

fn main() -> FalkorResult<()> {
    let connection_info = std::env::var("FALKORDB_CONNECTION")
        .unwrap_or_else(|_| "falkor://127.0.0.1:6379".to_string());

    // Opt this client into replica reads by default. Drop `with_read_preference` (or pass
    // `ReadPreference::Primary`) to keep every read on the primary.
    let client = FalkorClientBuilder::new()
        .with_connection_info(connection_info.as_str().try_into()?)
        .with_read_preference(ReadPreference::PreferReplica)
        .build()?;

    // `replica_reads_available()` reports capability (a replica pool exists), while
    // `read_preference()` reports the client's default policy.
    if client.replica_reads_available() {
        println!("Replica connections are available; read-only queries may be served from them.");
    } else {
        println!("No readable replicas detected; read-only queries use the primary.");
    }
    println!("Default read preference: {:?}", client.read_preference());

    let mut graph = client.select_graph("readonly_example");

    // Writes always go to the primary.
    graph
        .query("CREATE (:Greeting {text: 'hello'})")
        .execute()?;

    // This read-only query follows the client default (PreferReplica) — served from a replica when
    // one is available, otherwise transparently from the primary.
    let mut greetings = graph
        .ro_query("MATCH (g:Greeting) RETURN g.text")
        .execute()?;
    for row in greetings.data.by_ref() {
        println!("replica-preferring read: {row:?}");
    }

    // A read-your-writes path can force the primary for a single query, ignoring the client default.
    let mut fresh = graph
        .ro_query("MATCH (g:Greeting) RETURN g.text")
        .primary_only()
        .execute()?;
    for row in fresh.data.by_ref() {
        println!("primary (fresh) read: {row:?}");
    }

    graph.delete()?;

    Ok(())
}

Source: examples/readonly_replica.rs — compiled in CI.

The ergonomic details that make it safe to live with

  • It degrades gracefully. PreferReplica transparently falls back to the primary when no replica exists — a single-node dev box, say — so the same code runs in every environment. You don't write one path for laptops and another for production.
  • Capability and policy are separate questions. replica_reads_available() tells you whether a replica pool even exists; read_preference() tells you the client's default policy. Conflating "can I?" with "will I?" is how you get confusing behavior, so they're two honest accessors.
  • Asking for a replica on a write fails loudly. Request a replica for a writable query and you get a real error, not a quietly-wrong route to a node that can't serve it.

That's a wrap

Eight posts, one theme: push correctness into the type system and the API defaults, so the easy path is the safe path. Typed parameters, honest rows, streaming async, an embedded server, temporal math, retries that can't double-write, privacy-safe telemetry, and now reads that don't accidentally read the past. That's a month.

Every snippet you read across this series is a real file the CI compiled, and the benchmark in Part 4 is a real harness you can re-run. That wasn't an accident — it's the same principle as the code: make the trustworthy thing the default thing.

Thanks for reading the dev log. The code's on GitHub; come argue with me about any of it.