Production-grade: retries, tracing spans, metrics, and error hints

A client library spends most of its life in production, where the network is hostile and the only thing you have at 3am is whatever the library decided to tell you. This batch of work was all about that life: surviving transient failures, explaining what's happening, and — when something does break — pointing at the fix.

Retries that can't duplicate a write

Retries are a loaded gun. Re-issue a read and worst case you did some redundant work; re-issue a write and you might have just charged a customer twice. So the retry policy here has one inviolable rule: writes are never retried.

sequenceDiagram
    autonumber
    participant App
    participant C as falkordb client
    participant DB as FalkorDB
    App->>C: ro_query(...).execute()
    C->>DB: attempt 1
    DB--xC: ConnectionDown (transient)
    Note over C: read-only + transient? eligible.<br/>backoff: 50ms, 100ms, 200ms ... capped at 1s
    C->>C: sleep(backoff), re-borrow a healed connection
    C->>DB: attempt 2
    DB-->>C: result set
    C-->>App: Ok(rows)
    Note over App,DB: a write (query()) in the same spot is NEVER retried —<br/>it fails fast so it can't be applied twice

It's off by default — without a policy, every operation runs exactly once, same as before. When you do opt in, only read-only / idempotent operations that fail with a transient connection error get retried, with bounded exponential backoff. Here's the real examples/retry.rs:

/*
 * Copyright FalkorDB Ltd. 2023 - present
 * Licensed under the MIT License.
 */

//! Demonstrates the opt-in retry policy that re-issues eligible operations on transient
//! connection failures.
//!
//! A [`RetryPolicy`] is **disabled by default**, so a client built without one attempts every
//! operation exactly once (the previous behavior). When a policy is configured, read-only /
//! idempotent operations that fail with a transient connection error are automatically retried
//! with bounded backoff. **Writes are never retried**, so enabling a policy can never duplicate a
//! write.
//!
//! Set `FALKORDB_CONNECTION` to point at another server (for example `falkor://127.0.0.1:6379`);
//! it defaults to a local single node.

use falkordb::{Backoff, FalkorClientBuilder, FalkorResult, RetryPolicy};
use std::time::Duration;

fn main() -> FalkorResult<()> {
    let connection_info = std::env::var("FALKORDB_CONNECTION")
        .unwrap_or_else(|_| "falkor://127.0.0.1:6379".to_string());

    // Retry read-only operations on transient connection failures: up to 4 attempts total
    // (1 initial try + 3 retries), with exponential backoff starting at 50ms and capped at 1s.
    let policy = RetryPolicy::read_only()
        .max_attempts(4)
        .backoff(Backoff::exponential(Duration::from_millis(50)).max_delay(Duration::from_secs(1)));

    let client = FalkorClientBuilder::new()
        .with_connection_info(connection_info.as_str().try_into()?)
        .with_retry_policy(policy)
        .build()?;

    let mut graph = client.select_graph("retry_example");

    // Writes are NEVER auto-retried, even with a policy enabled, so a write can never be applied
    // twice. Classification is by the API you call: `query()` is always treated as a write.
    graph
        .query("CREATE (:Greeting {text: 'hello'})")
        .execute()?;

    // Read-only queries ARE eligible for retry. If the connection drops transiently here, the
    // client re-borrows a healed connection and re-issues the query (up to `max_attempts`), instead
    // of surfacing the error on the first blip. Use `ro_query()` (not `query()`) for retryable reads.
    let mut greetings = graph
        .ro_query("MATCH (g:Greeting) RETURN g.text")
        .execute()?;

    for row in greetings.data.by_ref() {
        println!("read-only result: {row:?}");
    }

    graph.delete()?;

    Ok(())
}

Source: examples/retry.rs — compiled in CI.

The classification is by the API you call, which is what makes it safe: query() is always treated as a write and never retried; ro_query() is eligible. There's no heuristic sniffing your Cypher to guess intent — you told the client what this is by which method you called, and it believes you. Enabling a policy can therefore never turn one write into two. That property mattered more to me than any amount of cleverness.

Spans that tell you something, without leaking everything

With the tracing feature, every query- and procedure-execution span carries structured, low-cardinality fields you can actually slice on — the graph (db.namespace), the command (db.operation.name), whether it was read-only, the connection strategy, and the server-reported execution time and returned row count. The naming is OpenTelemetry-aligned on purpose, so this drops into your existing dashboards instead of inventing private conventions.

/*
 * Copyright FalkorDB Ltd. 2023 - present
 * Licensed under the MIT License.
 */

//! Demonstrates the `tracing` span enrichment (run with `--features tracing`).
//!
//! With the `tracing` feature enabled, the query- and procedure-execution spans carry structured,
//! low-cardinality fields you can slice on — the graph (`db.namespace`), the command
//! (`db.operation.name`), whether it is read-only, the connection strategy, and a privacy-safe
//! `db.query.fingerprint`. The raw query text and parameter values are never recorded by default.
//!
//! This installs a simple `tracing-subscriber` that prints each span (and its fields) as it closes,
//! so you can see the enriched execution spans for the write and the read below. Run with:
//! `cargo run --example observability --features tracing`.

use falkordb::{FalkorClientBuilder, FalkorResult};
use tracing_subscriber::fmt::format::FmtSpan;

fn main() -> FalkorResult<()> {
    // Print each span and its recorded fields when it closes. The execution spans are at TRACE
    // level, so enable TRACE to see them (a real app would scope this to the `falkordb` target).
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::TRACE)
        .with_span_events(FmtSpan::CLOSE)
        .with_target(true)
        .init();

    let connection_info = std::env::var("FALKORDB_CONNECTION")
        .unwrap_or_else(|_| "falkor://127.0.0.1:6379".to_string());

    // `with_query_logging(true)` would additionally record the raw Cypher as `db.query.text`;
    // it is off by default for privacy, so only the redacted fingerprint is recorded.
    let client = FalkorClientBuilder::new()
        .with_connection_info(connection_info.as_str().try_into()?)
        .build()?;

    let mut graph = client.select_graph("observability_example");

    // A write: its span carries `db.operation.name = "GRAPH.QUERY"`, `db.falkordb.read_only = false`.
    graph
        .query("CREATE (:Movie {title: 'The Matrix', year: 1999})")
        .execute()?;

    // A read: its span carries `db.operation.name = "GRAPH.RO_QUERY"`, `read_only = true`, and a
    // `db.query.fingerprint` that is independent of the inlined `'The Matrix'` literal.
    let mut titles = graph
        .ro_query("MATCH (m:Movie) WHERE m.title = 'The Matrix' RETURN m.year")
        .execute()?;

    for row in titles.data.by_ref() {
        println!("read-only result: {row:?}");
    }

    graph.delete()?;

    Ok(())
}

Source: examples/observability.rs — compiled in CI.

Here's the design decision I'll defend hardest: the raw query text and parameter values are not recorded by default. Instead the span carries a db.query.fingerprint — a redacted identity for the query shape that's independent of the inlined literals. So WHERE m.title = 'The Matrix' and WHERE m.title = 'Heat' share a fingerprint, and you can aggregate "this query shape is slow" without spraying user data — or secrets — across your tracing backend. You can opt into full query logging with with_query_logging(true) when you're debugging locally, but you have to ask. Observability should never be the reason you failed an audit.

Metrics and actionable errors

Two more pieces round it out:

• A metrics feature emits bounded counters and histograms — executions, retries, in-flight requests, pool-wait time. "Bounded" is the key word: the cardinality is fixed, so your metrics bill doesn't explode because someone parameterized a label with a user ID. Plays directly into the metrics ecosystem.
• FalkorDBError::mitigation_hint() — because an error that only tells you what went wrong is half an error. The hint tells you what to do about it. It's the difference between ConnectionDown and "ConnectionDown — the server may be restarting or unreachable; check connectivity and consider a retry policy." Future-me, staring at a log line, is the customer for that sentence.

None of this is glamorous. All of it is the difference between a library you use and a library you trust. The finale, Part 8, is about trust of a subtler kind: reading from replicas without accidentally reading the past.