Skip to content

Consistency Models: Spend Latency Only Where Correctness Needs It

Choose consistency per operation, preserve the guarantees users rely on, and use replication lag and latency measurements to tune the rest.

Ekanatha Reddy Urivakili9 min1,700 words

The speed–consistency decision is per operation

Consistency in distributed systems and databases is often viewed as a monolithic, binary choice: either strong consistency for correctness or eventual consistency for speed. This black-and-white framing simplifies reality but frequently misleads system design choices.

Different operations within the same system have varying correctness requirements. For example:

  • A user updating their profile and immediately viewing the changes demands strong freshness guarantees.
  • Publicly available catalog data, which updates frequently but tolerates stale views, can leverage eventual consistency.

Furthermore, the same operation may need different consistency levels in distinct user journeys or client contexts. This flexibility allows latency and throughput optimizations without sacrificing user experience fidelity where it matters.

At the heart of these decisions is the trade-off in replication setups: a replica may respond faster but may lag behind the primary’s recent writes. Reading from it is only appropriate if the application tolerates the potential staleness.

The useful question is therefore not “How do we make the system eventually consistent?” It is “Which stale reads are acceptable, for whom, and for how long?”

Consistency guarantees overview

Consistency LevelDefinitionLatency ImpactUse Case Examples
Strong consistencyReads reflect the latest completed write according to system order.Higher latency, less throughputFinancial transactions, user profile updates
Eventual consistencyReads may be stale temporarily, but replicas converge eventually if no new updates.Lower latency, higher throughputProduct catalogs, social media timelines
Session guaranteesGuarantees such as read-your-writes within a session, without global consistency.Moderate, session-scoped latencyUser profiles, cart contents, user dashboards
Causal consistencyObserves cause-and-effect ordering between operations across distributed nodes.Variable latencyCollaborative editing, event stream processing

Each database or system might interpret these guarantees slightly differently, particularly around transaction semantics. Importantly, they extend beyond the database—application logic, routing, caching, and retry mechanisms all influence the effective guarantee observed.

Model the path from write to read

Consider an archetypal primary-and-replica architecture:

  1. A write commits successfully on the primary node.
  2. The write asynchronously replicates to one or more read replicas.
  3. A read can be served from either the primary or a replica.

Because replication is asynchronous, the write may not have reached some replicas at the time the read arrives. Reading from a replica offers latency and scalability benefits but risks returning stale data.

How a consistency-aware read is routed

flowchart LR
  W["Write acknowledged by primary"] --> R["Replication in flight"]
  R --> P["Replica has replicated the write"]
  C["Read request"] --> D{"Required consistency guarantee?"}
  D -- "Read-your-writes or fresh read" --> M["Read from primary or wait for replica to catch up"]
  D -- "Stale read acceptable" --> S["Read from replica immediately"]
  M --> O["Return current or confirmed-up-to-date value"]
  S --> O
  P --> S

In practice:

  • Reading the replica immediately trades freshness for speed.
  • Waiting for replication confirmation or reading the primary trades latency for consistency.

Whether to accept stale reads or pay the price of waiting depends on your operation's correctness needs.

Use the narrowest guarantee that meets the requirement

Consistency guarantees impose costs, including higher latency, increased load on the primary, or added complexity. To maximize overall system performance and user experience, start with your actual application-level invariants and select the least strict guarantee preserving them.

ScenarioConsistency GuaranteeNotes
Read immediately after user's own writeRead-your-writes (primary reads)Ensures user sees own updates; critical for edit-confirm flows.
Data tolerating minor stalenessEventual consistency (replica reads)Suitable for public content, logs, analytics views
Cross-request consistency neededSession guarantees with tokensUse freshness/version tokens to coordinate read ordering
Coordinated multi-entity readsTransactional or snapshot isolationPrevents inconsistent state between related domain objects
Durability vs visibility requirementStrong durability via commit logsVisibility may lag even if durable; choose path based on user expectations

For example, consider an e-commerce app: address update confirmation uses primary reads to see recent data, while product listing pages tolerate a few seconds' lag and can read from replicas.

A minimal policy: choose the source deliberately

Before automating complex consistency protocols, a minimal operational step is to make the read source selection explicit to the caller or calling subsystem. Even a simple toggle between primary and replica reads is a valuable lever.

var primary = new Dictionary<int, string> { [42] = "New address" };
var replica = new Dictionary<int, string> { [42] = "Old address" };

string Read(int id, bool mustSeeMyWrite)
{
    var source = mustSeeMyWrite ? primary : replica;
    return source.TryGetValue(id, out var value) ? value : "Not found";
}

Console.WriteLine(Read(42, mustSeeMyWrite: true));  // New address
Console.WriteLine(Read(42, mustSeeMyWrite: false)); // Old address

This toy example explicitly exposes the risk in replica reads: returning stale data is a deliberate trade-off, not an oversight.

API consumers and system architects can use this leverage to design user journeys that accept eventual consistency where suitable and protect consistency-sensitive flows.

A practical ASP.NET Core read policy with PostgreSQL

In a real environment, implementing explicit read policies over a PostgreSQL primary/replica setup can be done using connection strings and routing via a consistency parameter.

This example uses:

  • The Npgsql driver.
  • Named connection strings Primary and Replica.
  • A fully replicated products(id int, name text) table.
using Npgsql;

var builder = WebApplication.CreateBuilder(args);
var primaryConn = builder.Configuration.GetConnectionString("Primary")
    ?? throw new InvalidOperationException("Missing Primary connection string.");
var replicaConn = builder.Configuration.GetConnectionString("Replica")
    ?? throw new InvalidOperationException("Missing Replica connection string.");

builder.Services.AddSingleton<DbSources>(_ => new DbSources(
    NpgsqlDataSource.Create(primaryConn), NpgsqlDataSource.Create(replicaConn)));
builder.Services.AddSingleton<ProductQueries>();

var app = builder.Build();

app.MapGet("/products/{id:int}", async Task<IResult> (
    int id, string? consistency, ProductQueries queries, CancellationToken ct) =>
{
    var mode = consistency?.ToLowerInvariant() ?? "eventual";
    if (mode is not ("eventual" or "primary"))
        return Results.BadRequest("Use consistency=eventual or consistency=primary.");

    var product = await queries.FindAsync(id, usePrimary: mode == "primary", ct);
    return product is null ? Results.NotFound() : Results.Ok(product);
});

app.Run();

sealed record Product(int Id, string Name);

sealed class DbSources(NpgsqlDataSource primary, NpgsqlDataSource replica)
    : IAsyncDisposable
{
    public NpgsqlDataSource Primary { get; } = primary;
    public NpgsqlDataSource Replica { get; } = replica;

    public async ValueTask DisposeAsync()
    {
        await Primary.DisposeAsync();
        await Replica.DisposeAsync();
    }
}

sealed class ProductQueries(DbSources sources)
{
    public async Task<Product?> FindAsync(int id, bool usePrimary, CancellationToken ct)
    {
        var dataSource = usePrimary ? sources.Primary : sources.Replica;
        await using var connection = await dataSource.OpenConnectionAsync(ct);
        await using var command = new NpgsqlCommand(
            "SELECT id, name FROM products WHERE id = @id", connection);
        command.Parameters.AddWithValue("id", id);
        await using var reader = await command.ExecuteReaderAsync(ct);

        return await reader.ReadAsync(ct)
            ? new Product(reader.GetInt32(0), reader.GetString(1))
            : null;
    }
}

Tune with measurements, not a blanket downgrade

Changing consistency is a performance lever only if it moves actual bottlenecks. A naive switch to replica reads without measuring can lead to unexpected consequences:

  • Cross-region reads introduce high latency.
  • Connection pool exhaustion on replicas leads to fallback floods to the primary.
  • Slow replicas multiply tail latency.

What to measure and monitor:

MetricDescriptionTarget & Alert Threshold
Replication lag (tail percentile)Time delay between primary commit and replica visibilityAlert if lag > freshness budget for any operation
Read latency per consistency policyMedian and tail latency for reads from primary & replicaTail latency exceeding SLO or baseline
Fallback rate to primaryPercentage of replica reads falling back to primaryAlert on spikes indicating replica unavailability
Stale read rateFraction of reads returning older than required versionDefine acceptable business error budget
Error rateErrors per read type or consistency policyAlert on elevated error rates

Observability for routing decisions

Ensure each response logs:

  • Consistency mode requested (primary or eventual).
  • Actual source used (primary or replica).
  • Replication lag at read time, where measurable.
  • Any fallback or retry events.

Logging this data using a structured format enables long-term trend analysis and incident diagnosis.

{
  "timestamp": "2024-06-18T15:33:21Z",
  "requestId": "cf1b1234-5678-90ab-cdef-123456789abc",
  "userId": 123,
  "endpoint": "/products/42",
  "consistencyRequested": "eventual",
  "sourceUsed": "replica",
  "replicationLagMs": 135,
  "fallbackOccurred": false,
  "latencyMs": 215
}

Distributed tracing with OpenTelemetry (C#)

Capture end-to-end request context and annotate consistency routing:

using OpenTelemetry.Trace;

var tracer = TracerProvider.Default.GetTracer("ProductAPI");

async Task<Product?> FindAsyncWithTracing(int id, bool usePrimary, CancellationToken ct)
{
    using var span = tracer.StartActiveSpan("ReadProduct",
          SpanKind.Internal, out var currentSpan);

    currentSpan.SetAttribute("db.system", "postgresql");
    currentSpan.SetAttribute("db.statement", "SELECT id, name FROM products WHERE id = @id");
    currentSpan.SetAttribute("read.source", usePrimary ? "primary" : "replica");

    // Example: Measure or set replication lag attribute if available
    // currentSpan.SetAttribute("replication.lag.ms", lagMs);

    var product = await FindAsync(id, usePrimary, ct);
    if (product == null)
        currentSpan.SetStatus(Status.Error.WithDescription("Product not found"));
    else
        currentSpan.SetStatus(Status.Ok);

    return product;
}

Failure modes and trade-offs

Understanding the operational risks of multi-consistency routing is critical:

RiskDescriptionMitigation
Replica lag spikesPeriods of high replication lag break freshness contractsMonitor lag, blacklist replicas exceeding freshness thresholds, use circuit breakers
Primary fallback overloadFallback traffic overloads primary, impacting writesRate-limit retries, prioritize primary for writes, use health-aware routing
Inconsistent sessionsSwitching between primary and replica breaks read-your-writesUse session affinity or freshness tokens to maintain consistency
Lost causal contextSequential reads return conflicting dataEnforce causal ordering via application logic or session guarantees
Misleading API namesAmbiguous terms like “strong” confuse clients and developersUse explicit naming: read-your-writes, primary-read, etc., with clear documentation

A practical rollout

Systematically introducing multi-consistency routing for optimal latency and correctness requires discipline:

  1. List critical invariants and freshness requirements

    Document all user-visible correctness contracts, such as when read-your-writes semantics are mandatory or when some staleness is acceptable.

  2. Define consistency contracts

    Specify for each operation:

    • Consistency guarantee type
    • Scope (session, global, per request)
    • Freshness budget (max allowable replica lag/time)
    • Timeout and fallback behavior
  3. Instrument before changing routing

    Collect metrics on latency, replica lag, source selected, stale reads, and fallback events to set baselines.

  4. Gradually introduce replica routing

    Initially route only low-risk, stale-tolerant workloads to replicas. Monitor for correctness anomalies and latency improvements.

  5. Expand with evidence

    Keep freshness-sensitive workloads on primary until confidence in correctness and performance gains permits migration.

Considerations between consistency models and their trade-offs

Consistency ModelLatencyWrite Throughput ImpactRead FreshnessComplexitySuitability
Strong (linearizable)HighestHighest (writer bottleneck)Reads see all writesComplex distributed consensusCritical transactions, financial systems
Read-your-writesModerateModerateReads guarantee writer's own latest writesModerateUser session state, profile edits
EventualLowestHighReads may be staleSimplePublic data, logs, analytics
CausalVariableDepends on capture/storagePreserves cause-effect orderMore complex trackingCollaborative environments, event-driven flows

This table helps select the minimum consistency required to keep latency costs under control.

Observability: capacity to observe correctness and performance trade-offs

Reliably implementing multi-consistency routing requires observability covering correctness errors as well as performance:

What to log

  • Consistency policy requested and actual source used.
  • Replication lag at read time (and lag distribution).
  • Latency per request and per consistency mode (including tail latency).
  • Fallback and retry counts.
  • Stale read counts or inconsistencies detected by application logic.

Key metrics (RED & USE)

MetricDescriptionMeasurement Type
Request Rate (RED)Number of reads per consistency modeCounter, per endpoint
Error Rate (RED)Number of stale reads or failed consistency checksCounter, alertable
Duration (RED)Read latency distribution (median + tail)Histogram, per mode
Replica CPU/Memory (USE)Resource utilization on replicasGauge
Replication Lag (USE)Replication lag distribution and max lagGauge, alertable

Example SLI/SLO for a read-your-writes flow

Service Level IndicatorService Level ObjectiveAlert Condition
Read latency (primary read)95th percentile < 100msP95 latency > 150ms for 5 consecutive minutes
Read-your-writes consistency rate> 99.9% reads reflect latest writeStale read rate > 0.1% over 10 mins
Replica lag< 50msLag > 50ms sustained for > 1 min

Alerts worth paging

  • Replica lag exceeds allowed freshness threshold persistently.
  • Primary fallback rate spikes indicating replica failure or overload.
  • Stale read rate exceeds budget in sensitive user flows.
  • Unusually high tail latency on primary or replica reads affecting SLIs.

Conclusion

Balancing latency and correctness in distributed systems requires operational nuance. Consistency must be chosen per operation, reflecting real user invariants, not as a blanket system-wide toggle.

The architecture must:

  • Explicitly expose read policies.
  • Make consistency requirements observable and measurable.
  • Use the narrowest necessary guarantee to preserve correctness.
  • Tune policies driven by empirical latency, lag, and correctness data.
  • Handle failure modes with circuit breakers, timeouts, and fallback limits.

This pragmatic approach lets engineers spend latency where user-visible correctness requires it, preserving throughput and cost efficiencies elsewhere.


References:


This article upgrades the concepts from a brief overview into a comprehensive operational guide, suitable for senior engineers designing real-world distributed systems with explicit latency-consistency trade-offs.