The speed–consistency decision is per operation
Consistency in distributed systems and databases is often viewed as a monolithic, binary choice: either strong consistency for correctness or eventual consistency for speed. This black-and-white framing simplifies reality but frequently misleads system design choices.
Different operations within the same system have varying correctness requirements. For example:
- A user updating their profile and immediately viewing the changes demands strong freshness guarantees.
- Publicly available catalog data, which updates frequently but tolerates stale views, can leverage eventual consistency.
Furthermore, the same operation may need different consistency levels in distinct user journeys or client contexts. This flexibility allows latency and throughput optimizations without sacrificing user experience fidelity where it matters.
At the heart of these decisions is the trade-off in replication setups: a replica may respond faster but may lag behind the primary’s recent writes. Reading from it is only appropriate if the application tolerates the potential staleness.
The useful question is therefore not “How do we make the system eventually consistent?” It is “Which stale reads are acceptable, for whom, and for how long?”
Consistency guarantees overview
| Consistency Level | Definition | Latency Impact | Use Case Examples |
|---|---|---|---|
| Strong consistency | Reads reflect the latest completed write according to system order. | Higher latency, less throughput | Financial transactions, user profile updates |
| Eventual consistency | Reads may be stale temporarily, but replicas converge eventually if no new updates. | Lower latency, higher throughput | Product catalogs, social media timelines |
| Session guarantees | Guarantees such as read-your-writes within a session, without global consistency. | Moderate, session-scoped latency | User profiles, cart contents, user dashboards |
| Causal consistency | Observes cause-and-effect ordering between operations across distributed nodes. | Variable latency | Collaborative editing, event stream processing |
Each database or system might interpret these guarantees slightly differently, particularly around transaction semantics. Importantly, they extend beyond the database—application logic, routing, caching, and retry mechanisms all influence the effective guarantee observed.
Model the path from write to read
Consider an archetypal primary-and-replica architecture:
- A write commits successfully on the primary node.
- The write asynchronously replicates to one or more read replicas.
- A read can be served from either the primary or a replica.
Because replication is asynchronous, the write may not have reached some replicas at the time the read arrives. Reading from a replica offers latency and scalability benefits but risks returning stale data.
How a consistency-aware read is routed
flowchart LR
W["Write acknowledged by primary"] --> R["Replication in flight"]
R --> P["Replica has replicated the write"]
C["Read request"] --> D{"Required consistency guarantee?"}
D -- "Read-your-writes or fresh read" --> M["Read from primary or wait for replica to catch up"]
D -- "Stale read acceptable" --> S["Read from replica immediately"]
M --> O["Return current or confirmed-up-to-date value"]
S --> O
P --> SIn practice:
- Reading the replica immediately trades freshness for speed.
- Waiting for replication confirmation or reading the primary trades latency for consistency.
Whether to accept stale reads or pay the price of waiting depends on your operation's correctness needs.
Use the narrowest guarantee that meets the requirement
Consistency guarantees impose costs, including higher latency, increased load on the primary, or added complexity. To maximize overall system performance and user experience, start with your actual application-level invariants and select the least strict guarantee preserving them.
Recommended guidance:
| Scenario | Consistency Guarantee | Notes |
|---|---|---|
| Read immediately after user's own write | Read-your-writes (primary reads) | Ensures user sees own updates; critical for edit-confirm flows. |
| Data tolerating minor staleness | Eventual consistency (replica reads) | Suitable for public content, logs, analytics views |
| Cross-request consistency needed | Session guarantees with tokens | Use freshness/version tokens to coordinate read ordering |
| Coordinated multi-entity reads | Transactional or snapshot isolation | Prevents inconsistent state between related domain objects |
| Durability vs visibility requirement | Strong durability via commit logs | Visibility may lag even if durable; choose path based on user expectations |
For example, consider an e-commerce app: address update confirmation uses primary reads to see recent data, while product listing pages tolerate a few seconds' lag and can read from replicas.
A minimal policy: choose the source deliberately
Before automating complex consistency protocols, a minimal operational step is to make the read source selection explicit to the caller or calling subsystem. Even a simple toggle between primary and replica reads is a valuable lever.
var primary = new Dictionary<int, string> { [42] = "New address" };
var replica = new Dictionary<int, string> { [42] = "Old address" };
string Read(int id, bool mustSeeMyWrite)
{
var source = mustSeeMyWrite ? primary : replica;
return source.TryGetValue(id, out var value) ? value : "Not found";
}
Console.WriteLine(Read(42, mustSeeMyWrite: true)); // New address
Console.WriteLine(Read(42, mustSeeMyWrite: false)); // Old address
This toy example explicitly exposes the risk in replica reads: returning stale data is a deliberate trade-off, not an oversight.
API consumers and system architects can use this leverage to design user journeys that accept eventual consistency where suitable and protect consistency-sensitive flows.
A practical ASP.NET Core read policy with PostgreSQL
In a real environment, implementing explicit read policies over a PostgreSQL primary/replica setup can be done using connection strings and routing via a consistency parameter.
This example uses:
- The Npgsql driver.
- Named connection strings
PrimaryandReplica. - A fully replicated
products(id int, name text)table.
using Npgsql;
var builder = WebApplication.CreateBuilder(args);
var primaryConn = builder.Configuration.GetConnectionString("Primary")
?? throw new InvalidOperationException("Missing Primary connection string.");
var replicaConn = builder.Configuration.GetConnectionString("Replica")
?? throw new InvalidOperationException("Missing Replica connection string.");
builder.Services.AddSingleton<DbSources>(_ => new DbSources(
NpgsqlDataSource.Create(primaryConn), NpgsqlDataSource.Create(replicaConn)));
builder.Services.AddSingleton<ProductQueries>();
var app = builder.Build();
app.MapGet("/products/{id:int}", async Task<IResult> (
int id, string? consistency, ProductQueries queries, CancellationToken ct) =>
{
var mode = consistency?.ToLowerInvariant() ?? "eventual";
if (mode is not ("eventual" or "primary"))
return Results.BadRequest("Use consistency=eventual or consistency=primary.");
var product = await queries.FindAsync(id, usePrimary: mode == "primary", ct);
return product is null ? Results.NotFound() : Results.Ok(product);
});
app.Run();
sealed record Product(int Id, string Name);
sealed class DbSources(NpgsqlDataSource primary, NpgsqlDataSource replica)
: IAsyncDisposable
{
public NpgsqlDataSource Primary { get; } = primary;
public NpgsqlDataSource Replica { get; } = replica;
public async ValueTask DisposeAsync()
{
await Primary.DisposeAsync();
await Replica.DisposeAsync();
}
}
sealed class ProductQueries(DbSources sources)
{
public async Task<Product?> FindAsync(int id, bool usePrimary, CancellationToken ct)
{
var dataSource = usePrimary ? sources.Primary : sources.Replica;
await using var connection = await dataSource.OpenConnectionAsync(ct);
await using var command = new NpgsqlCommand(
"SELECT id, name FROM products WHERE id = @id", connection);
command.Parameters.AddWithValue("id", id);
await using var reader = await command.ExecuteReaderAsync(ct);
return await reader.ReadAsync(ct)
? new Product(reader.GetInt32(0), reader.GetString(1))
: null;
}
}
Tune with measurements, not a blanket downgrade
Changing consistency is a performance lever only if it moves actual bottlenecks. A naive switch to replica reads without measuring can lead to unexpected consequences:
- Cross-region reads introduce high latency.
- Connection pool exhaustion on replicas leads to fallback floods to the primary.
- Slow replicas multiply tail latency.
What to measure and monitor:
| Metric | Description | Target & Alert Threshold |
|---|---|---|
| Replication lag (tail percentile) | Time delay between primary commit and replica visibility | Alert if lag > freshness budget for any operation |
| Read latency per consistency policy | Median and tail latency for reads from primary & replica | Tail latency exceeding SLO or baseline |
| Fallback rate to primary | Percentage of replica reads falling back to primary | Alert on spikes indicating replica unavailability |
| Stale read rate | Fraction of reads returning older than required version | Define acceptable business error budget |
| Error rate | Errors per read type or consistency policy | Alert on elevated error rates |
Observability for routing decisions
Ensure each response logs:
- Consistency mode requested (
primaryoreventual). - Actual source used (primary or replica).
- Replication lag at read time, where measurable.
- Any fallback or retry events.
Logging this data using a structured format enables long-term trend analysis and incident diagnosis.
{
"timestamp": "2024-06-18T15:33:21Z",
"requestId": "cf1b1234-5678-90ab-cdef-123456789abc",
"userId": 123,
"endpoint": "/products/42",
"consistencyRequested": "eventual",
"sourceUsed": "replica",
"replicationLagMs": 135,
"fallbackOccurred": false,
"latencyMs": 215
}
Distributed tracing with OpenTelemetry (C#)
Capture end-to-end request context and annotate consistency routing:
using OpenTelemetry.Trace;
var tracer = TracerProvider.Default.GetTracer("ProductAPI");
async Task<Product?> FindAsyncWithTracing(int id, bool usePrimary, CancellationToken ct)
{
using var span = tracer.StartActiveSpan("ReadProduct",
SpanKind.Internal, out var currentSpan);
currentSpan.SetAttribute("db.system", "postgresql");
currentSpan.SetAttribute("db.statement", "SELECT id, name FROM products WHERE id = @id");
currentSpan.SetAttribute("read.source", usePrimary ? "primary" : "replica");
// Example: Measure or set replication lag attribute if available
// currentSpan.SetAttribute("replication.lag.ms", lagMs);
var product = await FindAsync(id, usePrimary, ct);
if (product == null)
currentSpan.SetStatus(Status.Error.WithDescription("Product not found"));
else
currentSpan.SetStatus(Status.Ok);
return product;
}
Failure modes and trade-offs
Understanding the operational risks of multi-consistency routing is critical:
| Risk | Description | Mitigation |
|---|---|---|
| Replica lag spikes | Periods of high replication lag break freshness contracts | Monitor lag, blacklist replicas exceeding freshness thresholds, use circuit breakers |
| Primary fallback overload | Fallback traffic overloads primary, impacting writes | Rate-limit retries, prioritize primary for writes, use health-aware routing |
| Inconsistent sessions | Switching between primary and replica breaks read-your-writes | Use session affinity or freshness tokens to maintain consistency |
| Lost causal context | Sequential reads return conflicting data | Enforce causal ordering via application logic or session guarantees |
| Misleading API names | Ambiguous terms like “strong” confuse clients and developers | Use explicit naming: read-your-writes, primary-read, etc., with clear documentation |
A practical rollout
Systematically introducing multi-consistency routing for optimal latency and correctness requires discipline:
-
List critical invariants and freshness requirements
Document all user-visible correctness contracts, such as when read-your-writes semantics are mandatory or when some staleness is acceptable.
-
Define consistency contracts
Specify for each operation:
- Consistency guarantee type
- Scope (session, global, per request)
- Freshness budget (max allowable replica lag/time)
- Timeout and fallback behavior
-
Instrument before changing routing
Collect metrics on latency, replica lag, source selected, stale reads, and fallback events to set baselines.
-
Gradually introduce replica routing
Initially route only low-risk, stale-tolerant workloads to replicas. Monitor for correctness anomalies and latency improvements.
-
Expand with evidence
Keep freshness-sensitive workloads on primary until confidence in correctness and performance gains permits migration.
Considerations between consistency models and their trade-offs
| Consistency Model | Latency | Write Throughput Impact | Read Freshness | Complexity | Suitability |
|---|---|---|---|---|---|
| Strong (linearizable) | Highest | Highest (writer bottleneck) | Reads see all writes | Complex distributed consensus | Critical transactions, financial systems |
| Read-your-writes | Moderate | Moderate | Reads guarantee writer's own latest writes | Moderate | User session state, profile edits |
| Eventual | Lowest | High | Reads may be stale | Simple | Public data, logs, analytics |
| Causal | Variable | Depends on capture/storage | Preserves cause-effect order | More complex tracking | Collaborative environments, event-driven flows |
This table helps select the minimum consistency required to keep latency costs under control.
Observability: capacity to observe correctness and performance trade-offs
Reliably implementing multi-consistency routing requires observability covering correctness errors as well as performance:
What to log
- Consistency policy requested and actual source used.
- Replication lag at read time (and lag distribution).
- Latency per request and per consistency mode (including tail latency).
- Fallback and retry counts.
- Stale read counts or inconsistencies detected by application logic.
Key metrics (RED & USE)
| Metric | Description | Measurement Type |
|---|---|---|
| Request Rate (RED) | Number of reads per consistency mode | Counter, per endpoint |
| Error Rate (RED) | Number of stale reads or failed consistency checks | Counter, alertable |
| Duration (RED) | Read latency distribution (median + tail) | Histogram, per mode |
| Replica CPU/Memory (USE) | Resource utilization on replicas | Gauge |
| Replication Lag (USE) | Replication lag distribution and max lag | Gauge, alertable |
Example SLI/SLO for a read-your-writes flow
| Service Level Indicator | Service Level Objective | Alert Condition |
|---|---|---|
| Read latency (primary read) | 95th percentile < 100ms | P95 latency > 150ms for 5 consecutive minutes |
| Read-your-writes consistency rate | > 99.9% reads reflect latest write | Stale read rate > 0.1% over 10 mins |
| Replica lag | < 50ms | Lag > 50ms sustained for > 1 min |
Alerts worth paging
- Replica lag exceeds allowed freshness threshold persistently.
- Primary fallback rate spikes indicating replica failure or overload.
- Stale read rate exceeds budget in sensitive user flows.
- Unusually high tail latency on primary or replica reads affecting SLIs.
Conclusion
Balancing latency and correctness in distributed systems requires operational nuance. Consistency must be chosen per operation, reflecting real user invariants, not as a blanket system-wide toggle.
The architecture must:
- Explicitly expose read policies.
- Make consistency requirements observable and measurable.
- Use the narrowest necessary guarantee to preserve correctness.
- Tune policies driven by empirical latency, lag, and correctness data.
- Handle failure modes with circuit breakers, timeouts, and fallback limits.
This pragmatic approach lets engineers spend latency where user-visible correctness requires it, preserving throughput and cost efficiencies elsewhere.
References:
- Npgsql Documentation
- PostgreSQL Replication
- OpenTelemetry .NET SDK
- Microsoft Docs: ASP.NET Core Performance Best Practices
- Brewer’s CAP Theorem: CAP Theorem Wiki
This article upgrades the concepts from a brief overview into a comprehensive operational guide, suitable for senior engineers designing real-world distributed systems with explicit latency-consistency trade-offs.