The theorem is about a partition, not a permanent choice
The CAP theorem describes a fundamental limitation on distributed systems that replicate data across nodes: if a network partition prevents those nodes from communicating, the system cannot guarantee both consistency and availability simultaneously for every operation during that partition. It must either reject or delay some operations, or answer without being able to ensure that all clients observe one coherent global state.
This insight is often misunderstood and oversimplified by the slogan “pick two.” That saying treats consistency, availability, and partition tolerance as three orthogonal attributes from which you pick two; CAP actually explains why during a partition, a system must choose between consistency and availability. Network partitions are inevitable in a distributed setting — they are not an optional deployment mode or a rare failure event. Thus the trade-off is unavoidable, and must be explicit in your design for robustness.
Here are the precise definitions CAP uses (see Gilbert and Lynch’s original paper for formal details):
-
Consistency (C): Also called linearizability, consistency means that all operations appear to take effect in a global total order. Reads always see the most recent committed write, enforcing a single-copy semantics for the data.
-
Availability (A): Every request received by a non-failing node eventually receives a non-error response. Returning an explicit failure or refusing the request does not count as available under CAP — the system must answer successfully.
-
Partition Tolerance (P): The system continues to function despite arbitrary message loss or delay that partitions nodes into disconnected subsets. Because network failures are unavoidable at scale, systems must tolerate partitions to be distributed and resilient.
CAP’s consistency guarantee is about operation ordering visibility globally, not every application-level invariant or database constraint. A system might maintain local correctness properties or eventual convergence without being CAP consistent. Likewise, availability in CAP is a liveness property that excludes returning failure responses.
A two-replica example
Imagine a bank balance stored at two replicas, A and B. Both currently say 100 units. Then a network partition isolates A and B from each other. Two clients make conflicting withdrawal requests concurrently: one at A requests 20 units withdrawn, the other at B requests 30.
-
Consistency-first design: Replica A refuses or delays the withdrawal if it cannot coordinate with B to confirm the operation. It waits or returns an error to maintain a single global state. This sacrifices availability during the partition.
-
Availability-first design: Replica A immediately accepts the withdrawal and replies successfully. Meanwhile, B independently accepts its withdrawal. The two replicas now differ — the system must resolve the conflicts after the partition heals, potentially losing some updates.
No retry policy can fully solve this. Retrying a failed write at a replica isolated on the other side cannot know the status of previous attempts at the first replica. Idempotency helps avoid duplicates but not divergent committed state.
How the request reaches the trade-off
The decision flow for a replicated write when coordination becomes impossible looks like this:
flowchart TD
Request["Client request reaches replica A"] --> Check["Can A coordinate with required replicas?"]
Check -->|Yes| Commit["Coordinate and commit consistently"]
Check -->|No| Policy{"Policy during partition"}
Policy -->|Consistency first| Reject["Wait or return unavailable"]
Policy -->|Availability first| Local["Accept locally and reconcile later"]During a partition or suspected partition, a node must decide what to do. It cannot reliably detect a partition or distinguish a failed node from a slow network, so policy drives behavior under uncertainty.
A minimal executable illustration
This minimal .NET example models the high-level CAP trade-off: if the peer replica is unreachable, either refuse the write or accept the write locally. It isolates the fundamental decision, without replication or network code.
var partitioned = true;
var replicaA = "balance:100";
var replicaB = "balance:100";
if (partitioned)
{
Console.WriteLine("Consistency-first: write not acknowledged.");
}
else
{
replicaA = "balance:80";
replicaB = replicaA;
Console.WriteLine("Write committed on both replicas.");
}
Console.WriteLine($"A={replicaA}; B={replicaB}");
Notice this models only the trade-off point: refusing a write when coordination is impossible, or accepting locally and risking divergence.
A small runnable policy model
This .NET example expands the model. It simulates two replicas, a simple string register, and two policies: consistency-first rejects writes without peer coordination, availability-first accepts writes locally. It is intentionally minimal: no concurrency or durable storage — just the policy decision in code.
enum WritePolicy
{
ConsistencyFirst,
AvailabilityFirst
}
sealed class TwoReplicaRegister
{
private string _a = "open";
private string _b = "open";
public bool TryWrite(
string replica,
string value,
bool peerReachable,
WritePolicy policy)
{
if (policy == WritePolicy.ConsistencyFirst && !peerReachable)
{
return false;
}
if (replica == "A")
{
_a = value;
}
else if (replica == "B")
{
_b = value;
}
else
{
throw new ArgumentException("Replica must be A or B.", nameof(replica));
}
if (peerReachable)
{
_a = value;
_b = value;
}
return true;
}
public string Read(string replica) => replica switch
{
"A" => _a,
"B" => _b,
_ => throw new ArgumentException("Replica must be A or B.", nameof(replica))
};
}
var cp = new TwoReplicaRegister();
var cpAccepted = cp.TryWrite("A", "closed", false, WritePolicy.ConsistencyFirst);
Console.WriteLine($"Consistency-first accepted: {cpAccepted}");
var ap = new TwoReplicaRegister();
var apAccepted = ap.TryWrite("A", "closed", false, WritePolicy.AvailabilityFirst);
Console.WriteLine($"Availability-first accepted: {apAccepted}");
Console.WriteLine($"A reads {ap.Read("A")}; B reads {ap.Read("B")}");
Here the availability-first write returns success but replicas diverge; the consistency-first write rejects the isolated attempt.
What “CP” and “AP” mean in practice
Engineers often use "CP" and "AP" to describe systems or operations, but these are shorthand for behavior during a network partition, not permanent product categories.
| Label | Typical Behavior | What It Implies | Real-World Examples |
|---|---|---|---|
| CP | Requires coordination/quorum for writes; rejects or blocks if unavailable. | Correct linearizable states but potential unavailability under partition. | ZooKeeper, HDFS, Consul (RAFT mode) |
| AP | Accepts writes locally without coordination; risks divergence. | High availability but potential inconsistency requiring reconciliation. | Cassandra, Dynamo-style stores, Riak |
| Flexible | Offers tunable consistency; different policies per operation or API. | Allows hybrid guarantees depending on business needs. | MongoDB, CosmosDB, CockroachDB |
- CP systems use leader election, majority quorums, or leases to ensure linearizability during connectivity.
- AP systems embrace eventual consistency and conflict resolution, optimizing availability at the cost of temporary divergence.
- Many modern databases provide selectable consistency modes, offering both strategies within a single system.
Understanding CAP requires examining exact operation contracts — what acknowledgement means, which replicas must respond, what read guarantees apply, and what happens on failover or partition healing. Never trust marketing or single-label claims.
Choosing a policy for application data
The choice between consistency-first and availability-first policies should be grounded in business semantics and data correctness needs, not on abstract “strong” vs. “weak” ideas.
-
Favor coordination for operations enforcing invariants. For example: financial debits, inventory reservations, unique constraints, ownership transfers. Here, rejecting or delaying requests during a partition avoids incompatible outcomes.
-
Favor availability for mergeable, tolerant work. Examples include draft edits, telemetry, “likes,” or user preferences where temporary divergence can be merged and resolved later.
-
Make uncertainty visible at the API boundary. For example, if a write is locally accepted but not globally committed, the API should clearly signal the potential for later reconciliation, rather than implying global durability.
-
Separate endpoints by invariant. Different operations or data may use different policies. For example, a product listing can afford stale reads, but the payment processing endpoint requires coordination.
Below is a practical decision checklist when choosing policies for replicated data:
| Data Type / Use Case | Consistency Priority | Available Strategy | Rationale |
|---|---|---|---|
| Account balances, payments | High | CP coordination with quorum or leader election | Must prevent double spending or overdraft |
| Inventory reservations | High | CP or transactional coordination | Must avoid overselling |
| User-generated content drafts | Medium/Low | Availability-first with conflict resolution (last writer wins, CRDTs) | Allows responsiveness, conflicts manageable |
| Metrics, telemetry, logs | Low | AP with eventual consistency, merge resets | Eventual consistency suffices |
Common mistakes and pitfalls
-
“We choose CA.” A single node is indeed consistent and available, but not partition-tolerant. The distributed CAP trade-offs apply once you replicate across networked nodes.
-
“Eventual consistency equals CAP availability.” Eventual consistency only promises convergence given enough time and no new writes. It does not ensure every request is answered during a partition.
-
“A timeout proves the peer is down.” Network timeouts are insufficient evidence to distinguish a partition from slow requests or transient issues. Aggressive assumptions can cause unnecessary unavailability or inconsistent failover.
-
“We will just merge later.” Conflict resolution must be explicitly defined for your data model. Naive last-write-wins might discard valid updates; sums or counters can be incorrect without commutativity guarantees.
-
“CAP covers every distributed systems trade-off.” CAP focuses on consistency and availability during partitions, but latency, durability, operational complexity, recovery time, and consistency outside partitions are also critical design concerns.
Trade-offs at a glance
The following table summarizes trade-offs between consistency-first (CP) and availability-first (AP) approaches under a partition:
| Factor | Consistency-first (CP) | Availability-first (AP) |
|---|---|---|
| Availability during partition | Reduced: may reject or delay requests | High: serves all requests locally |
| Consistency guarantee | Strong linearizability | Eventual consistency, may diverge |
| User experience | May see errors or stalled requests during partitions | Writes appear responsive with potential conflicting states |
| Complexity | Complex consensus protocols (Paxos, Raft, etc.) required | Conflict resolution and merge logic needed |
| Data correctness risk | Low during partition | High: risk of lost or conflicting updates |
| Recovery after partition | Usually automatic recovery due to coordination | Requires manual or algorithmic reconciliation |
| Use cases | Financial, inventory, unique constraints | Social media, collaborative editing, telemetry |
Understanding these trade-offs helps align system design with business priorities, driving explicit policy choices rather than vague labels.
Observability for CAP systems
Monitoring and observability are vital to verify that your system behaves as intended under partitions and to troubleshoot issues.
What to log
- Operation metadata: request ID, client ID, timestamp, operation type (read/write), replica ID
- Coordination attempts: latency and outcome of quorum or leader coordination
- Partition detection or network errors: timeouts, failed connections, suspect peer state
- Rejected operations (consistency-first): reason for unavailability, policy in effect
- Locally accepted but uncoordinated writes (availability-first): conflict flags, reconciliation status
- State transitions: leader election, quorum membership changes
Example structured JSON log from a consistency-first write attempt failing due to unavailable quorum:
{
"timestamp": "2024-06-15T10:32:12Z",
"level": "WARN",
"service": "payment-api",
"operation": "WriteBalanceUpdate",
"replica": "A",
"requestId": "abc123",
"peerReachable": false,
"policy": "ConsistencyFirst",
"result": "RejectedUnavailable",
"message": "Write rejected due to unavailable quorum"
}
Key metrics
Use combined RED and USE metrics to measure system health and performance under partitions:
| Metric | Description | Target / Alert Condition |
|---|---|---|
| Request Rate (RED) | Number of read and write requests per second | Sudden drops or spikes indicating availability issues |
| Error Rate (RED) | Proportion of failed or rejected requests | Alert if errors exceed baseline or SLA threshold |
| Duration / Latency (RED) | Operation latency percentiles (p95, p99) | Alert on increased latency indicating coordination delays |
| CPU/Memory (USE) | Resource utilization on replicas | Alert on resource exhaustion risking unresponsiveness |
| Network I/O (USE) | Message rate and dropped packets | Alert on unusual partitions or network failures |
| Leader Stability | Frequency of leader changes in consensus | Alert if frequent elections disrupt consistency |
Distributed tracing with OpenTelemetry (.NET example)
Instrument your coordination and API layers to trace request paths and latencies. Here is an example to create a span around a write attempt:
using OpenTelemetry;
using OpenTelemetry.Trace;
var tracerProvider = Sdk.CreateTracerProviderBuilder()
.AddSource("MyDistributedSystem")
.AddConsoleExporter()
.Build();
var tracer = tracerProvider.GetTracer("MyDistributedSystem");
using var span = tracer.StartActiveSpan("TryWrite");
span.SetAttribute("replica", "A");
span.SetAttribute("policy", "ConsistencyFirst");
span.SetAttribute("peerReachable", false);
bool result = TryWrite(replica: "A", value: "closed", peerReachable: false, policy: WritePolicy.ConsistencyFirst);
span.SetAttribute("result", result ? "accepted" : "rejected");
span.End();
Instrumenting calls with tracing lets you correlate failed writes with network issues or slow consensus rounds for rapid troubleshooting.
Typical SLI and SLO definitions
| SLI | Description | SLO Example |
|---|---|---|
| Successful Write Rate | Percentage of client writes accepted without error | 99.9% availability over rolling 30 days |
| Consistent Reads | Percentage of reads returning linearizable values | 99.99% during normal operation |
| Request Latency | 95th percentile latency for reads/writes | < 150 ms for 95% of requests |
| Partition Recovery Time | Time to restore full coordination after partition | < 5 minutes |
Alerts to page on
- High rate of request rejections due to unavailable quorum (CP systems)
- Sustained packet loss or network partitions between replicas
- Leader election flaps causing temporary service unavailability
- Sudden divergence detected in availability-first systems
- Replica unresponsiveness beyond acceptable timeout windows
Observability ensures your system enforces CAP expectations and surfaces unexpected partition consequences before impacting customers.
CAP theorem is a principle during partitions, not a fixed label. It forces us to make explicit choices about which way to bend: consistency or availability. Understanding this trade-off, quantifying it in policies, and operating systems with clear observability are essential for resilient distributed design.