Distributed Consistency: CAP, Raft, and Consensus

Let us start with a classic production incident:

The fiber link between two data centers became unstable, cutting the network connection between the primary and standby databases for 30 seconds. During those 30 seconds, both machines believed they were the "primary" and accepted writes independently. When the network recovered, the engineers faced two conflicting datasets. This is a textbook split-brain scenario.

If you build distributed systems, you will run into this kind of problem sooner or later. Multiple machines form one system, but networks fail, machines crash, and clocks drift. On such an unreliable foundation, how do you make the data "look as though only one copy exists"? That is what consistency and consensus are intended to solve.

This article explains four things thoroughly: what the CAP theorem actually says, and how it is commonly misunderstood; the spectrum of consistency guarantees; how Raft reaches consensus in plain language; and how engineers choose among primary-replica replication, Quorum, and CRDT in practice.

I. Why Consistency Is Unavoidable

A standalone database does not need to worry about consistency: one copy is written to disk, and that is the copy it reads. Distributed systems, however, inherently have multiple replicas: for high availability, so another machine can take over if one fails; for low latency, by placing data close to users; and for read throughput, by reading from multiple replicas in parallel.

As soon as data is replicated across multiple machines, problems arise:

• After writing to one machine, when can another machine read the data? Immediately, or several seconds later?
• If two machines receive different writes simultaneously, which one wins?
• If both sides accept writes during a network outage, how should those writes be reconciled afterward?

Replicas are the root of every problem in distributed systems. Consistency answers the question: "Can multiple replicas behave externally as a single system at any given moment?"

There is no such thing as "perfect consistency" in engineering; that would effectively be a standalone system. The real task is to balance "stronger consistency" against "greater availability and better performance." CAP defines the theoretical limit of this tradeoff.

II. The Truth About CAP: Stop Calling It "Pick Two"

The most common misconception is: "CAP means choosing two out of consistency, availability, and partition tolerance." This leads some people to confidently declare, "Our system is CA." That statement is almost always wrong.

CAP is accurately stated as follows: when a network partition occurs, you can choose either consistency or availability, but not both.

• C (Consistency): Reads always return the latest write.
• A (Availability): Every request receives a non-error response, although the response is not guaranteed to contain the latest data.
• P (Partition Tolerance): The network may lose, partition, or delay messages.

The key insight is: P is not optional; it is a physical fact. Networks will partition. Switches become unstable, fiber cables get severed, and machine network interfaces become overloaded. A distributed system cannot pretend that P does not exist.

The real tradeoff therefore looks like this:

No partition (normal conditions): C and A can coexist; everyone is happy.
Partition occurs (network fails): Either reject writes (C, sacrifice A),
                                  or let each side write independently (A, sacrifice C).

CAP should be understood as a "consistency/availability tradeoff during a partition," not as "choosing two out of three." There is no genuinely CA distributed system, unless it is a standalone machine.

In engineering practice:

• Most traditional relational database sharding middleware preserves C during a partition by rejecting writes on the split-brain side.
• Systems such as Cassandra and Dynamo preserve A during a partition by accepting writes first and converging later through mechanisms such as vector clocks and read repair.
• Most modern systems, including ZooKeeper, etcd, and consul, follow the CP approach: consensus protocols guarantee strong consistency at the cost of availability for the minority partition.

In one sentence: CAP is not a tool for selecting a system; it is a tool for understanding the limits of a system's behavior under failure.

III. The Consistency Spectrum: It Is Not Black and White

C in CAP is a binary concept, but consistency in the real world is a continuous spectrum. From strongest to weakest, it can be divided into roughly four levels:

The stronger the guarantee, the greater the implementation cost: coordination, locking, and cross-node RPCs are required, increasing latency. Weaker guarantees offer better performance and availability, but the application layer must tolerate "reading stale data."

An engineering judgment: eventual consistency is sufficient for the vast majority of internet applications. Only a few scenarios, such as account balances, inventory deductions, and distributed locks, require linearizability. Do not demand global strong consistency from the outset; you will only create problems for yourself.

IV. Raft: A Consensus Protocol Explained in Plain Language

How do you implement linearizability? With a consensus algorithm that enables a group of machines to agree on "what should be written next." Raft is currently the most widely used consensus protocol. etcd, Consul, TiKV, Redis Cluster's gossip, and Kubernetes state all rely on it.

Raft replaced Paxos as the de facto standard for teaching and engineering for one central reason: it was designed for understandability. It divides consensus into three clear subproblems: leader election, log replication, and safety.

1. Leader Election

A Raft cluster has only one Leader at any given time, and all write requests go through it. The election process is:

Node starts / Leader becomes unreachable (heartbeat timeout)
        ↓
  Increment term by 1 and become a Candidate
        ↓
  Vote for itself and send RequestVote to other nodes in parallel
        ↓
  Receive a majority (N/2 + 1) of votes → become Leader
        ↓
  Send periodic heartbeats to maintain authority; other nodes become Followers

The key design is majority voting (quorum). Not every node needs to agree; only a majority is required. A five-node cluster can still elect a leader after losing two nodes. If it loses three, it cannot elect a Leader and the cluster stops serving requests. This is the cost of a CP system.

2. Log Replication

When the Leader receives a write request, it does not simply write the data and return. It follows a replication process:

Client → Leader writes to local log (uncommitted)
              ↓
        Leader replicates to all Followers in parallel
              ↓
        A majority of Followers replicate successfully → Leader marks it "committed"
              ↓
        Leader returns success to Client and asynchronously tells Followers to commit

Why a majority rather than every node? Because a "majority plus a majority" must overlap: any two majorities share at least one node. This guarantees that a committed log entry cannot be overwritten, providing safety. If every node were required, one slow node could stall writes across the entire cluster.

3. The Complete Lifecycle of a Write Request

Client → Leader:  SET x=5
Leader:  append log [term=3, idx=7, SET x=5]
Leader → Followers:  AppendEntries RPC
Followers:  write log, return OK
Leader:  receive acknowledgments from 3/5 nodes (including itself) → commit
Leader:  apply to state machine (x=5), return success to Client
Leader → Followers:  next heartbeat includes commitIndex; Follower commits

This mechanism guarantees that committed logs are never lost, all nodes eventually become consistent, and data remains continuous when the Leader changes. The cost is that every write requires a cross-node RPC, resulting in latency an order of magnitude higher than on a standalone machine. That is the "price tag" of strong consistency.

V. Paxos: The Founding Father You Only Need to Know Exists

Before Raft, the de facto standard for consensus algorithms was Paxos (Lamport 1998). It is mathematically elegant and supported by complete proofs, and both Google Chubby and Spanner are based on it. It has only one problem: it is extremely difficult to understand. Its "prepare phase," "accept phase," and proposal number mechanics leave readers thoroughly confused, while engineering implementations vary widely because Multi-Paxos has no standard implementation.

Raft's author, Diego Ongaro, was frustrated by Paxos and deliberately designed an algorithm that was "functionally equivalent but easier to understand." In engineering practice, understanding Raft is enough. Leave Paxos for reading papers and showing off in interviews. When you see Paxos, it is sufficient to know that it is Raft's ancestor, theoretically stronger but harder to engineer.

VI. How Real Systems Make the Tradeoff

With the principles established, let us examine practice. Three typical engineering tradeoffs correspond to three levels of consistency:

1. Asynchronous Primary-Replica Replication (Eventual Consistency, Availability First)

MySQL semi-synchronous replication, Redis primary-replica replication, and ES replicas essentially belong to this category. The Leader writes locally and returns immediately, then pushes the log to replicas asynchronously.

• Advantage: Write latency is low. Although a small amount of data may be lost if the Leader fails, the system as a whole remains available.
• Risk: Writes that have not yet been replicated when the Leader fails will be lost, and reads may return stale data during asynchronous replication.
• Suitable for: Logs, user behavior, caches, and search indexes; scenarios that can tolerate data delays of several seconds.

2. Quorum / NWR (Tunable Consistency)

Dynamo, Cassandra, and Riak give users the choice through the NWR model:

• N: Total number of replicas
• W: Number of replicas that must succeed for a write
• R: Number of replicas that must respond to a read

As long as W + R > N, reads and writes are guaranteed to overlap, providing strong consistency. For a typical N=3 configuration:

• W=2, R=2: Strong consistency with moderate latency, the mainstream configuration.
• W=1, R=1: Eventual consistency with the highest performance.
• W=3, R=1: Suitable for write-heavy, read-light scenarios.

The essence of Quorum is "tunable consistency": the same storage system can use different configurations for different workloads. This is a highly practical engineering design.

3. CRDT (Conflict-Free Data Types, Eventual Consistency Without Coordination)

CRDT (Conflict-free Replicated Data Type) takes another approach: the data structure itself guarantees that the final result will be consistent regardless of merge order.

Typical applications include collaborative documents, online documents, Notion, Yjs, RedisCRDTs, and Riak Map/Set. You and another person can edit the same passage simultaneously, apply changes locally, and merge them in the background without conflicts because data structures such as G-Counter and LWW-Set mathematically guarantee convergence.

The cost of CRDT is limited expressive power because not every operation can be merged without conflicts. Its advantage is the complete absence of coordination costs, maximizing availability. CRDT is the preferred choice for collaboration, social, and IM state synchronization.

VII. ACID vs BASE: Two Philosophies

The database world has ACID (Atomicity, Consistency, Isolation, Durability), which emphasizes strict transactional consistency. The distributed systems world has BASE (Basically Available, Soft state, Eventually consistent), which emphasizes sacrificing strong consistency for availability and performance.

These philosophies are not opposites. They represent tradeoffs at different levels:

• ACID transactions within a single database ensure that data inside the database remains correct.
• Distributed transactions across databases and services, such as 2PC and TCC, provide global strong consistency, but they are heavy, slow, and error-prone, so use them cautiously in production.
• The BASE approach accepts a softly consistent intermediate state and uses compensation, idempotency, and messages for eventual reconciliation.

The mainstream engineering approach is Saga + idempotency + message queue: split a long transaction into local transactions and events, let each service commit independently, and use compensation to roll back failures. This approach is ten times more practical than 2PC and is the current de facto standard for consistency in microservices.

VIII. Do Not Demand Global Strong Consistency

Let us end with some practical judgment. A simple principle guides consistency choices: choose according to business value, not technical purism.

• Money, inventory, permissions: Strong consistency, specifically linearizability, is mandatory. A one-yuan discrepancy in an account balance is an incident. This is not where you cut costs; use Raft / database transactions / TCC.
• User profiles, product information: Eventual consistency is sufficient. Nobody will complain if a new avatar takes one or two seconds to appear.
• Social feeds, comments, likes: Eventual consistency + CRDW-like merging. Users cannot perceive the strength of the consistency guarantee, but the system can handle enormous traffic.
• Counters, rankings, statistics: Eventual consistency + asynchronous aggregation. Nobody cares about a 0.1% accuracy difference, but people do care about an order-of-magnitude difference in throughput.

Global strong consistency is a luxury. Every strongly consistent write requires cross-node coordination, amplifies latency, and compromises availability. Use it where it is genuinely needed, and confidently use eventual consistency everywhere else.

Summary

Let us condense the main points:

1. Replicas are the root of distributed systems problems. Consistency answers the question, "How can multiple replicas behave like one?" This is not technical purism; it is the baseline for business correctness.
2. The truth about CAP is that it describes "the tradeoff during a partition," not choosing two out of three. P is a physical fact. Under normal conditions, you can have both C and A; only when a partition occurs must you choose between CP and AP. Most storage systems choose CP.
3. Consistency is a continuous spectrum: linearizability is the strongest and most expensive, while eventual consistency is the weakest and fastest. Do not demand global strong consistency from the outset; choose the level according to business value.
4. Raft has three elements: leader election, log replication, and majority voting. Once you understand that "any two majorities must overlap," you understand why Raft is both safe and tolerant of minority failures. Paxos is the ancestor; knowing that it exists is enough.
5. Let money guide the choice: use strong consistency for financial scenarios, eventual consistency + CRDT / Saga for social and content systems, and "local transactions + messages + compensation" instead of 2PC for cross-service transactions.

One final line to remember:

The essence of distributed consistency is not eliminating conflicts, but facing them honestly: use strong consistency when money is involved, eventual consistency when traffic is involved, and consciously choose CP or AP when partitions occur. Strong consistency is a luxury, not an everyday necessity.

In the next article, we will discuss "Distributed Transactions: Saga, TCC, and the Transactional Outbox" and see how to make cross-service consistency more dependable.