Consistency & Availability

The eternal trade-off. Learn why you can't have it all (CAP), how to handle the 99% of time when things ARE working (PACELC), and how to tune your truth.

What you will learn

• Grasp the tragedy of the CAP Theorem (Pick 2 of 3)
• Understand why 'Partition Tolerance' is not optional
• Master PACELC: The modern refinement of CAP
• Use Quorums (R + W > N) to mathematically tune consistency

Consistency & Availability

In a single-computer world, the truth is simple. If you write x = 5 to your hard drive, and then read x, it is 5.

In a distributed system, "Truth" is a slippery concept. If you have data replicated across New York, London, and Tokyo... and a user in New York sets x = 5... does the user in Tokyo see 5 instantly?

• If yes, you have Consistency.
• If no, you have Eventual Consistency.

This chapter explores the physics that prevent us from having it all.

1. The CAP Theorem

Brewer's CAP Theorem is the E=mc² of distributed systems. It states that a distributed data store can only provide two of the following three guarantees:

1. Consistency (C)

"Every read receives the most recent write or an error." If I write to Node A, and you read from Node B immediately after, you must get my write. If Node B hasn't updated yet, it shouldn't return old data; it should fail or wait.

2. Availability (A)

"Every request receives a (non-error) response, without the guarantee that it contains the most recent write." The system must always reply. It’s allowed to say "x=4" (old data), but it’s not allowed to say "System Error" or hang indefinitely.

3. Partition Tolerance (P)

"The system continues to operate despite an arbitrary number of messages being dropped or delayed by the network." If the cable between New York and London is cut, the system keeps running.

Brewer's CAP Theorem is the most famous law in distributed systems. It states you can only have two of three:

2. The Great Misunderstanding: "Pick Two"

You will often hear: "You can pick separate C, A, or P. Pick any two!"

This is wrong. In a distributed system over a network (like the Internet), Partition Tolerance (P) is mandatory. You cannot choose to have perfect networks. Cables will be cut. Routers will fail.

So your real choice is: When a Partition occurs (P), do you choose C or A?

The CP System (Consistency over Availability)

"I would rather return an ERROR than give you wrong data."

• Scenario: An ATM network.
• Partition: The ATM loses connection to the bank server.
• Behavior: The ATM refuses to give you money. It says "Service Unavailable". It cannot verify your balance, so it chooses to fail safely (Consistency) rather than guess and risk an overdraft.

The AP System (Availability over Consistency)

"I would rather give you OLD data than give you an ERROR."

• Scenario: Facebook Feed.
• Partition: The US datacenter loses connection to the Europe datacenter.
• Behavior: You post a photo in the US. Your friend in Europe doesn't see it yet. That's fine. It's better that they can still scroll their feed (Availability) than show a "Service Down" page just because one photo is missing.

3. Beyond CAP: The PACELC Theorem

CAP is great, but it has a flaw: It only discusses what happens when things break (Network Partitions). But 95% of the time, the network is fine! What trade-offs do we make then?

Enter PACELC (pronounced "Pass-Elk").

If P (Partition): Choose A (Availability) or C (Consistency). Else (No Partition): Choose L (Latency) or C (Consistency).

This adds a nuance: Latency.

When the network is working perfectly, you still have a choice:

1. High Consistency: You write to Node A. Before you say "Success", Node A must replicate to Node B and Node C. This takes time. (High Latency).
2. Low Latency: You write to Node A. It says "Success" immediately. It replicates to B and C in the background. (Low Consistency).

Example: DynamoDB

Amazon's DynamoDB allows you to choose at runtime:

• ConsistentRead: true -> Uses more RCU (money) and takes longer (Latency), but guarantees truth.
• ConsistentRead: false -> Faster (Latency), cheaper, but might return data from 1 second ago.

4. Tuning Consistency: The Quorum Formula

How do databases actually enforce these rules? It's not magic; it's simple math called Quorums.

$ R + W > N $

• N: The total number of replicas (e.g., 3 nodes).
• W: The number of nodes that must verify a Write.
• R: The number of nodes that must verify a Read.

Scenario: N = 3 (3 Replicas)

1. Strong Consistency (R+W > N)

We set W=2 and R=2. $2 + 2 = 4$ (which is

• Why it works: By Pigeonhole Principle, there must be an overlap. At least one node in your "Read Group" was also in the "Write Group". You are guaranteed to see the latest data.

2. High Performance (Eventual Consistency)

We set W=1 and R=1. $1 + 1 = 2$ (which is

• Behavior: You write to Node A. You read from Node C. Node C hasn't heard from Node A yet. You get old data.
• Benefit: Blazing fast. Node A didn't have to wait for anyone.

5. Summary

The Architect's Decision Matrix

1. Money: Banking, Inventory, Billing? -> CP (Strong Consistency). You cannot double-spend money.
2. Likes/Comments: Social Media, Analytics? -> AP (Eventual Consistency). Speed is more important than absolute truth.
3. Tuning: Use Quorums to adjust the dial between speed and truth depending on the specific query.