(Image generated by AI)
The Golden Rule of Distributed Systems
When designing modern systems, we rarely rely on a single database running on a single computer. We use distributed systems—clusters of databases running across multiple servers, often in different cities, communicating over a network.
When you build a distributed database, you are immediately bound by an unbreakable law of computer science: The CAP Theorem.
Formulated by computer scientist Eric Brewer, the CAP Theorem states that a distributed data store can only simultaneously provide two out of the following three guarantees:
- Consistency
- Availability
- Partition Tolerance
Let’s break down what these terms actually mean in plain English.
The 3 Pillars
1. Consistency (C)
Every read receives the most recent write, or an error. If User A updates their profile name from “John” to “Johnny”, and User B views the profile one millisecond later, User B must see “Johnny”. All nodes in the database cluster must agree on the exact same data at the exact same time.
2. Availability (A)
Every request receives a non-error response, without the guarantee that it contains the most recent write. The system is always online and ready to answer. If a server is running, it will give you an answer, even if that answer is slightly outdated.
3. Partition Tolerance (P)
The system continues to operate despite an arbitrary number of messages being dropped or delayed by the network between nodes. Networks fail. Cables get cut. Routers break. If Server 1 and Server 2 can no longer communicate with each other (a partition), the overall system must keep running.
Why You Must “Choose Two”
Here is the secret of the CAP Theorem: On the internet, Partition Tolerance (P) is not a choice, it is a requirement.
Because you are communicating over a network you do not control, network partitions will happen. Therefore, you are actually forced to choose between Consistency (CP) or Availability (AP) when a partition occurs.
Scenario: The Network Breaks
Imagine a database cluster with Server A in New York and Server B in London. They are constantly syncing data. Suddenly, the transatlantic internet cable gets cut. A network partition has occurred. Server A and Server B can no longer talk to each other.
Now, a user in New York connects to Server A and changes their password. A second later, the user’s phone in London connects to Server B and tries to log in.
Option 1: You chose Consistency (CP) If your database prioritizes Consistency, Server B realizes it cannot talk to Server A to verify if the password was recently changed. Because it cannot guarantee it has the most consistent data, Server B returns an error (or shuts down). Result: Data is perfectly accurate, but the system is entirely unavailable for the user in London.
Option 2: You chose Availability (AP) If your database prioritizes Availability, Server B realizes it cannot talk to Server A, but says, “I’ll just use the old password I have on file.” It allows the user to log in with the old password. Result: The system is highly available (no errors!), but the data is completely inconsistent between New York and London.
Real-World Examples
You must choose your database based on the business requirements of your application.
When to choose CP (Consistency over Availability)
Banking and Financial Apps. If you have $100 in your account, and you withdraw $100 from an ATM in New York, the database must immediately be consistent before your wife tries to withdraw $100 from an ATM in London. If a network partition happens, the bank would much rather deny the transaction (Unavailable) than allow you to overdraft (Inconsistent). Databases: MongoDB, HBase.
When to choose AP (Availability over Consistency)
Social Media Feeds and E-Commerce Carts. If you post a picture on Instagram, it is completely fine if your friend in Australia doesn’t see it on their feed for another 5 minutes (Eventual Consistency). The primary goal is that Instagram is always online, never returning an error, and always allowing you to scroll. Databases: Cassandra, DynamoDB, CouchDB.
Conclusion
You cannot cheat the CAP theorem. As a software engineer, your job is not to build a system that has 100% of all three traits, but to understand your business requirements and choose the right trade-off when the network inevitably fails.