Distributed Systems

Distributed systems run computation across multiple machines that communicate over a network to present a unified interface. Your web app is already a distributed system: client, server, database, cache, queue - each component can fail independently, and the network between them is unreliable.

The core difficulty is that messages can be delayed, duplicated, or lost. Machines crash and restart. Clocks drift between nodes. Two nodes can hold inconsistent views of the world at the same instant. Most backend complexity traces back to these constraints.

Why it matters in Engineering: Every senior engineer makes decisions shaped by distributed systems fundamentals - even when not building infrastructure. When is it safe to retry? How do you design for partial failure? When does eventual consistency cause product bugs? These questions surface constantly in API design, database choices, and service architecture.

Core Concepts

CAP Theorem
A distributed system can only guarantee two of three properties: Consistency, Availability, Partition tolerance. Since network partitions are inevitable, you are really choosing between consistency and availability during a failure.

Eventual Consistency
Nodes may temporarily hold different views of data but converge to the same state over time. Many distributed databases - DynamoDB, Cassandra - are eventually consistent by default.

Consensus
Getting multiple nodes to agree on a value or event sequence. Protocols like Raft and Paxos solve this. Kubernetes uses etcd (Raft-based) for its control plane.

Idempotency
Operations that can be retried without side effects. Critical when designing APIs and message handlers in any distributed context.

Observability
Distributed failures are hard to debug without structured logs, metrics, and traces. See Observability for how these tools work together.