Sharding

Sharding is the practice of horizontally partitioning a dataset across multiple database instances so each shard holds a subset of the data and serves a subset of the load. Sharding is the standard answer when a single database server cannot keep up with storage, read throughput, or write throughput.

How it works

A shard key (one or more fields) determines which shard each row or document belongs to. A router or proxy uses the shard key to dispatch each query to the right shard, gather partial results from many shards if needed, and combine them. The shard key choice is the most consequential design decision in a sharded system.

Common partitioning strategies

• Hash sharding. Apply a hash to the shard key; distributes data evenly; loses range-query efficiency.
• Range sharding. Each shard owns a contiguous key range; efficient range scans; risks hotspots if keys are time-ordered.
• List or zone sharding. Hand-assign key values or buckets to shards; useful for geographic placement or tenant isolation.

Common pitfalls

• Hot shards. A poor key concentrates load on one shard.
• Cross-shard queries. Joins and transactions that span shards cost much more than single-shard operations.
• Re-sharding. Changing the shard key after data is loaded requires migration.

Where sharding shows up

• Application-level: Vitess (for MySQL), Citus (for PostgreSQL), client-side sharding.
• Native: MongoDB sharded clusters, Cassandra, ScyllaDB, DynamoDB partitions, CockroachDB ranges.
• Cache-level: Redis Cluster hash slots.