Tech • AI • Robotics • Game
Database Sharding Explained for Beginners (with Hands-On Labs)
5/10TL;DR
Database sharding splits rows across multiple machines to scale write-heavy systems beyond the limits of a single primary database.
KEY POINTS
Why sharding becomes necessary
A small social app such as Chirp can begin with one application server and one database, handling user records and short posts with simple reads and writes. As usage grows from hundreds to thousands, teams typically scale up by renting larger machines. But once the user base reaches the millions, a single database can hit hard limits on throughput and storage.
Reads can be eased with replicas
Slow reads often appear first. Profile pages that once loaded in about 30 milliseconds can degrade to roughly 3 seconds when too many queries pile up on one machine. Replicas help by keeping exact copies of the data and serving read traffic, while also improving failover if the primary goes down.
Writes remain tied to the primary
Replication does not solve write saturation because multiple machines cannot safely accept conflicting updates to the same row without risking inconsistency. A post with 99 likes could incorrectly stay at 100 if two simultaneous writes hit different replicas. For that reason, the primary remains the source of truth, and heavy write traffic eventually turns it into the system bottleneck.
What sharding changes
Sharding, also called horizontal scaling, spreads rows across several ordinary machines instead of relying on one larger server. Each shard holds only part of the data but keeps the same schema as the others. Splitting users across four shards, for example, creates four write targets instead of one, letting the system handle far more concurrent writes.
Shard keys determine success or failure
The column used to place data is the shard key. A good shard key should have high cardinality, distribute load evenly, match common access patterns, stay stable over time, and preserve collocation of related records. User ID fits those requirements better than fields like country, signup year, or username, which can create skew, scatter-gather queries, or costly data moves.
Hashing maps any key to a shard
When the shard key is not naturally numeric, a hash function turns values such as usernames or email addresses into numbers. The same input must always produce the same output, which is why unstable runtime hashes are unsuitable for sharding. Systems such as MongoDB support hashed sharding to spread adjacent values widely and keep distribution more even.
Other sharding strategies exist
Range sharding works well when most queries target value ranges such as dates, but it can overload the newest shard with fresh writes. Directory sharding uses a lookup table to place large tenants manually, at the cost of adding a critical routing dependency. Geographic sharding stores data by region for speed or legal compliance, often followed by another method such as hashing within each region.
Routing is a core part of the design
Every request needs a routing layer that computes the target shard and forwards traffic correctly. That logic can live inside each application service, in a central proxy, or in the database layer itself. App-side routing is fast but hard to update everywhere; proxies are simpler to manage but can add latency and become bottlenecks; database-managed routing reduces application work but limits flexibility.
Hotspots create the celebrity problem
Even well-balanced shards can fail under uneven popularity. A celebrity with 40 million followers can overload one shard while others stay lightly used. Common responses include moving that account to a dedicated shard through a directory rule, serving most reads from a cache, and spreading write-heavy counters with salting, where one hot key is split into multiple logical keys across existing shards.
Simple modulo sharding breaks during growth
A common beginner rule is to hash a key, divide by the number of shards, and use the remainder. It works at first but causes painful resharding when capacity is added. Moving from 4 to 5 shards can change the destination for most records, forcing a large-scale migration even though the underlying data has not changed.
Consistent hashing reduces data movement
Consistent hashing places shards on a hash ring and sends each key to the first shard found clockwise from its hash value. When a shard is added, only part of the ring moves, rather than most of the dataset. To avoid uneven ownership, systems place each physical shard on the ring multiple times as virtual nodes, improving balance and reducing the blast radius when machines are added or removed.
CONCLUSION
Sharding is the main path for scaling database writes, but it introduces difficult choices around shard keys, routing, hotspots, and rebalancing. Designs that work at small scale can become costly later, which is why careful planning around distribution and growth is essential from the start.
Ask a question