The day you actually shard

Before you start

• Databases

  Queues and partitions sit around a primary database. That lesson covers the key and the index.

  Open lesson

Why it matters

The shortener’s primary has grown to 20,000 creates per second and 30 TB, beyond one machine’s comfortable write and disk limits. Reads are already cached, so adding read capacity does not remove either wall, while a poor data split could make every redirect search many places. The larger layout must preserve redirect as a single-key lookup.

Creates have grown to 20,000 a second and the primary’s disk and write throughput are the wall. Reads are already cached.

Input

writes_per_sec = 20_000, storage_tb = 30, max_writes = 5_000, max_tb = 4

Output

8

For 20,000 creates/s, 30 TB, a 5,000-write/s limit, and 4 TB per machine, use eight shards keyed by code; x7Kp hashes directly to one shard instead of making redirects search owner-based shards.

At 5,000 writes per second per machine, throughput needs four shards. At 4 TB per machine, 30 TB needs eight, so disk sets the count. Each of eight shards receives about 2,500 writes per second and stores about 3.75 TB.

Hash the short code to choose one of those eight shards on both create and redirect. The caller already has x7Kp, so it can find the same shard directly. Owner-based queries become harder and need an index or fanout; that is the cost of preserving the hottest lookup.