Planning a Shard Split with VReplication

Splitting one hot shard is mostly a planning exercise: by the time you type Reshard create, the boundary, the target capacity, and the tablet placement should already be settled, because VReplication will faithfully copy whatever you point it at — including a bad plan.

Context

This page sits under Resharding Workflows and Shard Splits, which covers running the workflow end to end; here the focus is everything that must be true before the copy starts. The boundary you choose is constrained by the keyspace’s partitioning model — how the sharding key maps into the keyspace-ID space is defined in Understanding Vitess Keyspace Partitioning Models — and the placement of the new tablets follows the failure-domain rules established across Vitess Sharding Architecture & Topology Design. Planning does not change those inputs; it consumes them.

Choosing the split boundary

A shard named -80 owns keyspace IDs from zero up to 0x80. The default and safest split is the midpoint bisection: -80 becomes -40 and 40-80, because 0x40 is exactly half of 0x80. Under a hash vindex, keyspace IDs are uniformly distributed, so a midpoint boundary splits both the stored rows and the incoming write rate close to evenly — which is the whole point of the exercise.

The midpoint stops being the right choice when the key distribution is skewed. If load-testing or per-range row counts show that most traffic lands in the lower half of -80, a midpoint split leaves -40 still hot and 40-80 nearly idle. In that case you deliberately bisect off-centre — carving -60 and 60-80, for example — so the busy sub-range gets its own shard. You can measure where the mass sits before committing by counting rows per candidate range on the source primary.

Midpoint versus skew-aware bisection of shard -80 Two number lines over the same keyspace-ID range. The top line splits at the midpoint 0x40 into two equal shards. The bottom line splits off-centre at 0x60 to give a busy lower range its own shard. Midpoint -40 40-80 00 40 80 Skew-aware -60 (busy) 60-80 00 60 80

Whichever boundary you pick, keep the hexadecimal digits aligned to a clean power-of-two prefix where you can — -40, -60, -c0 are all valid single-nibble boundaries. Odd boundaries like -4a are legal but make the next split harder to reason about, so reserve them for genuine hotspot isolation.

To measure where the mass actually sits, count rows by the high bits of the keyspace ID on the source primary before committing to a boundary. Vitess exposes the computed keyspace ID, so you can bucket rows into candidate sub-ranges and see the real distribution rather than guessing:

-- On the source shard's primary: how many rows fall in each candidate half?
-- keyspace_id() returns the vindex-computed id; HEX its first byte to bucket.
SELECT
  CASE WHEN HEX(LEFT(keyspace_id, 1)) < '40' THEN '-40' ELSE '40-80' END AS candidate_shard,
  COUNT(*) AS rows_in_range
FROM orders
GROUP BY candidate_shard;

If the two buckets are within a few percent of each other, the midpoint is safe; if one holds most of the rows and — more importantly — most of the writes, shift the boundary toward the busy side so the hot sub-range gets a shard to itself. Row count is a proxy; where you can, weight the decision by measured write rate per range, since it is write pressure, not stored size, that made the shard hot.

Sizing target capacity and timing

Two shards that together carry the source’s load are not automatically enough. Plan capacity against the post-split peak each child will see, not the pre-split average of the parent. If -80 peaks at 8,000 writes/second and the split is even, each child must comfortably serve 4,000/second plus the copy write amplification while the reshard is running — the copy itself lands on the target primaries at the same time production writes do.

Timing follows from that. The copy phase reads the entire source range and writes it into the targets, so schedule it to start in a traffic trough and expect it to run for hours on a large table. Estimate the copy duration from the source table’s on-disk size and your measured VReplication copy throughput on a comparable table; a 400-million-row table tells you nothing from a 200-row rehearsal. Leave replica headroom on both source and target so the throttler is not pausing the copy for the entire window.

Storage is the other half of capacity. Each target primary must hold its slice of the source data plus the binary logs generated during the copy, and a target that runs out of disk mid-copy fails the stream and forces a restart. Provision each child’s disk for its post-split data set with generous headroom, and confirm the source has room for the reverse-stream binary logs that accumulate after the primary switch. Under-provisioning disk is the quiet failure that surfaces hours into a copy, when the least time is available to fix it.

Rehearsing the plan on staging

A reshard plan is cheap to rehearse and expensive to get wrong in production. On a staging keyspace loaded with representative row counts, run the full create → copy → VDiff → switchtraffic → reversetraffic → complete sequence and measure the two numbers you cannot get any other way: the real copy throughput on your hardware and the empirical write-freeze duration at the primary switch. Those numbers size the production window. A rehearsal on a 200-row table validates command syntax and nothing else — the throttling and lag behaviour that decide whether the plan converges only appear at production-like scale.

Provisioning target tablets across failure domains

The split does not create tablets — it copies into shards you have already stood up. Before create, each target shard needs a full tablet set placed so no single zone loss removes its quorum: a PRIMARY, at least one promotable REPLICA in a different zone, and an RDONLY for batch and as a copy source. A child shard whose primary and only replica share a rack has the same non-availability the parent would have had.

# Confirm both target shards exist and carry a healthy tablet set
# across distinct cells before starting the copy.
vtctldclient GetTablets --keyspace commerce --shard -40
vtctldclient GetTablets --keyspace commerce --shard 40-80

Every listed shard should show a PRIMARY plus replicas in cells that map to different failure domains. If a target shard is missing a tablet type or has co-located tablets, fix placement now; adding a promotable replica after cutover is a scramble under load.

The Reshard create invocation

With boundary, capacity, and placement settled, the launch is a single command. Read the copy from replica,rdonly so the source primary is not carrying the scan, and set --on-ddl STOP so an in-flight schema change halts the stream instead of silently diverging the target:

vtctldclient Reshard --workflow split80 --target-keyspace commerce create \
  --source-shards '-80' --target-shards '-40,40-80' \
  --tablet-types 'replica,rdonly' \
  --on-ddl 'STOP' \
  --defer-secondary-keys

--defer-secondary-keys copies rows into the targets without their secondary indexes and builds those indexes after the bulk copy, which is markedly faster on wide tables. The routing layer is untouched by create; queries keep flowing through the VTGate routing layer to the source until you later switch traffic.

The workflow name (split80 here) is not cosmetic — it is the handle every subsequent command references, so choose one that encodes what the split does and stays unique within the keyspace. Vitess derives the reverse workflow’s name from it (split80_reverse), and your monitoring and orchestration will key off it, so a vague name like reshard1 becomes a liability the moment a second reshard is in flight. Record the exact create invocation, the chosen boundary, and the target tablet aliases in your change record before running it; if the copy has to be diagnosed at 3am, the plan you wrote down is the reference for what “correct” looks like.

Edge cases and gotchas

  • Uneven key distribution. A hash vindex distributes keyspace IDs uniformly, so a midpoint split balances load; a range-style or naturally skewed key may not. Measure per-range row counts before assuming the midpoint is balanced, and bisect off-centre if the mass is lopsided.
  • In-flight DDL during the copy. A schema migration on the source mid-copy can leave the target with a different table shape. Freeze schema changes for the copy window, and set --on-ddl STOP so a stray migration stops the stream loudly rather than corrupting it quietly.
  • Insufficient target capacity. If a child primary cannot absorb its post-split write rate plus copy amplification, the throttler pauses the copy repeatedly and the reshard never converges. Provision the targets for peak, not average, before starting.
  • Sequence tables and unique keys. If the source uses a Vitess sequence for auto-increment, confirm the sequence keyspace and the targets agree on the next value; a split does not re-seed sequences, and a stale value causes duplicate-key errors after cutover.
  • Copy source too weak. Reading the copy from a single lightly-provisioned RDONLY can bottleneck throughput. Ensure the tablet types you name in --tablet-types have the IO headroom to feed the copy.

Verification

Before advancing past the copy, confirm every stream has finished copying and is only tailing the binary log within your lag threshold:

vtctldclient Workflow --keyspace commerce show --workflow split80

Every stream should report state Running (not Copying) with lag inside the throttle threshold, and both target shards should appear with a caught-up position. Only then is the plan proven enough to move on to verifying data integrity and switching traffic.

← Back to Resharding Workflows and Shard Splits