Resharding Workflows and Shard Splits: Driving VReplication Reshard from Copy to Cutover

Resharding is the operation that lets a live Vitess keyspace change its shard boundaries without a maintenance outage — most commonly splitting one overloaded shard into two so write and storage pressure is redistributed. The mechanism is VReplication: Vitess copies rows from the source shard’s range into newly provisioned target shards, keeps them current by tailing the binary log, proves the copy is byte-for-byte correct with VDiff, and only then moves serving traffic across in carefully ordered phases. Nothing about the operation is atomic at the storage layer, so the discipline is entirely in sequencing — copy before verify, verify before switch, and always keep a reverse path alive until the change has proven itself under production load. This guide is written for MySQL SREs and platform engineers who own the cutover and are accountable when a shard split stalls at 2am. It builds directly on the abstractions defined in Vitess Sharding Architecture & Topology Design and assumes you have already decided why and where to split.

Prerequisites

Before you launch a Reshard workflow against a production keyspace, confirm the following:

  • Vitess 16+ for the modern vtctldclient Reshard command surface. Vitess 19+ tightens VDiff v2 semantics and the --tablet-types handling shown here; the examples assume 19 or newer. Older releases used vtctlclient Reshard with a slightly different argument order.
  • MySQL 8.0 primaries with ROW-based binary logging and GTIDs enabled on every source and target shard. VReplication tails the binary log with binlog_row_image=FULL, so a source with MINIMAL row images or GTIDs disabled cannot be resharded online.
  • Target tablets already provisioned and healthy across distinct failure domains. The split does not create tablets; it copies into shards you have already stood up. Each target shard needs a PRIMARY, at least one promotable REPLICA, and ideally an RDONLY, placed so no single zone loss removes a shard’s quorum — the placement discipline from Designing Horizontal Shard Topologies.
  • A correct VSchema whose primary vindex maps the sharding key into the keyspace-ID space. Resharding a keyspace whose partitioning model is wrong just relocates the problem; the trade-offs between range, hash, and lookup distribution are worked through in Understanding Vitess Keyspace Partitioning Models.
  • Headroom on the source replicas. The copy is read-heavy on the source and write-heavy on the targets; the replication lag and throttling signals you will watch throughout must have slack before you start, not after lag has already breached its threshold.

The reshard lifecycle end to end

The workflow moves through five phases, and each has a distinct rollback posture. During copy and verify the source still serves all traffic, so aborting is free. Once SwitchTraffic begins moving reads and then writes, rollback means running the reverse workflow rather than simply cancelling. The diagram maps the full path and the phased traffic switch, including the --reverse safety stream that keeps a path back to the source open after cutover.

The VReplication Reshard lifecycle from source shard to cutover A left-to-right pipeline of five stages: source shard, VReplication copy into two target shards, VDiff verify, phased SwitchTraffic, and complete with cleanup. A dashed reverse stream loops from the switch stage back to the source shard. Source shard -80 VReplication copy chunked + binlog tail VDiff verify row-count + content SwitchTraffic phased Complete cleanup drops source -40 40-80 target shards SwitchTraffic order 1 · rdonly + replica 2 · primary --reverse safety stream (writes can move back)

How the Reshard workflow moves data

A Reshard workflow is a set of VReplication streams, one per target shard, each reading from the source shard’s PRIMARY (or an RDONLY, to spare the primary) and writing into the target’s PRIMARY. Every stream does two things at once. It runs a copy phase, walking the source table in primary-key-ordered chunks and inserting only the rows whose keyspace ID falls inside the target shard’s range, and it runs a replicate phase, tailing the source binary log from the GTID position captured when the copy started so that concurrent writes are applied to the target as they happen. When the copy finishes and the replicate phase has caught the target up to the source’s current GTID, the stream is caught up and the target is a live, current replica of its slice of the source.

The reason a split is always a clean bisection is arithmetic. Shard ranges are expressed in hexadecimal over the keyspace-ID space: -80 covers keyspace IDs from zero up to (but excluding) 0x80, and splitting it produces -40 (zero to 0x40) and 40-80 (0x40 to 0x80). Every row that was in -80 lands in exactly one of the two children because the union of the child ranges is identical to the parent range and they do not overlap. This is why practitioners provision shard counts as powers of two and split down the middle: -80 → -40,40-80; 80- → 80-c0,c0-. A non-power-of-two split (say carving -80 into -60 and 60-80) is legal but leaves you with uneven ranges that complicate the next split, so it is reserved for deliberately rebalancing a skewed key distribution.

Because the state of every stream lives in the target’s _vt.vreplication table and is surfaced through vtctldclient Workflow ... show, the operation is observable and resumable. A VTTablet restart mid-copy resumes from the persisted GTID position rather than restarting the copy from row zero — the same durability property that makes native Online DDL restart-safe.

The reason reads switch before writes is not convention but blast-radius control. Switching rdonly and replica traffic exercises the new shards’ routing, serving graph entries, and query plans against real load while the source primary still owns every write — so if the VSchema or a target tablet is misconfigured, the symptom is a degraded analytics query on a shard you can afford to lose, not a lost write. Only once the low-risk classes are proven does the primary move. And because SwitchTraffic --tablet-types primary starts the reverse streams automatically, the write cutover is itself reversible right up until complete — the reverse streams replicate every post-cutover write on the new shards back to the retained source, so ReverseTraffic can restore the old topology’s write path in one command. Understanding that read-switch, write-switch, and reverse are three distinct, individually reversible steps is what separates a controlled reshard from a hopeful one.

Running a shard split step by step

The sequence below splits commerce/-80 into -40 and 40-80. Substitute your own keyspace, workflow name, and ranges.

1. Create the workflow

Reshard ... create records the workflow in the topology and starts the VReplication streams. The --tablet-types flag chooses which source tablet type the streams read from; reading from replica,rdonly keeps copy load off the source primary.

vtctldclient Reshard --workflow split80 --target-keyspace commerce create \
  --source-shards '-80' --target-shards '-40,40-80' \
  --tablet-types 'replica,rdonly'

2. Watch the copy reach the running state

Poll the workflow until every stream leaves Copying and enters Running (copy done, now only tailing the binary log). The --include-logs output shows per-stream position and lag.

vtctldclient Workflow --keyspace commerce show --workflow split80

Do not proceed to verification until every stream reports Running with lag inside your throttle threshold. A stream still Copying has not finished walking the source table, and a VDiff against it will report false mismatches for rows not yet copied.

3. Verify with VDiff

VDiff compares source and target row-by-row and produces a report. Create the diff, then read it back; a clean report shows zero mismatched, extra, or missing rows on every table.

vtctldclient VDiff --workflow split80 --target-keyspace commerce create
vtctldclient VDiff --workflow split80 --target-keyspace commerce show last

Never run SwitchTraffic before a VDiff reports clean. The full mechanics of reading the report, choosing row-count versus full-content mode, and handling transient mismatches are covered in Verifying Data Integrity After Resharding.

4. Switch reads first (rdonly and replica)

Move the low-risk traffic classes before the primary. Switching rdonly and replica reads routes analytics and read-replica traffic to the new shards while writes still land on the source, so any routing problem shows up on read paths you can tolerate degrading.

vtctldclient Reshard --workflow split80 --target-keyspace commerce switchtraffic \
  --tablet-types 'rdonly,replica'

Watch query success rate and latency on the new shards for several minutes. The routing change is served by the stateless VTGate routing layer, which re-plans against the updated serving graph the moment the switch commits.

5. Switch writes (primary)

Once reads are healthy, move the primary. This is the real cutover: VTGate briefly stops writes to the source, waits for the target streams to reach the final source position, flips the serving graph so the targets own the range, and starts the reverse streams.

vtctldclient Reshard --workflow split80 --target-keyspace commerce switchtraffic \
  --tablet-types 'primary'

6. Keep the reverse stream alive, then complete

After the primary switch, Vitess runs split80_reverse streams that replicate writes from the new shards back to the old source, so ReverseTraffic can move writes back instantly if the new shards misbehave. Leave this safety window open long enough to trust the change under real load, then finalize:

# If the new shards look wrong, roll writes back to the source:
vtctldclient Reshard --workflow split80 --target-keyspace commerce reversetraffic

# Once confident, tear down streams and drop the source shard's data:
vtctldclient Reshard --workflow split80 --target-keyspace commerce complete

complete stops the reverse streams, removes the workflow records, and drops the now-unused data from the source shard. After this point the reverse path is gone; a mistake means a fresh reshard the other way, not a one-command rollback.

Configuration reference

The flags below dominate the behaviour, safety, and duration of a reshard. Defaults favour safety; the recommended column is production-oriented for a large sharded fleet.

Flag / setting Command Type Default Recommended (production)
--source-shards / --target-shards Reshard create string — power-of-two bisection: -80 → -40,40-80
--tablet-types Reshard create list in_order:REPLICA,PRIMARY replica,rdonly so copy load avoids the source primary
--tablet-types switchtraffic list all switch rdonly,replica first, then primary separately
--on-ddl Reshard create enum IGNORE STOP so in-flight DDL cannot silently diverge the copy
--defer-secondary-keys Reshard create bool false true on large tables to add secondary indexes after copy
--max-replication-lag-allowed switchtraffic duration 30s 10s–30s; the switch waits for streams inside this bound
--timeout switchtraffic duration 30s raise for hot tables where the write-freeze wait runs long
VReplication throttler threshold tablet flag duration 1s 1s–5s; pauses copy on target/replica lag
--auto-start Reshard create bool true keep true; set false only to stage streams before copy

The most damaging misconfiguration is leaving --on-ddl at IGNORE while a schema migration runs on the source during the copy: the change may reach the source but not the in-flight stream, so the target ends up with a different table shape and VDiff fails at the worst time. Setting it to STOP makes the stream halt on a DDL event so you notice before cutover. The second-most-common error is switching primary before rdonly,replica — legal, but it moves the highest-risk traffic first and gives you no gentle signal.

Failure modes specific to resharding

Copy never catches up (persistent lag). Symptom: a stream stays in Copying or oscillates in Running with lag pinned above threshold; copy throughput near zero. Root cause: the throttler is pausing the copy because target or replica lag exceeds the limit — the copy is generating more write volume than the targets can absorb. Mitigation: let the throttler self-heal; if lag never recovers, reduce copy concurrency, move the copy source to an RDONLY, or run the heavy copy in an off-peak window. Watch the replication lag and throttling metrics to distinguish a healthy throttle from a target that is simply undersized.

VDiff reports a mismatch. Symptom: the VDiff report shows non-zero mismatched or missing rows. Root cause: usually a stream still copying, a source DDL applied mid-copy, or genuine drift from a --on-ddl IGNORE divergence. Mitigation: confirm every stream is Running first; if the diff was run too early, re-diff after catch-up. A persistent mismatch on a caught-up stream means the copy is wrong — do not switch. Diagnosing and re-diffing is detailed in the verification guide linked above.

SwitchTraffic hangs at the write freeze. Symptom: the primary switch does not return and eventually times out. Root cause: streams could not reach the final source GTID within --max-replication-lag-allowed, or a long-running transaction on the source held the write freeze open past --timeout. Mitigation: raise --timeout, clear the blocking transaction, and confirm lag is inside the allowed bound before retrying. The switch is idempotent — a timed-out switch leaves the source serving, so retry is safe.

Post-switch regret (reverse needed). Symptom: after the primary switch, the new shards show elevated errors, hot-spotting, or an application bug tied to the new ranges. Root cause: a problem invisible during read-only verification, e.g. a query that only fans out badly once writes are distributed. Mitigation: run reversetraffic to move writes back to the source immediately — this works only because you have not yet run complete. Treat complete as the point of no return and do not run it until the reverse window has held.

Driving the reshard from Python

A production reshard is rarely a hand-typed sequence — a controller submits it, polls until the streams are caught up, gates the switch on a clean VDiff, and only then advances. The control-plane surface is vtctldclient, which a Python orchestrator shells out to; reading persisted state before every action keeps a restarted controller idempotent so a crash between steps never double-switches.

import json, subprocess

def workflow_streams(keyspace: str, workflow: str) -> list[dict]:
    out = subprocess.run(
        ["vtctldclient", "Workflow", "--keyspace", keyspace,
         "show", "--workflow", workflow, "--format", "json"],
        capture_output=True, text=True, check=True,
    )
    return json.loads(out.stdout or "{}").get("workflows", [])

def all_streams_caught_up(keyspace: str, workflow: str, max_lag_s: int = 10) -> bool:
    """True only when every stream is Running (copy done) within the lag bound."""
    for wf in workflow_streams(keyspace, workflow):
        for shard_streams in wf.get("shard_streams", {}).values():
            for s in shard_streams.get("streams", []):
                if s.get("state") != "Running":
                    return False
                if s.get("max_v_replication_lag", 0) > max_lag_s:
                    return False
    return True

# Gate the traffic switch on both catch-up AND a clean VDiff, never one alone.
if all_streams_caught_up("commerce", "split80"):
    launch_vdiff_then_switch("commerce", "split80")

The load-bearing invariant is that the switch is gated on two independent signals — every stream Running within the lag bound, and a VDiff that came back clean — because either alone can be green while the other is not. A controller that switches on catch-up without a diff can cut over onto silently corrupt data; one that diffs but ignores lag can trip on transient mismatches. The same subprocess pattern drives the switch, reverse, and complete commands, each recording its outcome so the run is auditable.

Verification

Confirm the split landed; do not assume it. First, every stream should be gone and the serving graph should show the new shards owning the range:

# Streams should be finalized; new shards should be serving
vtctldclient Workflow --keyspace commerce show --workflow split80
vtctldclient GetShards commerce

Then confirm routing is actually targeted through VTGate, not scattering across old and new shards. A keyed lookup that formerly hit -80 should now resolve to exactly one child shard:

-- Through VTGate: a keyed read must resolve to a single new shard
SELECT customer_id FROM orders WHERE customer_id = 42;

Finally, watch the post-cutover signals for several minutes: per-shard QPS should show the source shard’s former load split roughly in half across the two children, replica lag on both children should sit clear of its threshold, and the _vt.vreplication tables should be empty. Only once load is genuinely balanced and lag is stable should you run complete and retire the reverse safety window.

← Back to Vitess Sharding Architecture & Topology Design