Verifying Data Integrity After Resharding

VReplication copies rows quietly and reports success on its own throughput, but “the stream is running” is not “the data is correct” — the only thing that proves a reshard is safe to cut over is a VDiff that compares source and target row-by-row and comes back clean.

Context

This page sits under Resharding Workflows and Shard Splits, between the copy phase and SwitchTraffic. Verification is the gate: no traffic moves through the VTGate routing layer to the new shards until VDiff confirms the target holds exactly the rows the source does for its range. A clean diff is what turns an invisible copy into a trustworthy one, and skipping it is how silent corruption reaches production during a cutover.

What VDiff actually compares

VDiff walks the source and target tables in parallel over the workflow’s tables, hashing rows and comparing them in primary-key order. For a reshard it checks each target shard against the slice of the source range it owns, and it accounts for the boundary — a row that belongs to -40 is expected on -40 and nowhere else. The report it produces breaks down, per table, into rows that matched, rows that mismatched (present on both sides but different), rows extra on the target, and rows missing from the target. A correct copy shows all four non-match counts at zero.

Because a reshard’s streams keep tailing the binary log while you diff, VDiff takes a consistent snapshot: it briefly records the source GTID position, waits for the target to reach it, and diffs against that fixed point so concurrent writes do not produce phantom differences. This is why you run VDiff only after the streams are in the Running state — a stream still Copying has not finished, and every un-copied row would report as missing.

VDiff v2 (the default from Vitess 14 onward) runs inside the target tablets rather than in the vtctld process, which is what makes it viable on large tables: the comparison is checkpointed to _vt.vdiff tables, so a diff that is interrupted by a tablet restart resumes from its last checkpoint rather than restarting from the first row. The work is distributed across the target shards, each diffing its own range against the source in parallel, so adding shards does not linearly lengthen the diff. That checkpointing is also what lets you time-box a diff and resume it later, which matters on tables too large to compare in one maintenance window.

Creating and reading a diff

Create the diff, then read the report. VDiff runs asynchronously, so create returns a UUID and show reports progress and results.

# Start the diff for the reshard workflow
vtctldclient VDiff --workflow split80 --target-keyspace commerce create

# Read the most recent diff's report
vtctldclient VDiff --workflow split80 --target-keyspace commerce show last

The report you want to see reports every table completed with zero mismatches. Pull it as JSON when you are scripting the gate so a controller can decide whether to proceed:

vtctldclient VDiff --workflow split80 --target-keyspace commerce show last --format json

In the JSON, each table carries RowsCompared, MismatchedRows, ExtraRowsTarget, ExtraRowsSource, and a HasMismatch boolean. The gate is simple: if any table has HasMismatch: true or any non-zero extra/mismatched count, do not switch traffic. Treat a green report as necessary but time-bound — a diff proves the state at the moment it ran, so run it close to the switch, not hours before.

Row-count versus full-content mode

VDiff defaults to a full-content comparison: it hashes and compares every column of every row, which is the only mode that catches value-level corruption. That thoroughness costs IO and time proportional to table size. On very large tables you have two ways to bound the cost:

  • Row-count comparison (--only-pks reduces the compared payload toward keys, and count-oriented checks) confirms cardinality quickly — the target has the right number of rows in its range — without hashing every column. It catches missing and extra rows but not a wrong value in an otherwise-present row, so it is a fast pre-check, never the final gate.
  • Sampling (--max-diff-duration to time-box, and limiting rows) lets a huge table produce a partial signal on a schedule you control.

The safe pattern on a large fleet is a fast count check across all tables first, then a full-content diff on the tables whose values actually matter for correctness. A full-content pass is the one you trust before moving the primary.

# Time-box a full-content diff on a large table so it cannot run unbounded
vtctldclient VDiff --workflow split80 --target-keyspace commerce create \
  --tables orders \
  --max-diff-duration 30m

Restricting the diff to specific tables with --tables is how you keep the verification window bounded and focused. Diff the small reference tables in one fast pass, then diff each large table individually so a single huge table’s diff does not block the report on everything else. When a diff is time-boxed and does not finish, it stops at a checkpoint; re-running create on the same tables continues rather than starting over, so a multi-window verification of a very large table is a series of resumed diffs.

Automating the diff as a cutover gate

In a scripted reshard the diff is a gate, not a manual eyeball. A controller creates the diff, polls show until the state is completed, and refuses to advance to SwitchTraffic unless every table came back clean. Reading the JSON report is what turns “looks fine” into a hard decision:

import json, subprocess, time

def vdiff_clean(keyspace: str, workflow: str, poll: int = 20) -> bool:
    """Create a diff, wait for completion, return True only if no table mismatched."""
    subprocess.check_call(
        ["vtctldclient", "VDiff", "--workflow", workflow,
         "--target-keyspace", keyspace, "create"])
    while True:
        out = subprocess.run(
            ["vtctldclient", "VDiff", "--workflow", workflow,
             "--target-keyspace", keyspace, "show", "last", "--format", "json"],
            capture_output=True, text=True, check=True)
        report = json.loads(out.stdout or "{}")
        rows = report.get("TableSummary", report)  # shape varies by version
        state = report.get("State", "")
        if state == "completed":
            tables = rows.values() if isinstance(rows, dict) else rows
            return all(int(t.get("MismatchedRows", 0)) == 0
                       and int(t.get("ExtraRowsTarget", 0)) == 0
                       and int(t.get("ExtraRowsSource", 0)) == 0
                       for t in tables)
        time.sleep(poll)

The controller calls vdiff_clean immediately before the traffic switch and aborts the run if it returns False. Running the diff close to the switch matters — a clean report an hour before cutover does not prove the state at cutover, so the gate belongs in the switch path, not upstream of it.

Handling mismatches before SwitchTraffic

A non-zero mismatch is a stop sign, but not always a real corruption — most mismatches on a live reshard are transient and resolve once the target catches up. Work through them in order:

  1. Confirm the streams are caught up. Run Workflow show and verify every stream is Running with low lag. A diff run against a still-copying or lagging stream reports differences that are just un-applied writes.
  2. Re-diff. Create a fresh VDiff and read the new report. A transient mismatch from lag disappears; a real one persists at the same rows.
  3. Inspect the offending rows. If a mismatch survives a re-diff on a caught-up stream, the report identifies the tables and keys. A recurring cause is a schema change applied to the source mid-copy under --on-ddl IGNORE, leaving the target with a different table shape — the fix is to stop and restart the affected stream cleanly, not to switch anyway.
  4. Never switch on a red report. Moving traffic to a target that failed its diff copies the corruption into production and removes the source as a reference.
# Re-run the diff after confirming streams are caught up
vtctldclient Workflow --keyspace commerce show --workflow split80
vtctldclient VDiff --workflow split80 --target-keyspace commerce create

A useful discipline is to classify every mismatch before acting on it. A mismatch that appears on the first diff, then vanishes on a re-diff of a caught-up stream, was lag — harmless. A mismatch that persists at the same primary keys across two clean-stream diffs is structural and needs a root cause before you switch. A mismatch whose row set changes between diffs usually means the stream is still moving (not truly caught up) — go back and confirm the workflow state rather than chasing individual rows. Only the persistent, stable mismatch is a genuine correctness defect, and the correct response to it is to stop the affected stream, drop and recreate the target’s copy for that table, and re-diff — never to switch traffic in the hope it resolves itself.

It is worth being precise about what a clean VDiff does and does not prove. It proves the target holds exactly the rows the source held for its range at the snapshot GTID, down to column values in full-content mode. It does not prove that routing is correct, that the VSchema sends future reads to the right shard, or that writes arriving after the snapshot are handled — those are validated by the traffic-switch steps and post-cutover monitoring, not by the diff. Treat the diff as the data-correctness gate and the switch verification as the routing gate; both are required, and neither substitutes for the other.

Edge cases and gotchas

  • Diffing under heavy write load. High write rates widen the window between the source snapshot GTID and the target catching up to it, so a diff can take much longer or repeatedly report transient mismatches. Diff in a traffic trough where you can, and read MismatchedRows as suspect until a re-diff on a quiet moment confirms it.
  • Large-table sampling gives false confidence. A sampled or row-count-only diff that comes back clean has not proven value-level correctness. Do not let a fast count check stand in for the full-content pass on tables where a wrong value is unacceptable.
  • Transient mismatches from replication lag. If the target stream is behind, recently-written rows legitimately differ for a moment. Always gate the diff on the stream being Running with lag inside threshold before believing a mismatch is real. The replication lag and throttling signals tell you whether a mismatch is lag or corruption.
  • Unbounded diffs on huge tables. Without --max-diff-duration, a full-content diff on a multi-hundred-million-row table can run for many hours and hold IO. Time-box large tables and diff them individually rather than launching one unbounded diff across the whole workflow.
  • Diffing the wrong direction after reverse. Once you have switched and the reverse streams are live, a VDiff compares in the reverse direction. Be explicit about which workflow (split80 versus split80_reverse) you are diffing so the report means what you think.

Verification

The verification of resharding is the diff, so the concrete check is a clean report on every table immediately before the traffic switch:

vtctldclient VDiff --workflow split80 --target-keyspace commerce show last --format json \
  | jq -r '.[] | [.TableName, .RowsCompared, .MismatchedRows, .ExtraRowsTarget, .ExtraRowsSource] | @tsv'

Every table should show a non-zero RowsCompared and zeros across MismatchedRows, ExtraRowsTarget, and ExtraRowsSource. A single non-zero mismatch column means the target does not yet faithfully hold its range — resolve it and re-diff before running SwitchTraffic.

← Back to Resharding Workflows and Shard Splits