Enforcing a Global Cutover Barrier

The unsafe moment in any multi-shard schema change is not the row copy — it is the instant one shard swaps to the new table while another is still serving the old one, and the barrier exists to make sure that instant never arrives.

Context

This page sits under Coordinating Multi-Shard Schema Migrations, within the broader practice of Online DDL Orchestration & Migration Coordination. Coordination has two halves: pacing the copy so every shard makes progress, and gating the cutover so they all switch together. This is the second half — the barrier that separates “all shards have finished copying” from “all shards have switched traffic.” Automating it is the job of the Python control loop; understanding why it must exist, and what breaks without it, is the job of this page.

The default behaviour of Vitess Online DDL is to cut each shard over as soon as its own copy is caught up. On an unsharded keyspace that is exactly right. On a sharded keyspace it is a trap: shards finish copying at different times — a shard with a 400-million-row table finishes long after one with 4 million — so uncoordinated completion means the keyspace serves a mix of old and new table shapes for however long the slowest shard lags the fastest. The barrier removes that window.

The concept: caught-up is not cut-over

A migration under the vitess strategy passes through two distinct milestones that operators often conflate. Caught-up means the shard has copied all existing rows and its VReplication stream has drained the backlog of concurrent writes — the shadow table is a faithful, live copy. Cut-over means the shard has performed the atomic RENAME that swaps the shadow table in and starts serving the new schema. Between the two, the shard is fully prepared but still serving the old table.

Normally cut-over follows caught-up automatically and immediately. The barrier is the deliberate insertion of a wait between them: submit with --postpone-completion, and every shard reaches caught-up and then holds, indefinitely, until an explicit completion command releases it. Because completion is now a separate, single action you control, you can withhold it until the slowest shard has also reached caught-up — at which point one fleet-wide command swaps them all in a tight window instead of a staggered one.

Global cutover barrier across four shards Four caught-up-but-held shards converge on a single barrier that opens only when all have arrived; one completion command then swaps every shard together. -40 caught up · held 40-80 caught up · held 80-c0 caught up · held c0- caught up · held barrier — closed until all 4 caught up OnlineDDL complete one call, released once all shards swap together

Solution: postpone, wait for the barrier, then complete once

The barrier is three steps: submit with completion postponed, poll until every shard reports caught-up, then issue one completion for the whole keyspace.

Submit with the barrier held. --postpone-completion is what makes every shard stop at caught-up instead of swapping:

vtctldclient ApplySchema \
  --ddl-strategy "vitess --postpone-completion --singleton" \
  --migration-context "orders-add-fc-2026q3" \
  --sql "ALTER TABLE orders ADD COLUMN fulfilment_center_id BIGINT UNSIGNED NULL" \
  commerce

Under --postpone-completion, a shard that has finished copying and draining its backlog reports migration_status = complete in the sense of ready and holding — caught up, not yet swapped. The distinction between that held state and a truly swapped-in migration is important enough that the full state vocabulary is catalogued in decoding Online DDL migration states.

Now the barrier check: block until every shard is caught up, and refuse to proceed if any shard has failed. This is a logical AND across the fleet — one straggler keeps the gate closed for everyone.

import json
import subprocess
import time

TERMINAL_FAIL = {"failed", "cancelled"}


def shard_statuses(keyspace: str, uuid: str) -> dict[str, str]:
    proc = subprocess.run(
        ["vtctldclient", "OnlineDDL", "show", keyspace, uuid, "--json"],
        capture_output=True, text=True, check=True, timeout=30,
    )
    return {r["shard"]: r["migration_status"] for r in json.loads(proc.stdout or "[]")}


def wait_for_barrier(keyspace: str, uuid: str, expected_shards: int,
                     poll: int = 15, deadline_s: int = 4 * 3600) -> None:
    """Return only when ALL shards are caught up and held. Raise on failure or timeout."""
    started = time.monotonic()
    while True:
        states = shard_statuses(keyspace, uuid)
        stragglers = {s: st for s, st in states.items() if st in TERMINAL_FAIL}
        if stragglers:
            raise RuntimeError(f"barrier aborted — shards failed: {stragglers}")
        ready = [s for s, st in states.items() if st == "complete"]
        if len(states) >= expected_shards and len(ready) == len(states) and states:
            return                                   # gate opens: every shard is caught up
        if time.monotonic() - started > deadline_s:
            not_ready = {s: st for s, st in states.items() if st != "complete"}
            raise TimeoutError(f"barrier not reached; stragglers: {not_ready}")
        time.sleep(poll)

Only once wait_for_barrier returns do you release the cutover — with a single command that Vitess fans out to every shard, swapping them in one tight window:

# Fires the cutover on EVERY shard of the keyspace at once.
vtctldclient OnlineDDL complete commerce orders-add-fc-2026q3

The ordering is the whole safety property: postpone holds every shard at the edge, the barrier check confirms none is still copying, and the single complete collapses the mixed-schema window from “slowest-minus-fastest shard copy time” (possibly hours) down to the few seconds it takes the fan-out to issue N renames.

Edge cases and gotchas

  • A straggler shard. One shard lagging — a hot table, a throttled copy, a slow replica — holds the barrier closed for the whole fleet, which is correct but can stall the migration indefinitely. The deadline_s converts an indefinite stall into a raised TimeoutError naming the stragglers. Do not respond by completing the shards that are ready; that reintroduces exactly the mixed-schema window the barrier exists to prevent. Instead, investigate the straggler (often throttler backpressure) and either let it catch up or cancel the whole migration.
  • Timeout policy. Size the barrier deadline to the empirical copy time of the largest shard plus generous margin, not to a fixed hour. A deadline shorter than the slowest shard’s copy guarantees a spurious timeout on every large migration; a deadline with no upper bound lets a genuinely wedged migration hold resources forever. Pair the deadline with an alert so a straggler surfaces to an operator rather than silently expiring.
  • Mixed-schema window if the barrier is skipped. Omitting --postpone-completion is the classic mistake: each shard cuts over the moment it finishes copying, so for the duration between the fastest and slowest shard the keyspace serves two table shapes. Any query that fans across shards — especially a cross-shard transaction — can then observe both the old and new column set at once, producing errors or silently wrong results that are extremely hard to trace after the fact.
  • Partial cutover during the fan-out. Even with the barrier, the single complete fans out N renames, and one can fail (a metadata-lock wait, a lost tablet). The result is some shards swapped and some not — a mixed window reopened. Treat the fan-out as all-or-nothing: on any shard’s rename failure, re-swap the already-cut shards back to their retained originals (possible only because --retain-online-ddl-tables kept them) and re-queue the failed shard, rather than leaving the keyspace split.
  • Completing before the barrier is confirmed. Calling complete off a partial read — three of four shards complete, the fourth not yet reporting — cuts over early. Gate completion on the full shard count (expected_shards), never on “all the shards I happened to hear from.”
  • Releasing into the wrong time window. The barrier decouples submission from cutover, which is what lets you copy ahead of a maintenance window and complete inside it — but that only works if you actually gate the release on the clock as well as the barrier, as covered in scheduling DDL windows across multiple timezones.

Verification

Confirm every shard was actually held at the barrier before you complete, then confirm they all swapped together after. Before completion, every shard should be caught-up-and-held and none should have swapped:

# Pre-completion: every shard "complete" in the held/postponed sense, none renamed yet
vtctldclient OnlineDDL show commerce orders-add-fc-2026q3 --json \
  | jq -r '.[] | [.shard, .migration_status, .postpone_completion] | @tsv'

Every row should show migration_status = complete with postpone_completion = true — the signature of a shard waiting at the gate. After issuing complete, verify the cutover timestamps fall within a tight window rather than spreading across the copy duration:

vtctldclient OnlineDDL show commerce orders-add-fc-2026q3 --json \
  | jq -r '.[] | [.shard, .completed_timestamp] | @tsv'

The max-minus-min of completed_timestamp across shards should be seconds, not minutes — a wide spread means the barrier did not hold and shards swapped as they finished. Finally read the new column back through VTGate to confirm it resolves uniformly on every shard, not just the ones that happened to cut over first.

← Back to Coordinating Multi-Shard Schema Migrations