Decoding Online DDL Migration States

Every branch in a DDL controller is a decision about a migration_status value, so misreading one state — treating a paused migration as a stuck one, or a held one as a finished one — is how automation either abandons a healthy change or cuts over before it should.

Context #

This page is the vocabulary reference beneath Tracking Migration Progress and State Machines, part of Online DDL Orchestration & Migration Coordination. The state machine describes how a migration moves; this page defines what each state means and which transitions are legal, so the poller that reads these values and the barrier check that gates on them are working from the same definitions.

Each shard tracks its own migration_status in _vt.schema_migrations, exposed through SHOW VITESS_MIGRATIONS and vtctldclient OnlineDDL show. On a sharded keyspace a single logical migration is therefore N status values, one per shard, and the migration as a whole is only as advanced as its least-advanced shard. Reading the states correctly is the prerequisite for every aggregation the automation performs.

The state machine #

A migration submitted with the vitess strategy walks a small, well-defined graph. It is admitted as queued, scheduled to ready, executes as running, and — depending on outcome and whether completion was postponed — ends in complete, failed, or cancelled. Two conditions overlay the running state without being distinct terminal states: the migration can be throttled (paused by backpressure, still running) and it can be postponed (caught up but held at the cutover gate).

The states, one by one #

queued — the migration has been admitted to _vt.schema_migrations on the shard but has not started executing. It is waiting for the scheduler, often because a --singleton or --singleton-context constraint is serializing it behind another migration on the same table. Actionable: you can cancel it cleanly here, since no copy has begun.

ready — the scheduler has picked the migration up and it is about to run; a brief transitional state between queued and running. Seeing a migration sit in ready for a long time usually means the tablet is busy or a resource gate is closed.

running — the migration is actively copying rows through VReplication and tailing the binary log for concurrent writes. This is where a migration spends nearly all of its wall-clock time. A migration can remain running while throttled: the throttler has paused the copy because of replica lag, but the migration has not left running — its throughput is simply near zero until backpressure clears.

complete — the copy is caught up and, unless completion was postponed, the atomic cutover has swapped the new table in. This is the success terminal state. The crucial subtlety: under --postpone-completion, a shard reports complete when it is caught up and held at the cutover gate — ready, but not yet swapped. The same status value therefore means “done and serving new schema” in the default case and “ready and waiting” in the postponed case; the postpone_completion flag on the row disambiguates them, and the cutover barrier relies on exactly this distinction.

failed — the migration hit an unrecoverable error (a copy error, a lost tablet, a cutover that could not complete). Terminal. The migration will not resume on its own; the retained artifact tables let you investigate before cleaning up, and the change must be re-submitted as a new migration.

cancelled — an operator (or the automation) issued OnlineDDL cancel. Terminal, and distinct from failed in that it was intentional, not an error. Cancelling a running migration stops the copy and discards the shadow table.

Two conditions are not standalone terminal states but modify running or complete:

• throttled — the migration is running but paused by the throttler. It resumes automatically when lag recovers. Never treat throttled as stuck or failed; treat it as backpressure doing its job.
• postponed — the migration reached caught-up under --postpone-completion and is holding at complete without swapping. It waits for an explicit OnlineDDL complete to cut over.

State reference table #

Reading the states in automation #

The only reliable way to branch is to group the raw values by intent, exactly as a controller does — terminal-failure aborts, terminal-success (with the postpone flag) either releases or completes, everything else keeps polling:

TERMINAL_FAIL = {"failed", "cancelled"}
SUCCESS = {"complete"}

def classify(row: dict) -> str:
    status = row["migration_status"]
    if status in TERMINAL_FAIL:
        return "abort"
    if status in SUCCESS:
        # "complete" is ambiguous: postponed means held, not swapped.
        return "held" if row.get("postpone_completion") else "done"
    return "running"        # queued, ready, running, throttled — keep polling

Reducing a whole keyspace to one verdict is then a fold over per-shard classifications: any abort aborts; all held means the barrier is reached and the cutover can be released; all done means the migration is finished; anything else means keep waiting.

Edge cases and gotchas #

• complete does not always mean cut over. The single biggest decoding error is treating complete as “schema is live.” Under --postpone-completion it means “ready and holding.” Always read the postpone_completion flag alongside the status, or the barrier logic will conclude the change is done while every shard is actually still serving the old schema.
• Throttled is not stuck. A migration pinned at running with zero throughput looks stuck but is usually throttled by design. Check the throttler metric before intervening — cancelling a throttled migration throws away real copy progress for no reason.
• Per-shard divergence. Different shards can legitimately be in different states at the same instant — one running, one complete (held), one throttled. The migration’s true state is the aggregate, never any single shard’s value; a controller that reads one shard and generalizes will misjudge the fleet.
• failed versus cancelled. Both are terminal, but only failed indicates something went wrong. Alerting that pages on cancelled will fire every time an operator deliberately aborts a migration; page on failed, log cancelled.
• A migration missing entirely. SHOW VITESS_MIGRATIONS returning no row for a UUID on a shard is not a state — it means the migration was never admitted there, or the UUID is wrong. Distinguish “no row” from any status value; the poller treats fewer-shards-than-expected as not-yet-ready, not as success.
• Re-submitting after a terminal state. failed and cancelled migrations do not resume; there is no transition out of a terminal state. Recovery is always a new migration (with a fresh UUID), which is why crash-safe automation keys on a deterministic context rather than expecting to reattach to a dead one.

Verification #

Read the per-shard states directly and confirm they match what your automation believes. Through VTGate, SHOW VITESS_MIGRATIONS exposes the status and the postpone flag together:

-- Per-shard status and whether each shard is merely held at the cutover gate
SHOW VITESS_MIGRATIONS LIKE 'orders-add-fc-2026q3'\G

Inspect the migration_status and postpone_completion columns per shard: every shard running means keep waiting, every shard complete with postpone_completion = 1 means the barrier is reached and holding, and every shard complete with postpone_completion = 0 means the cutover has actually happened. Cross-check the same values from the control plane, which is the source the automation reads:

vtctldclient OnlineDDL show commerce orders-add-fc-2026q3 --json \
  | jq -r '.[] | [.shard, .migration_status, .postpone_completion] | @tsv'

The two views must agree; a discrepancy means you are reading a stale VTGate or a wrong UUID. A shard reporting failed here is the signal to inspect its migration’s error message before any cleanup, since the retained tables are your only forensic trail once the migration is gone.

Related #

• Tracking Migration Progress and State Machines — the progress model these status values drive.
• Polling Migration Status with vtctldclient — the poller that classifies and aggregates these states across shards.
• Enforcing a Global Cutover Barrier — how the held complete state gates a fleet-wide cutover.

← Back to Tracking Migration Progress and State Machines