Polling Migration Status with vtctldclient

The single most-run piece of any DDL automation is the status poll — and the naive version, a bare subprocess.run in a while True, is exactly the code that hangs a pipeline at 3am when vtctldclient blocks on a slow topology server or prints a warning line ahead of its JSON.

Context

This page zooms in on the poll step of the control loop built in Automating DDL Pipelines with Python, which sits within the broader practice of Online DDL Orchestration & Migration Coordination. The loop’s other steps — submit, persist, complete, verify — each run a handful of times; the poller runs continuously for the entire life of the migration, often for hours, so its robustness dominates the pipeline’s reliability. A poller that misreads one shard’s status, or dies on one malformed line, either stalls the migration or, worse, releases the global cutover barrier early.

The command it wraps is vtctldclient OnlineDDL show <keyspace> <uuid> --json, which returns one JSON object per shard for the given migration. On a sharded keyspace that is N objects — the poller’s job is to read all N, reduce them to a single verdict, and do so without ever blocking the loop indefinitely or crashing on output it did not expect.

The concept: a poll is an unreliable read, not a function call

Treating vtctldclient like an in-process function is the root of every poller bug. It is an external binary invoked over a subprocess boundary, and each of the three things it does — run, print, and exit — can fail independently. It can hang (topology-server slowness), so every call needs a hard timeout. It can print non-JSON to standard output ahead of the payload (a deprecation notice, a connection warning), so parsing must tolerate leading noise. It can report fewer shards than exist during a brief window right after submission, before every shard has registered its migration row, so aggregation must distinguish “not all shards reporting yet” from “all shards caught up.” And it can simply not be on PATH on the host the controller was rescheduled to, which should be a loud, immediate failure rather than a silent retry loop.

The correct mental model is a poll returning one of four outcomes: terminal-success (every shard complete), terminal-failure (any shard failed/cancelled), transient (a recoverable error — timeout, parse failure, partial reporting — that should back off and retry), and still-running (a valid read that is simply not done yet). The poller’s structure follows directly from those four cases.

Solution: a robust poll_until_terminal()

The implementation separates three concerns: invoking the command safely, parsing its output defensively, and aggregating per-shard statuses into a verdict. Keeping them separate is what makes each testable and each failure attributable.

import json
import shutil
import subprocess
import time
from dataclasses import dataclass

# Per-shard migration_status values, grouped by what the poller should do.
TERMINAL_SUCCESS = {"complete"}
TERMINAL_FAILURE = {"failed", "cancelled"}
# Everything else ("queued", "ready", "running", throttled/postponed) means keep polling.


class PollTransientError(Exception):
    """A recoverable poll failure: timeout, non-JSON output, or partial reporting."""


class PollConfigError(Exception):
    """An unrecoverable setup failure, e.g. vtctldclient missing from PATH."""


@dataclass
class Verdict:
    done: bool                 # True once a terminal state is reached
    ok: bool                   # True only if terminal AND every shard complete
    statuses: dict[str, str]   # {shard: migration_status}


def _run_show(keyspace: str, uuid: str, timeout_s: int) -> str:
    if shutil.which("vtctldclient") is None:
        raise PollConfigError("vtctldclient not found on PATH")
    try:
        proc = subprocess.run(
            ["vtctldclient", "OnlineDDL", "show", keyspace, uuid, "--json"],
            capture_output=True, text=True, timeout=timeout_s,
        )
    except subprocess.TimeoutExpired as e:
        raise PollTransientError(f"vtctldclient timed out after {timeout_s}s") from e
    if proc.returncode != 0:
        # Non-zero exit is transient (topo blip) unless it names a hard error.
        raise PollTransientError(f"vtctldclient exit {proc.returncode}: {proc.stderr.strip()}")
    return proc.stdout


def _parse_rows(raw: str) -> list[dict]:
    """Tolerate leading non-JSON noise by scanning for the first '[' or '{'."""
    if not raw or not raw.strip():
        return []                        # no rows yet — shards not registered
    text = raw.strip()
    start = min((i for i in (text.find("["), text.find("{")) if i != -1), default=-1)
    if start == -1:
        raise PollTransientError(f"no JSON in vtctldclient output: {text[:120]!r}")
    try:
        payload = json.loads(text[start:])
    except json.JSONDecodeError as e:
        raise PollTransientError(f"malformed JSON from vtctldclient: {e}") from e
    if isinstance(payload, dict):        # single-shard keyspaces return one object
        payload = [payload]
    return payload


def _aggregate(rows: list[dict], expected_shards: int) -> Verdict:
    statuses = {r["shard"]: r["migration_status"] for r in rows}
    # Any failed/cancelled shard is terminal and NOT ok — abort the whole migration.
    if any(st in TERMINAL_FAILURE for st in statuses.values()):
        return Verdict(done=True, ok=False, statuses=statuses)
    # Success requires the FULL fleet reporting, all complete. Partial reporting
    # (fewer rows than shards) is deliberately NOT treated as success.
    fully_reported = len(statuses) >= expected_shards
    all_complete = bool(statuses) and all(st in TERMINAL_SUCCESS for st in statuses.values())
    if fully_reported and all_complete:
        return Verdict(done=True, ok=True, statuses=statuses)
    return Verdict(done=False, ok=False, statuses=statuses)


def poll_until_terminal(keyspace: str, uuid: str, expected_shards: int,
                        base_interval: int = 15, max_interval: int = 120,
                        deadline_s: int = 6 * 3600, timeout_s: int = 30) -> Verdict:
    """Poll until every shard is terminal, backing off on transient errors.
    Raises TimeoutError past the deadline; raises PollConfigError on setup faults."""
    started = time.monotonic()
    backoff = base_interval
    while True:
        try:
            rows = _parse_rows(_run_show(keyspace, uuid, timeout_s))
            verdict = _aggregate(rows, expected_shards)
            backoff = base_interval                 # success resets the backoff
            if verdict.done:
                return verdict
        except PollTransientError as e:
            # Do not give up on a blip; grow the interval and try again.
            backoff = min(backoff * 2, max_interval)
            print(f"transient poll error, backing off {backoff}s: {e}")
        if time.monotonic() - started > deadline_s:
            raise TimeoutError(f"migration {uuid} not terminal within {deadline_s}s")
        time.sleep(backoff)

The design decisions worth calling out:

  • Every invocation has a timeout. subprocess.run without one is an unbounded block; TimeoutExpired is caught and reclassified as transient so the loop backs off rather than dies.
  • Parsing scans for the first JSON delimiter rather than assuming stdout starts with [. A single deprecation line printed ahead of the payload would otherwise raise JSONDecodeError on every poll.
  • expected_shards gates success. Aggregation refuses to declare success until it has heard from the whole fleet, which closes the partial-reporting window where three of four shards are complete and the fourth has not yet registered a row.
  • Backoff is per-error, reset on success. Transient failures widen the interval up to max_interval; a clean read snaps it back to base_interval so steady-state polling stays responsive.
  • vtctldclient missing is a PollConfigError, not transient. There is no point backing off and retrying a binary that is not installed — fail loudly so the controller’s supervisor surfaces it.

You get expected_shards once, cheaply, from the topology at the start of the migration:

def shard_count(keyspace: str) -> int:
    out = subprocess.run(["vtctldclient", "GetShards", keyspace],
                         capture_output=True, text=True, check=True, timeout=30)
    return len([ln for ln in out.stdout.splitlines() if ln.strip()])

From poller to progress reporter

A poll that only returns a terminal verdict wastes most of the information Vitess already hands you. Each shard’s row from OnlineDDL show carries a progress percentage and an eta_seconds estimate alongside migration_status, and surfacing them turns a silent multi-hour wait into an observable one. The cheapest useful upgrade is to have the loop emit a fleet summary on every tick — the minimum across shards is what actually governs when the barrier can be reached, because the migration is bottlenecked by its slowest shard:

def progress_summary(rows: list[dict]) -> dict:
    """Fleet-level progress: the migration is only as far along as its slowest shard."""
    pcts = [float(r.get("progress", 0.0)) for r in rows]
    etas = [int(r.get("eta_seconds", 0)) for r in rows if r.get("eta_seconds")]
    return {
        "shards_reporting": len(rows),
        "min_progress": min(pcts) if pcts else 0.0,   # gates the barrier
        "max_progress": max(pcts) if pcts else 0.0,
        "worst_eta_s": max(etas) if etas else None,    # longest remaining copy
    }

Logging min_progress and worst_eta_s each tick makes a stalled shard obvious long before the deadline fires — a fleet stuck at 62% for twenty minutes is a throttled or wedged shard, not a slow one, and the number tells you which shard to look at. Exporting the same summary as a gauge (a Prometheus metric, a log field a dashboard scrapes) lets an on-call engineer watch an unattended migration without shelling into the controller. Keep the summary derived purely from the rows the poller already fetched; do not issue a second OnlineDDL show just to compute it, or you double the control-plane load for no new information.

Edge cases and gotchas

  • Non-JSON on stdout. Warnings, deprecation notices, or a connection banner can precede the payload. _parse_rows scans for the first [/{; if you see persistent parse failures, run the raw command by hand and check whether the noise is on stdout (parse around it) or stderr (already ignored).
  • Partial shard reporting right after submit. For a few seconds after ApplySchema, OnlineDDL show may return fewer objects than the keyspace has shards. Gating success on expected_shards is what stops the poller from declaring an all-complete verdict off a two-of-four read and releasing the barrier early.
  • vtctldclient not on PATH. After a controller is rescheduled onto a fresh host, the binary may be absent. The explicit shutil.which check turns that into an immediate PollConfigError instead of an infinite transient-retry loop that looks like a stuck migration.
  • Transient non-zero exits. A topology-server hiccup can make vtctldclient exit non-zero for one poll and succeed the next. Treating non-zero as transient (with backoff) rather than fatal keeps a healthy migration from being abandoned over a momentary control-plane blip — but cap the total wait with deadline_s so a genuinely wedged migration does not poll forever.
  • A shard stuck in running forever. The poller will happily loop until deadline_s because running is a valid non-terminal state. The deadline is what converts “stuck” into a raised TimeoutError the operator can act on; size it to the largest table’s expected copy time plus generous margin, not to a fixed hour.
  • Throttled and postponed sub-states read as non-terminal. A migration paused by the throttler or holding under --postpone-completion is not complete and not failed, so the poller correctly keeps waiting. Do not add those sub-states to TERMINAL_SUCCESS — their meanings are catalogued in decoding Online DDL migration states.
  • --json shape differs for one shard. An unsharded or single-shard keyspace may return a lone object rather than an array; _parse_rows normalizes a bare object into a one-element list so aggregation is uniform.

Verification

Confirm the poller agrees with Vitess’s own view before trusting it in the loop. Run the underlying command by hand and diff the shard set against the topology:

# Raw per-shard JSON the poller consumes
vtctldclient OnlineDDL show commerce <migration_uuid> --json | jq -r '.[] | [.shard, .migration_status] | @tsv'

# The shard set the poller expects to hear from
vtctldclient GetShards commerce

The shards printed by OnlineDDL show should, once the migration is past its initial registration window, exactly match GetShards. If OnlineDDL show lists fewer, the poller is correct to keep waiting; if it lists a shard GetShards does not, the migration UUID is stale or from a prior topology and the controller should reload it. Cross-check the same reduction from SQL — SHOW VITESS_MIGRATIONS LIKE '<uuid>' through VTGate should show the identical per-shard migration_status values the poller aggregated.

← Back to Automating DDL Pipelines with Python