Throttling and Cutover Control: Pacing the Copy and Governing the Swap in Sharded Online DDL

A managed Online DDL migration on a sharded keyspace does exactly two dangerous things, and both are governed by a distinct control. First it reads and rewrites potentially hundreds of millions of rows per shard while production traffic keeps writing — that write volume can bury replicas if it is not paced. Second it swaps the finished shadow table in for the live table under a brief write lock — that swap can stall queries or, worse, land on some shards and not others. Vitess exposes one control for each hazard: the lag throttler applies backpressure to the row copy so replicas never fall further behind than you permit, and the cutover barrier (postponed completion plus the --cutover-threshold write-lock window) lets you hold every shard at the caught-up-but-not-swapped state and then release the swap on your own signal. This page is for MySQL SREs and Python orchestration builders who own those two knobs on a live fleet. It sits within Online DDL Orchestration & Migration Coordination and assumes you have already chosen the native engine over an external one; if that choice is still open, resolve it first in Vitess Native Online DDL vs External Tools.

Prerequisites #

Before any of the commands below behave as documented, confirm the following:

• Vitess 19 or newer. The OnlineDDL throttle/unthrottle verbs, the throttler’s check-self semantics, and the current --cutover-threshold flag all assume a recent release. Older builds expose a coarser throttling surface and different flag names.
• ROW-based binary logging with GTIDs on every shard primary. The native engine drives a VReplication stream that tails the binlog, so binlog_format=ROW and gtid_mode=ON are non-negotiable on each MySQL instance the migration touches.
• A reachable control plane via vtctldclient, and enough topology access to run vtctldclient GetKeyspaces and reach every VTTablet. The throttler and cutover commands are issued through this path.
• A throttler that already has a lag signal. The lag throttler reads replication lag from the shard’s replicas; if a keyspace has no replicas, or the tablet throttler is disabled, the copy will run unthrottled. Verify with vtctldclient GetThrottlerStatus before you trust the gate.
• Assumed knowledge: how a migration fans out per shard and moves through its state machine — the phase model lives under the parent guide — and how to read replica lag and the throttler’s own metrics from Monitoring Replication Lag and Throttling, which is the observation side of everything described here.

Two Controls, Two Hazards #

The throttler and the cutover barrier act at opposite ends of the migration’s life. The throttler is a continuous, self-healing gate that sits in front of the copy loop for the entire duration of the row copy: it lets writes through when replicas are healthy and closes when they lag, opening again on its own once lag recovers. The cutover barrier is a one-shot, operator-driven gate that sits between “the copy is caught up” and “traffic is served from the new table”: with --postpone-completion set, every shard reaches the ready state and waits there until you issue complete. The write-lock at the actual swap is separately bounded by --cutover-threshold, which caps how long Vitess will hold the metadata lock before it gives up and retries. The lifecycle below shows where each control intercepts the migration.

The migration only spends real wall-clock time in two places: the running/throttled copy phase, whose duration the throttler stretches to protect replicas, and the ready phase, whose duration you control by choosing when to release the barrier. Everything else is fast. Getting throttling and cutover right is therefore the whole operational art of a large migration — the copy is just plumbing.

The Lag Throttler: Backpressure on the Copy #

The throttler is a shared Vitess primitive, not a DDL-specific one: the same mechanism paces VReplication streams for resharding. During a migration each VTTablet running the copy asks the throttler, before every batch of copied rows, whether it is safe to proceed. The throttler answers by checking replication lag against a configured threshold. If the aggregated lag on the shard’s replicas is under the threshold, the check passes and the batch copies; if lag is over, the check fails, the copy pauses, and the migration’s migration_status flips to throttled. No batch is lost — the stream simply stops requesting work until the next check passes, then resumes from its persisted position.

Two scoping flags matter. By default the throttler answers a self check — “is this tablet’s own replication healthy” — but a migration copy runs on the primary and cares about the replicas that will serve reads. Running the check with --throttle-check-as-check-self makes the migration evaluate the tablet’s own lag view rather than the aggregate app-facing check, which is the correct scope when you want DDL paced independently of the application throttle. The threshold itself is a duration: the throttler treats any replica lagging beyond it as a reason to close the gate.

You can also drive the gate by hand. vtctldclient OnlineDDL throttle <keyspace> <uuid> forces a specific migration into the throttled state regardless of lag — useful when an unrelated incident means you want the copy to stop consuming I/O immediately. vtctldclient OnlineDDL unthrottle <keyspace> <uuid> releases that manual hold and returns control to the automatic lag check. A manually-thrown throttle and an automatic lag throttle look identical in migration_status; the distinction lives in the throttler status, which reports the reason.

# Force this migration to stop copying right now, regardless of lag
vtctldclient OnlineDDL throttle commerce a1b2c3d4_...

# Return it to automatic lag-based pacing
vtctldclient OnlineDDL unthrottle commerce a1b2c3d4_...

The critical property is that throttling is safe by default: a migration that sits throttled for hours is not failing, it is waiting. The danger is only that it can wait forever if the threshold is set tighter than the replicas can ever satisfy — the tuning of that threshold is the whole subject of a dedicated page below.

The Cutover Barrier and the Write-Lock Window #

Once the copy catches up, Vitess is ready to swap the shadow table in for the live table. Without intervention it does this immediately, per shard, the moment each shard’s copy finishes — which on a sharded keyspace means shards cut over at different times and the keyspace briefly serves two table shapes at once. --postpone-completion prevents that. Submitted with postponement, every shard copies to caught-up and then holds in the ready state indefinitely. Nothing swaps. You now have a global barrier: you can wait until every shard reports ready, confirm the fleet is healthy, and then release all of them together with a single complete. This is the mechanism the fleet-wide coordination in Coordinating Multi-Shard Schema Migrations is built on.

# Release the postponed cutover on every shard of the migration at once
vtctldclient OnlineDDL complete commerce a1b2c3d4_...

The swap itself is not free. To rename the shadow table in and the original out atomically, Vitess must briefly acquire a write lock so that no transaction is mid-flight against the table shape it is replacing. --cutover-threshold bounds how long it will wait to grab that lock. Inside the window Vitess drains and blocks writes, performs the atomic rename, and releases; if it cannot acquire the lock within the threshold — usually because a long-running transaction is holding the table’s metadata lock — it aborts this cutover attempt and retries, rather than blocking traffic unboundedly. A larger threshold raises the chance a busy table cuts over on the first try but lengthens the worst-case write stall; a smaller threshold keeps stalls short at the cost of more retries on hot tables. It is a latency-versus-progress trade, and the right value depends on how long your longest normal transaction runs.

Revert is the inverse of complete and is possible only because the swap does not immediately destroy the original. When a migration completes, the old table is renamed aside and retained — not dropped — for the duration of --retain-online-ddl-tables. Within that window vtctldclient OnlineDDL revert <uuid> swaps the retained original back in, undoing the schema change. Revert is itself an Online DDL migration with its own UUID and its own per-shard fan-out, so it obeys the same barrier discipline. The full mechanics and the sharp limits on what revert can and cannot undo are covered in Reverting a Completed Online DDL Migration.

Step-by-Step: Submit Postponed, Watch the Throttle, Complete, Verify #

1. Submit with the barrier held #

Attach --postpone-completion to the strategy so no shard cuts over on its own, and --singleton so a second migration cannot race the same table. Use a stable --migration-context so every shard’s job is grouped:

vtctldclient ApplySchema \
  --ddl-strategy "vitess --postpone-completion --singleton" \
  --migration-context "orders-add-region-2026q3" \
  --sql "ALTER TABLE orders ADD COLUMN region_id BIGINT UNSIGNED NULL" \
  commerce

The command returns a migration_uuid. Capture it — every subsequent control acts on it.

2. Watch the copy and its throttle state #

Poll the per-shard status. During the copy you will see shards oscillate between running and throttled as replica lag rises and falls; that oscillation is healthy, not a fault:

vtctldclient OnlineDDL show commerce orders-add-region-2026q3

If a shard sits throttled far longer than the copy should take, inspect the throttler to see whether the block is automatic (lag) or manual:

vtctldclient GetThrottlerStatus commerce/-80

3. Release the cutover once every shard is ready #

Do not complete until all shards report ready (caught up and postponed). Completing while a shard is still running cuts over only the ready shards and leaves the rest — a partial cutover. Once the barrier is fully populated, release the fleet:

vtctldclient OnlineDDL complete commerce orders-add-region-2026q3

4. Verify the swap and hold the artifact #

Confirm every shard reached complete, then leave the retained original in place until you are confident (do not cleanup immediately, or you forfeit the revert path):

vtctldclient OnlineDDL show commerce orders-add-region-2026q3

Configuration Reference #

These are the flags that determine throttling and cutover behaviour. Defaults favour safety; the recommended column is oriented at a large sharded fleet.

The two misconfigurations that cause real incidents are predictable. Setting the lag threshold tighter than replicas can ever satisfy under copy load leaves a migration throttled indefinitely — it never reaches ready, the barrier never populates, and the change silently stalls. Setting --cutover-threshold very high on a table with long transactions turns the swap into a multi-second write stall visible to the application as a latency spike. Neither is a crash; both are quiet, which is why they get missed.

Failure Modes #

Stuck in throttled, never reaching ready. Symptom: one or more shards report migration_status = throttled for far longer than the copy volume justifies; copy throughput is near zero; the throttler status shows an automatic lag block. Root cause: the lag threshold is unachievable — the replicas cannot keep lag under the threshold while absorbing the copy write volume, so the gate never opens long enough to finish. Mitigation: confirm the block is lag-driven (not a manual throttle left in place), then either raise the threshold to a value the replicas can meet, reduce copy concurrency, or move the heavy copy to an off-peak window. Do not unthrottle blindly — that only removes a manual hold and does nothing about automatic lag pacing.

Cutover lock timeout. Symptom: the migration stays ready after complete is issued; logs show repeated cutover attempts; performance_schema shows metadata-lock waits on the target table. Root cause: a long-running transaction or open cursor holds the table’s metadata lock, so Vitess cannot acquire the write lock within --cutover-threshold and keeps retrying. Mitigation: find and end the blocking transaction (or wait it out), then the next retry succeeds. If long transactions on this table are normal, raise --cutover-threshold modestly and schedule the cutover for a quiet window rather than fighting the lock.

Partial cutover across shards. Symptom: OnlineDDL show reports some shards complete and others still ready or running; queries that span both see two table shapes. Root cause: complete was issued before the barrier was fully populated, or a shard-local cutover failure fired after the fan-out began. Mitigation: treat the cutover fan-out as all-or-nothing. If a shard fails to swap, immediately revert the shards that already cut over (possible only because retention kept their originals alive), then re-queue the failed shard. This is most dangerous for cross-shard reads, so resolve it fast.

Manual throttle left engaged. Symptom: a migration sits throttled with healthy replicas and near-zero lag. Root cause: someone ran OnlineDDL throttle during an incident and never unthrottled; the automatic check would pass, but the manual hold overrides it. Mitigation: vtctldclient OnlineDDL unthrottle clears it; add a check to your orchestration that alerts when a migration is throttled while lag is well under threshold.

Verification #

Confirm the outcome rather than assuming it. Every shard should report a terminal complete with no leftover artifact tables and the throttler sitting clear of its threshold:

# Every shard should show migration_status = complete
vtctldclient OnlineDDL show commerce orders-add-region-2026q3

# Throttler should report no active throttle and lag under threshold
vtctldclient GetThrottlerStatus commerce/-80

Then confirm the schema is actually visible through the router — present on disk is not the same as routed correctly:

-- Through VTGate: the new column must resolve on every routed shard
SELECT region_id FROM orders WHERE order_id = 4820117;

Finally, watch the post-cutover signals for a few minutes: query-plan p99 latency (optimizers recompile against the new definition and the rebuilt table’s buffer pool is cold), the artifact-table count (should return to baseline once you cleanup), and replica lag (should settle now that the copy write volume is gone). Those signals are exactly what the observability side is built to surface — wire them up in Monitoring Replication Lag and Throttling before you run a large migration, not after.

Related #

• gh-ost vs pt-osc for Vitess DDL — how the two external tools’ throttling and cutover knobs compare when the native strategy is not the fit.
• Tuning the Online DDL Throttler — choosing a lag threshold and check interval that protect replicas without stalling the copy forever.
• Reverting a Completed Online DDL Migration — using the retained original table to undo a completed swap, and the limits of what revert can restore.
• Coordinating Multi-Shard Schema Migrations — holding and releasing the fleet-wide barrier that postponed completion makes possible.
• Vitess Native Online DDL vs External Tools — why native throttling and cutover are control-plane operations rather than something you reconstruct per shard.

← Back to Online DDL Orchestration & Migration Coordination · Related area: Monitoring Replication Lag and Throttling