Tuning the Online DDL Throttler

The throttler has exactly one job — pause the migration copy when replicas fall behind — and exactly one setting that decides whether it does that job well: the lag threshold. Set it wrong in either direction and the migration either never finishes or breaches your replica-lag SLO.

Context

The throttler is one of the two safety controls described in Throttling and Cutover Control for Online DDL; this page is only about tuning it. Tuning it well is a closed loop with the observation side: you set a threshold, watch how the copy and replica lag actually behave against it, and adjust. The metrics that close that loop — the throttled/running transition counts, per-shard lag, and copy throughput — are surfaced in Monitoring DDL Lag During Migrations. Treat this page as the “decide” half and that page as the “observe” half.

The concept: a threshold is a bet against your replicas

The throttler works by asking, before each batch of copied rows, whether replication lag on the shard is under the threshold. Under: the batch copies and the migration stays running. Over: the batch is held, the migration flips to throttled, and the copy waits for the next check to pass. The threshold is therefore a bet: you are asserting that your replicas can absorb the copy’s extra write volume while staying under this much lag. Two things go wrong when the bet is bad.

Set the threshold too tight — say 500ms on replicas that lag 2s the instant the copy applies real pressure — and the check almost never passes. The migration spends its life throttled, throughput hovers near zero, and it never reaches the caught-up ready state. The migration is not failing; it is waiting, forever, which is worse because nothing alerts.

Set the threshold too loose — say 30s on a fleet whose read path assumes sub-second freshness — and the check almost always passes. The copy runs full-tilt, replicas fall tens of seconds behind, and stale reads breach SLO even though the migration itself looks perfectly healthy. The right value is the widest threshold your read path can actually tolerate, and no wider.

Three lag-threshold regimes for the Online DDL throttler A vertical replica-lag axis with an SLO ceiling line near the top and an induced-copy-lag line near the bottom. Three columns show a too-tight threshold below induced lag (always throttled), a good threshold in the band between induced lag and the SLO ceiling (copy proceeds within SLO), and a too-loose threshold above the SLO ceiling (SLO breach). replica lag replica-lag SLO ceiling lag the copy induces safe threshold band too tight always throttled well chosen copy runs, SLO held too loose SLO breach

Choosing the threshold and check interval

Start from your read-path freshness requirement, not from a Vitess default. If reads tolerate up to 3s of staleness, the threshold ceiling is 3s; pick a working value a little under it — 2s — so there is headroom before the SLO line. Then run the migration and watch whether the copy still makes steady progress at that threshold. If it does, you are done. If it spends most of its time throttled, your replicas cannot absorb the copy at that lag budget, and the fix is to give the copy more headroom (a slightly higher threshold, if the SLO allows) or reduce the copy pressure, not to keep tightening.

The check interval governs how quickly the gate reacts, not how tolerant it is. A short interval (~1s) means the throttler notices lag rising and closes the gate promptly, and notices recovery and reopens promptly — which keeps lag hugging the threshold closely. A long interval lets lag overshoot between checks. Leave the interval short; it is the threshold, not the interval, that you tune for the SLO trade-off.

You can throttle a specific migration by hand while you tune. vtctldclient OnlineDDL throttle forces the migration to stop copying regardless of lag; unthrottle returns it to automatic lag pacing:

# Manually pause this migration's copy while you investigate a lag spike
vtctldclient OnlineDDL throttle commerce a1b2c3d4_...

# Hand control back to the automatic lag-based throttler
vtctldclient OnlineDDL unthrottle commerce a1b2c3d4_...

A manual throttle and an automatic lag throttle both show migration_status = throttled; only the throttler status distinguishes them. That distinction is the single most common source of confusion during tuning — see the edge cases.

Reading throttled and running transitions

Do not judge the throttler by a single status read; judge it by the transition pattern over time. Poll per-shard status and watch how the migration moves between running and throttled:

vtctldclient OnlineDDL show commerce a1b2c3d4_...

Three patterns tell you three different things. Frequent oscillation between running and throttled with steady overall progress is the throttler working exactly as intended — it is riding the threshold, letting the copy through in the gaps and backing off on the peaks. Permanently throttled with no progress means the threshold is unachievable; the copy will never finish. Permanently running during a heavy copy, with replica lag climbing, means the threshold is too loose and the gate is effectively never closing — check whether lag is quietly breaching SLO. To see why a shard is throttled, read the throttler directly:

vtctldclient GetThrottlerStatus commerce/-80

App-vs-DDL throttle scope

The same throttler serves two audiences with different needs, and conflating them is a tuning trap. The app throttle protects the whole primary from the application’s own write pressure, using an aggregate check. The DDL throttle exists specifically to pace this migration’s copy against the replicas that will serve reads. Submitting the migration with --throttle-check-as-check-self scopes its check to the tablet’s own lag view rather than the aggregate app-facing check, so DDL pacing is decided independently of the application’s throttle state:

vtctldclient ApplySchema \
  --ddl-strategy "vitess --postpone-completion --throttle-check-as-check-self" \
  --sql "ALTER TABLE orders ADD COLUMN region_id BIGINT UNSIGNED NULL" \
  commerce

The practical consequence: with self-scoped checks, a threshold you tune for the migration only affects the migration. Without it, a migration and the app can throttle each other in ways that make the transition pattern hard to read — you tighten the migration’s budget and see the app’s throttle state instead. Decide the scope before you tune the number.

There is a second reason scope matters during an incident. If the application is already throttled — say a traffic spike pushed it into its own backpressure — a migration sharing that aggregate check inherits the throttle whether or not replica lag from the copy warrants it. That is sometimes what you want (pause the copy while the primary is under stress) and sometimes not (the copy was the only thing keeping to its lag budget and the app spike is unrelated to it). Self-scoped checks let you reason about the two independently; the aggregate check ties their fates together. Neither is universally correct, which is exactly why the flag exists — pick the coupling deliberately rather than discovering it during a page.

Tuning against a whole fleet, not one shard

A threshold is set per migration but experienced per shard, and shards are not identical. One shard’s replicas may sit on busier hardware, carry a hotter slice of the keyspace, or serve a region whose read traffic never fully quiets. A threshold that keeps the median shard riding the gate cleanly can leave the busiest shard permanently throttled — so the migration’s true completion time is set by its worst shard, not its average. When you read transitions, read them per shard and watch the tail: if fifteen shards oscillate healthily and one sits pinned at throttled, the fix is targeted at that shard (more copy headroom during its region’s trough, or a temporarily relaxed threshold for that run) rather than a fleet-wide loosening that would breach SLO everywhere else. This is the same worst-shard-dominates logic that governs the fleet-wide cutover barrier; the throttler just surfaces it earlier, during the copy.

Edge cases and gotchas

  • Threshold too tight → never completes. The clearest failure: permanently throttled, throughput near zero, ready never reached, and nothing alerts because “waiting” is not “failing”. Alert explicitly on a migration throttled beyond an expected copy-duration budget.
  • Threshold too loose → SLO breach. The migration looks perfectly healthy while replicas drift past the freshness SLO. The throttler cannot save a read path from a threshold set above its tolerance — the number is the SLO promise.
  • Manual throttle mistaken for lag throttle. A migration sitting throttled with healthy replicas and near-zero lag is almost always a leftover manual throttle from an incident. unthrottle clears it; automatic lag pacing would have let it run. Always check GetThrottlerStatus before assuming lag is the cause.
  • No replicas, no signal. On a keyspace with no replicas the lag check has nothing to read and the copy runs effectively unthrottled. Verify a lag source exists before you rely on the threshold to protect anything.
  • Check interval blamed for tolerance. Widening the check interval does not make the throttler more tolerant of lag — it just makes it react more slowly and overshoot further. Tune the threshold for tolerance; keep the interval short.
  • Tuned on staging row counts. Throttle behaviour on a 200-row table tells you nothing about a 400-million-row table — induced lag scales with copy volume. Tune against representative data.

Verification

Confirm the throttler actually held the SLO during the copy, not just that the migration finished. Cross-check the migration’s own throttle transitions against measured replica lag over the copy window:

# Migration should reach ready; throttler should report no permanent block
vtctldclient OnlineDDL show commerce a1b2c3d4_...
vtctldclient GetThrottlerStatus commerce/-80

Peak replica lag across the copy window should stay under your SLO ceiling, and the migration should have reached ready in a time consistent with your copy-volume estimate — not far longer, which would mean it spent the run throttled. If either check fails, adjust the threshold and re-run against representative data; the per-shard lag and throttle-event series to chart are detailed in Monitoring DDL Lag During Migrations.

← Back to Throttling and Cutover Control for Online DDL