Monitoring DDL Lag During Migrations

While an Online DDL copy is running, replication lag rising and the copy pausing is usually the throttler doing exactly its job — the operational skill is reading the metrics well enough to tell that healthy backpressure apart from a migration that has genuinely stalled and needs a human.

Context #

A long schema migration deliberately generates replication lag: the VReplication copy applies rows in bulk, replicas fall behind, the throttler pauses the copy, replicas catch up, the copy resumes. On a lag dashboard that looks alarming — a sawtooth of lag climbing to threshold and copy throughput dropping to zero — yet nothing is wrong. The failure this page addresses is the opposite mistake to over-alerting: an operator who has learned to ignore migration-time lag also misses the run that has actually wedged. Distinguishing the two is a matter of correlating lag with progress, not watching lag alone.

This is the observation counterpart to tuning; where tuning the Online DDL throttler decides how aggressively the copy should be paused, this page decides how you watch the result and when to page. It builds directly on the lag-and-throttle loop in monitoring replication lag and throttling, and the migration’s own phase model — the states a run passes through — is defined in tracking migration progress and state machines. It is part of the broader practice of observability and operations for sharded Vitess.

Which metrics move, and how they should move together #

Three families of signal move during a migration, and their relationship is the diagnosis:

• Replica lag (vttablet_replication_lag_seconds) rises as the copy applies rows and falls when the throttler pauses it. Under healthy throttling it oscillates around the throttle threshold — a sawtooth, not a monotone climb.
• Throttler rejections (rate(vttablet_throttler_check_total{code="429"}[5m])) rise in anti-phase with copy throughput: when lag is over threshold the copy is refused, so rejections climb and throughput drops.
• VReplication progress — the copied-row count and the stream’s own lag — must keep advancing across pauses. Progress is the discriminator: a healthy run pauses but its cumulative progress still climbs over any multi-minute window; a stalled run shows progress flat while everything else looks superficially similar.

The VReplication copy exposes its own counters distinct from the tablet’s replica-lag gauge. vttablet_vreplication_count and the copy-phase row counters advance as rows are applied, and the stream reports a vreplication_lag — how far behind the source binlog position the stream is — which is different from the replica’s mysqld replication lag. Correlating the stream’s row progress with the migration’s reported SHOW VITESS_MIGRATIONS progress percentage is what turns “lag is high” into “the copy is or is not advancing.”

The discriminator: correlate lag with progress #

The rule that separates the two signatures is compact: high lag plus advancing progress is healthy; high lag plus flat progress is a stall. A throttled-but-healthy copy is expected to sit at high reject rate — that is the throttler protecting @replica reads — so alerting on reject rate or lag alone guarantees false pages during every large migration. The alert must additionally require that progress has not moved.

VReplication progress is observable two ways. From the control plane, the migration’s own progress percentage:

# Progress percentage and copy state per shard for a running migration
vtctldclient OnlineDDL show --json commerce <migration_uuid> \
  | jq -r '.[] | [.shard, .migration_status, .progress] | @tsv'

From Prometheus, the stream’s cumulative counters, whose increase() over a window is the forward-motion test:

# Rows applied by the migration's VReplication copy in the last 10m, per shard
sum by (shard) (increase(vttablet_vreplication_query_count{keyspace="commerce"}[10m]))

A stalled copy shows this increase() at or near zero across a multi-minute window while replica lag stays high — the exact conjunction the naive lag-only view cannot see.

A PromQL alert that fires only on a real stall #

Encode the discriminator directly: fire only when the throttler is rejecting and the copy has made no progress for a sustained window. The and between two vectors is what suppresses the healthy-throttling false positive.

# Alert: copy is throttled AND has not advanced for 15m — a genuine stall
(
  sum by (keyspace, shard) (rate(vttablet_throttler_check_total{code="429"}[5m])) > 0
)
and
(
  sum by (keyspace, shard) (increase(vttablet_vreplication_query_count[15m])) < 1
)

The left vector is true whenever the throttler is refusing the copy — true for both healthy and stalled runs. The right vector is true only when the copy applied effectively nothing in fifteen minutes. Their intersection is the stall: rejected and not moving. During a healthy migration the left vector fires constantly but the right vector is false, so no page is raised; the alert stays silent through the entire expected sawtooth and speaks only when forward motion genuinely stops.

The distinction from the tuning page is worth restating: tuning changes how much the copy is throttled, and would change the amplitude and frequency of the healthy sawtooth on the left panel above; this alert is indifferent to that amplitude because it keys on progress, not on lag magnitude. That is why an observe-side alert survives a re-tune without adjustment.

Healthy versus stalled at a glance #

Edge cases and gotchas #

• Progress percentage can plateau legitimately near the end. The copy phase reaches ~100% and the stream enters the catch-up/verify phase where the row counter barely moves but the migration is still healthily draining binlog events. Cross-check migration_status — running in the final phase is fine; treat a flat counter as a stall only while progress is well below 100%.
• A paused-by-design migration is not a stall. A run submitted with --postpone-completion sits caught-up-but-not-cut-over on purpose, and one you explicitly throttled via the control plane is intentionally frozen. The alert above keys on 429 reject rate, which is not raised by an intentionally postponed run — but an operator eyeballing progress should confirm the run was not deliberately held before escalating.
• The stream lag and the replica lag are different numbers. vreplication_lag is how far the copy stream trails the source binlog; vttablet_replication_lag_seconds is how far a replica trails its primary. A migration can drive replica lag high while its own stream lag is low. Alert on replica lag for read-safety, but diagnose copy health with the stream’s own counters.
• Per-shard skew. One shard can stall while the rest of the fleet progresses — often the shard with the hottest table or the least replica headroom. Keep the alert grouped by (keyspace, shard) so a single wedged shard pages rather than being averaged away by healthy siblings.
• Counter resets on tablet restart. A VTTablet restart zeroes vttablet_vreplication_query_count; increase() handles the reset, but a very short evaluation window straddling a restart can briefly read low. A 15m window comfortably absorbs a restart without a false stall.

Verification #

Confirm the alert discriminates correctly by watching both series through a deliberately throttled copy on staging. Lower the throttle threshold so the copy is forced to pause frequently, start a migration on a table with real row volume, and watch that replica lag sawtooths and the reject rate stays high while the progress counter keeps climbing:

# During the throttled staging run: reject rate should be high, progress still advancing
watch -n5 'vtctldclient OnlineDDL show --json commerce <migration_uuid> \
  | jq -r ".[] | [.shard, .progress, .migration_status] | @tsv"'

If the progress column advances on every poll, the run is healthy and the stall alert must stay silent — verify it does. Then simulate a stall (for example, hold a metadata lock on the target table so the copy cannot advance) and confirm the same alert fires within its 15-minute window. An alert that stays quiet through the healthy run and fires on the induced stall is correctly tuned to the observe-side question this page exists to answer.

Related #

• Monitoring Replication Lag and Throttling — the lag-and-throttle feedback loop these migration signals ride on.
• Tuning the Online DDL Throttler — the tune-side counterpart that sets how aggressively the copy is paused.
• Tracking Migration Progress and State Machines — the per-shard state model whose progress field the stall discriminator reads.

← Back to Monitoring Replication Lag and Throttling