Monitoring Replication Lag and Throttling Across a Sharded Vitess Fleet

Replication lag is the single most load-bearing health signal in a sharded MySQL deployment: it decides whether @replica reads are safe to serve, whether a shard’s replicas can be promoted without data loss, and whether a background row copy is allowed to keep writing. Vitess turns that signal into a control input through the tablet throttler, a lag-driven gate that every VReplication stream and Online DDL copy must ask permission from before it applies another batch. This guide is for MySQL SREs and platform engineers who need to observe that loop end to end — how a VTTablet measures and exposes lag, how the throttler consumes it to pause and resume copies, and how to tell a healthy backpressure event apart from a genuine incident. It is the observation counterpart to the tuning work in throttling and cutover control, and it sits inside the broader practice of observability and operations for sharded Vitess.

Prerequisites

Before the mechanics below are actionable, confirm the following about your environment:

  • Vitess 16+ for the current tablet-throttler surface. The throttler was substantially reworked around Vitess 16–17: it now runs on every tablet by default and drives Online DDL and VReplication throttling through the same check API. Examples here assume 17 or newer; on older builds the --enable-lag-throttler flag and per-keyspace enablement differ.
  • MySQL 8.0 primaries and replicas with GTIDs and semi-sync configured on every shard. The throttler reads replication lag from the replicas it discovers through the topology, so a shard with no healthy replica has no lag signal to gate on.
  • A Prometheus scrape already reaching each VTTablet, or at least the ability to add one. Every tablet exposes /metrics and a /debug/vars JSON endpoint; the throttler additionally exposes /throttler/check and /throttler/check-self.
  • Assumed knowledge: how a shard’s replicas are provisioned and which one is promotable, covered in configuring VTTablet for high availability; and that a shard split or resharding run is itself a large VReplication copy subject to the same throttle, described in resharding workflows and shard splits.
  • A representative write workload on staging. Throttler behaviour on an idle keyspace tells you nothing — lag only builds under the write volume that a real copy competes with.

How the Signal Flows: Lag, Check, Decision, Feedback

The throttler is not a background reaper that kills slow copies; it is a request-time gate. Each copy loop — a VReplication stream applying rows, or an Online DDL migration copying a chunk — calls the throttler before it applies the next batch and obeys the answer. The throttler answers by comparing the freshest replication lag it has measured against a configured threshold. When lag is under threshold it returns 200 OK and the copy proceeds; when lag is over threshold it returns 429 Too Many Requests and the copy sleeps briefly and re-asks. That single request/response, repeated thousands of times a minute, is the entire feedback loop.

The lag-driven throttle feedback loop A cycle diagram. A copy loop asks the throttler for permission before each batch. The throttler compares measured replica lag against a threshold and returns 200 or 429. A 200 lets the copy apply writes, which raises lag; a 429 pauses the copy, which lets the replica drain and lag fall. The two outcomes form a self-correcting loop around the lag measurement. Shard primary mysqld — accepts writes Replica VTTablet measures replication lag publishes lag gauge binlog Tablet throttler lag vs threshold /throttler/check lag Copy loop VReplication / Online DDL copy check before batch 200 OK — lag under threshold apply batch applies writes → raises replication lag 429 — lag over threshold pause & re-ask pause lets replica drain → lag falls → next check allowed

The elegance of the design is that nothing needs to actively stop a runaway copy — the copy stops itself by respecting the 429, and because pausing the copy removes write pressure, lag falls, and the next check is allowed. The loop self-corrects around the lag measurement. What an operator monitors, then, is not one number but the relationship between three: measured replica lag, the throttle-check reject rate, and copy throughput. When all three are stable the loop is healthy even if the copy is intermittently paused; when lag climbs while the reject rate is pinned at 100% and throughput is zero, the loop has stopped self-correcting and something outside it needs attention.

Where Lag Comes From and How VTTablet Exposes It

A VTTablet running on a replica computes replication lag continuously as part of its health check and republishes it in several places. The canonical measure is heartbeat lag: Vitess writes a periodic heartbeat row on the shard primary (governed by --heartbeat_interval and --heartbeat_on_demand_duration), and each replica measures the delta between the newest heartbeat it has applied and the current time. Heartbeat lag is preferred over MySQL’s native Seconds_Behind_Master because it does not falsely read zero on an idle replica and does not spike to a huge number simply because the primary went quiet.

That lag value surfaces on three endpoints that matter for observability:

# 1. Prometheus scrape — the replication lag gauge, in seconds
curl -s http://shard-80-replica-a.internal:15100/metrics | grep -i replication_lag

# 2. Structured vars — the same value plus health state, as JSON
curl -s http://shard-80-replica-a.internal:15100/debug/vars | jq '.ReplicationLagSeconds'

# 3. The throttler's own view: check THIS tablet against its threshold
curl -s "http://shard-80-replica-a.internal:15100/throttler/check-self?app=test"

The distinction between the last two calls is the crux of per-shard vs fleet-wide observation. check-self asks a single tablet “are you lagged?” and is what you scrape to build a per-tablet lag panel. The plain check endpoint, served by the shard primary’s throttler, asks the aggregate question “is this shard healthy enough to accept more copy load?” — it inspects the lag of the shard’s replicas and returns the shard-level verdict that a copy actually gates on. A per-shard lag view is therefore built from check-self scrapes across every tablet; a fleet lag view is the same gauge aggregated with a max by (keyspace, shard) in PromQL. The throttler decision that governs copies is the check verdict, exposed as throttler check-total counters you can rate over time.

The reason the aggregation must be max and not avg deserves emphasis, because it is a common dashboard error. A shard’s read safety is governed by its worst replica, not its average: if three replicas are healthy and one is ten seconds behind, the shard cannot safely serve @replica reads from the lagged one, and the throttler — configured to gate on the shard aggregate — will act on that worst case. An avg-based panel hides the very replica that is about to force a throttle event, so a shard can look green on the dashboard right up to the moment its copy stalls. Aggregate with max by (keyspace, shard) and, separately, keep a per-tablet panel so you can see which replica is the laggard. When resharding, remember that the split’s own VReplication copy is a first-class consumer of this same throttle — a shard split running concurrently with an Online DDL migration means two copy loops competing for the same lag budget, which is why resharding runs are usually scheduled clear of large schema changes.

Step-by-Step: Wiring the Scrape and the Throttler

1. Confirm the throttler is enabled for the keyspace

On current Vitess the throttler runs by default, but its enforcement and threshold are keyspace-level settings you should set explicitly rather than inherit:

vtctldclient UpdateThrottlerConfig \
  --enable --threshold 1.5 --check-as-check-self \
  commerce

--threshold 1.5 sets the lag ceiling to 1.5 seconds; --enable turns on enforcement so that VReplication and Online DDL actually obey the verdict rather than merely reporting it.

2. Verify the throttler answers on every shard

Ask each shard’s primary throttler for a verdict directly. A healthy shard returns HTTP 200; a lagged one returns 429 with the offending metric:

# Ask the shard's throttler whether a copy app may proceed
curl -s -o /dev/null -w '%{http_code}\n' \
  "http://shard-80-primary.internal:15100/throttler/check?app=vreplication"

3. Scrape the lag gauge and throttler counters into Prometheus

Point the scrape at every tablet’s /metrics, keeping keyspace, shard, and tablet_type as labels so per-shard and fleet views are both derivable from the same series:

scrape_configs:
  - job_name: vttablet
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets:
          - shard-80-replica-a.internal:15100
          - shard-80-primary.internal:15100
        labels:
          keyspace: commerce
          shard: "-80"

4. Build the per-shard and fleet lag views

The per-shard view is the raw gauge; the fleet view collapses replicas to the worst case per shard, which is what actually gates a copy:

# Per-shard, per-tablet replica lag in seconds
vttablet_replication_lag_seconds{keyspace="commerce"}

# Fleet view: worst replica lag per shard — the number the throttler gates on
max by (keyspace, shard) (vttablet_replication_lag_seconds{keyspace="commerce"})

5. Track the throttle-check reject rate

A rising reject rate is the throttler doing its job; you monitor its shape, not merely its presence:

# Fraction of throttler checks rejected (429) over 5m, per shard
sum by (shard) (rate(vttablet_throttler_check_total{code="429"}[5m]))
  / sum by (shard) (rate(vttablet_throttler_check_total[5m]))

Configuration and Metric Reference

The settings and series below dominate how the loop behaves and what you can see of it. Defaults favour safety; the recommended column is production-oriented for a large sharded fleet where copies compete with live write traffic.

Setting / metric Where Type Default Recommended / meaning
--throttle_threshold VTTablet duration 1s 1s–5s; the lag ceiling above which copies are rejected. Raise only if replicas are provisioned for the extra load
--throttle_check_as_check_self VTTablet bool false true when copies should gate on the local tablet’s lag rather than the shard aggregate
--heartbeat_interval VTTablet duration 1s 250ms–1s; finer interval gives lower-latency lag detection at a small write cost
--heartbeat_on_demand_duration VTTablet duration 0 (always on) Set (e.g. 5s) to emit heartbeats only when a lag consumer is active, saving idle writes
--throttler-config-via-topo VTTablet bool true Keep on so UpdateThrottlerConfig propagates through the topology rather than per-tablet flags
vttablet_replication_lag_seconds metric (gauge) seconds — Heartbeat-based replica lag; the primary input to every throttle decision
vttablet_throttler_check_total metric (counter) count — Throttle checks by result code; rate(...{code="429"}) is the reject rate
vttablet_throttled_count metric (counter) count — Times a copy app was throttled; correlates with copy-throughput dips

Failure Modes Specific to Lag and Throttling

Lag storm. Symptom: max by (shard) lag climbs on several shards at once, throttler reject rate saturates at 100%, and all VReplication/Online DDL throughput drops to zero fleet-wide. Root cause: an external write surge (bulk import, a backfill job, a hot batch) is saturating replica apply on many shards simultaneously, so the throttler correctly refuses every copy — but the copies are now indefinitely starved. Mitigation: the throttler is working as designed; the fix is upstream. Shed or reschedule the write surge, or temporarily raise --throttle_threshold on shards whose replicas have headroom. Do not disable the throttler to “unstick” the copy — that trades a paused copy for @replica reads that silently go stale.

Throttler flapping. Symptom: the reject-rate series oscillates rapidly between 0% and 100% on a minute cadence, copy throughput is sawtoothed, and lag hovers exactly at threshold. Root cause: the threshold sits right at the shard’s steady-state lag, so each resumed batch pushes lag just over, triggering a reject, which lets lag fall just under, resuming again. Mitigation: widen the operating margin — either lower copy concurrency so a resumed batch does not overshoot, or raise the threshold by a second so the loop has hysteresis. The tuning trade-offs behind that choice are their own discipline, separate from the observation covered here.

Throttler disabled or blind. Symptom: copies run at full speed and never throttle, but replica lag on the shard climbs unbounded and @replica reads start returning stale data. Root cause: the throttler was disabled (--enable off), the shard has no healthy replica for it to measure, or the app name issuing copies was added to the throttler’s exempt list. Mitigation: confirm UpdateThrottlerConfig shows --enable for the keyspace, confirm at least one replica is SERVING, and audit the exempt-app list. A shard with a single replica that is itself down has no lag signal, so the throttler fails open — provision a promotable replica per the HA guide before running heavy copies.

Stale check-self from a promoted replica. Symptom: after a reparent, a tablet reports zero lag but is actually a fresh primary no longer replicating. Root cause: a former replica became PRIMARY; heartbeat lag against itself is meaningless. Mitigation: always filter lag panels by tablet_type="replica" so a promoted tablet does not inject a misleading zero into the fleet view.

Observing the Loop from Python

Dashboards are for humans; controllers need the same signals in code so that an automated migration driver can refuse to launch heavy copies into a shard that is already lagging. VTTablet exposes the throttler verdict over plain HTTP, so a controller can gate its own actions on the live shard health rather than on a stale scrape. The pattern is to poll the shard check endpoint and treat a non-200 as “do not add load right now”:

import requests

def shard_accepts_copy(primary_host: str, app: str = "online-ddl", port: int = 15100) -> bool:
    """True when the shard's throttler will admit more copy load for this app."""
    resp = requests.get(
        f"http://{primary_host}:{port}/throttler/check",
        params={"app": app}, timeout=2,
    )
    # 200 = under threshold; 429 = lagged and refusing; anything else = unhealthy
    return resp.status_code == 200

def fleet_ready(primaries: dict[str, str], app: str = "online-ddl") -> dict[str, bool]:
    """Map each shard to whether it will currently admit copy load."""
    return {shard: shard_accepts_copy(host, app) for shard, host in primaries.items()}

A controller that consults fleet_ready before submitting or resuming a migration never fights the throttler — it simply waits until every shard it is about to load returns True, then acts. This is the same idempotent read-before-act discipline used across the site’s orchestration surface: the controller derives its decision from live shard state instead of assuming the fleet is where it left it. For the parallel view of the routing layer’s health, the query-latency signals that pair with these lag signals are covered in monitoring VTGate query latency, and the specific act of reading these series while a copy is in flight is the subject of monitoring DDL lag during migrations.

Two guardrails make this robust. First, give the request a short timeout — a throttler that does not answer within a couple of seconds is itself a signal of an unhealthy tablet, and you want the controller to treat a timeout as “not ready,” never as “ready by default.” Second, use a distinct app name per workload (online-ddl, vreplication, a bespoke backfill name) so the throttler’s per-app accounting — and any exempt-list entries — apply to exactly the workload you intend, and so the vttablet_throttled_count attribution in your dashboards points at the right consumer.

Verification

Confirm the loop is both measuring and enforcing, not merely one of the two. First, prove the throttler will actually reject under lag by inducing a check while a copy is running and reading the verdict:

# During an active migration copy, the shard check should flip to 429 under load
watch -n2 'curl -s -o /dev/null -w "%{http_code}\n" \
  "http://shard-80-primary.internal:15100/throttler/check?app=online-ddl"'

Then confirm the observable series line up with the endpoint: the Prometheus vttablet_replication_lag_seconds for the shard’s replica should track the same value check-self reports, and every 429 you saw on the endpoint should appear as an increment in vttablet_throttler_check_total{code="429"}. If the gauge and the endpoint disagree, the scrape is hitting the wrong port or a stale target. A correct system shows lag, reject rate, and copy throughput moving together as one self-correcting loop — the exact signature you learn to read in the two guides below.

← Back to Observability & Operations for Sharded Vitess