Key VTTablet Prometheus Metrics to Watch

A single VTTablet exports several hundred Prometheus series, and paging through all of them during an incident is how you miss the one that matters — this page names the handful that actually predict trouble on a sharded fleet and the threshold at which each becomes alert-worthy.

Context #

Every tablet exposes /metrics for Prometheus and /debug/vars for structured JSON, and the temptation is to graph everything. The discipline that keeps a dashboard useful is choosing a small set of leading indicators — signals that move before a user-visible incident rather than after. This guide narrows the export to that set. It sits under monitoring replication lag and throttling, which covers the lag-and-throttle loop in depth, and inside the wider practice of observability and operations for sharded Vitess. The pool-sizing and durability flags these metrics reflect are set per the guidance in configuring VTTablet for high availability.

A VTTablet is a proxy in front of exactly one mysqld, so its metrics fall into four natural families: query-serving (what the tablet is asked to do), pool (the finite resources it uses to do it), mysqld (the health of the backing MySQL), and throttler/replication (whether the shard is healthy enough to accept background load). Watching one metric from each family catches the large majority of tablet-level incidents.

The metrics that predict trouble #

Query-serving rate and errors. vttablet_queries_total counts queries served, broken out by plan and table; its rate is your per-shard QPS, and a sudden divergence between shards on the same keyspace is the earliest sign of a routing hotspot. Pair it with vttablet_query_error_counts, whose rate should be near zero — a step increase almost always precedes a user-visible error budget burn. The companion histogram vttablet_query_durations (or vttablet_queries in --histogram form on older builds) gives you tablet-side p99, which isolates whether latency is born at the tablet or upstream at the router.

Transaction and connection pool saturation. These are the metrics that turn a latency blip into an outage, because a full pool means new work simply waits. vttablet_transaction_pool_available reports free slots in the transaction pool; when it sits at or near zero while vttablet_transaction_pool_wait_count climbs, transactions are queueing for a connection and every write is paying that wait. The same pair exists for the read pool. The classic pool-exhaustion signature is available-at-zero plus a rising wait count plus flat or falling query rate — throughput collapses precisely because the pool cannot hand out a connection, not because MySQL is slow.

mysqld health. vttablet_mysql_conn_count (and the related mysql_connapp pool series) track connections the tablet holds open to its mysqld; a value pinned at the configured --db-conn-pool-size means the tablet cannot open more and is the ceiling behind pool waits. vttablet_mysql_seconds histograms per-call MySQL latency as seen by the tablet, which cleanly separates “MySQL got slow” from “the tablet got busy.” A rise here with a flat query rate points at mysqld — a cold buffer pool, a lock, or disk pressure — not at Vitess.

Replication lag. vttablet_replication_lag_seconds is the heartbeat-based lag gauge described in the parent guide; on a replica tablet it is the single most important series to alert on, because it gates both @replica read safety and throttler decisions. Filter it to tablet_type="replica" so a promoted primary’s meaningless self-lag never pollutes the view.

Throttler checks. vttablet_throttler_check_total, labelled by result code, lets you compute the reject rate covered in monitoring DDL lag during migrations. A steady low reject rate during a copy is healthy backpressure; a rate pinned at 100% with no copy progress is a stalled shard. Its counterpart vttablet_throttled_count attributes throttling to the app that was refused, which is how you tell an Online DDL copy being paused apart from a resharding stream being paused when both are labelled by app.

The four families are not equally weighted, and the fastest way to build an incorrect dashboard is to treat them as one flat list. Query-serving metrics answer what is the tablet being asked to do; pool metrics answer can it afford to do it; mysqld metrics answer is the storage engine keeping up; and throttler/replication metrics answer is the shard healthy enough for background work. When an incident starts, you read them in roughly that order — a query-rate spike is the trigger, a pool at zero is the constraint that turned the spike into an outage, mysqld latency tells you whether the backing store contributed, and the lag/throttler pair tells you whether a background copy was competing for the same headroom. Wiring one panel per family in that order turns a wall of series into a short diagnostic sequence.

Building a minimal alert set from the families #

You do not alert on all four families equally. Two of them — pool saturation and replication lag — are causal signals that fire before users notice; the other two — query rate/errors and mysqld latency — are mostly diagnostic, useful for root-causing after a causal alert has already fired. A defensible minimal page-worthy set is exactly three alerts per shard: transaction-pool availability at zero with a rising wait count (the write path is queueing), replica lag over the degraded threshold (reads are about to stop), and query error rate stepping off its zero baseline (something is failing outright). Everything else in the table below belongs on the dashboard you open after one of those three pages you, not on a pager of its own — adding more pagers dilutes the signal and trains the on-call to ignore them.

The reason pool metrics rank above raw latency is causality: a pool that has run dry causes the latency you would otherwise alert on, and it does so a few seconds earlier, so paging on the pool catches the incident before the latency histogram has even shifted. Latency alerts, by contrast, fire once damage is already user-visible. Wherever you can alert on the cause instead of the symptom, prefer the cause.

Reference table #

Metric names below are the current Vitess exposition; see the version-rename note that follows the table. Types are the Prometheus type; the alert condition is a starting point to tune against measured load, not a universal constant.

Edge cases and gotchas #

• Metric renames across Vitess versions. The exposition has churned. Older builds published counters without the _total suffix (vttablet_queries), used TransactionPoolAvailable in /debug/vars camelCase rather than the snake-case Prometheus form, and named the lag gauge differently before heartbeat lag became the default. When a panel goes blank after an upgrade, diff the raw /metrics output rather than assuming the tablet is down — the series was almost certainly renamed. Pin dashboard queries to the exposition of the version you actually run and re-check them on every major upgrade.
• Counters need rate(), gauges do not. vttablet_queries_total is a monotonic counter — graphing it raw shows an ever-rising line that tells you nothing. Always wrap counters in rate() or increase(); only gauges like vttablet_transaction_pool_available and vttablet_replication_lag_seconds are meaningful as instantaneous values.
• Pool-exhaustion looks like a MySQL problem but is not. When vttablet_transaction_pool_available is zero, tablet-side latency (vttablet_query_durations) spikes even though vttablet_mysql_seconds stays flat — MySQL is fine; the tablet just cannot hand out a connection. Reading only the query-latency panel misdiagnoses this as a database slowdown and sends you tuning the wrong layer. The available-plus-wait_count pair is what disambiguates it.
• Aggregate carefully across tablet types. A sum over vttablet_replication_lag_seconds without a tablet_type filter mixes replicas (which have real lag) with primaries (which report zero), diluting the signal. Always aggregate lag as max by (keyspace, shard) over replicas only.
• High-cardinality labels. vttablet_queries_total is labelled by table and plan; on a keyspace with thousands of tables this explodes series count and can overwhelm Prometheus. Drop the table label at scrape time if you do not need per-table breakdowns.
• Per-shard, not per-tablet, is the unit of alerting. A keyspace has many tablets but a fixed set of shards, and it is the shard that has an SLO. Alerts that key on individual tablet instances page twice for one problem (primary and replica both fire) and go silent when a tablet is replaced during a reparent. Aggregate by (keyspace, shard, tablet_type) and alert on the shard, so a routine tablet swap does not resolve-then-refire the same alert.
• /debug/vars and /metrics can disagree briefly. The JSON vars endpoint and the Prometheus exposition are sampled independently, so during a fast transient (a pool draining and refilling within a scrape interval) the two can show different values. Trust the Prometheus series for alerting and reserve /debug/vars for point-in-time debugging on a single tablet.

Verification #

Run a PromQL smoke query to confirm the core series exist and carry the labels your dashboards assume, before you rely on any alert built on them:

# Smoke test: every scraped tablet should return each core series with keyspace/shard labels
count by (__name__) (
  {__name__=~"vttablet_(queries_total|transaction_pool_available|mysql_seconds_count|replication_lag_seconds|throttler_check_total)",
   keyspace="commerce"}
)

Every listed metric should return a non-zero count; a metric returning nothing means either the scrape is not reaching those tablets or the series was renamed in your Vitess version. As a second check, confirm the lag gauge on a known replica matches what the tablet’s own check-self endpoint reports — if the Prometheus value and the endpoint disagree, the scrape target or port is wrong, and every alert built on that series is unreliable.

Related #

• Monitoring Replication Lag and Throttling — the lag-and-throttle feedback loop these metrics measure.
• Monitoring DDL Lag During Migrations — reading the same throttler and lag series specifically while a copy runs.
• Configuring VTTablet for High Availability — the pool and durability flags these metrics reflect.

← Back to Monitoring Replication Lag and Throttling