Observability & Operations for Sharded Vitess: Instrumenting a Distributed MySQL Fleet

Running a sharded Vitess deployment means operating dozens or hundreds of moving parts — routers, tablets, replication streams, a topology store, and an orchestrator — where the failure of any one is invisible from inside a single MySQL instance. This reference is written for database platform engineers, MySQL SREs, and Python automation builders who own the pager for that fleet and need to turn its raw metric surfaces into golden signals, service-level objectives, and actionable alerts. Observing Vitess is not the same problem as observing a monolithic database: the signal you care about — “is the customer’s query fast and correct?” — is spread across a routing tier that fans out, a storage tier partitioned by key range, and a control plane that reshapes both underneath live traffic. This guide establishes the metric taxonomy, the collection pipeline, and the operational discipline that make that distributed state legible.

Everything on this site that changes the shape of the fleet — the VTGate routing architecture, VTTablet high-availability configuration, and Online DDL orchestration — emits telemetry that this observability layer consumes. The diagram below shows how each Vitess component exposes a Prometheus-format /metrics endpoint, how a scrape pipeline pulls them into a time-series database, and how that store feeds both dashboards and alert routing.

Why observing a sharded fleet is a different problem #

On a single MySQL server, one exporter and a handful of dashboards give you a near-complete picture: connections, buffer-pool hit rate, replication lag to a known replica, slow-query log. The instance is the unit of observation and the unit of failure at once. Vitess breaks that identity apart. A single logical query can touch one shard or every shard; a single logical table lives as N physical tables on N primaries; and the health of “the database” is now the aggregate of many independent MySQL processes plus the routing tier in front of them. Four consequences follow, and each shapes how the fleet must be instrumented.

The unit of observation is a label, not a host. Every useful Vitess metric is dimensioned by keyspace, shard, and tablet_type (and often plan or table). A p99 latency number with those labels stripped off is meaningless — it averages a healthy shard with a degraded one and hides exactly the partial failure you need to see. The whole collection pipeline exists to preserve those dimensions from the tablet’s /metrics endpoint all the way to the alert expression.

Aggregate health hides tail health. Fleet-wide “queries per second is normal” can coexist with one shard at 100% replica lag and a scatter query timing out on it. Because a scatter-gather query’s latency is bounded by its slowest participant, the arithmetic mean of shard health is the wrong statistic; you observe the fleet by its worst shard, not its average shard.

The control plane mutates the topology under you. Resharding, reparents, and Online DDL all change which tablet serves which range while traffic flows. An observability stack that assumes a static target list will silently stop scraping a tablet that was reparented, or double-count a shard mid-split. Service discovery has to track topology, not a static config file.

Correctness is now observable and must be observed. In a monolith a query either runs or errors. In Vitess a query can succeed while quietly degrading into a scatter across all shards, or land on a stale replica, or observe two schema versions mid-cutover. These are correctness-adjacent regressions with no error code, visible only as a shift in the ratio of targeted to scattered plans or a divergence between shards — which is why plan-type and consistency metrics are first-class signals here, not afterthoughts.

The metric surfaces: what each component exposes #

Every Vitess binary serves an HTTP status port (commonly :15000 for VTGate, :15100+ for VTTablet) that renders internal counters as a Prometheus exposition when built with the --emit_stats / --stats_backend plumbing, or simply via the /metrics handler. Knowing which counters live behind which endpoint is the map you navigate during an incident.

• VTGate exposes routing and query-plane telemetry: vtgate_api histograms (latency by operation and db_type), vtgate_queries_processed and vtgate_queries_routed counters dimensioned by plan type, vtgate_api_error_counts by error code, and connection/pool gauges. This surface answers “how fast and how scattered is routing?” and is dissected in depth under monitoring VTGate query latency.
• VTTablet exposes the storage-tier truth: vttablet_query_counts and vttablet_query_error_counts by table and plan, transaction-pool and connection-pool utilization, mysql semi-sync status, and — critically for sharded operations — the replication lag and health-check gauges. These are catalogued in key VTTablet Prometheus metrics to watch.
• VTOrc exposes the failover control loop: counters for detected problems, recoveries attempted and completed, reparent durations, and the current view of each shard’s primary. This is the surface that tells you why a shard’s primary moved and how long the window of no-writable-primary lasted.
• VReplication exposes stream progress: per-stream copy phase, rows copied, the transaction timestamp lag between source and target (vreplication_lag), and stream state. During a shard split or an Online DDL row copy this is the surface that reports whether the workflow is advancing or stalled.
• The tablet throttler exposes the backpressure signal: the currently observed replication lag against its threshold, and the fraction of throttle checks that were rejected. Because both Online DDL and VReplication obey the throttler, its metrics are the single place to see whether the fleet is deliberately slowing a background copy to protect live traffic.

The important property is that these surfaces are layered but correlated: a VTGate latency spike, a VTTablet lag gauge climbing, and a throttler rejecting checks are frequently three views of one event — a background copy saturating a shard’s replicas. Observability design is largely about wiring those views together so the on-call sees the cause, not just the symptom.

The four observability domains #

Rather than a flat list of hundreds of counters, treat the fleet as four domains. Each has a primary metric surface, a golden-signal question, and an owning subsystem. This is the mental model the dashboards and alerts are organized around.

Routing latency is the customer-facing golden signal. It is measured at VTGate because that is the only tier that sees the whole query lifecycle — plan, fan-out, gather. The subtlety is that a single latency number conflates two very different populations: targeted single-shard queries and scatter-gather queries. Observing them separately, and watching the ratio between them, is the substance of monitoring VTGate query latency; a rising scatter fraction is the classic silent regression.

Replication and throttling health is the storage-tier golden signal. Replica lag governs whether replicas can serve @replica reads and whether they are safe failover candidates; the throttler governs whether background work is being deliberately slowed to protect that lag. These two are inseparable — the throttler exists precisely to keep lag inside its threshold — and are treated together in monitoring replication lag and throttling.

Migration progress is the change-management signal. When an Online DDL row copy or a resharding workflow is in flight, the fleet is in a transient state that must be tracked to completion; the observable state model those metrics reflect is defined in tracking migration progress and state machines. A migration that stalls silently ties up resources and blocks the next change.

Topology and failover events is the availability signal. Every shard must have exactly one healthy writable primary; VTOrc counters expose how often that invariant broke and how quickly it was restored. A reparent that took 40 seconds is a 40-second write outage on that shard, invisible unless you measure it.

Building the collection pipeline: Prometheus, Grafana, Alertmanager #

Prometheus is the natural fit because Vitess exposes native Prometheus exposition and because the fleet is defined by topology that Prometheus service discovery can consume. The load-bearing design decision is how targets are discovered. A static target list rots the first time a tablet is reparented or a shard is split. Drive discovery from the same source of truth Vitess uses — the topology server — or from the orchestration platform (Kubernetes pod labels, Consul services) so that a tablet appears and disappears from the scrape set as the topology changes.

# prometheus.yml — discover tablets via the orchestration platform, keep the
# shard/type dimensions as labels so every downstream query can slice by them.
scrape_configs:
  - job_name: vttablet
    metrics_path: /metrics
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_component]
        regex: vttablet
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_keyspace]
        target_label: keyspace
      - source_labels: [__meta_kubernetes_pod_label_shard]
        target_label: shard
      - source_labels: [__meta_kubernetes_pod_label_tablet_type]
        target_label: tablet_type
  - job_name: vtgate
    metrics_path: /metrics
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_component]
        regex: vtgate
        action: keep

Preserving keyspace, shard, and tablet_type as target labels via relabel_configs is what lets every dashboard and alert downstream slice by shard without re-deriving it. From there, Grafana reads the same Prometheus for dashboards — the concrete panel construction is worked through in building a VTGate latency dashboard — and Alertmanager receives fired alerts and routes them by severity and team. Recording rules pre-aggregate the expensive expressions (fleet-wide p99 over all shards’ histograms) so dashboards and alerts read a cheap pre-computed series instead of recomputing a histogram_quantile over thousands of buckets on every evaluation.

Golden signals and SLOs per layer #

Translate the four domains into golden signals — latency, traffic, errors, saturation — and attach a service-level objective to each layer. SLOs are what convert a wall of graphs into a small number of yes/no health questions the on-call can reason about at 3am.

Two rules keep these honest. First, SLO the tail, not the mean — a p50 SLO on a scatter workload will look green while the p99 that the customer actually feels is on fire. Second, SLO per shard and per keyspace, then roll up — the fleet-wide number is for reporting; the per-shard number is for paging, because that is where partial failure hides.

Operational concerns: cardinality and retention #

The same labels that make Vitess metrics useful are the ones that blow up your time-series database if left unbounded. Cardinality is the product of every label’s distinct values: keyspace × shard × tablet_type × plan × table × the histogram’s bucket count, multiplied across every scraped target. On a fleet with 128 shards, a per-table per-plan histogram can generate hundreds of thousands of active series from a single metric family.

• Never let unbounded values become labels. Query text, user, or per-connection identifiers must never appear as label values; they turn one metric into millions of series and can take Prometheus down harder than the incident you were trying to observe.
• Drop high-cardinality labels you will not alert on. If you never page on per-table latency, use a metric_relabel_config to drop the table label at scrape time rather than storing it for every table on every shard.
• Use recording rules to pre-aggregate. A recording rule that rolls vtgate_api_bucket up to per-keyspace p99 lets you retain the cheap aggregate at long resolution and the expensive raw histogram at short retention.
• Tier retention by resolution. Keep raw high-cardinality series for days (incident forensics), downsampled aggregates for months (capacity trends). A single flat 400-day retention over full-cardinality histograms is the most common way a Vitess monitoring stack runs out of disk.

The uncomfortable trade-off is that dropping a label for cardinality reasons removes your ability to slice by it during an incident. Resolve it deliberately per metric: keep shard always (it is how partial failure surfaces), keep plan on VTGate (it is how scatter regressions surface), and be willing to drop table unless a specific table has an SLO.

Failure modes and recovery #

Observability itself fails, and its failures are especially dangerous because they blind you precisely when you most need sight. Each scenario below pairs a root cause with a mitigation checklist.

Observability blind spot after a topology change. Symptom: a shard’s metrics vanish from dashboards after a reparent or split; alerts on it stop firing (and look “green” by absence). Root cause: static scrape targets did not follow the topology; Prometheus is scraping a tablet that no longer serves that range, or not scraping a newly promoted one.

• Confirm service discovery is topology-driven, not a hand-maintained target list.
• Add an alert on up == 0 and on expected shard count — page when the number of scraped primaries per keyspace drops, so a missing shard is loud, not silent.
• After any resharding workflow, verify the new shards appear in the scrape set before declaring the split done.

Cardinality explosion. Symptom: Prometheus ingestion latency climbs, prometheus_tsdb_head_series spikes, queries time out, then the server OOMs. Root cause: a new label with unbounded values (often an accidental query/user dimension, or a new keyspace multiplying an already-wide metric).

• Identify the offending series with topk(10, count by (__name__)({__name__=~".+"})).
• Drop the label at scrape time with metric_relabel_configs; do not wait for a code fix.
• Set a per-target sample limit (sample_limit) so one misbehaving exporter cannot take the server down.

Missing scatter-query visibility. Symptom: latency SLO is green, but customers report intermittent slowness; no single shard looks unhealthy. Root cause: latency is being measured only in aggregate, so a rising fraction of scatter-gather queries — each bounded by the slowest shard — is averaged away.

• Split the latency histogram by plan type and by targeted-vs-scatter, per the scatter-ratio alerting approach.
• Alert on the scatter ratio, not just absolute latency; the ratio moves before the latency SLO breaches.

Throttler-induced stall mistaken for a hang. Symptom: a migration’s vreplication_lag is flat and copy throughput is zero; on-call suspects a hung workflow. Root cause: the throttler is deliberately pausing the copy because replica lag exceeded threshold — the system is working as designed.

• Correlate the stalled stream with the throttler’s rejected-check metric before intervening.
• If lag never recovers, the fix is capacity or scheduling, not restarting the workflow.

Python integration: pulling metrics and running SLO checks #

The two surfaces from the topology reference reappear here. Metrics are HTTP text on each component’s /metrics endpoint, and control-plane state is available through vtadmin. A controller that gates deployments or runbook steps on real health scrapes the former and queries the latter. Parsing the Prometheus exposition format directly keeps the check dependency-light and lets it run anywhere the endpoint is reachable:

import re
import urllib.request

_SAMPLE = re.compile(r'^(?P<name>[a-zA-Z_:][\w:]*)(?P<labels>\{[^}]*\})?\s+(?P<val>[-\d.eE+]+)$')

def scrape(endpoint: str) -> list[tuple[str, dict, float]]:
    """Fetch and parse a Prometheus /metrics endpoint into (name, labels, value)."""
    text = urllib.request.urlopen(endpoint, timeout=5).read().decode()
    out = []
    for line in text.splitlines():
        if not line or line.startswith("#"):
            continue
        m = _SAMPLE.match(line)
        if not m:
            continue
        labels = {}
        if m.group("labels"):
            for pair in m.group("labels")[1:-1].split(","):
                k, _, v = pair.partition("=")
                labels[k.strip()] = v.strip().strip('"')
        out.append((m.group("name"), labels, float(m.group("val"))))
    return out

def scatter_ratio(vtgate_metrics: str) -> float:
    """Fraction of routed reads that scattered, straight from a VTGate endpoint."""
    samples = scrape(vtgate_metrics)
    scattered = total = 0.0
    for name, labels, val in samples:
        if name == "vtgate_queries_routed":
            total += val
            if labels.get("plan", "").lower().startswith("scatter"):
                scattered += val
    return scattered / total if total else 0.0

A CI gate or a pre-cutover check reads that ratio and refuses to proceed when the fleet is unhealthy, the same idempotent, state-before-action discipline the orchestration guides use. For control-plane facts — is any migration in flight, does every shard have a primary — query vtadmin rather than scraping, because it already aggregates topology across cells:

import json
import urllib.request

def vtadmin_get(base: str, path: str) -> dict:
    req = urllib.request.Request(f"{base.rstrip('/')}/{path.lstrip('/')}")
    return json.loads(urllib.request.urlopen(req, timeout=5).read())

def every_shard_has_primary(base: str, keyspace: str) -> bool:
    """Availability gate: refuse a risky action unless the topology is whole."""
    data = vtadmin_get(base, f"api/tablets")
    primaries = {
        t["shard"]
        for t in data.get("tablets", [])
        if t.get("keyspace") == keyspace and t.get("tablet_type") == "PRIMARY"
        and t.get("state") == "SERVING"
    }
    shards = {t["shard"] for t in data.get("tablets", []) if t.get("keyspace") == keyspace}
    return shards and shards == primaries

These building blocks compose into the automation that runs throughout this site: a deploy pipeline that scrapes the scatter ratio before shipping a query change, a migration controller that will not release a cutover barrier while a shard is lagging, and an SLO reporter that reads per-shard histograms nightly. Observability for sharded Vitess is not a dashboard you look at after an incident — it is the instrumentation that lets automation reason about the fleet’s health continuously, and the discipline of preserving the shard-level dimensions that make partial failure visible before it becomes total.

Related #

• Monitoring VTGate Query Latency — measuring targeted versus scatter latency at the router and reading the histogram surfaces.
• Monitoring Replication Lag and Throttling — the storage-tier golden signals, replica-lag SLOs, and the throttler backpressure loop.
• Building a VTGate Latency Dashboard — the concrete Grafana panels and PromQL for the routing-latency domain.
• Key VTTablet Prometheus Metrics to Watch — the storage-tier counters and gauges that make replica and pool saturation legible.

← Back to shardedtopology.org home