Building a VTGate Latency Dashboard

A latency dashboard for a sharded keyspace has one job the generic MySQL dashboard cannot do: show the targeted and scatter-gather query populations side by side so a rising tail is attributable, not just visible.

Context

This page turns the histogram surface described in monitoring VTGate query latency into concrete Grafana panels, and it is the dashboard layer of the broader observability and operations for sharded Vitess practice. The metrics come from VTGate’s /metrics endpoint; the assumption throughout is that Prometheus already scrapes them with the keyspace and plan labels preserved. The goal is a single board an on-call engineer can open during a latency page and, within one screen, tell whether the problem is fan-out, a slow shard, planning, or errors.

Panel layout: four rows, one question each

A useful board is organized top-to-bottom from customer impact to root cause, so the eye travels from “is it bad?” to “why?”. The layout below is what the PromQL in the next section fills.

Panel layout for a VTGate latency dashboard Four stacked rows. Top row: three stat tiles for p99 targeted latency, scatter ratio, and error ratio. Second row: a full-width latency time series showing p50, p95 and p99. Third row: two half-width panels, latency by plan type and query rate by plan type. Bottom row: a full-width table of slowest keyspace and table pairs. p99 targeted latency stat tile scatter ratio stat tile · threshold color error ratio stat tile API latency time series — p50 · p95 · p99 full width · templated by keyspace latency by plan type targeted vs scatter query rate by plan type QPS stacked table: slowest keyspace / table pairs sorted descending by p99

The PromQL, panel by panel

Every quantile panel follows the same rule: aggregate the bucket rate with sum by (le, ...) first, then apply histogram_quantile once. Wherever possible the panels read the recording rules from the parent guide (vtgate:api_latency_p99:5m, vtgate:scatter_ratio:5m) so the board is cheap to render; the raw expressions are shown here so the panels are self-documenting.

Stat tile — p99 latency (top-left). Point at the recording rule, templated by the dashboard’s $keyspace variable:

vtgate:api_latency_p99:5m{keyspace="$keyspace"}

Stat tile — scatter ratio (top-center), with a threshold color. Amber above 5%, red above 10%:

vtgate:scatter_ratio:5m{keyspace="$keyspace"}

Stat tile — error ratio (top-right):

sum(rate(vtgate_api_error_counts{keyspace="$keyspace"}[5m]))
/
sum(rate(vtgate_queries_processed{keyspace="$keyspace"}[5m]))

Time series — p50 / p95 / p99 (row two). Three queries on one panel; only the quantile argument changes:

histogram_quantile(0.50, sum by (le) (rate(vtgate_api_bucket{keyspace="$keyspace"}[5m])))
histogram_quantile(0.95, sum by (le) (rate(vtgate_api_bucket{keyspace="$keyspace"}[5m])))
histogram_quantile(0.99, sum by (le) (rate(vtgate_api_bucket{keyspace="$keyspace"}[5m])))

Latency by plan type (row three, left). The targeted-vs-scatter split. Because vtgate_api_bucket is not itself labeled by plan, approximate the split by faceting on the db_type/operation you have and overlaying the scatter QPS; where your build exposes a plan label on the latency histogram, group by it directly:

histogram_quantile(0.99,
  sum by (le, plan) (rate(vtgate_api_bucket{keyspace="$keyspace"}[5m])))

Query rate by plan type (row three, right). A stacked graph that makes fan-out visible as area:

sum by (plan) (rate(vtgate_queries_processed{keyspace="$keyspace"}[5m]))

Table — slowest keyspace/table pairs (row four). Use an instant query with topk so the panel lists offenders newest-worst-first:

topk(10,
  histogram_quantile(0.99,
    sum by (le, keyspace, table) (rate(vttablet_query_bucket[5m]))))

This last panel deliberately crosses into VTTablet’s per-table histogram, because attributing a slow scatter to a specific hot table is where the router’s view runs out and the storage tier’s begins.

A Grafana panel as JSON

Grafana panels are portable JSON, so a board can be code-reviewed and version-controlled rather than hand-built. The scatter-ratio stat tile with graded thresholds:

{
  "type": "stat",
  "title": "Scatter ratio",
  "datasource": { "type": "prometheus", "uid": "${DS_PROM}" },
  "targets": [
    {
      "expr": "vtgate:scatter_ratio:5m{keyspace=\"$keyspace\"}",
      "legendFormat": "{{keyspace}}",
      "refId": "A"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "percentunit",
      "decimals": 1,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": null },
          { "color": "orange", "value": 0.05 },
          { "color": "red", "value": 0.10 }
        ]
      }
    }
  },
  "options": {
    "reduceOptions": { "calcs": ["lastNotNull"] },
    "colorMode": "background",
    "graphMode": "area"
  },
  "gridPos": { "h": 4, "w": 8, "x": 8, "y": 0 }
}

The percentunit unit renders 0.05 as 5%, and colorMode: background makes a breach impossible to miss at a glance. Define a dashboard templating variable keyspace with the query label_values(vtgate_queries_processed, keyspace) so the whole board pivots between keyspaces from one dropdown.

Edge cases and gotchas

  • Quantile over-aggregation. Reversing the order — avg(histogram_quantile(...)) across shards — produces a number with no statistical meaning. Always sum by (le) the bucket rate() first, quantile once. This is the single most common way a Vitess latency board lies.
  • Rate window vs scrape interval. A rate(...[1m]) over a 15s scrape yields jagged lines and NaN gaps when one scrape is missed. Keep the window at four or more scrape intervals ([5m] for 15s) on every quantile panel; mismatched windows across panels make two graphs disagree for no real reason.
  • Recording rules or the board is slow. histogram_quantile over a 128-shard fleet’s raw buckets, recomputed on every refresh and every viewer, is expensive. Point panels at recording rules; keep raw expressions only in ad-hoc explore sessions.
  • $keyspace = All explodes the quantile. A templating variable set to “All” sums buckets across keyspaces, blending distributions with different shapes. Prefer a single-keyspace default and a separate fleet-rollup row that reads a per-keyspace recording rule, not a raw all-keyspace quantile.
  • Heatmap beats a line for distribution shifts. A p99 line hides bimodality (two clusters of latency). For the main latency panel, a Grafana heatmap sourced directly from vtgate_api_bucket reveals a second mode — the scatter population separating from the targeted one — that a single percentile line flattens.
  • Stale le buckets after a config change. Changing the histogram’s bucket boundaries mid-retention leaves old le series that histogram_quantile will happily but wrongly interpolate across. After a bucket change, scope panels to a time range after the change.

Verification

Confirm the board reflects reality by injecting a known scatter and watching every row respond together:

# Fire a scatter, then a burst of targeted reads.
for i in $(seq 1 50); do
  mysql -h vtgate.internal -P 15306 commerce \
    -e "SELECT COUNT(*) FROM orders WHERE status='shipped';" >/dev/null
done

Within one scrape-plus-rate window the scatter-ratio tile should turn amber, the “query rate by plan type” panel should show a SelectScatter spike, and the p99 line should lift while p50 barely moves — the fingerprint of a scatter regression. If the tiles stay green after a confirmed scatter burst, the recording rules are not evaluating or the scrape dropped the plan label; re-check the rule group interval and the scrape relabeling before trusting the board in an incident.

← Back to Monitoring VTGate Query Latency