Monitoring VTGate Query Latency Across a Sharded Keyspace

VTGate is the only tier that observes a query’s entire lifecycle — parse, plan, shard fan-out, and gather — so its latency histograms are where the customer-facing “is the database fast?” question is actually answered. The problem specific to a sharded keyspace is that a single latency number is a blend of two populations with wildly different cost curves: targeted single-shard queries whose latency tracks one shard, and scatter-gather queries whose latency is bounded by the slowest of every shard. Averaging them together hides exactly the regression you need to catch. This guide covers how to read the vtgate_api histogram surface, how to separate targeted from scattered latency, how to reason about the p99 tail, and how to attribute where in the routing path a slow query spent its time. It is the routing-latency domain of observability and operations for sharded Vitess, examined at the depth needed to build alerts and dashboards on top of it.

Prerequisites #

Before the metrics below are meaningful, confirm the following about your environment:

• Vitess 14+ with Prometheus stats enabled. VTGate must be started with --emit_stats --stats_backend prometheus (or the equivalent --stats_backend for your build) so the histogram families are exposed on the /metrics handler of the status port (default :15000).
• A Prometheus server scraping VTGate with dimensions preserved. The keyspace and plan labels must survive scrape-time relabeling — the whole point of this exercise collapses if they are dropped. Target discovery and relabeling are set up in the parent observability pipeline.
• --normalize_queries=true (the default). Normalization turns literals into bind variables so that structurally identical queries share one plan and one histogram, instead of exploding into per-literal series.
• Working knowledge of the routing model. You should understand how the stateless VTGate routing layer decides between a targeted single-shard plan and a scatter-gather plan based on whether the sharding key appears in the WHERE clause — that distinction is the axis every metric here is sliced along.
• A representative query mix. Latency percentiles from a synthetic single-query benchmark tell you nothing about a production blend of targeted point reads and analytical scatters.

The latency surface: what VTGate exposes #

VTGate records query timings as Prometheus histograms — cumulative bucket counters, not pre-computed percentiles. The families that matter for latency are:

• vtgate_api_bucket — the core latency histogram, dimensioned by operation (e.g. Execute, StreamExecute) and db_type (the tablet type served). Each _bucket series carries an le (less-than-or-equal) label; you reconstruct a percentile at query time with histogram_quantile over these buckets. The companion vtgate_api_count and vtgate_api_sum give you request count and total time for computing a mean.
• vtgate_queries_processed — a counter of queries by plan type (Select, SelectScatter, SelectEqualUnique, Insert, and so on). This is how you separate the targeted population from the scattered one.
• vtgate_queries_routed — a counter of the number of shard destinations touched, dimensioned by plan. It rises faster than queries_processed whenever a query fans out, so the ratio between them is a direct measure of average fan-out.
• vtgate_api_error_counts — errors by code, needed to keep a latency SLO honest (a fast error is not a fast success).

The load-bearing insight is that plan type is the dimension that separates the two latency populations. SelectEqualUnique is a targeted single-shard read; SelectScatter fans out to all shards. Reading vtgate_api_bucket without slicing by plan-type context averages them; the entire monitoring strategy is about keeping them apart.

Where latency originates in the routing path #

A VTGate latency number is a sum of distinct phases, and knowing which phase dominates points at a different fix. The diagram below traces one query through the router and back.

Three contributors, three different remedies:

• Planning is cheap on a cache hit and expensive on a miss (full parse and plan build). A latency floor that rises with query diversity rather than load points at plan-cache pressure — check the plan-cache hit rate and whether --normalize_queries is on.
• Tablet round-trip is the network hop plus the shard’s own execution. For a targeted query this is the whole story; for a scatter it is the maximum across shards, so one slow shard drags the whole query’s tail.
• Gather / sort / LIMIT is VTGate-side merge cost, which grows with result-set size and shard count. A scatter that returns many rows before applying LIMIT spends real CPU here.

Reading the histogram: targeted versus scatter #

The single most useful transformation is to compute the p99 separately for the targeted and scattered plan populations. Because vtgate_api_bucket is not itself split by plan, the practical approach is to alert and dashboard on the plan-type QPS from vtgate_queries_processed alongside the overall latency quantile, and to use the routed/processed ratio to confirm fan-out. The core percentile expression:

# Overall p99 API latency per keyspace over a 5-minute window.
histogram_quantile(
  0.99,
  sum by (keyspace, le) (rate(vtgate_api_bucket[5m]))
)

Aggregating the _bucket series with sum by (le) before histogram_quantile is mandatory — computing the quantile per-series and averaging is statistically meaningless. Separate the populations by plan type from the processed counter:

# Scatter-gather query rate vs total read rate per keyspace.
sum by (keyspace) (rate(vtgate_queries_processed{plan=~"Select.*Scatter"}[5m]))
/
sum by (keyspace) (rate(vtgate_queries_processed{plan=~"Select.*"}[5m]))

A latency graph that is flat while this scatter ratio climbs is the early-warning signature of a routing regression — the mean has not moved yet, but the fraction of expensive queries has. Turning that ratio into a page is the subject of alerting on scatter-query ratio.

Step-by-step: instrumenting VTGate latency #

1. Confirm the histogram is exposed #

Curl the status port and check the histogram family is present with its le buckets:

curl -s http://vtgate-0.internal:15000/metrics | grep '^vtgate_api_bucket' | head

If nothing returns, VTGate was not started with the Prometheus stats backend — restart it with --emit_stats --stats_backend prometheus before going further.

2. Scrape with keyspace and plan dimensions preserved #

The scrape job must keep the labels that make the split possible. In prometheus.yml:

scrape_configs:
  - job_name: vtgate
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets: ['vtgate-0.internal:15000', 'vtgate-1.internal:15000']
    metric_relabel_configs:
      # Keep the histogram and plan counters; drop unrelated high-cardinality families.
      - source_labels: [__name__]
        regex: 'vtgate_(api_bucket|api_count|api_sum|queries_processed|queries_routed|api_error_counts)'
        action: keep

3. Add recording rules for the expensive quantiles #

histogram_quantile over a wide fleet is costly to evaluate on every dashboard refresh. Pre-compute it:

groups:
  - name: vtgate_latency
    interval: 30s
    rules:
      - record: vtgate:api_latency_p99:5m
        expr: |
          histogram_quantile(0.99,
            sum by (keyspace, le) (rate(vtgate_api_bucket[5m])))
      - record: vtgate:scatter_ratio:5m
        expr: |
          sum by (keyspace) (rate(vtgate_queries_processed{plan=~"Select.*Scatter"}[5m]))
          /
          sum by (keyspace) (rate(vtgate_queries_processed{plan=~"Select.*"}[5m]))

4. Verify from the client side once #

Cross-check the histogram against an out-of-band measurement so you trust the numbers. A tiny Python probe that times a known targeted query through VTGate gives you ground truth to compare the p50 against:

import time
import pymysql

conn = pymysql.connect(host="vtgate.internal", port=15306, db="commerce")
samples = []
for _ in range(500):
    t0 = time.perf_counter()
    with conn.cursor() as cur:
        cur.execute("SELECT id FROM orders WHERE customer_id = %s", (42,))
        cur.fetchall()
    samples.append((time.perf_counter() - t0) * 1000)  # ms
samples.sort()
print(f"client p50={samples[len(samples)//2]:.1f}ms p99={samples[int(len(samples)*0.99)]:.1f}ms")

If the client p50 for this targeted read diverges sharply from vtgate:api_latency_p99:5m filtered to the same keyspace, the gap is client-side (driver, network) rather than routing — a useful triangulation.

Configuration and metric reference #

Failure modes specific to latency monitoring #

Quantile over-aggregation. Symptom: a p99 dashboard shows an implausibly smooth line that never spikes even during a known incident. Root cause: histogram_quantile was applied per-series and then averaged, or the buckets were summed with the wrong grouping, smearing the distribution. Mitigation: always sum by (le, ...) the rate() of the buckets first, then take the quantile once.

Averaged populations hide scatter regressions. Symptom: latency SLO stays green while customers report slowness. Root cause: targeted and scatter latency are pooled into one number, so a growing scatter fraction is diluted. Mitigation: track the scatter ratio as a first-class series and alert on it independently of absolute latency.

Rate window too short for the scrape interval. Symptom: the p99 series is jagged with gaps or NaNs. Root cause: a rate([1m]) over a 15s scrape has too few samples per window; a single missed scrape produces a gap. Mitigation: use a rate window of at least four scrape intervals ([5m] for a 15s scrape) so each window always spans multiple samples.

Plan-cache thrash inflates the floor. Symptom: p50 latency rises with query diversity, not load; the plan-cache hit rate drops. Root cause: unnormalized literals or a too-small plan cache force re-planning. Mitigation: confirm --normalize_queries=true and size the plan cache to the working set of distinct query shapes.

Error-as-fast-success. Symptom: p99 latency improves during an incident. Root cause: queries are failing fast, and failed requests are short — a lower latency that is actually worse. Mitigation: pair every latency panel with an error-ratio panel from vtgate_api_error_counts so a latency drop caused by errors is unmistakable.

Verification #

Confirm the pipeline reports what the router actually did. Trigger a known scatter and a known targeted query, then check that the plan counters and the latency quantile both moved in the expected direction:

# Force a scatter (no sharding-key predicate) and a targeted read, then read back the counters.
mysql -h vtgate.internal -P 15306 commerce \
  -e "SELECT COUNT(*) FROM orders WHERE status='shipped'; SELECT id FROM orders WHERE customer_id=42;"

curl -s http://vtgate-0.internal:15000/metrics \
  | grep -E 'vtgate_queries_processed\{.*plan="Select(Scatter|EqualUnique)"'

The SelectScatter counter should have advanced by one and SelectEqualUnique by one. In Prometheus, vtgate:scatter_ratio:5m for the keyspace should reflect the injected scatter, and vtgate:api_latency_p99:5m should show the scatter’s higher tail. If the recording rules are flat after a confirmed scatter, the scrape relabeling dropped a needed label — revisit step 2.

Related #

• Building a VTGate Latency Dashboard — the concrete Grafana panels and PromQL that render this latency surface.
• Alerting on Scatter-Query Ratio — turning the scatter fraction into a Prometheus alert before latency breaches.
• Monitoring Replication Lag and Throttling — the storage-tier signal that explains why one shard’s tablet round-trip, and thus a scatter’s tail, went slow.
• VTGate Routing Architecture Deep Dive — the plan types and fan-out behavior that this latency monitoring measures.

← Back to Observability & Operations for Sharded Vitess · Related area: Vitess Sharding Architecture & Topology Design