Async VSchema Validation Workflows

Applying a routing change directly to a live keyspace is one of the few operations in a sharded Vitess topology with no natural rollback: once VTGate reloads the active VSchema, every subsequent query is planned against the new definition. A dropped vindex binding, a renamed sequence table, or a primary vindex that no longer matches the shard key does not raise a clean error — it silently degrades correct point queries into scatter-gather reads or, worse, routes writes to the wrong shard. Async VSchema validation workflows resolve this by moving verification off the critical path: routing definitions are treated as version-controlled artifacts and checked deterministically in an isolated simulation before any ApplyVSchema command reaches the production control plane. This page shows how to build that out-of-band validation layer, gate it in CI/CD, and catch the specific failure classes that break query routing under load. It is a companion to Automating VSchema Sync with Python Scripts, which owns the apply-and-reconcile half of the pipeline.

Prerequisites #

Before building the validation layer, confirm the following are in place:

• Vitess 16.0 or later. ApplyVSchema --dry-run and the strict JSON validation path used below are stable from v16; earlier releases accept some malformed payloads that later panic VTGate.
• A committed baseline VSchema per keyspace. Validation is a diff operation — you compare a candidate against the last-known-good artifact. Store each keyspace’s VSchema as JSON under version control, not only in the topology server.
• Read access to the topology server. The simulator seeds its mock shard layout from the live serving graph (GetKeyspace, FindAllShardsInKeyspace) so the offline model matches production ranges. Read-only credentials are sufficient.
• Working knowledge of the routing model. You should be comfortable with the abstractions in VSchema Configuration & Routing Rule Management — keyspaces, primary vs. lookup vindexes, sequence tables, and routing rules — and with how the VTGate routing layer turns a VSchema into an execution plan.
• Python 3.11+ for the orchestration harness. The concurrency patterns below assume asyncio.TaskGroup, which lands in 3.11.
• jsonschema and the Vitess VSchema JSON schema available to the runner, so structural checks happen before any network call.

How Out-of-Band Validation Works #

The workflow is a four-stage funnel, and each stage rejects a distinct class of defect so the expensive stages never run on input the cheap stages could have caught.

Stage 1 — static grammar and schema validation. The candidate VSchema is parsed as JSON and validated against the canonical Vitess VSchema schema with jsonschema. This catches malformed payloads, unknown vindex types, and missing required fields with zero infrastructure — the class of error that otherwise reaches VTGate as a reload panic.

Stage 2 — mock topology construction. The validator reads the live serving graph from the topology server and builds an in-memory model of every keyspace, its shard ranges, and its current table-to-vindex bindings. Because the model mirrors production ranges, structural checks — orphaned tables, conflicting primary vindexes, a column_vindexes entry pointing at an undeclared vindex — run against the real shard layout rather than a guess.

Stage 3 — simulated query planning. For each representative query in a fixture set, the validator computes the shard set the candidate VSchema would target and compares it to the baseline. This is where routing regressions surface: a query that resolved to a single shard under the baseline but scatters under the candidate is flagged as a latency regression even though the JSON is perfectly valid. Cross-shard join paths are checked here too, confirming that every lookup vindex referenced by a join still resolves without a broadcast.

Stage 4 — control-plane dry run. Only candidates that clear the offline stages are submitted to ApplyVSchema --dry-run, which asks the live vtctld to plan the change and report what it would do without persisting it. This is the final authoritative check: it exercises the same code path production will, but commits nothing.

The property that makes the workflow “async” is that stages 1–3 never touch live traffic and stage 4 never mutates state, so the entire funnel runs on every pull request, in parallel across keyspaces, decoupled from any deployment window.

Step-by-Step Implementation #

The steps below assume the candidate VSchema lives at vschema/commerce.json in the repository and that the last-known-good baseline is retrievable from the topology server.

1. Validate structure before anything else #

Run the structural gate first — it needs no cluster access and fails in milliseconds.

import json
from jsonschema import Draft7Validator

def validate_structure(candidate_path: str, schema_path: str) -> list[str]:
    with open(candidate_path) as f:
        candidate = json.load(f)          # raises on malformed JSON
    with open(schema_path) as f:
        vschema_schema = json.load(f)

    validator = Draft7Validator(vschema_schema)
    errors = [
        f"{'/'.join(map(str, e.path))}: {e.message}"
        for e in validator.iter_errors(candidate)
    ]
    return errors

A non-empty return list means the payload is rejected before it can reach a mock router or the control plane.

2. Seed a mock topology from the live serving graph #

Pull the real shard ranges so the simulation reflects production, not an idealized layout.

# Enumerate shards for the keyspace under validation
vtctldclient GetKeyspace commerce
vtctldclient FindAllShardsInKeyspace commerce --format=json > topo/commerce_shards.json

The shard boundaries in commerce_shards.json (for example -40, 40-80, 80-c0, c0-) define the key ranges the simulated vindex output must fall within. A candidate whose primary vindex would produce keyspace IDs outside any declared range is structurally impossible to route and is rejected here.

3. Simulate query plans concurrently across keyspaces #

Use asyncio to plan the fixture query set against every keyspace in parallel. Each task is CPU-light (in-memory planning) but there may be dozens of keyspaces, so structured concurrency keeps wall-clock time flat as the fleet grows.

import asyncio

async def plan_keyspace(keyspace: str, candidate: dict, fixtures: list[str]) -> dict:
    shards = load_shard_ranges(keyspace)          # from step 2
    regressions = []
    for query in fixtures:
        baseline_targets = plan_against(BASELINE[keyspace], query, shards)
        candidate_targets = plan_against(candidate, query, shards)
        if candidate_targets != baseline_targets:
            regressions.append({
                "query": query,
                "baseline": sorted(baseline_targets),
                "candidate": sorted(candidate_targets),
            })
    return {"keyspace": keyspace, "regressions": regressions}

async def simulate_all(candidates: dict[str, dict], fixtures: dict[str, list[str]]):
    async with asyncio.TaskGroup() as tg:
        tasks = [
            tg.create_task(plan_keyspace(ks, cand, fixtures[ks]))
            for ks, cand in candidates.items()
        ]
    return [t.result() for t in tasks]

A regression where candidate targets every shard but baseline targeted one is the canonical scatter-gather introduction — the most common way a valid VSchema still ships a latency incident.

4. Dry-run against the control plane #

Candidates that survive simulation are submitted to the live vtctld in dry-run mode. Nothing is persisted; the command returns the plan Vitess would execute.

vtctldclient ApplyVSchema \
  --vschema="$(cat vschema/commerce.json)" \
  --dry-run \
  commerce

A non-zero exit or a diff that touches tables outside the intended change set fails the gate. This step catches semantic problems the offline model cannot — for instance, a sequence table reference that is valid JSON and structurally sound but points at an unsharded keyspace that no longer exists.

5. Gate promotion in CI/CD #

Wire the four stages into the pipeline so promotion is automatic only on a clean run. The dry run is the boundary between validation and the real ApplyVSchema performed by the sync layer described in Automating VSchema Sync with Python Scripts.

validate-vschema:
  stage: verify
  script:
    - python -m vschema_validate structure vschema/commerce.json
    - python -m vschema_validate topology --keyspace commerce
    - python -m vschema_validate simulate --keyspace commerce --fixtures fixtures/commerce.sql
    - vtctldclient ApplyVSchema --vschema="$(cat vschema/commerce.json)" --dry-run commerce
  rules:
    - changes: [vschema/**/*.json]

Each command is independently runnable, so an engineer can reproduce any failed stage locally with the exact invocation the pipeline used.

Configuration Reference #

The flags below govern how quickly a promoted VSchema takes effect and how much routing state each component holds. Set them consistently across the fleet so the dry-run environment behaves like production.

Leave --vschema_ddl_authorized_users unset in production if you apply VSchema exclusively through vtctldclient/vtadmin; enabling in-band ALTER VSCHEMA widens the surface that bypasses the validation gate entirely.

Failure Modes #

Each scenario below is one the funnel is designed to catch, with the signal that identifies it.

Silent scatter-gather promotion. A candidate drops or renames the primary vindex on a hot table. The JSON is valid and the dry run succeeds, so only stage 3 catches it: the fixture query that previously hit one shard now returns every shard in candidate_targets. Symptom in production if it escapes: vtgate_queries_processed for the ScatterGather plan type climbs while p99 latency on the affected keyspace rises. Mitigation: block promotion on any query whose target-shard count increases versus baseline.

Orphaned lookup vindex. A table’s column_vindexes references a lookup vindex whose backing table was removed in the same change. Stage 2 flags the dangling reference against the mock topology before any join is simulated. Symptom if it escapes: VTGate logs vindex ... not found and cross-shard joins fail to plan. Mitigation: validate that every referenced vindex name resolves to a declared entry, and that lookup backing tables still exist in the serving graph.

Sequence table drift. An auto-increment table points at a sequence in an unsharded keyspace that a prior migration renamed. Valid JSON, valid structure — only ApplyVSchema --dry-run (stage 4) reaches the sequence resolution path and reports the broken reference. Symptom if it escapes: inserts fail with cannot find sequence. Mitigation: keep the dry run mandatory; never promote on offline stages alone.

Stale baseline comparison. The workflow diffs against a baseline that no longer matches the live topology because an out-of-band change was applied manually. Stage 3 then reports phantom regressions or, worse, masks a real one. Symptom: simulation results that disagree with vtctldclient GetVSchema. Mitigation: re-seed the baseline from the topology server at the start of every run (step 2) rather than trusting a cached copy — and forbid manual ApplyVSchema outside the pipeline. This class of drift is the same coordination hazard that Online DDL orchestration manages for schema changes, and the two pipelines must agree on a single source of truth.

Concurrency masking a failure. A bare asyncio.gather(..., return_exceptions=True) swallows a planning exception in one keyspace and the gate passes green. Mitigation: use asyncio.TaskGroup (as in step 3), which propagates the first failure and cancels siblings, so a broken keyspace fails the whole run.

Verification #

Confirm the workflow is actually gating changes, not just running.

1. Prove the offline gate rejects bad input. Commit a VSchema with a deliberately dangling vindex reference and confirm stage 1 or 2 fails without contacting the live cluster:

   python -m vschema_validate structure vschema/commerce_broken.json
   # expect: non-zero exit, "column_vindexes/0/name: 'missing_vindex' not declared"
   

2. Confirm the dry run is authoritative. Run the promoted VSchema through the dry run and diff it against what is live:

   vtctldclient ApplyVSchema --vschema="$(cat vschema/commerce.json)" --dry-run commerce
   vtctldclient GetVSchema commerce > /tmp/live.json
   diff <(jq -S . vschema/commerce.json) <(jq -S . /tmp/live.json)
   

   After a successful promotion the diff is empty; a non-empty diff means the live control plane and the artifact have drifted and the baseline must be re-seeded.

3. Watch the routing metric that a regression would move. After promotion, confirm the scatter-gather plan rate on the keyspace is flat:

   # Prometheus: rate of scatter plans should not step up post-deploy
   rate(vtgate_queries_processed{plan="ScatterGather",keyspace="commerce"}[5m])
   

   A step change here after a green pipeline run is the signal that the fixture set missed a query shape — extend the fixtures and re-validate.

A validation workflow that passes all three checks provides the deterministic promotion path this layer exists to deliver: structural correctness proven offline, semantic correctness proven by dry run, and routing behaviour proven by a metric that can only move if a regression slipped the fixtures.

Related #

• Automating VSchema Sync with Python Scripts — the apply-and-reconcile pipeline that consumes the artifacts this workflow validates.
• Mastering VSchema Syntax and Structure — the grammar and structural rules the static stage checks against.
• Configuring Lookup Vindexes for Cross-Shard Joins — why simulated join paths must resolve every lookup vindex without a broadcast.
• Dynamic Routing Rules and Query Rewriting — runtime routing overrides that the same validation funnel should cover.
• Coordinating Multi-Shard Schema Migrations — aligning VSchema promotion with in-flight DDL so the two never contradict the topology.

← Back to VSchema Configuration & Routing Rule Management