Debugging VSchema Routing Rule Conflicts

When a query lands on the wrong keyspace or silently fans out to every shard, the cause is almost always two routing rules disagreeing about where a table reference should resolve.

Where This Fits #

Routing rules are the redirection layer that lets a sharded topology move traffic without redeploying the application, and they are authored and applied as described in Dynamic Routing Rules and Query Rewriting. This page owns the diagnosis half of that workflow: what happens when the rule set you applied is well-formed but wrong, so vtctldclient ApplyRoutingRules accepts it yet queries resolve to the wrong target under live traffic. The mechanics assume you are comfortable with the abstractions in VSchema Configuration & Routing Rule Management — keyspaces, primary and lookup vindexes, and tablet types — and with how the VTGate routing layer compiles a VSchema plus rules into an execution plan.

What a “Conflict” Actually Is #

A malformed rule set is rejected at apply time — duplicate from_table keys, a to_tables pointing at a syntactically invalid target, or JSON that fails to parse never reaches the topology server. Those are easy. The conflicts that reach production are semantic: two rules that are each individually valid but that, taken together, make a single table reference resolvable in more than one way. VTGate resolves references deterministically at table-resolution time (stage 4 of the pipeline, before planning), so the “conflict” is never a crash — it is a resolution you did not intend.

Four collision classes account for nearly every incident:

• Ambiguous unqualified reference. A bare orders is legal in more than one keyspace, and a from_table: "orders" rule redirects it globally. If a second keyspace also serves an orders table, the rule silently captures traffic that was meant to stay local.
• Tablet-type precedence gaps. A commerce.orders@rdonly rule exists but the unqualified commerce.orders rule is missing or points elsewhere, so rdonly reads and primary reads for the same logical table diverge onto different keyspaces.
• Table-rule vs. keyspace-rule overlap. A keyspace routing rule redirects all of commerce, while a table routing rule redirects commerce.orders somewhere else. The table rule wins for that one table, which is easy to forget when you reason about the keyspace rule in isolation.
• Redirect onto a vindex-less target. The rule resolves cleanly, but the destination keyspace’s VSchema has no vindex for the query’s WHERE column, so a formerly single-shard query degrades into a scatter across all shards.

The key mental model: qualified rules are matched before unqualified ones, and table rules are matched before keyspace rules. A conflict is what happens when the rule you think applies is shadowed by a higher-precedence rule you forgot was there.

Isolating the Collision #

Debugging is a three-move sequence: read the live rules, ask VTGate how it actually resolves the query, then diff the applied rules against your version-controlled baseline to find the offending entry.

1. Read the live rules #

Routing rules, shard routing rules, and keyspace routing rules are three separate global objects. A conflict can live in any of them, so pull all three.

vtctldclient --server localhost:15999 GetRoutingRules
vtctldclient --server localhost:15999 GetShardRoutingRules
vtctldclient --server localhost:15999 GetKeyspaceRoutingRules

2. Ask VTGate how it resolves the query #

VEXPLAIN PLAN compiles a statement through the full resolution pipeline and reports the keyspace, shard, and Route operator it lands on — after routing rules apply. This is ground truth: it tells you where the query actually goes, not where the rules imply it should.

VEXPLAIN PLAN SELECT * FROM orders WHERE id = 42;

A correctly targeted read shows the intended keyspace and a Route of type EqualUnique or Equal. Two failure signatures matter. If the keyspace is wrong, a higher-precedence rule is shadowing the one you expected — proceed to the diff. If the operator is Scatter, the reference resolved to a keyspace whose VSchema lacks a vindex for the predicate column. To check the tablet-type path, prefix the session and re-run:

USE commerce@rdonly;
VEXPLAIN PLAN SELECT * FROM orders WHERE id = 42;

If the @rdonly result diverges from the primary path in a way you did not intend, you have a tablet-type precedence gap.

3. Diff against the baseline #

Every rule set should be committed as a last-known-good artifact (this is why step 1 of the authoring workflow captures a baseline). Diffing the live object against it surfaces the exact entry that introduced the collision.

vtctldclient --server localhost:15999 GetRoutingRules > routing_rules.live.json
diff <(jq -S . routing_rules.baseline.json) <(jq -S . routing_rules.live.json)

For overlap detection that a plain diff cannot see — two entries whose from_table targets collide once tablet-type suffixes and keyspace rules are accounted for — a short Python pass catches them before they ship. Platform teams typically wire this into a pre-apply CI gate, in the same spirit as Async VSchema Validation Workflows validate schema promotion out of band.

import json
from collections import defaultdict

def find_conflicts(rules_json: str) -> list[str]:
    """Flag routing-rule entries that can shadow one another.

    A conflict is any pair of rules whose *base* table (the part before an
    optional @tablet_type suffix) matches while their target keyspaces differ,
    or any bare (unqualified) from_table that could resolve in >1 keyspace.
    """
    rules = json.loads(rules_json).get("rules", [])
    by_base = defaultdict(list)
    problems = []

    for rule in rules:
        src = rule["from_table"]
        base, _, ttype = src.partition("@")           # split off @rdonly / @replica
        targets = {t.split(".")[0] for t in rule["to_tables"]}  # target keyspaces
        by_base[base].append((ttype or "primary", targets))

        if "." not in base:                            # unqualified reference
            problems.append(f"unqualified from_table '{src}' can match multiple keyspaces")

    for base, entries in by_base.items():
        target_sets = {frozenset(t) for _, t in entries}
        if len(target_sets) > 1:                       # same base, divergent targets
            detail = ", ".join(f"{tt}->{sorted(ts)}" for tt, ts in entries)
            problems.append(f"divergent targets for '{base}': {detail}")

    return problems

if __name__ == "__main__":
    import subprocess
    live = subprocess.check_output(
        ["vtctldclient", "--server", "localhost:15999", "GetRoutingRules"],
        text=True,
    )
    for p in find_conflicts(live):
        print("CONFLICT:", p)

The script reports two things a diff misses: any unqualified from_table that could match in more than one keyspace, and any base table whose @rdonly, @replica, and primary rules point at different keyspaces — the precedence gap that produces split-brain result sets.

Edge Cases and Gotchas #

• Stale plan cache masks a fixed rule. After you correct and re-apply the rules, VTGate reloads on its watch tick, but plans compiled moments earlier keep serving the old target for a few seconds. A VEXPLAIN immediately after apply can still show the wrong keyspace — re-run it after the reload converges rather than assuming the fix failed.
• Shard routing rules override table routing rules per shard. A table rule can look correct while a leftover shard routing rule from an earlier Reshard silently redirects one shard’s copy of the table. Always pull GetShardRoutingRules when a conflict affects only some of a table’s rows.
• The bare and qualified forms of one table need to move together. Applying commerce.orders without also updating orders (the unqualified form an application may emit) leaves half the traffic on the old target. Author both in the same document.
• A redirect can pass VEXPLAIN yet still scatter under real predicates. VEXPLAIN PLAN SELECT ... WHERE id = 42 may resolve EqualUnique while a production query filtering on a non-vindex column scatters. Trace with the predicates your workload actually uses, not a synthetic primary-key lookup.
• Multi-element to_tables is only valid mid-migration. Outside an active MoveTables that is keeping two copies consistent, a two-target to_tables is a conflict, not a redirect — it will read from both and return duplicated or inconsistent rows.
• Keyspace routing rules are all-or-nothing. They redirect every table in the keyspace. A single per-table exception must be expressed as a higher-precedence table rule, and forgetting that exception is a common source of “one table went to the wrong place” incidents.
• Rule / DDL races leave VTGate planning on stale metadata. If a schema change lands on the target keyspace after the rule flips but before the VSchema reload finishes, resolution is briefly correct while planning is wrong. Sequence rule cutovers behind Online DDL orchestration so DDL propagation completes first.

Verification #

Once the offending rule is corrected and re-applied, confirm the fix two ways. First, re-run VEXPLAIN PLAN for the affected query (and its @rdonly variant) and confirm both resolve to the intended keyspace with an EqualUnique or Equal operator. Second, watch the scatter counter in Prometheus:

rate(vtgate_queries_processed{plan="Scatter"}[5m])

A conflict that was degrading a targeted query into a fan-out shows here as an elevated scatter rate; a correct fix returns it to its baseline. Pair that with vtgate_queries_routed broken out by keyspace — the keyspace that was wrongly capturing traffic should trend to zero as the intended keyspace absorbs it. Sustained routing correctness under load is covered in Optimizing VIndex Performance for High QPS.

FAQ #

Why did ApplyRoutingRules succeed if my rules conflict? #

Apply-time validation only rejects malformed rules — unparseable JSON, duplicate from_table keys, or targets it cannot recognize. Semantic conflicts (ambiguous unqualified references, tablet-type precedence gaps, table-vs-keyspace overlap) are legal individually, so they pass validation and only surface at resolution time under traffic.

A query resolves to the right keyspace but still scatters. Is that a routing-rule conflict? #

Not directly — the rule did its job. The target keyspace’s VSchema lacks a vindex for your predicate column, so the planner cannot narrow to a shard. Add a matching primary or lookup vindex on the target before switching traffic, and guard the gateway with --no_scatter so future misroutes fail loudly instead of fanning out.

How do I tell whether a table or a shard rule is misrouting? #

If the misrouting affects the whole table, inspect GetRoutingRules and GetKeyspaceRoutingRules. If it affects only some rows or only one key range, a GetShardRoutingRules entry — often a leftover from a prior Reshard — is the more likely culprit.

Related #

• Dynamic Routing Rules and Query Rewriting — how the resolution pipeline works and how to author and apply rules safely.
• Mastering VSchema Syntax and Structure — the keyspace, table, and vindex definitions that routing rules redirect between.
• Async VSchema Validation Workflows — validate rule and VSchema changes out of band before they reach production.

← Back to Dynamic Routing Rules and Query Rewriting