Circuit Breaking for Degraded Shards

A scatter query is only as fast as its slowest shard, so a single degraded primary — swapping, IO-starved, or mid-reparent — can hold every fan-out query open until it times out, turning one shard’s problem into a keyspace-wide latency incident unless something sheds the load fast.

Context

This page sits under Implementing Fallback Routing for Shard Outages, which covers graceful degradation of the routing layer; here the focus is the specific pattern of circuit breaking — cutting requests to a shard that is failing slowly rather than waiting on it. The behaviour lives at the boundary owned by the VTGate routing layer, where a scatter-gather query fans out to every shard and blocks on the last response. A slow shard is more dangerous than a dead one: a dead shard fails fast and frees the connection, while a slow shard holds pool slots and threads until a timeout fires, so the goal is to make “slow” fail as quickly as “dead.”

Why a slow shard stalls the whole scatter

When VTGate executes a scatter query, it dispatches to every shard in parallel and gathers results before returning to the client. If nine shards answer in 3ms and the tenth takes 40 seconds because its primary is reparenting, the client waits ~40 seconds — the fan-out is bounded by the tail, not the median. Worse, every in-flight scatter query holds a connection to the slow shard for that whole window, so a steady request rate quickly exhausts the connection pool, and queries that would have been targeted (and healthy) start queuing behind the backlog. The failure amplifies: one shard’s latency becomes the gateway’s latency becomes the application’s.

Circuit breaking interrupts that chain in two layers. First, bound how long any single shard is allowed to hold a request, so a slow shard is abandoned rather than waited on. Second, when a shard is known to be transiently unavailable — a planned or detected reparent — buffer briefly instead of erroring, so a short blip does not surface as failed queries. Above both, an application-level breaker stops sending work that will predictably fail.

Healthy scatter versus an open circuit that sheds a degraded shard Left panel: a healthy fan-out to four shards returns at median latency. Right panel: a timeout trips on the degraded shard, cutting it from the gather so the query returns from the healthy shards. Healthy scatter returns at median latency VTGate -40 40-80 80-c0 c0- all shards answer → fast gather Open circuit timeout trips, degraded shard shed VTGate -40 40-80 80-c0 degraded c0- tripped shard shed → return from healthy

Bounding per-shard time at VTGate and VTTablet

The first line of defence is timeouts that fail a slow shard fast. Two flags matter, and their ordering is what makes them work together. VTGate’s --mysql_server_query_timeout bounds the client-visible time for a query; the tablet’s --queryserver-config-query-timeout bounds how long mysqld is allowed to run the query on the shard. Keep the tablet timeout below the gateway timeout, so a runaway query is killed at the shard and its connection freed before VTGate gives up — if the tablet timeout is higher, VTGate abandons the request while mysqld keeps executing, burning the connection you were trying to reclaim.

# VTGate: bound the client-visible query time
vtgate --mysql_server_query_timeout 30s

# VTTablet: shed work at the shard just under the gateway bound
vttablet --queryserver-config-query-timeout 25s

Set these so the healthy tail has comfortable headroom but a genuinely stuck shard is abandoned in tens of seconds, not minutes. A 30-second gateway timeout with a 25-second tablet timeout means a degraded shard costs a scatter at most ~30 seconds instead of blocking indefinitely, and the tablet reclaims its own connection at 25.

Buffering during reparent

A reparent — planned promotion or VTOrc-driven failover — leaves a shard with no writable primary for a short window. Erroring every write during that window turns a routine failover into visible application errors. VTGate request buffering absorbs it: instead of failing, it holds writes to the affected shard for a bounded window and replays them once the new primary is serving.

vtgate \
  --enable_buffer \
  --buffer_size 20000 \
  --buffer_max_failover_duration 20s \
  --buffer_min_time_between_failovers 1m

--buffer_max_failover_duration caps how long buffering persists before it gives up and errors — set it a little above your measured reparent time so a normal failover is invisible but a genuinely stuck shard is not buffered forever. Buffering is a short-blip smoother, not a substitute for the timeouts above; a shard that is degraded rather than reparenting will exhaust the buffer and should trip the circuit.

Application and orchestration circuit breakers

Timeouts and buffering are per-request; a circuit breaker is stateful. When the error or timeout rate to a shard crosses a threshold, the breaker “opens” and the client stops sending requests that will predictably fail, returning a fast fallback (cached data, a degraded response, or a targeted retry to another traffic class) instead. After a cooldown it “half-opens,” lets a probe through, and closes again if the shard has recovered. This keeps a degraded shard from consuming request slots that healthy work needs.

import time

class ShardBreaker:
    def __init__(self, fail_threshold=5, reset_after=30):
        self.fail_threshold = fail_threshold
        self.reset_after = reset_after
        self.failures = 0
        self.opened_at = None

    def allow(self) -> bool:
        if self.opened_at is None:
            return True                       # closed: normal traffic
        if time.monotonic() - self.opened_at >= self.reset_after:
            return True                       # half-open: let one probe through
        return False                          # open: shed load fast

    def record(self, ok: bool):
        if ok:
            self.failures = 0
            self.opened_at = None             # recovered: close the circuit
        else:
            self.failures += 1
            if self.failures >= self.fail_threshold:
                self.opened_at = time.monotonic()

The application wraps shard-scoped calls in allow()/record(), so a shard that starts timing out is skipped after a few failures rather than retried into the ground. Keep the breaker per shard, not global — a global breaker that opens on one degraded shard also sheds the healthy ones, converting a partial outage into a total one. The state should be keyed by shard identity so each shard’s circuit trips and recovers independently.

Where the breaker lives is a design choice. A breaker inside the application process reacts fastest and can return a domain-specific fallback, but every process maintains its own state and they trip at slightly different times. A breaker in a shared orchestration or service-mesh layer gives one consistent view of shard health at the cost of a hop. Many teams run both: fast local breakers for immediate shedding, and a slower orchestration-level signal — driven off the same VTGate and VTTablet health metrics that feed alerting — that can proactively drain a shard known to be reparenting before per-request failures even accumulate. For scatter-heavy workloads, though, the highest-leverage move is upstream: reducing how often a query fans out at all keeps a single degraded shard off the critical path in the first place, so a breaker rarely has to fire.

Edge cases and gotchas

  • Timeout inversion. If the tablet query timeout exceeds the gateway timeout, VTGate abandons the request while mysqld keeps running it, so the connection you meant to reclaim stays busy. Always keep the tablet timeout under the gateway timeout.
  • Buffer sized for a stuck shard. Buffering is for short reparents. If --buffer_max_failover_duration is set high enough to cover a degraded shard, buffered writes pile up and then all fail at once. Size it to real failover time and let the circuit breaker handle sustained degradation.
  • Breaker that never half-opens. A cooldown with no probe path leaves a recovered shard shut out. Ensure the half-open state actually lets a request through so the circuit can close again.
  • Scatter with LIMIT still needs all shards. Circuit-breaking a shard out of a scatter changes results (a COUNT(*) or ordered LIMIT becomes wrong). Shed a shard only where a partial answer is acceptable, or fail the query explicitly rather than returning a silently incomplete result.
  • Retries amplifying the storm. A client that retries every timeout immediately multiplies load on the very shard that is drowning. Pair breakers with backoff so retries do not become a self-inflicted denial of service.

Verification

Confirm the circuit actually sheds a degraded shard rather than waiting on it. Inject or observe a slow shard and check that scatter latency is bounded by the timeout, not the shard:

# Watch VTGate's per-query timing and error/buffer counters while a shard is slow
vtctldclient GetTablets --keyspace commerce --tablet-type primary
curl -s http://vtgate.internal:15000/debug/vars \
  | jq '{buffered: .BufferRequestsBuffered, timeouts: .QueryTimings}'

Under a degraded shard, buffered-request counts should rise during a reparent and settle, query timeouts should fire near the configured bound rather than open-ended, and end-to-end scatter latency should plateau at the gateway timeout instead of climbing with the slow shard. If scatter latency tracks the degraded shard’s response time, the circuit is not opening — recheck the timeout ordering and breaker thresholds.

← Back to Implementing Fallback Routing for Shard Outages