Setting Retry Budgets for Flaky USB-Serial Links

Cheap FTDI, CP210x, and CH340 USB-serial bridges fail intermittently under thermal drift, marginal cabling, and USB power dips — the question is not how long to wait between retries but how many failures you are willing to absorb before declaring the link dead. This guide sizes a retry budget and a circuit breaker for those flaky bridges: a per-command attempt ceiling, a per-session error budget, a sliding-window retry rate, and a breaker that trips when the failure rate crosses a threshold. It sits under the parent Timeout Handling & Retry Logic collection inside the Serial, USB, and GPIB Communication Workflows stack, and it is the deliberate counterpart to implementing exponential backoff for serial timeout handling: backoff decides the shape of the wait, the budget decides when to stop trying and escalate.

Scope: Budgeting Retries, Not Shaping Them

The backoff curve answers a single-exchange question — given that this *IDN? timed out, how long until I re-issue it. A budget answers a fleet-and-session question that no per-exchange delay can: across thousands of commands over an eight-hour unattended run, at what point does the pattern of failures stop looking like a noisy-but-alive bridge and start looking like a cable that has fallen out. Those are orthogonal controls. You can run aggressive fast retries (a cheap backoff curve) under a tight budget, or patient long retries under a generous one; conflating them produces the worst failure of flaky-link handling — a supervisor that retries forever because each individual exchange eventually succeeded, while throughput has quietly collapsed to a tenth of nominal.

Two assumptions bound the scope. First, the wrapped command is idempotent — a query, a measurement trigger, a status read — so the budget layer is free to re-issue it; non-idempotent actuation is escalated on the first ambiguous timeout by the retry engine in the parent collection, never budgeted here. Second, transport parameters (baud, port.timeout, the FTDI latency timer) are already pinned per PySerial Configuration & Tuning; a mis-tuned latency timer inflates the failure rate and will trip the breaker for the wrong reason. The budget layer consumes the same SerialTimeoutError the backoff handler raises and decides, at a higher altitude, whether the link as a whole is still worth talking to.

The Error Budget and the Breaker Trip Condition

A retry budget is two numbers layered over the attempt ceiling. The innermost bound is the familiar per-command cap: no single command gets more than max_attempts tries. Above it sits a per-session error budget — a tolerance for how many commands may exhaust their attempts across a rolling window before the link is judged unhealthy. If a session issues N commands over a window and you accept a failure fraction ε, the allowed count of fully-failed commands is:

 $allowed_failures = ⌊ ε \cdot N ⌋, ε \in (0, 1)$

A typical bench value is ε = 0.02 — two percent of commands may burn their whole attempt budget and still be tolerated as ordinary bridge flakiness. The window N is what makes this a rate rather than a lifetime count: a link that fails ten commands in the first minute is dying; a link that fails ten commands spread across a full day is merely cheap. The circuit breaker enforces exactly that rate. Over a sliding window W of the most recent outcomes, let f be the number of failures and w the number of samples; the breaker trips — moves from closed to open — when the observed failure rate reaches the threshold θ:

 $trip ⟺ \frac{f}{w} \geq θ for w \geq w_{m i n}, θ \in (0, 1)$

The w_min guard is not optional: without a minimum sample count, the first two commands of a session failing gives f/w = 1.0 and opens the breaker before the link has had a chance to prove itself. Set θ above ε — the error budget tolerates steady-state flakiness, the breaker fires only when the rate spikes well past it (for example ε = 0.02 steady-state, θ = 0.5 trip). Once open, the breaker rejects calls immediately for a cooldown T_open, then admits a single probe in the half-open state; the probe’s success closes the breaker and the probe’s failure re-opens it for another cooldown. That half-open probe is the mechanism that distinguishes a flaky link (probe succeeds, resume) from a dead one (probe fails, stay open, escalate).

The breaker cycles closed → open → half-open: a failure rate past θ trips it open, a cooldown admits one half-open probe, and the probe's outcome either resumes normal traffic or re-opens the breaker and escalates.

A RetryBudget and CircuitBreaker Around a Command Executor

The implementation layers three bounds over any command executor. RetryBudget owns the per-command attempt ceiling and the per-session error budget; CircuitBreaker owns the sliding-window failure rate and the closed/open/half-open lifecycle. Both use time.monotonic() so a wall-clock adjustment mid-run cannot corrupt the cooldown, and the window is a deque of recent outcomes rather than a lifetime counter, so the failure rate — not an ever-growing total — drives the trip decision.

from __future__ import annotations

import logging
import time
from collections import deque
from dataclasses import dataclass, field
from enum import Enum
from typing import Callable, Deque, Tuple, TypeVar

logger = logging.getLogger(__name__)

T = TypeVar("T")


class LinkDeadError(RuntimeError):
    """Raised when the breaker is open (or a half-open probe failed)."""


class BudgetExhaustedError(RuntimeError):
    """Raised when the per-session error budget is spent."""


class BreakerState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


@dataclass
class RetryBudget:
    """Per-command attempt ceiling plus a per-session error budget.

    ``epsilon`` is the tolerated fraction of *fully failed* commands over a
    session of ``expected_commands`` (N); once that many commands exhaust their
    attempts the link is judged unhealthy and the session must stop.
    """

    max_attempts: int = 4
    epsilon: float = 0.02
    expected_commands: int = 5000
    _failed_commands: int = field(default=0, init=False)

    @property
    def allowed_failures(self) -> int:
        """floor(epsilon * N) — the per-session failure ceiling."""
        return int(self.epsilon * self.expected_commands)

    def record_command_failure(self) -> None:
        """Charge one fully-failed command against the session budget."""
        self._failed_commands += 1
        if self._failed_commands > self.allowed_failures:
            raise BudgetExhaustedError(
                f"session error budget spent: {self._failed_commands} failed "
                f"commands > allowed {self.allowed_failures} "
                f"(epsilon={self.epsilon}, N={self.expected_commands})"
            )


@dataclass
class CircuitBreaker:
    """Sliding-window breaker with closed/open/half-open hysteresis.

    Trips to OPEN when the failure rate f/w over the most recent ``window``
    outcomes reaches ``theta`` (with at least ``min_samples`` observations).
    After ``cooldown`` seconds it admits exactly one HALF_OPEN probe whose
    outcome either closes the breaker or re-opens it.
    """

    window: int = 40
    theta: float = 0.5
    min_samples: int = 12
    cooldown: float = 10.0
    state: BreakerState = BreakerState.CLOSED
    _events: Deque[bool] = field(default_factory=lambda: deque(maxlen=40), init=False)
    _opened_at: float = field(default=0.0, init=False)

    def __post_init__(self) -> None:
        # Keep the deque bound aligned with the configured window length.
        self._events = deque(maxlen=self.window)

    def _failure_rate(self) -> Tuple[float, int]:
        w = len(self._events)
        if w == 0:
            return 0.0, 0
        return sum(1 for ok in self._events if not ok) / w, w

    def allow(self) -> bool:
        """Whether a call may proceed; advances OPEN → HALF_OPEN on cooldown."""
        if self.state is BreakerState.OPEN:
            if time.monotonic() - self._opened_at >= self.cooldown:
                self.state = BreakerState.HALF_OPEN
                logger.info("breaker cooldown elapsed; admitting half-open probe")
                return True
            return False
        return True  # CLOSED or HALF_OPEN both admit a call

    def record(self, success: bool) -> None:
        """Feed one outcome into the window and update the state."""
        if self.state is BreakerState.HALF_OPEN:
            if success:
                self.state = BreakerState.CLOSED
                self._events.clear()  # fresh window after recovery
                logger.info("half-open probe ok; breaker closed")
            else:
                self.state = BreakerState.OPEN
                self._opened_at = time.monotonic()
                logger.warning("half-open probe failed; breaker re-opened")
            return

        self._events.append(success)
        rate, w = self._failure_rate()
        if (
            self.state is BreakerState.CLOSED
            and w >= self.min_samples
            and rate >= self.theta
        ):
            self.state = BreakerState.OPEN
            self._opened_at = time.monotonic()
            logger.error("breaker tripped: f/w=%.2f >= theta=%.2f (w=%d)", rate, self.theta, w)


def call_guarded(
    executor: Callable[[], T],
    budget: RetryBudget,
    breaker: CircuitBreaker,
    *,
    op_name: str = "cmd",
) -> T:
    """Run one idempotent command under the retry budget and breaker.

    The ``executor`` is expected to embed its own per-attempt backoff (see the
    exponential-backoff handler); ``call_guarded`` retries it up to
    ``max_attempts`` times, then charges a fully-failed command against the
    session budget and feeds the outcome to the breaker.

    Raises:
        LinkDeadError: The breaker is open, or a half-open probe failed.
        BudgetExhaustedError: The per-session error budget is spent.
    """
    if not breaker.allow():
        raise LinkDeadError(f"{op_name}: breaker OPEN, shedding call")

    for attempt in range(budget.max_attempts):
        try:
            result = executor()
        except (OSError, TimeoutError) as exc:
            logger.debug("%s attempt %d failed: %s", op_name, attempt + 1, exc)
            continue
        breaker.record(success=True)
        return result

    # Every attempt for this command failed: one charge against both bounds.
    breaker.record(success=False)
    try:
        budget.record_command_failure()
    except BudgetExhaustedError:
        raise
    if breaker.state is BreakerState.OPEN:
        raise LinkDeadError(f"{op_name}: link failing, breaker tripped open")
    raise TimeoutError(f"{op_name}: exhausted {budget.max_attempts} attempts")

The division of labour is deliberate: the executor embeds the backoff curve (the sibling handler), RetryBudget caps how much failure a session absorbs, and CircuitBreaker sheds load the instant the rate spikes so a dying bridge stops consuming the whole budget one slow retry at a time. When these guarded calls feed an Async Command Queuing System, the queue holds one breaker per device so a single flaky bridge opens its own circuit without stalling healthy instruments on the same host.

Verifying the Breaker Opens Then Half-Opens

The behaviour worth pinning is the full cycle: a burst of failures must open the breaker, a cooldown must admit exactly one probe, and a successful probe must close it. Inject the burst deterministically with a scriptable executor and drive the breaker with a patched clock so the test does not actually sleep the cooldown.

import itertools

import pytest

from link_budget import (BreakerState, BudgetExhaustedError, CircuitBreaker,
                         LinkDeadError, RetryBudget, call_guarded)


def _scripted(outcomes):
    """Executor that fails (raises) or succeeds per the given iterable."""
    seq = iter(outcomes)

    def executor():
        if next(seq):
            return b"OK"
        raise TimeoutError("simulated bridge stall")

    return executor


def test_breaker_opens_then_half_opens(monkeypatch) -> None:
    clock = itertools.count(0.0, 1.0)
    monkeypatch.setattr("link_budget.time.monotonic", lambda: next(clock))

    budget = RetryBudget(max_attempts=1, epsilon=0.5, expected_commands=100)
    breaker = CircuitBreaker(window=20, theta=0.5, min_samples=12, cooldown=10.0)

    # 12 straight failures: f/w = 1.0 >= theta once min_samples is reached.
    executor = _scripted([False] * 12)
    for _ in range(12):
        with pytest.raises((TimeoutError, LinkDeadError)):
            call_guarded(executor, budget, breaker, op_name="poll")
    assert breaker.state is BreakerState.OPEN

    # Within the cooldown the breaker sheds calls without touching the link.
    with pytest.raises(LinkDeadError):
        call_guarded(_scripted([True]), budget, breaker, op_name="poll")
    assert breaker.state is BreakerState.OPEN

    # Advance past cooldown: one probe is admitted; a success closes the breaker.
    for _ in range(11):
        next(clock)  # burn monotonic ticks to exceed the 10 s cooldown
    call_guarded(_scripted([True]), budget, breaker, op_name="poll")
    assert breaker.state is BreakerState.CLOSED


def test_session_budget_stops_a_dead_cable() -> None:
    budget = RetryBudget(max_attempts=1, epsilon=0.02, expected_commands=500)
    breaker = CircuitBreaker(min_samples=10_000)  # disable breaker for this test
    executor = _scripted([False] * 100)
    with pytest.raises((BudgetExhaustedError, LinkDeadError)):
        for _ in range(100):
            try:
                call_guarded(executor, budget, breaker, op_name="poll")
            except TimeoutError:
                continue

On a live rig the same signature is visible in the logs: a healthy flaky bridge prints occasional attempt N failed lines that never accumulate to a trip, whereas a failing one prints a dense run ending in breaker tripped followed by a single admitting half-open probe per cooldown. If the probe line repeats every cooldown without ever logging breaker closed, the link is dead, not flaky — that is the escalation trigger, and the LinkDeadError should route through Error Code Categorization as a terminal fault rather than a retryable one.

Failure Modes When Sizing the Budget

Budget too generous, masking a dead cable. Set epsilon at 0.2 or expected_commands at a million and the session budget effectively never trips; a cable that fell out at hour two limps on retrying forever while the dataset fills with nothing. The tell is throughput collapse without an escalation: commands still eventually succeed after burning every attempt, so no single command raises, yet the completion rate has fallen off a cliff. Diagnose by logging the ratio of attempts-consumed to commands-completed; when it approaches max_attempts, the budget is too loose regardless of whether any exception has fired. Keep epsilon near real bridge flakiness (1–3%) and let the breaker, not the session budget, be the fast trip.

Breaker flapping without half-open hysteresis. If a probe failure is allowed to immediately re-admit traffic — or if the window is never cleared on recovery — the breaker oscillates open/closed every few commands, alternately shedding and hammering a marginal link and producing a sawtooth throughput. The fix is the half-open discipline in record: exactly one probe decides the transition, and a successful probe clears the window so residual failures from the open period do not instantly re-trip. Diagnose by counting state transitions per minute; more than a handful means the cooldown is too short or the window is carrying stale failures across a recovery.

Per-command versus per-session budget confusion. Treating max_attempts as the whole story lets a session with a 4-attempt cap retry forever in aggregate — every command independently gets four fresh tries, so a link failing 40% of commands never stops. The attempt ceiling bounds one exchange; only the session error budget and the breaker bound the run. The signature is a run that never escalates despite a visibly high steady-state failure rate. Always pair the per-command cap with a per-session allowed_failures derived from ε · N, and make sure a fully failed command — not each attempt — is what charges the session budget.

Re-enumeration silently resetting the counters. When a CP210x or CH340 bridge drops off the USB bus and reappears at a new /dev/ttyUSB*, the naive reflex is to construct a fresh RetryBudget and CircuitBreaker for the reopened handle — which zeroes the failure history and hides a bridge that is re-enumerating every few minutes. Each re-enumeration looks like a clean start, so the breaker never accumulates enough samples to trip. Bind the budget and breaker to the device identity (USB VID/PID/serial), not to the file handle, and carry the failure window across a reopen; a bridge that re-enumerates repeatedly should itself count as failures against the budget. This is where budget state and the port-identity tracking from PySerial Configuration & Tuning must share a key.

Timeout Handling & Retry Logic — the parent collection covering timeout semantics, the bounded retry engine, and the escalation path this budget layer plugs into.
Implementing exponential backoff for serial timeout handling — the per-exchange backoff curve the guarded executor embeds; this guide sizes the budget that stops it.
Error Code Categorization — mapping a LinkDeadError to a terminal fault versus a SerialTimeoutError to a retryable one.
PySerial Configuration & Tuning — bind budgets to USB VID/PID identity and pin the latency timer so the breaker trips on real faults, not tuning artefacts.
Async Command Queuing Systems — hold one breaker per device so a flaky bridge opens its own circuit without stalling healthy instruments.

← Back to Timeout Handling & Retry Logic

References

Python collections.deque documentation — the bounded sliding window backing the breaker’s failure-rate calculation.
Python time.monotonic documentation — the monotonic clock used for the cooldown so wall-clock adjustments cannot corrupt the open-state timer.
pyserial API documentation — port identity, reopen, and buffer semantics relevant to carrying budget state across a re-enumeration.