Instrument Control

Securing Lab Networks for Instrument Control Systems

A segmented control network stops hostile or noisy traffic at the boundary, but it does nothing for the process that actually holds the socket. The moment an orchestration worker opens a connection to an LXI oscilloscope or a SCPI-over-TCP power supply, the security posture of the run is in the hands of the client: how it validates the endpoint, how it sequences commands, and how it behaves when a permitted-but-degraded path stalls mid-transaction. This guide covers the host-side hardening that rides on top of network isolation — a deterministic, finite-state TCP control client with explicit error boundaries, bounded reconnection, and buffer-safe framing — so that a firewall hiccup or a half-open connection becomes a caught, attributable fault instead of a device-under-test left energized while the scheduler retries into a dead socket.

Baseline Constraints & Scope

This page assumes the network layer is already hardened as described in the parent guide on Security Boundaries & Network Isolation: instruments live on a dedicated control VLAN behind a DENY ALL firewall, sessions are pinned by literal IP, and only allow-listed control ports are reachable. What follows is the complementary client-side layer — the code that runs on the control host after the zone broker has authorized a target. It applies to any instrument reachable over TCP on the segmented VLAN: LXI scopes and analyzers on HiSLIP (4880), cleartext SCPI raw sockets (5025), and legacy GPIB or serial devices fronted by a serial-to-Ethernet bridge. Sessions established through a VISA Resource Manager Setup share the same requirements; the raw-socket client here is what you reach for when you need transaction-level control over framing and state that a VISA backend abstracts away.

The design goal is determinism under partial failure. A control client is not a generic network client: it must never leave an actuator in an undefined state, must never let an unhandled asyncio exception unwind through an experimental sequence, and must map every transport outcome to an explicit, queryable state. Before the byte stream reaches this layer it should already be normalized by the Protocol Abstraction Layers beneath it, and every command it emits should be a validated string from Command Set Standardization, so the client reasons only about connection lifecycle, framing, and timeouts.

Finite-State Sequencing & Bounded Backoff

The client is modeled as an explicit finite state machine with five states — IDLE, CONNECTING, READY, EXECUTING, FAULT — and legal transitions only between adjacent phases of a transaction. Commands are accepted only in READY; a timeout or unexpected close drives the machine to FAULT, from which the only exit is an explicit teardown back to IDLE. This is what prevents a cancelled coroutine or a race in the scheduler from issuing a second write while a prior read is still draining the socket.

Instrument control finite state machine Five states are shown as rounded boxes. IDLE (socket closed) transitions on connect() to CONNECTING (bounded backoff), which either reaches READY (accepts commands) on established, or drops to FAULT on max retries. From READY a command moves to EXECUTING (one in flight) and a response returns to READY. A timeout or reset from EXECUTING drops to FAULT (fail closed). The only exit from FAULT is safe_close(), which returns the machine to IDLE. Normal transitions are solid accent-coloured arrows; fault transitions are dashed amber arrows. INSTRUMENT CONTROL FSM connect() established command response max retries timeout / reset safe_close() IDLE socket closed CONNECTING bounded backoff READY accepts commands EXECUTING one in flight FAULT fail closed normal transition fault transition

Reconnection uses bounded exponential backoff with randomized jitter. Deterministic fixed delays cause a fleet recovering from a shared firewall event to reconnect in lockstep — a thundering herd that hammers the DENY ALL rule and can itself trigger rate limiting. The delay before retry $n$ is capped and jittered:

where $b_0$ is the base delay, $b_{\max}$ the ceiling, and $U(-j, j)$ a uniform sample bounding jitter to $\pm j$ (here $j = 0.1$). The min clamps unbounded growth so recovery latency stays predictable, while the jitter decorrelates retries across instruments. This is the same bounded curve used for transport-level retries in Timeout Handling & Retry Logic; the dedicated derivation and tuning matrix live in Implementing Exponential Backoff for Serial Timeout Handling.

Production-Ready Implementation

The client below is fully typed and runnable on Python 3.11+ using only the standard library. It enforces the FSM, caps the read buffer to prevent a runaway response from exhausting memory, enables TCP keepalive so a silently dropped path surfaces as a reset rather than an indefinite hang, and isolates every network exception behind structured logging and explicit state transitions.

from __future__ import annotations

import asyncio
import logging
import random
import socket
from dataclasses import dataclass
from enum import Enum
from typing import Optional

logger = logging.getLogger("secure_instrument")


class InstrumentState(str, Enum):
    IDLE = "idle"
    CONNECTING = "connecting"
    READY = "ready"
    EXECUTING = "executing"
    FAULT = "fault"


@dataclass(frozen=True)
class ConnectionConfig:
    """Immutable transport policy for one instrument endpoint."""

    host: str                       # literal IP, pinned by the zone policy
    port: int                       # allow-listed control port (5025 / 4880)
    timeout_s: float = 5.0          # per-operation deadline
    max_retries: int = 3
    base_backoff_s: float = 1.0
    max_backoff_s: float = 10.0
    jitter: float = 0.1             # +/- fraction of the computed delay
    termination: bytes = b"\n"
    max_response_bytes: int = 1 << 20  # hard cap: 1 MiB per response


class BoundaryFault(Exception):
    """A transport fault that must fail closed, never silently retry."""


class SecureInstrumentClient:
    """Deterministic, finite-state TCP control client for a segmented lab VLAN.

    The client accepts commands only in the READY state and drives itself to
    FAULT on any timeout, reset, or oversized response. It never leaves the
    socket in an ambiguous state: a fault always triggers a safe teardown.
    """

    def __init__(self, config: ConnectionConfig) -> None:
        self.config = config
        self.state = InstrumentState.IDLE
        self._reader: Optional[asyncio.StreamReader] = None
        self._writer: Optional[asyncio.StreamWriter] = None

    async def connect(self) -> None:
        """Open a keepalive TCP connection with bounded, jittered backoff."""
        if self.state is not InstrumentState.IDLE:
            raise BoundaryFault(f"connect refused in state {self.state.value}")

        self.state = InstrumentState.CONNECTING
        for attempt in range(1, self.config.max_retries + 1):
            try:
                self._reader, self._writer = await asyncio.wait_for(
                    asyncio.open_connection(self.config.host, self.config.port),
                    timeout=self.config.timeout_s,
                )
                self._enable_keepalive()
                self.state = InstrumentState.READY
                logger.info("connected %s:%d", self.config.host, self.config.port)
                return
            except (asyncio.TimeoutError, ConnectionError, OSError) as exc:
                logger.warning("connect attempt %d/%d failed: %s",
                               attempt, self.config.max_retries, exc)
                if attempt == self.config.max_retries:
                    self.state = InstrumentState.FAULT
                    raise BoundaryFault(
                        f"max retries to {self.config.host}:{self.config.port}"
                    ) from exc
                await asyncio.sleep(self._backoff_delay(attempt))

    def _backoff_delay(self, attempt: int) -> float:
        raw = min(self.config.base_backoff_s * 2 ** (attempt - 1),
                  self.config.max_backoff_s)
        return raw * (1 + random.uniform(-self.config.jitter, self.config.jitter))

    def _enable_keepalive(self) -> None:
        sock = self._writer.get_extra_info("socket") if self._writer else None
        if sock is not None:
            sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

    async def execute(self, command: str) -> str:
        """Send one command and return the framed response, decoded and stripped."""
        if self.state is not InstrumentState.READY:
            raise BoundaryFault(f"execute refused in state {self.state.value}")
        assert self._reader is not None and self._writer is not None

        self.state = InstrumentState.EXECUTING
        payload = command.encode() + self.config.termination
        try:
            self._writer.write(payload)
            await asyncio.wait_for(self._writer.drain(), timeout=self.config.timeout_s)
            response = await asyncio.wait_for(
                self._read_framed(), timeout=self.config.timeout_s
            )
            self.state = InstrumentState.READY
            return response.rstrip(self.config.termination).decode(errors="replace")
        except (asyncio.TimeoutError, ConnectionError, BoundaryFault, OSError) as exc:
            self.state = InstrumentState.FAULT
            logger.error("execute fault on %r: %s", command, exc)
            await self._safe_close()
            raise BoundaryFault(f"command {command!r} failed: {exc}") from exc

    async def _read_framed(self) -> bytes:
        """Read until the termination byte, refusing to buffer past the cap."""
        buffer = bytearray()
        while not buffer.endswith(self.config.termination):
            chunk = await self._reader.read(4096)  # type: ignore[union-attr]
            if not chunk:
                raise ConnectionResetError("instrument closed connection mid-response")
            buffer.extend(chunk)
            if len(buffer) > self.config.max_response_bytes:
                raise BoundaryFault(
                    f"response exceeded {self.config.max_response_bytes} bytes "
                    "with no terminator (framing mismatch)"
                )
        return bytes(buffer)

    async def _safe_close(self) -> None:
        writer, self._writer, self._reader = self._writer, None, None
        if writer is not None and not writer.is_closing():
            try:
                writer.close()
                await writer.wait_closed()
            except OSError as exc:
                logger.warning("teardown error: %s", exc)

    async def close(self) -> None:
        await self._safe_close()
        self.state = InstrumentState.IDLE

Validation & Verification in a Live Lab

Prove the client behaves correctly against a real instrument on the control VLAN before it drives an unattended run. Each step below produces an observable indicator you can assert against.

  1. Confirm the boundary permits only the client’s traffic. Run sudo tcpdump -i eth0 -nn "tcp port 5025 or tcp port 4880" on the gateway during a connect. You should see SYN packets only from the authorized control host; any SYN from a telemetry or admin subnet means the firewall rule is wrong, not the client.
  2. Force a fault and assert the transition. Drop the port mid-session with sudo iptables -A INPUT -p tcp --dport 5025 -j DROP, then issue a command. The client must land in FAULT within timeout_s, emit an execute fault log line, and raise BoundaryFault — never an uncaught asyncio.TimeoutError. Remove the rule with -D to restore.
  3. Verify framing against the real terminator. Manually reproduce the transaction with nc -v <host> 5025, type *IDN? and press Enter. Compare the instrument’s terminator to config.termination; a mismatch is the single most common cause of a hung read and is caught here by the max_response_bytes cap rather than by an OOM.
  4. Measure the backoff envelope. Log time.monotonic() around each asyncio.sleep (monotonic readings are immune to NTP and DST adjustments) and confirm successive delays follow $\min(b_{\max}, b_0 2^{n-1})$ with spread inside $\pm 10%$. Flat, identical delays across instruments mean the jitter is not wired in and a thundering herd is still possible.

Instrument the client so every state change is queryable: a FAULT count per endpoint, correlated against the firewall deny log by timestamp and target, is what an auditor needs to prove no unvalidated path to an actuator existed during a run.

Failure Modes & Edge Cases

Four failure modes are specific to a hardened control client and are worth rehearsing deliberately:

  • Unbounded response buffer. A firmware bug or a framing mismatch can stream bytes that never contain the terminator, growing the read buffer until the worker is OOM-killed and the run dies silently. The max_response_bytes cap converts this into a fast, attributable BoundaryFault. Diagnose a suspected case with grep "framing mismatch" control.log and cross-check the instrument’s actual terminator with nc.
  • Half-open TCP connection. If the instrument or an inline firewall drops state without a FIN, a naive client blocks forever on the next read. SO_KEEPALIVE (tuned at the OS level via net.ipv4.tcp_keepalive_time) forces the kernel to probe the peer so the dead path surfaces as a reset. Confirm the socket carries the option with ss -tio dst <host> and look for the keepalive timer.
  • FSM stuck in EXECUTING. A coroutine cancelled by the scheduler between write and _read_framed can leave the machine in EXECUTING with a half-sent command. Because execute refuses to run outside READY, the next command fails fast rather than interleaving; recovery is an explicit close() back to IDLE. Query for it with grep "refused in state executing" control.log.
  • Retry storm against DENY ALL. A misrouted target — an orchestration bug pointing at a jump host on :22 — retries into a non-allow-listed port and shows up as a firewall deny-log spike correlated with a run start. Transport faults should be classified through Error Code Categorization before recovery so a permanent boundary violation fails closed instead of looping.

When many instruments are driven across the boundary at once, arbitrate their connections and retries through Async Command Queuing Systems so a single stalled endpoint never blocks the scheduler or turns into a retry storm. Results pushed out of the control zone should be verified with Checksum/CRC Validation before they reach a LIMS, closing the loop between transport integrity and data integrity. The network-layer controls this client depends on — the zone model, allow-list port reference, and failover ruleset signing — are covered in full in the parent guide.

← Back to Security Boundaries & Network Isolation