Instrument Control

Configuring pyserial for High-Throughput Instrument Polling

High-throughput instrument polling demands deterministic latency, strict buffer management, and explicit error boundaries — properties the default pyserial constructor does not give you, because it is tuned for interactive terminal sessions rather than sustained automated acquisition. This page covers one narrow problem: how to configure a serial.Serial object and drive it so a single control thread can issue commands and drain fixed-size response frames at hundreds of hertz without dropping bytes, blocking indefinitely, or spinning the CPU on a virtual COM port. It is the readiness-driven counterpart to the timeout policy in Timeout Handling & Retry Logic and sits directly under PySerial Configuration & Tuning, which holds the full parameter matrix this page draws on.

The scope is deliberately tight. It applies to byte-oriented links — native RS-232, RS-485, and USB-CDC virtual COM ports exposed through FTDI, CP210x, or CH340 bridges — carrying fixed-length binary or delimited ASCII frames from DAQ cards, source-measure units, motion controllers, and benchtop meters. It assumes a request-response instrument (you write a command, the device answers) rather than a free-running telemetry stream, and it assumes 8N1 framing throughout. Instruments reached over VISA or a VISA Resource Manager session use a different transport layer and are out of scope here, though the readiness discipline transfers directly.

Poll Period Bounded by OS Receive-Buffer Fill Time

The governing constraint of high-rate polling is that the operating system’s receive buffer must be drained before it overflows. On an 8N1 line every byte costs 10 bit-times (1 start + 8 data + 1 stop), so the effective byte rate is the baud rate divided by ten. If the kernel receive buffer holds B_os bytes, the time for a continuously transmitting instrument to fill it from empty is:

The loop that reads the port must therefore drain it with a period T_poll strictly shorter than t_fill, or bytes are lost the moment in_waiting crosses the buffer ceiling and the kernel silently discards the overrun (or raises OSError: [Errno 11] Resource temporarily unavailable). The safe operating condition is:

At 115200 baud with the common 4096-byte kernel buffer, t_fill is roughly 355 ms — comfortable. At 921600 baud it collapses to about 44 ms, and any GC pause or scheduler preemption longer than that drops data. This is why the poll loop must be non-blocking and bounded rather than sleep-driven: a fixed time.sleep() interval that happened to work at one baud rate silently corrupts data at the next.

A second timing quantity governs latency rather than loss. A response frame of F bytes takes t_frame = 10F / baud to arrive on the wire once the instrument begins transmitting. The readiness loop should sample in_waiting at an interval far smaller than t_frame so per-cycle latency is dominated by the instrument’s processing delay, not by the host’s sampling granularity — but not so small that it busy-waits and thrashes a USB bridge. A micro-yield on the order of 0.5 ms is the practical floor on USB-CDC ports, where each in_waiting query crosses the USB stack. The design goal is a loop whose worst-case added latency is bounded and known, so it composes predictably when fed by Async Command Queuing Systems.

One bounded poll cycle: buffer occupancy against time A time axis runs left to right; the vertical axis is buffer occupancy in bytes. At t0 the host issues write() then flush(). During the instrument processing gap the buffer stays at zero and each in_waiting sample reads nothing (hollow ticks). When the response frame begins arriving on the wire over t_frame, each in_waiting sample reads bytes (filled ticks) and buffer occupancy climbs as a staircase. When occupancy crosses the frame_size threshold the loop returns the frame — well before the dashed response-deadline line and far below the OS receive-buffer ceiling B_os, which is only reached if the poll period exceeds the fill time t_fill equal to ten times B_os divided by baud. instrument processing (t_instr) frame on wire t_frame = 10·F / baud time → buffer occupancy (bytes) OS RX buffer ceiling B_os — overflow only if T_poll > t_fill = 10·B_os / baud frame_size threshold t0 · write() + flush() len(buffer) ≥ frame_size → return frame deadline t0 + response_timeout returns inside deadline in_waiting sampled every poll_interval (micro-yield): hollow = 0 bytes, filled = bytes read

A Bounded, Readiness-Driven Poller

Three configuration choices make the constructor fit for automated polling. Set timeout=0 for non-blocking reads so read(n) returns immediately with whatever is buffered rather than parking the control thread. Keep xonxoff=False unconditionally — software flow control injects 0x11/0x13 control bytes into binary payloads and corrupts them silently. Enable rtscts=True only when the instrument firmware genuinely drives the RTS/CTS lines; on a virtual COM port that lacks physical handshake routing it deadlocks. Finally, set exclusive=True on POSIX to take a TIOCEXCL advisory lock so a stray second open() cannot race the poller — the flag is POSIX-only and raises ValueError on Windows, where a COM port already opens exclusively by default.

The implementation below wraps those choices in a context manager, flushes stale buffers on entry, and drives a bounded readiness loop with a hard monotonic deadline. Each command-response cycle either returns a complete frame or fails fast; it never blocks on phantom data.

from __future__ import annotations

import time
from types import TracebackType
from typing import Optional

import serial


class PollTimeoutError(Exception):
    """Raised when a full frame does not arrive within the response deadline."""


class HighThroughputPoller:
    """Bounded, readiness-driven serial poller for high-rate instrument acquisition.

    Configures a ``serial.Serial`` object for non-blocking reads and drives a
    fixed-size request/response cycle whose worst-case latency is bounded by
    ``response_timeout``. Intended for 8N1 request-response instruments on
    RS-232/RS-485 or USB-CDC virtual COM ports.
    """

    def __init__(
        self,
        port: str,
        baudrate: int = 115200,
        frame_size: int = 64,
        max_retries: int = 3,
        response_timeout: float = 0.05,
        write_timeout: float = 0.5,
        poll_interval: float = 0.0005,
    ) -> None:
        self.port = port
        self.baudrate = baudrate
        self.frame_size = frame_size
        self.max_retries = max_retries
        self.response_timeout = response_timeout   # hard per-cycle read deadline
        self.write_timeout = write_timeout          # fail-fast write ceiling
        self.poll_interval = poll_interval          # micro-yield between in_waiting checks
        self._ser: Optional[serial.Serial] = None

    def __enter__(self) -> "HighThroughputPoller":
        self._ser = serial.Serial(
            port=self.port,
            baudrate=self.baudrate,
            timeout=0,                # non-blocking reads
            write_timeout=self.write_timeout,
            rtscts=False,             # enable only if firmware drives RTS/CTS
            xonxoff=False,            # never inject control chars into binary streams
            exclusive=True,           # POSIX advisory lock; drop on Windows if needed
        )
        self._ser.reset_input_buffer()
        self._ser.reset_output_buffer()
        return self

    def __exit__(
        self,
        exc_type: Optional[type[BaseException]],
        exc_val: Optional[BaseException],
        exc_tb: Optional[TracebackType],
    ) -> None:
        if self._ser is not None and self._ser.is_open:
            self._ser.reset_input_buffer()
            self._ser.reset_output_buffer()
            self._ser.close()

    def poll_frame(self, command: bytes) -> bytes:
        """Issue one command and return exactly ``frame_size`` response bytes.

        Retries up to ``max_retries`` times, flushing the input buffer between
        attempts so a partial frame from a failed cycle cannot corrupt the next.
        Raises ``PollTimeoutError`` if every attempt misses the deadline.
        """
        if self._ser is None or not self._ser.is_open:
            raise RuntimeError("Serial port not initialized or already closed.")

        last_error: Optional[Exception] = None
        for attempt in range(self.max_retries):
            try:
                self._ser.write(command)
                self._ser.flush()  # block until the OS hands the command to the UART

                deadline = time.monotonic() + self.response_timeout
                buffer = bytearray()
                while time.monotonic() < deadline:
                    waiting = self._ser.in_waiting
                    if waiting:
                        buffer.extend(self._ser.read(waiting))
                        if len(buffer) >= self.frame_size:
                            return bytes(buffer[: self.frame_size])
                    time.sleep(self.poll_interval)  # yield; avoid CPU thrash on USB-CDC

                # Deadline hit without a complete frame — discard the partial.
                self._ser.reset_input_buffer()
                last_error = PollTimeoutError(
                    f"Frame incomplete: {len(buffer)}/{self.frame_size} bytes "
                    f"within {self.response_timeout:.3f}s"
                )
            except (serial.SerialException, OSError) as exc:
                self._ser.reset_input_buffer()
                last_error = exc

            time.sleep(0.005 * (attempt + 1))  # linear inter-attempt backoff

        raise PollTimeoutError(
            f"Poll failed after {self.max_retries} attempts: {last_error}"
        )

Two properties make this deterministic. in_waiting never blocks, so the loop’s timing is set entirely by the monotonic deadline, not by driver behaviour; and the input buffer is flushed on every failure path, so a partial frame from one cycle can never bleed into the next and desynchronise framing. When the buffer accumulates more than frame_size bytes, only the first frame_size are returned and the remainder is discarded on the next entry — appropriate for strict request-response instruments where a surplus signals an echo or a stale reply.

Validating Poll Timing Against a Live Instrument

Confirm the poller before trusting it in a run. The cheapest check is a loopback: short pins 2 and 3 on a DE-9 (or use a socat PTY pair on Linux) so every written command echoes back, and assert the round-trip returns the bytes you sent within the deadline.

# socat -d -d pty,raw,echo=0 pty,raw,echo=0  → gives two linked /dev/pts/N devices
with HighThroughputPoller("/dev/pts/5", baudrate=115200, frame_size=8) as poller:
    echo = poller.poll_frame(b"\x01\x02\x03\x04\x05\x06\x07\x08")
    assert echo == b"\x01\x02\x03\x04\x05\x06\x07\x08"

Against real hardware, three indicators tell you the timing budget holds. Time a batch of poll_frame calls and divide: the achieved rate should sit at or just under 1 / (t_instrument + t_frame), where t_instrument is the device’s command-processing delay from its datasheet. A large shortfall means the loop is blocking somewhere it should not. Second, watch ser.in_waiting sampled just before each write — it should read zero on a healthy request-response link; a persistent nonzero value means responses are arriving faster than they are drained and you are approaching the t_fill ceiling. Third, on Linux, run cat /proc/tty/driver/serial or attach strace -e trace=read,ioctl -p <pid> and confirm ioctl(FIONREAD) (the syscall behind in_waiting) never returns a value near your kernel buffer size; if it does, lower the poll period or raise frame_size alignment. Frames that clear this stage flow onward to Binary & ASCII Format Parsing and, for binary payloads carrying a checksum trailer, through Checksum & CRC Validation before they are trusted.

Failure Modes Specific to High-Rate Polling

These are the faults that surface only under sustained polling — distinct from the connect-time and configuration errors covered on the parent PySerial Configuration & Tuning page.

USB-CDC scheduling jitter starves a fast loop. USB bridges coalesce interrupts and buffer against a latency timer — FTDI defaults to 16 ms — so a single frame is often delivered as two host reads, and per-poll latency spikes far above the wire time. Lower the FTDI latency timer to 1–2 ms (echo 1 | sudo tee /sys/bus/usb-serial/devices/ttyUSB0/latency_timer on Linux) and disable USB selective suspend for the hub (echo on > /sys/bus/usb/devices/.../power/control). Confirm the fix by logging the count of in_waiting iterations per frame: it should drop toward one after the timer is lowered.

OS receive buffer overflow at high baud. When T_poll exceeds t_fill — a GC pause, a logging call, or a scheduler preemption inside the loop — the kernel buffer overruns and bytes vanish, showing up downstream as a frame that fails its checksum rather than as an exception here. Diagnose by sampling in_waiting and alarming when it climbs past half the kernel buffer size; the sustainable fix is to shorten the loop body, raise thread priority, or move the drain to a dedicated thread. Do not paper over it by enlarging frame_size alone.

Control-character injection from stray XON/XOFF. If xonxoff is enabled anywhere in the driver stack, a 0x13 (XOFF) byte inside a binary payload silently pauses transmission and a 0x11 (XON) resumes it, corrupting the stream without any error. Log a hex dump of the first 16 bytes of each frame; recurring 0x11/0x13 at frame boundaries is the signature. Assert xonxoff=False on the open port and verify the underlying tty flags with stty -a -F /dev/ttyUSB0 (the ixon/ixoff flags must be off).

Driver-level stall masked as instrument silence. Python’s non-blocking read cannot distinguish a device that has gone quiet from a driver that has wedged. Wrap the port in a select.poll() loop (POSIX) to separate genuine instrument silence from OS scheduling delay, and treat a repeated deadline miss as a transport fault to be classified, not retried blindly. Route those faults through Error Code Categorization so a wedged bridge is escalated while a one-off timeout is retried under the exponential backoff policy for serial timeouts rather than hammered.

← Back to PySerial Configuration & Tuning

References