Parsing IEEE 488.2 Definite-Length Block Data in Python

SCPI waveform and measurement queries such as CURV?, FETC?, and :WAVeform:DATA? return their binary payload wrapped in an IEEE 488.2 arbitrary block: a #<nd><length> ASCII preamble whose digits declare exactly how many payload bytes follow. Get the byte count wrong by one and every struct decode downstream shifts; scan for a newline inside the block and a raw sample byte of 0x0A truncates the read. This guide, a focused companion within Binary & ASCII Format Parsing, pins down the block-framing layer only — reading the preamble, consuming the declared byte count across fragmented transport reads, and stripping the trailing terminator — independent of how the payload itself is later decoded.

Block Framing Scope & the Definite-Length Contract

Within the Data Capture, Validation & Metadata Sync pipeline, block framing sits one layer below payload decoding. The concern here is purely the envelope: given a stream of bytes arriving from a VISA Resource Manager read or a raw PySerial port, extract the payload region exactly, and hand it to a decoder without touching its contents. What that payload means — little-endian f, big-endian d, packed int16 ADC counts — is resolved separately in Decoding IEEE 754 Floats from Instrument Binary Blocks, which takes over precisely where this page stops.

The IEEE 488.2 standard defines two arbitrary-block forms. The definite-length form is # then a single digit nd giving the number of length digits, then those nd ASCII digits giving the payload byte count, then the payload, then a message terminator. A 2048-byte block reads #42048 followed by 2048 raw bytes: #, then 4 (four length digits follow), then 2048, then the data. The indefinite-length form is #0 followed immediately by the payload and terminated only by the final newline with EOI asserted — it carries no byte count, so it can only be consumed by reading until end-of-message. A definite block is self-describing and safe to slice; an indefinite block forces you to trust the terminator, and any payload byte equal to the terminator will corrupt the boundary. Most instruments let you force the definite form with a FORMat:BORDer / header-mode command, and production code should demand it.

Header Arithmetic & Total Frame Width

The preamble is fixed-cost and computable before a single payload byte is allocated. Reading the first two bytes gives you everything needed to size the read. Let the ASCII digit after # be nd (one character, 1–9), let the next nd characters form the decimal string digits, and let T be the width of the trailing terminator (\n, or \r\n, or zero if the transport signals EOI out of band):

$$\begin{array}{rl}\text{payload\_len}& =\mathrm{int}(\text{digits}),\text{digits}\in \{\text{0}\text{–}\text{9}{\}}^{nd}\\ \text{total\_frame}& =\underset{\#\text{and}nd}{\underbrace{2}}+\underset{\text{length digits}}{\underbrace{nd}}+\text{payload\_len}+\underset{\text{terminator}}{\underbrace{T}}\end{array}$$

For #42048, nd = 4, payload_len = 2048, and with a single-byte \n terminator the frame occupies 2 + 4 + 2048 + 1 = 2055 bytes on the wire. The #0 indefinite case is the degenerate nd = 0, where payload_len is undefined and total_frame collapses to a read-until-EOI loop. Because payload_len is known the instant the preamble is parsed, the parser never scans the payload for a delimiter and never speculatively over-reads — it asks the transport for exactly the remaining count.

The nd digit counts the length digits, and int(digits) fixes the payload width before any byte is read — so a definite block is sliced deterministically, while the indefinite #0 form has no count and breaks if a payload byte equals the terminator.

Reading Exactly N Bytes Across Chunked Transport Reads

A single #42048 block almost never arrives in one transport read. A serial FIFO flushes on a threshold, a TCP/LXI socket coalesces or splits writes, and PyVISA hands back controller-sized chunks. The parser therefore cannot assume one read() yields the whole preamble, let alone the whole payload. The discipline is a two-phase pull: first accumulate until the two fixed preamble bytes and then the nd length digits are present, then issue repeated reads until the exact declared count has been collected. The reader is abstracted to a single callable so the same code drives a serial.Serial.read(n), a socket recv, or a PyVISA read_bytes(n) without change.

Copy
import struct
from typing import Callable, Protocol

class Reader(Protocol):
    """Any transport exposing a blocking read of up to n bytes."""
    def __call__(self, n: int) -> bytes: ...

class BlockFramingError(Exception):
    """Raised on a malformed IEEE 488.2 arbitrary-block envelope."""

def _read_exact(reader: Reader, n: int) -> bytes:
    """Pull exactly n bytes, looping across fragmented transport reads.

    Raises BlockFramingError on premature end-of-stream (a read that
    returns b'') so an under-run surfaces here rather than as a short slice.
    """
    if n < 0:
        raise BlockFramingError(f"negative read length {n}")
    out = bytearray()
    while len(out) < n:
        chunk = reader(n - len(out))
        if not chunk:
            raise BlockFramingError(
                f"stream ended after {len(out)} of {n} bytes"
            )
        out.extend(chunk)
    return out

def read_definite_block(
    reader: Reader,
    terminator: bytes = b"\n",
    max_payload: int = 64_000_000,
) -> bytes:
    """Parse one IEEE 488.2 definite-length block and return its payload.

    Consumes '#<nd><length><payload><terminator>' from the reader, validating
    each stage. The returned bytes are the payload region only, with the
    trailing terminator stripped. Endianness and struct decoding are the
    caller's concern.
    """
    lead = _read_exact(reader, 1)
    if lead != b"#":
        raise BlockFramingError(f"expected '#' block leader, got {lead!r}")

    nd_byte = _read_exact(reader, 1)
    if not nd_byte.isdigit():
        raise BlockFramingError(f"non-digit nd field {nd_byte!r}")
    nd = int(nd_byte)
    if nd == 0:
        raise BlockFramingError(
            "indefinite-length block (#0): no byte count declared; "
            "use a read-until-EOI reader instead"
        )

    digits = _read_exact(reader, nd)
    if not digits.isdigit():
        raise BlockFramingError(f"non-digit length field {digits!r}")
    payload_len = int(digits)
    if payload_len == 0 or payload_len > max_payload:
        raise BlockFramingError(f"invalid payload length {payload_len}")

    payload = bytes(_read_exact(reader, payload_len))

    if terminator:
        trailer = _read_exact(reader, len(terminator))
        if trailer != terminator:
            raise BlockFramingError(
                f"expected terminator {terminator!r}, got {trailer!r}"
            )
    return payload

Two design choices carry the correctness. _read_exact treats an empty read as a hard error, so an under-run truncation raises inside the framing layer instead of returning a short payload that a later struct.unpack blames on the wrong stage. And the terminator is read and compared as its own step rather than stripped with .rstrip(); a blanket strip would silently eat a genuine trailing 0x0A that happens to be the last payload byte. nd == 0 is rejected explicitly, converting the ambiguous indefinite form into a caught, actionable error at the boundary.

Verifying Against Synthetic Blocks and Split Reads

Trust the parser only after it survives adversarial fragmentation on bytes you control, so a live-rig failure isolates to the instrument rather than your framing. Build a spec-compliant block, then wrap it in a reader that hands back one byte at a time — the worst-case boundary — and confirm the payload reassembles intact even when the count digits, payload, and terminator all straddle read boundaries.

Copy
def build_block(payload: bytes, terminator: bytes = b"\n") -> bytes:
    """Wrap payload in a definite-length IEEE 488.2 block."""
    digits = str(len(payload)).encode("ascii")
    return b"#" + str(len(digits)).encode("ascii") + digits + payload + terminator

class ChunkReader:
    """Reader that releases the buffer in fixed-size slices (1 = worst case)."""
    def __init__(self, data: bytes, chunk: int = 1) -> None:
        self._data, self._chunk, self._pos = data, chunk, 0
    def __call__(self, n: int) -> bytes:
        take = min(n, self._chunk, len(self._data) - self._pos)
        out = self._data[self._pos:self._pos + take]
        self._pos += take
        return out

def test_split_read_reassembly() -> None:
    # A payload that deliberately contains terminator-looking bytes.
    payload = struct.pack("<512f", *(float(i) for i in range(512)))
    assert b"\n" in payload and b"#" in payload  # decoy bytes present
    block = build_block(payload)

    for chunk in (1, 3, 7, 4096):          # byte-at-a-time through whole-frame
        reader = ChunkReader(block, chunk=chunk)
        got = read_definite_block(reader)
        assert got == payload, f"reassembly failed at chunk={chunk}"
    print("split-read reassembly OK:", len(payload), "bytes recovered")

On a live instrument, cross-check the parsed payload_len against the sample count the query should return — a CURV? on a scope with a 2048-point record must yield 2048 * bytes_per_point. Log the observed nd and byte count per query; a stable pairing confirms the header mode is locked to definite-length. If you see the count field change width between otherwise identical acquisitions, the instrument is auto-sizing nd with the record length, which is legal and precisely why the parser reads nd before the digits rather than assuming a fixed preamble.

Failure Modes & Diagnosis

Four failures are specific to arbitrary-block framing. Each maps to a symptom you can read off a log or a hex dump.

Payload byte collides with the terminator. A raw sample of 0x0A mid-block causes truncation only if you scan for the terminator instead of honoring the count. The definite-length parser is immune because it slices exactly payload_len bytes and reads the terminator as a fixed follow-on. If you observe short payloads, confirm no upstream stage is calling readline() or splitting on \n before read_definite_block sees the stream.

Indefinite-length block with no count. A #0 preamble carries no length; read_definite_block raises a BlockFramingError naming it rather than hanging or guessing. The fix is instrument-side: issue the format command that forces definite-length headers (for example set the response header mode on, or FORMat the transfer explicitly). Only fall back to a read-until-EOI loop when the device genuinely cannot emit a count, and never on a delimiter that can appear in the payload.

Under-read truncation. A read timeout or a closed socket mid-payload returns fewer bytes than declared. _read_exact catches the empty read and raises with the exact got of expected counts, so the fault is attributable instead of silently corrupting the next frame. Recovery is a resync: discard the partial, and align the transport read granularity and timeout to span at least one frame, the same partial-read discipline that governs high-throughput PySerial polling.

Digit-count off-by-one. Reading nd digits as nd - 1, or including the # in the count, shifts the payload boundary by one and every subsequent decode with it — the classic 3.4e38/NaN signature of a one-byte slip. Guard it by asserting len(digits) == nd (implicit in _read_exact(reader, nd)) and by validating that the recovered payload length is an exact multiple of the element size before handing off to the decoder.

Integrating the Framed Payload Downstream

read_definite_block is the envelope stage only; its output is opaque bytes. The immediate next step is decoding, worked in Decoding IEEE 754 Floats from Instrument Binary Blocks, where the byte order and element format are resolved. Where a block carries its own integrity trailer inside the payload, gate it through Checksum & CRC Validation before any value is trusted. For concurrent acquisition across an instrument array, wrap the reader behind an Async Command Queuing System so one slow FETC? cannot stall the shared loop — one framing call per session, since a block read is a stateful pull that must not interleave. The broader survey of frame anatomy, byte-order tables, and hybrid hex/ASCII responses lives in the parent Binary & ASCII Format Parsing guide.

Related

• Decoding IEEE 754 Floats from Instrument Binary Blocks — the decode stage that consumes this parser’s output.
• Binary & ASCII Format Parsing — frame anatomy and byte-order reference.
• VISA Resource Manager — the transport that delivers CURV?/FETC? responses.
• PySerial Configuration & Tuning — read granularity and timeouts that span a frame.
• Checksum & CRC Validation — integrity gating for payloads that carry a trailer.

← Back to Binary & ASCII Format Parsing

References

• IVI Foundation — IEEE 488.2 / SCPI specifications
• Python struct — binary data interpretation
• PyVISA — reading binary block data