Instrument Control

Structuring a New Lab Automation Project from Scratch

Laying out a greenfield Python lab-automation repository so it stays maintainable past its first three instruments: a layered package (drivers, transport, pipeline, config, tests), pinned dependencies, typed instrument config, and a single driver contract every device implements. This is the repository-and-conventions companion to End-to-End Lab Automation Pipeline Assembly, within the Instrument Control Architecture & Taxonomy section — where the assembly guide composes running stages, this one fixes the folders, imports, and config that host them.

Scope: The Repository, Not the Running Pipeline

The failure this guide prevents is the six-month-old lab codebase that works but can no longer be changed. A serial address is hard-coded in three modules, the oscilloscope driver imports the pipeline orchestrator “just to reuse a helper,” dependencies float on whatever pip install last resolved, and the only integration test needs the real rig powered on — so nobody runs it. Each shortcut was locally reasonable and collectively they make the project unshippable. Structuring from scratch is the cheap insurance: decisions about package boundaries, dependency pinning, and the driver contract cost minutes on day one and are prohibitively expensive to retrofit once forty modules depend on the wrong shape.

The assumptions are narrow and deliberate. The target is a Python 3.11+ project managed with a virtual environment and a lockfile, controlling a mix of SCPI, serial, and GPIB instruments. The concern here is arrangement — where code lives, what may import what, and how instrument-specific facts enter the program — not the behaviour of any single stage. The lifecycle orchestrator, bounded queues, and drain/stop logic belong to the parent pipeline assembly guide; the concrete session, capture, and sink code belongs to its sibling, Wiring VISA, Serial, Async Capture, and LIMS into One Pipeline. This guide is the scaffold both of those drop into.

The Layering Rule as an Acyclic Import Constraint

The load-bearing convention is a strict layer order: app → pipeline → drivers → transport. Higher layers may import lower ones; a lower layer may never import a higher one. Transport knows about bytes and ports and nothing about instruments; drivers know about SCPI verbs and nothing about the pipeline; the pipeline sequences drivers and knows nothing about the CLI or scheduler that started it. This is exactly the discipline that makes Protocol Abstraction Layers work — the layer boundary is where the abstraction lives.

Formally, treat every module as a node and every import as a directed edge, and assign each layer an integer rank — transport 0, drivers 1, pipeline 2, app 3. The rule is a two-part constraint on the module import graph G = (V, E):

The first clause forbids upward edges — a transport module importing from pipeline is a rank-0 node pointing at a rank-2 node, which violates rank(v) ≤ rank(u). The second forbids cycles even within a permitted direction, because a cycle makes the layers un-orderable and import time non-deterministic. Together they guarantee the package has a valid topological order, which is what lets you reason about, test, and reuse any layer in isolation: a driver pulled into a test drags in transport but never the pipeline, so it can be exercised against a fake port with no orchestrator present. The constraint is mechanically checkable — a small grep in CI over each layer’s imports, shown below, turns “we agreed not to do that” into a build failure.

Layered project architecture with a one-directional import rule Four stacked layer bands from top to bottom: app (rank 3, entry points and CLI), pipeline (rank 2, stage orchestration), drivers (rank 1, per-instrument Driver implementations), and transport (rank 0, byte-level ports). Green downward arrows on the left show permitted imports flowing from each higher layer to lower layers. On the right, an amber dashed arrow points upward from transport to pipeline and is crossed out, labelled forbidden: creates a cycle and breaks the DAG. A config module sits to the side as a leaf every layer may read but which imports nothing internal. app rank 3 · CLI · scheduler · entry points pipeline rank 2 · stage orchestration · lifecycle drivers rank 1 · Driver implementations · SCPI verbs transport rank 0 · bytes · ports · VISA / serial handles imports allowed downward forbidden: upward edge breaks the DAG config typed leaf · read by all

Every internal import points down the stack or sideways to the typed config leaf; the single upward edge (amber) from transport into pipeline is the violation that reintroduces a cycle and makes the package un-orderable.

Step 1–3: Package Layout, Environment, and Dependency Pinning

Step 1 — lay out the package by layer, not by feature. Each top-level package maps to exactly one rank, and tests/ mirrors the tree so a test’s location tells you which layer it exercises. Keep config/ a pure leaf: it defines shapes and defaults but imports nothing from drivers, pipeline, or transport, so every layer can read it without creating an upward edge.

labctl/
├── pyproject.toml            # metadata + pinned deps, single source of truth
├── uv.lock                   # fully resolved lockfile, committed
├── instruments.toml          # per-rig addresses & tuning (not in code)
├── src/labctl/
│   ├── config/               # rank leaf: pydantic settings, TOML loader
│   │   ├── __init__.py
│   │   └── settings.py
│   ├── transport/            # rank 0: serial / VISA byte channels
│   │   ├── __init__.py
│   │   ├── serial_port.py
│   │   └── visa_session.py
│   ├── drivers/              # rank 1: one module per instrument family
│   │   ├── __init__.py
│   │   ├── base.py           # the Driver Protocol every driver implements
│   │   ├── mock.py           # in-memory driver for CI
│   │   └── keysight_dmm.py
│   ├── pipeline/             # rank 2: stage orchestration
│   │   └── __init__.py
│   └── app/                  # rank 3: CLI, scheduler, entry points
│       └── __main__.py
└── tests/
    ├── transport/  drivers/  pipeline/
    └── test_smoke.py         # imports package, builds a mock driver from config

Step 2 — one virtual environment, one lockfile, committed. Create an isolated environment (python -m venv .venv or uv venv) and never install into the system interpreter — instrument driver stacks pin awkward versions of pyvisa, pyserial, and vendor shims that must not leak between projects. Declare direct dependencies in pyproject.toml and resolve them into a committed lockfile (uv.lock, poetry.lock, or pip-compile’s requirements.txt). The lockfile is the reproducibility contract: a rig that passed acceptance in March must rebuild bit-for-bit in September.

Step 3 — pin with bounds, not with floats. Pin direct dependencies to a compatible range in pyproject.toml (pyvisa>=1.14,<1.15, pyserial==3.5, pydantic>=2.6,<3) and let the lockfile freeze the exact transitive graph. Unbounded specifiers are how a pyvisa point release silently changes read-termination defaults and every timeout in the lab shifts overnight. The upstream byte-level knobs those versions govern are covered in PySerial Configuration & Tuning; pinning is what keeps them from moving underneath you.

Step 4: Typed Config for Addresses and Tuning Constants

Every instrument address, baud rate, and timeout is data, not code. Put them in instruments.toml and validate them into typed objects at startup, so a malformed rig file fails loudly with a field-level error instead of surfacing as a cryptic VISA fault twenty minutes into a run. pydantic over tomllib gives you both the parse and the validation in one pass:

"""labctl.config.settings — typed instrument configuration.

A pure leaf module: it imports only the standard library and pydantic,
never anything from transport, drivers, or pipeline, so any layer may
read it without introducing an upward import edge.
"""
from __future__ import annotations

import tomllib
from pathlib import Path

from pydantic import BaseModel, Field, TypeAdapter


class InstrumentConfig(BaseModel, frozen=True):
    """One instrument's connection and tuning parameters."""

    name: str
    driver: str = Field(..., description="dotted path, e.g. 'labctl.drivers.mock:MockDriver'")
    address: str = Field(..., description="VISA resource string or serial device path")
    timeout_s: float = Field(2.0, gt=0, le=60)
    baud: int | None = Field(None, ge=1200, le=921_600)


class RigConfig(BaseModel, frozen=True):
    """The whole rig: a named collection of instruments."""

    rig_id: str
    instruments: list[InstrumentConfig]


def load_rig(path: Path) -> RigConfig:
    """Parse and validate ``instruments.toml`` into a frozen RigConfig.

    Raises:
        pydantic.ValidationError: if any field is missing or out of range.
        tomllib.TOMLDecodeError: if the file is not valid TOML.
    """
    with path.open("rb") as fh:
        raw = tomllib.load(fh)
    return TypeAdapter(RigConfig).validate_python(raw)

Because RigConfig is frozen, a validated rig config is immutable for the life of the process — no stage can quietly mutate an address mid-run, which keeps provenance honest when those values are later stamped onto records by Metadata Injection Workflows.

Step 5: The Driver Contract Every Instrument Implements

The single most valuable convention is a Driver contract expressed as a Protocol, so the pipeline depends on an interface rather than any concrete instrument. A runtime_checkable Protocol also lets the smoke test assert conformance without inheritance. This is the same seam a VISA Resource Manager hands live sessions across — the driver owns instrument semantics, the transport owns bytes.

"""labctl.drivers.base — the contract every instrument driver satisfies."""
from __future__ import annotations

import importlib
from typing import Protocol, runtime_checkable

from labctl.config.settings import InstrumentConfig


@runtime_checkable
class Driver(Protocol):
    """Uniform instrument interface the pipeline programs against."""

    name: str

    def open(self) -> None:
        """Acquire the transport handle. Must be idempotent."""

    def identify(self) -> str:
        """Return an instrument identity string (e.g. the ``*IDN?`` reply)."""

    def query(self, command: str) -> str:
        """Send a command and return its textual response."""

    def close(self) -> None:
        """Release the handle. Must tolerate being called after a failed open."""


def build_driver(cfg: InstrumentConfig) -> Driver:
    """Instantiate the driver named in ``cfg.driver`` and verify conformance.

    Raises:
        TypeError: if the constructed object does not satisfy the Driver Protocol.
    """
    module_path, _, cls_name = cfg.driver.partition(":")
    cls = getattr(importlib.import_module(module_path), cls_name)
    driver = cls(cfg)
    if not isinstance(driver, Driver):
        raise TypeError(f"{cfg.driver} does not satisfy the Driver Protocol")
    return driver

A MockDriver(cfg) in drivers/mock.py implements the same four methods against an in-memory dictionary of canned responses — it is what makes the whole stack testable without hardware, and it is the object the smoke test builds.

Step 6: A Smoke Test and CI That Run Without a Rig

Validation of the structure is a test that imports the package and builds a mock driver straight from a config file — no instrument required. If this passes in CI, the layering holds, the config parses, and the driver contract is satisfiable:

import tomllib

from labctl.config.settings import load_rig
from labctl.drivers.base import Driver, build_driver


def test_package_assembles_from_config(tmp_path) -> None:
    """The package imports cleanly and a mock driver builds from config."""
    cfg_file = tmp_path / "instruments.toml"
    cfg_file.write_text(
        'rig_id = "ci-rig"\n'
        "[[instruments]]\n"
        'name = "dmm0"\n'
        'driver = "labctl.drivers.mock:MockDriver"\n'
        'address = "MOCK::0::INSTR"\n'
        "timeout_s = 1.0\n"
    )
    rig = load_rig(cfg_file)
    driver = build_driver(rig.instruments[0])
    assert isinstance(driver, Driver)          # Protocol conformance
    driver.open()
    assert driver.identify()                    # non-empty identity
    driver.close()

Wire it into CI so the layering rule is machine-enforced. A pytest job runs the smoke test on every push, and a second guard greps each lower layer for forbidden upward imports so a leaky edge fails the build rather than rotting silently:

# CI guard: transport (rank 0) must never import drivers/pipeline/app.
! grep -rE 'from labctl\.(drivers|pipeline|app)|import labctl\.(drivers|pipeline|app)' src/labctl/transport/ \
  || { echo "layer violation: transport imports upward"; exit 1; }

Failure Modes in Project Structure

Circular imports from a leaky layer. A driver reaches into pipeline for a “handy” constant, the pipeline already imports that driver, and Python raises ImportError: cannot import name ... (most likely due to a circular import) — or worse, a half-initialised module that only fails under a specific import order. The signature is an error that appears when you add an unrelated entry point and the import graph reshuffles. Diagnose with the CI grep above; the fix is to push the shared constant down into config (the leaf every layer may read) rather than sideways or up.

Hard-coded addresses instead of config. A serial.Serial("/dev/ttyUSB0") literal buried in a driver works on the developer’s bench and breaks the moment the rig enumerates ports in a different order or moves to another host. The tell is a project that needs a code edit and redeploy to change instruments. Every address, baud, and timeout must arrive through the validated RigConfig; a grep -rn '/dev/tty\|::INSTR\|GPIB[0-9]' src/ that returns hits inside drivers/ or pipeline/ is the smell.

Un-pinned dependencies drifting. With floating specifiers, a fresh pip install months later resolves newer pyvisa/pyserial releases whose defaults differ, and a rig that was green goes red with no source change. The signature is “it broke and nobody touched it.” Commit the lockfile, pin direct deps to bounded ranges, and have CI install from the lock so the resolved graph is identical everywhere — reproducibility is a property you build in, not one you hope for.

Tests that require real hardware. If the only integration test opens a live GPIB session, CI cannot run it, coverage silently drops to zero, and regressions ship. The fix is the MockDriver and the transport seam: the smoke test and every unit test run against fakes, and hardware-in-the-loop tests are a separate, explicitly-marked suite (@pytest.mark.hardware) gated behind a rig being present. Structure the project so the default test run needs nothing but the source tree.

← Back to End-to-End Lab Automation Pipeline Assembly

References