Skip to content

livery.toolroom.bench._ingest

livery.toolroom.bench._ingest

Ingest verification: structural checks on a version's deployments, from any machine.

A new version is checked against the deployment already known before its pull request merges. The checks need no execution, so every host of the version is checked from whichever machine runs the refresh: each host's artifact is staged through the store, unpacked and looked at, and compared with the version before it.

The nine checks, in the order CHECKS lists them:

  • root: the declared root is in the archive (a stage refuses it otherwise, and the refusal is the finding).
  • entry-points: every declared entry point is a file in the tree.
  • paths: every declared path directory is in the tree.
  • shims: every shim's target is a file in the tree.
  • env: every env value naming a path under the install points at something in the tree.
  • entry-points-kept: no entry point the previous version declared for the host is gone.
  • hosts-kept: no host the previous version had lost its build.
  • exclusions: every exclusion pattern matches something in the unpacked tree.
  • stray-executables: no executable sits in a declared path directory without being a declared entry point or shim.

A finding names the check, the host and what was found. The structural diff of the paths added and removed since the previous version, per host, is the reviewer's summary. The executable checks, the entry point runs and the surface extracts, need a matching host and are the six-host point's business.

Classes:

Name Description
Finding

One check that did not pass, on one host.

HostDiff

The paths one host's tree gained and lost since the previous version.

Report

What the checks found for one version of one tool.

Functions:

Name Description
previous_with_hosts

The nearest earlier version that has a host, or empty when there is none.

verify

Run the nine checks on version of record for its hosts; the report.

summary

The report as lines: the verdict, each finding, and each host's diff.

Attributes:

Name Type Description
CHECKS

The nine structural checks, by name, in the order they run.

CHECKS module-attribute

CHECKS = ('root', 'entry-points', 'paths', 'shims', 'env', 'entry-points-kept', 'hosts-kept', 'exclusions', 'stray-executables')

The nine structural checks, by name, in the order they run.

Finding dataclass

Finding(check: str, host: str, detail: str)

One check that did not pass, on one host.

Attributes:

Name Type Description
check str

The check's name, from CHECKS.

host str

The host key checked; empty for a finding about the version as a whole.

detail str

What was found, in the words a reviewer acts on.

Methods:

Name Description
__str__

check instance-attribute

check: str

host instance-attribute

host: str

detail instance-attribute

detail: str

__str__

__str__() -> str
Source code in packages/toolroom-bench/src/livery/toolroom/bench/_ingest.py
def __str__(self) -> str:
    where = f" on {self.host}" if self.host else ""
    return f"{self.check}{where}: {self.detail}"

HostDiff dataclass

HostDiff(host: str, added: tuple[str, ...], removed: tuple[str, ...])

The paths one host's tree gained and lost since the previous version.

Attributes:

Name Type Description
host str
added tuple[str, ...]
removed tuple[str, ...]

host instance-attribute

host: str

added instance-attribute

added: tuple[str, ...]

removed instance-attribute

removed: tuple[str, ...]

Report dataclass

Report(name: str, version: str, previous: str, hosts: tuple[str, ...], findings: tuple[Finding, ...] = (), diffs: tuple[HostDiff, ...] = tuple())

What the checks found for one version of one tool.

Attributes:

Name Type Description
name str

The tool.

version str

The version checked.

previous str

The version it was compared with; empty for the first version with a host.

hosts tuple[str, ...]

The hosts checked, in the record's order.

findings tuple[Finding, ...]

Every check that did not pass; empty means every check passed on every host.

diffs tuple[HostDiff, ...]

Per host, the structural diff against the previous version; a host the previous version lacked diffs against nothing, so every path is added.

name instance-attribute

name: str

version instance-attribute

version: str

previous instance-attribute

previous: str

hosts instance-attribute

hosts: tuple[str, ...]

findings class-attribute instance-attribute

findings: tuple[Finding, ...] = ()

diffs class-attribute instance-attribute

diffs: tuple[HostDiff, ...] = field(default_factory=tuple)

passed property

passed: bool

Whether every check passed on every host.

previous_with_hosts

previous_with_hosts(record: Record, version: str) -> str

The nearest earlier version that has a host, or empty when there is none.

Source code in packages/toolroom-bench/src/livery/toolroom/bench/_ingest.py
def previous_with_hosts(record: Record, version: str) -> str:
    """The nearest earlier version that has a host, or empty when there is none."""
    before = ""
    for delta in record.deltas:
        if delta.version == version:
            return before
        if delta.hosts:
            before = delta.version
    return before

verify

verify(record: Record, version: str, *, store: Store, hosts: tuple[str, ...] = (), previous: str | None = None) -> Report

Run the nine checks on version of record for its hosts; the report.

Each host's artifact is staged through store into a scratch directory, which goes when the host's checks are done. hosts narrows to some of the version's hosts; previous names the version to compare with, the nearest earlier version with a host otherwise. A version with no host has nothing to check and passes with no diff.

Raises:

Type Description
RecordError

when the record does not track version.

Source code in packages/toolroom-bench/src/livery/toolroom/bench/_ingest.py
def verify(
    record: Record,
    version: str,
    *,
    store: Store,
    hosts: tuple[str, ...] = (),
    previous: str | None = None,
) -> Report:
    """Run the nine checks on *version* of *record* for its hosts; the report.

    Each host's artifact is staged through *store* into a scratch
    directory, which goes when the host's checks are done. *hosts*
    narrows to some of the version's hosts; *previous* names the
    version to compare with, the nearest earlier version with a host
    otherwise. A version with no host has nothing to check and passes
    with no diff.

    Raises:
        RecordError: when the record does not track *version*.
    """
    from livery.toolroom.store import StoreError, resolve

    delta = record.delta_for(version)
    before = previous_with_hosts(record, version) if previous is None else previous
    wanted = tuple(host for host in delta.hosts if not hosts or host in hosts)
    findings: list[Finding] = []
    diffs: list[HostDiff] = []
    if before:
        for host in record.hosts_of(before):
            if host not in delta.hosts:
                findings.append(
                    Finding(
                        "hosts-kept",
                        host,
                        f"{before} had a build and {version} has none",
                    )
                )
    for host in wanted:
        deployment = resolve(record, version, host)
        earlier = (
            resolve(record, before, host)
            if before and host in record.hosts_of(before)
            else None
        )
        with tempfile.TemporaryDirectory(prefix=f"ingest-{record.name}-") as scratch:
            into = Path(scratch) / "tree"
            into.mkdir()
            try:
                members = store.stage(
                    record.name, record.kind, version, deployment, into
                )
            except StoreError as error:
                findings.append(Finding("root", host, str(error)))
                continue
            findings.extend(_check_tree(host, deployment, earlier, members, into))
            current = _paths_in(into)
        if earlier is not None:
            with tempfile.TemporaryDirectory(
                prefix=f"ingest-{record.name}-"
            ) as scratch:
                into = Path(scratch) / "tree"
                into.mkdir()
                try:
                    store.stage(record.name, record.kind, before, earlier, into)
                except StoreError:
                    older: set[str] = set()
                else:
                    older = _paths_in(into)
        else:
            older = set()
        diffs.append(
            HostDiff(
                host,
                tuple(sorted(current - older)),
                tuple(sorted(older - current)),
            )
        )
    return Report(record.name, version, before, wanted, tuple(findings), tuple(diffs))

summary

summary(report: Report) -> list[str]

The report as lines: the verdict, each finding, and each host's diff.

Source code in packages/toolroom-bench/src/livery/toolroom/bench/_ingest.py
def summary(report: Report) -> list[str]:
    """The report as lines: the verdict, each finding, and each host's diff."""
    lines = [
        f"  {report.name} {report.version}: "
        + (
            f"every check passed on {len(report.hosts)} host(s)"
            if report.passed
            else f"{len(report.findings)} finding(s)"
        )
        + (
            f", against {report.previous}"
            if report.previous
            else ", the first version with a host"
        )
    ]
    lines.extend(f"    {finding}" for finding in report.findings)
    for diff in report.diffs:
        if not diff.added and not diff.removed:
            lines.append(f"    {diff.host}: no path added or removed")
            continue
        lines.append(
            f"    {diff.host}: +{len(diff.added)} -{len(diff.removed)} path(s)"
        )
        lines.extend(f"      + {path}" for path in diff.added[:20])
        if len(diff.added) > 20:
            lines.append(f"      + ... {len(diff.added) - 20} more")
        lines.extend(f"      - {path}" for path in diff.removed[:20])
        if len(diff.removed) > 20:
            lines.append(f"      - ... {len(diff.removed) - 20} more")
    return lines