Skip to content

livery.toolroom.bench._artifacts

livery.toolroom.bench._artifacts

Record a version's release artifacts, one per host, by downloading them.

A tool read from a forge tier has a release with one asset per host, and node publishes one build per host under its own release index. The record needs the asset's URL and its sha256 for every host before the store can supply the version, and the only honest hash is the hash of bytes the bench had: each asset is picked for its host from the release's asset list, downloaded once, hashed, landed in the bench's store so the ingest verification stages it without a second download, and written on the version line as artifacts[host].

A host the release publishes no asset for is absent from the version: not a hole, not a refusal. A tool on a package tier (PyPI, npm) has no artifacts to record, since its installer supplies it.

fm toolroom.artifacts <tool> records a version already read; the assembler calls the same function for every fresh version of a forge-tier tool.

Classes:

Name Description
ArtifactError

A version's artifacts could not be recorded; the message says why.

Recorded

What one version's artifact step found.

Functions:

Name Description
forge_of

The forge to list assets from, or empty for a tier that is not a forge.

lists_assets

Whether the driver's tier publishes per-host assets to record.

record_version

record with version's artifacts for every host that has an asset.

Attributes:

Name Type Description
FORGE_TIERS

The provision tiers whose releases publish per-host assets on a forge.

ASSET_TIERS

Every tier a version's artifacts can be recorded from: the forges,

FORGE_TIERS module-attribute

FORGE_TIERS = ('github', 'gitlab', 'gitea', 'bun')

The provision tiers whose releases publish per-host assets on a forge.

ASSET_TIERS module-attribute

ASSET_TIERS = (*FORGE_TIERS, 'nodejs', 'dotnet')

Every tier a version's artifacts can be recorded from: the forges, node's own release index, and Microsoft's for the .NET SDK.

ArtifactError

Bases: Exception

A version's artifacts could not be recorded; the message says why.

Recorded dataclass

Recorded(version: str, found: dict[str, tuple[str, int]], absent: tuple[str, ...])

What one version's artifact step found.

Attributes:

Name Type Description
version str

The version recorded.

found dict[str, tuple[str, int]]

Per host, the asset's name and its size in bytes.

absent tuple[str, ...]

The hosts the release publishes no asset for.

Methods:

Name Description
lines

The step as report lines, one per host.

version instance-attribute

version: str

found instance-attribute

found: dict[str, tuple[str, int]]

absent instance-attribute

absent: tuple[str, ...]

lines

lines(name: str) -> list[str]

The step as report lines, one per host.

Source code in packages/toolroom-bench/src/livery/toolroom/bench/_artifacts.py
def lines(self, name: str) -> list[str]:
    """The step as report lines, one per host."""
    out = [f"  {name} {self.version}: {len(self.found)} artifact(s) recorded"]
    for host, (asset, size) in self.found.items():
        out.append(f"    {host}: {asset} ({size // 1024} KB)")
    if self.absent:
        out.append(f"    no asset for {', '.join(self.absent)}")
    return out

forge_of

forge_of(driver: Driver) -> str

The forge to list assets from, or empty for a tier that is not a forge.

Source code in packages/toolroom-bench/src/livery/toolroom/bench/_artifacts.py
def forge_of(driver: Driver) -> str:
    """The forge to list assets from, or empty for a tier that is not a forge."""
    kind = driver.provision.kind
    if kind not in FORGE_TIERS:
        return ""
    return kind if kind in ("gitlab", "gitea") else "github"

lists_assets

lists_assets(driver: Driver) -> bool

Whether the driver's tier publishes per-host assets to record.

Source code in packages/toolroom-bench/src/livery/toolroom/bench/_artifacts.py
def lists_assets(driver: Driver) -> bool:
    """Whether the driver's tier publishes per-host assets to record."""
    return driver.provision.kind in ASSET_TIERS

record_version

record_version(record: Record, driver: Driver, version: str, tag: str, *, store: Store, hosts: tuple[str, ...] = HOSTS) -> tuple[Record, Recorded]

record with version's artifacts for every host that has an asset.

Lists the release at tag (the version when the tag is empty), picks one asset per host, downloads and hashes each, lands the bytes in store, and writes the artifacts on the version line. Hosts already recorded on the version are left as they are. The record's hosts gains every host recorded. The record validates itself as it is rebuilt, so a layout the record lacks refuses here, naming what is missing, and never on disk.

Raises:

Type Description
ArtifactError

for a driver on a package tier, a version the record does not track, a release with no asset for any host, or a host whose deployment does not resolve whole.

Source code in packages/toolroom-bench/src/livery/toolroom/bench/_artifacts.py
def record_version(
    record: Record,
    driver: Driver,
    version: str,
    tag: str,
    *,
    store: Store,
    hosts: tuple[str, ...] = HOSTS,
) -> tuple[Record, Recorded]:
    """*record* with *version*'s artifacts for every host that has an asset.

    Lists the release at *tag* (the version when the tag is empty),
    picks one asset per host, downloads and hashes each, lands the
    bytes in *store*, and writes the artifacts on the version line.
    Hosts already recorded on the version are left as they are. The
    record's `hosts` gains every host recorded. The record validates
    itself as it is rebuilt, so a layout the record lacks refuses
    here, naming what is missing, and never on disk.

    Raises:
        ArtifactError: for a driver on a package tier, a version the
            record does not track, a release with no asset for any
            host, or a host whose deployment does not resolve whole.
    """
    from livery.toolroom.bench import _provision
    from livery.toolroom.store import Artifact, RecordError

    if not lists_assets(driver):
        raise ArtifactError(
            f"{record.name} is read from the {driver.provision.kind!r} tier, which"
            " has no release assets to record; its installer supplies it"
        )
    if version not in {delta.version for delta in record.deltas}:
        raise ArtifactError(f"{record.name} does not track {version}")
    universal = ""
    if driver.provision.asset:
        # One file for every host, at a versioned address the release
        # lists no asset for: no listing is asked for and no host is
        # absent.
        universal = driver.provision.asset.format(
            repo=driver.provision.repo, tag=tag or version
        )
        assets = [(universal.rsplit("/", 1)[-1], universal)]
    elif driver.provision.kind in ("nodejs", "dotnet"):
        from livery.toolroom.bench._toolfetch import dotnet_assets, nodejs_assets

        assets = (
            nodejs_assets(version)
            if driver.provision.kind == "nodejs"
            else dotnet_assets(version)
        )
    else:
        forge = forge_of(driver)
        assets = _assets(record.name, forge, driver.provision.repo, version, tag)
    delta = record.delta_for(version)
    artifacts = dict(delta.artifacts)
    found: dict[str, tuple[str, int]] = {}
    absent: list[str] = []
    for host in hosts:
        if host in artifacts:
            continue
        try:
            asset, url = (
                assets[0] if universal else _provision._pick_asset(assets, host=host)
            )
        except _provision.ProvisionError:
            absent.append(host)
            continue
        try:
            data = _fetch(url)
        except OSError as error:
            raise ArtifactError(f"{record.name} {version}: {url}: {error}") from None
        digest = store.objects.put(data)
        artifacts[host] = Artifact(url, digest.encoded)
        found[host] = (asset, len(data))
    if not found and not delta.artifacts:
        raise ArtifactError(
            f"{record.name} {version}: the release at {tag or version} has no asset"
            f" for any of {', '.join(hosts)}; it has {_names(assets)}"
        )
    ordered = {host: artifacts[host] for host in HOSTS if host in artifacts}
    deltas = tuple(
        replace(d, artifacts=ordered) if d.version == version else d
        for d in record.deltas
    )
    known = tuple(host for host in HOSTS if host in record.hosts or host in ordered)
    try:
        # A record validates itself on construction: a host whose deployment
        # does not resolve whole is refused here, with the field named.
        updated = replace(record, hosts=known, deltas=deltas)
    except RecordError as error:
        raise ArtifactError(str(error)) from None
    return updated, Recorded(version, found, tuple(absent))