Skip to content

livery.workshop._fold

livery.workshop._fold

The gate job's fold: a platform's shards become one check leg on the run's refs.

A platform's check leg may run as several shards, each on a runner of its own, each leaving a per-run ref of its own (livery.workshop.RUNS) with its timing row and the files its entries left. Every reader after the gate job's first entries, the coverage union, the verified stamp and the speed marks, reads one leg per platform, keyed by the platform leg (livery.workshop.platform_leg). fold_run is that fold, run once in the gate job before any of them: it lists the run's refs, groups them by platform, proves that the shards of a platform are one leg's work, puts one folded ref under the platform leg's key, and drops the shards' refs. A platform whose one ref is one shard of one is left as it is.

The proof is the plan's first ground-truth contract: nothing is assumed from a shard that did not write. A platform's shards fold only when every shard the matrix produces left a ref with a row and the record of what it partitioned, every row names one scope, every record names one item set and one list of durations rows read under one bound, no item ran in two shards, and the items the shards ran are together the set they partitioned. Anything else is red by name: the platform, the shards and what differs. A shard whose record says it ran as one shard of one fell open on the record and ran the whole set; when every shard of the platform did, the first stands for the platform. A platform none of whose shards left a record partitioned nothing: the gate skipped on every shard, the tree already proved or nothing affected, or no step that reads the shard ran. Its shards fold on their scopes and files alone, with no partition to prove, and the line says so; one shard with a record beside one without is red by name. Shards whose partitions differ and whose records name two attempts of the run are a rerun of some jobs: the refusal names the attempts and says to rerun the whole run.

The folded row is the first shard's timing row (its tasks, steps and waits) with every shard's package sums added and its seconds per item merged, keyed by the platform leg and named by the platform's own display name (livery.workshop._speed.platform_display), with the shards listed under shard. Every other file of the shards' refs folds through the ci.fold query (livery.workshop._queries.FOLDERS): the extension whose leg entry leaves the file answers with the folder; a file nobody answers for is red by name.

A fold that dies midway is re-run as its own recovery: the shards' refs stay until the folded ref is put, a put replaces the folded ref in place, and a shard's ref a failed drop left behind is known by the stamp the folded row recorded for it and dropped on the next run.

Functions:

Name Description
expected_shards

The shard labels the matrix produces, by platform leg, where it splits one.

fold_run

Fold each platform's shards on run's refs into one leg, or refuse by name.

Attributes:

Name Type Description
NAMED

NAMED module-attribute

NAMED = 5

expected_shards

expected_shards(root: Path, run: RunContext) -> dict[str, tuple[str, ...]]

The shard labels the matrix produces, by platform leg, where it splits one.

From [ci] pull-request-shards on a pull request and [ci] shards otherwise (livery.workshop.pull_request_shards, livery.workshop.check_shards). A platform of one shard is left out. Empty without a root contract, as a test workspace may be.

Source code in packages/workshop/src/livery/workshop/_fold.py
def expected_shards(root: Path, run: RunContext) -> dict[str, tuple[str, ...]]:
    """The shard labels the matrix produces, by platform leg, where it splits one.

    From ``[ci] pull-request-shards`` on a pull request and ``[ci]
    shards`` otherwise ([livery.workshop.pull_request_shards][],
    [livery.workshop.check_shards][]). A platform of one shard is left
    out. Empty without a root contract, as a test workspace may be.
    """
    from livery.workshop._points import check_shards, pull_request_shards

    if not (root / "workshop.toml").is_file():
        return {}
    legs = (
        pull_request_shards(root) if run.event == "pull_request" else check_shards(root)
    )
    by_platform: dict[str, list[str]] = defaultdict(list)
    for leg in legs:
        by_platform[platform_leg(leg)].append(leg)
    return {
        platform: tuple(labels)
        for platform, labels in by_platform.items()
        if len(labels) > 1
    }

fold_run

fold_run(root: Path, run: RunContext, *, folders: Mapping[str, Folder] | None = None) -> None

Fold each platform's shards on run's refs into one leg, or refuse by name.

Lists the run's per-run refs and groups them by platform leg. A platform with one ref that is one shard of one is left as it is. A platform the matrix splits, or whose refs are shards', is proved and folded (_fold_platform), its line printed; with no platform sharded, one line says so. folders fold the files beside the row, by file name; the listed extensions' answers to the ci.fold query when absent.

Raises:

Type Description
Failed

When the refs cannot be listed or read, when a platform's shards are not one leg's work, naming the platform, the shards and what differs, when a file nobody folds is on a shard's ref, and when the folded ref cannot be put; the shards' refs then stay for the next run.

Source code in packages/workshop/src/livery/workshop/_fold.py
def fold_run(
    root: Path, run: RunContext, *, folders: Mapping[str, Folder] | None = None
) -> None:
    """Fold each platform's shards on *run*'s refs into one leg, or refuse by name.

    Lists the run's per-run refs and groups them by platform leg. A
    platform with one ref that is one shard of one is left as it is.
    A platform the matrix splits, or whose refs are shards', is proved
    and folded (`_fold_platform`), its line printed; with no platform
    sharded, one line says so. *folders* fold the files beside the
    row, by file name; the listed extensions' answers to the
    ``ci.fold`` query when absent.

    Raises:
        Failed: When the refs cannot be listed or read, when a
            platform's shards are not one leg's work, naming the
            platform, the shards and what differs, when a file nobody
            folds is on a shard's ref, and when the folded ref cannot
            be put; the shards' refs then stay for the next run.
    """
    store = state_store(root)
    keys = RUNS.listed(store, run.run_id)
    if keys is None:
        fail(
            f"{RUNS.prefix}{run.run_id}/*: the remote could not be listed; the fold"
            " proves nothing from refs it cannot read"
        )
    refs = [_read_ref(store, key) for key in keys]
    expected = expected_shards(root, run)
    # Grouped by the platform's slug, since a ref without a row has no
    # label but its key, which is one; named by the matrix's spelling
    # or the first row's.
    groups: dict[str, list[_Ref]] = defaultdict(list)
    names: dict[str, str] = {git.slug(platform): platform for platform in expected}
    for ref in refs:
        platform = platform_leg(ref.label)
        groups[git.slug(platform)].append(ref)
        if ref.row is not None:
            names.setdefault(git.slug(platform), platform)
    if folders is None:
        try:
            folders = answer(root, FOLDERS)
        except QueryError as error:
            fail(str(error))
    folded = 0
    for key in sorted(set(groups) | set(names)):
        platform = names.get(key, key)
        held = groups.get(key, [])
        shards = [ref for ref in held if ref.sharded]
        wanted = expected.get(platform, ())
        if not shards and not wanted and not any(ref.folded for ref in held):
            continue
        folded += 1
        _fold_platform(store, run, platform, shards, held, wanted, folders)
    if not folded:
        print("  fold: no check leg ran as shards; nothing to fold")