Skip to content

livery.extensions.coverage._union

livery.extensions.coverage._union

The gate job's half of coverage: every leg's parts and the records', as one union.

The gate job's judge, coverage.union, reads what each check leg left on its per-run ref, carries every unit a leg did not measure from the records, merges every unit's parts by the line format's rules, and writes the merged files to the workspace's union file, which the floors then judge (combine_union). It writes the run's units back onto the branch's record or main's.

Functions:

Name Description
expected_legs

The check legs the gate produces, and the ones run's event runs.

combine_union

Union the run's legs' parts with the records; the packages it judges.

Attributes:

Name Type Description
ABSENT

ABSENT module-attribute

ABSENT = 'absent'

expected_legs

expected_legs(root: Path, run: RunContext) -> tuple[list[str], list[str]]

The check legs the gate produces, and the ones run's event runs.

A pull request runs the legs [ci] pull-request-runners names, the first runner's alone while that key is absent, unless it carries livery.workshop.EVERY_RUNNER_LABEL; every other event runs them all. Both empty without a root contract, as a test workspace may be.

Source code in packages/extensions/coverage/src/livery/extensions/coverage/_union.py
def expected_legs(root: Path, run: RunContext) -> tuple[list[str], list[str]]:
    """The check legs the gate produces, and the ones *run*'s event runs.

    A pull request runs the legs ``[ci] pull-request-runners`` names,
    the first runner's alone while that key is absent, unless it
    carries [livery.workshop.EVERY_RUNNER_LABEL][]; every other event
    runs them all. Both empty without a root contract, as a test
    workspace may be.
    """
    from livery.workshop import EVERY_RUNNER_LABEL, check_legs, pull_request_legs

    if not (root / "workshop.toml").is_file():
        return [], []
    every = check_legs(root)
    narrowed = run.event == "pull_request" and EVERY_RUNNER_LABEL not in run.labels
    due = pull_request_legs(root) if narrowed else every
    return every, due

combine_union

combine_union(root: Path, packages: tuple[Package, ...]) -> tuple[Package, ...]

Union the run's legs' parts with the records; the packages it judges.

Each check leg's per-run ref carries the scope its gate ran and the units it measured, each unit's parts in the line format. A full leg measured every suite, a narrowed or measured leg the suites its marker names; a leg whose gate skipped (a tree already proved, or nothing affected) measured nothing. Every unit a leg did not measure is carried from a record on that leg, the branch's own before main's, at the unit's current closure identity, so the union is the same global union a full run produces and every package is judged. A unit no record can supply refuses by name: a leg skips a suite only when a record holds it, so a miss here is a leg that narrowed without the records, or a record moved on since, never a smaller union. Every unit's parts merge by the line format's rules: one file's readings merge at the content the checkout holds and by one measurer, a reading at an earlier content the checkout's object store holds is carried across the change line by line, and every reading left out is named. The merged files go to the union file, which the floors read; no union file outlives a run that did not write one.

The union is written back per leg: a pull request's run onto its branch's record, main's run onto main's. The leg's fresh units replace their rows, the rows carried from the other record are copied in, the target's own carried rows stand, and the rows of units that no longer exist go. On main's run the branch is the one the verified record names for the tree, so the merge copies the branch's record into main's. Each row written numbers the tests that reached its lines as the row it replaces does, and a unit the target lacks as the other record's row of it, so a branch's rows number them as main's do; each appends every test the run's rows of its unit name on any leg, so every leg's row numbers a test alike. A record that cannot be written prints why: the next run reruns what it cannot reuse, the safe direction.

A leg the contract names and a pull request did not run, one outside [ci] pull-request-runners, is carried whole from the records, so the narrowed run judges the same global union and a line only another platform reaches stays covered by main's measurement of it. A file the change touched keeps that leg's reading at every line the change left in place, carried across the diff; its reading on a line the change touched is dropped by name, so a changed line only that platform reaches is red until a run of every leg measures it.

A leg the run runs and that left no ref is red by name, never carried: it died before its write, or its write was refused, and the union of the legs that did write would judge the floors without the lines only its platform reaches.

Refuses by name when no leg left its parts, when a leg the run runs left no ref, when a leg's ref carries no readable file, when a leg names no scope the union reads, when a leg that ran a suite left its parts out, when a unit is neither the run's nor any record's, and when a part is off the line format, naming where it came from and the key.

Returns:

Type Description
Package

The judged packages, in packages order; empty when the

...

workspace has no unit to union, and then no union file is

tuple[Package, ...]

written.

Source code in packages/extensions/coverage/src/livery/extensions/coverage/_union.py
def combine_union(root: Path, packages: tuple[Package, ...]) -> tuple[Package, ...]:
    """Union the run's legs' parts with the records; the packages it judges.

    Each check leg's per-run ref carries the scope its gate ran and
    the units it measured, each unit's parts in the line format. A full
    leg measured every suite, a narrowed or measured leg the suites its
    marker names; a leg whose gate skipped (a tree already proved, or
    nothing affected) measured nothing. Every unit a leg did not
    measure is carried from a record on that leg, the branch's own
    before main's, at the unit's current closure identity, so the union
    is the same global union a full run produces and every package is
    judged. A unit no record can supply
    refuses by name: a leg skips a suite only when a record holds it,
    so a miss here is a leg that narrowed without the records, or a
    record moved on since, never a smaller union. Every unit's parts
    merge by the line format's rules: one file's readings merge at the
    content the checkout holds and by one measurer, a reading at an
    earlier content the checkout's object store holds is carried
    across the change line by line, and every reading left out is
    named. The merged files go to the union file, which the floors
    read; no union file outlives a run that did not write one.

    The union is written back per leg: a pull request's run onto its
    branch's record, main's run onto main's. The leg's fresh units
    replace their rows, the rows carried from the other record are
    copied in, the target's own carried rows stand, and the rows of
    units that no longer exist go. On main's run the branch is the one
    the verified record names for the tree, so the merge copies the
    branch's record into main's. Each row written numbers the tests
    that reached its lines as the row it replaces does, and a unit the
    target lacks as the other record's row of it, so a branch's rows
    number them as main's do; each appends every test the run's rows
    of its unit name on any leg, so every leg's row numbers a test
    alike. A record that cannot be written prints why: the next run
    reruns what it cannot reuse, the safe direction.

    A leg the contract names and a pull request did not run, one
    outside ``[ci] pull-request-runners``, is carried whole from the
    records, so the narrowed run judges the same global union and a
    line only another platform reaches stays covered by main's
    measurement of it. A file the change touched keeps that leg's
    reading at every line the change left in place, carried across the
    diff; its reading on a line the change touched is dropped by name,
    so a changed line only that platform reaches is red until a run of
    every leg measures it.

    A leg the run runs and that left no ref is red by name, never
    carried: it died before its write, or its write was refused, and
    the union of the legs that did write would judge the floors
    without the lines only its platform reaches.

    Refuses by name when no leg left its parts, when a leg the run
    runs left no ref, when a leg's ref carries no readable file, when
    a leg names no scope the union reads, when a leg that ran a suite
    left its parts out, when a unit is neither the run's nor any
    record's, and when a part is off the line format, naming where it
    came from and the key.

    Returns:
        The judged packages, in *packages* order; empty when the
        workspace has no unit to union, and then no union file is
        written.
    """
    from livery.extensions.coverage._parts import UNION
    from livery.extensions.coverage._record import (
        RUN_FILE,
        Leg,
        closure_id,
        put_record,
        row_name,
        run_contexts,
        run_legs,
    )
    from livery.workshop import LegScope, ci_run

    # The floors read the union file this run writes, and no other.
    (root / UNION).unlink(missing_ok=True)
    run = ci_run()
    if run is None:
        fail(
            "the union reads the run's per-run refs, and outside CI there is no"
            " run: the legs' parts are unioned and judged inside CI alone"
        )
    legs, why = run_legs(root, run)
    if why:
        fail(f"the run's per-run refs could not be read ({why}); nothing to union")
    if not legs:
        fail(
            f"no check leg left its parts on run {run.run_id}'s refs: every leg"
            f" puts its scope and its parts there ({RUN_FILE}) at its end, and"
            " a missing put is a red leg, never a smaller union"
        )
    every, due = expected_legs(root, run)
    present = {leg.label for leg in legs}
    for label in every:
        if label in present or label in due:
            continue
        print(
            f"  coverage: leg {label} did not run on this pull request"
            " ([ci] pull-request-runners); every suite carried from the records"
        )
        legs.append(Leg(label, label, ABSENT, (), {}))
    units = units_of(root, packages)
    keys = {unit.path: closure_id(root, packages, unit) for unit in units}
    bases, target = _record_bases(root, run)
    # Every unit the union merges, with where its parts came from.
    merged: list[tuple[Unit, str]] = []
    reused = 0
    ran = 0
    writes: list[
        tuple[
            str,
            dict[str, Unit],
            dict[str, tuple[str, Unit]],
            Record,
            dict[str, Unit],
        ]
    ] = []
    for leg in sorted(legs, key=lambda item: item.key):
        if leg.why:
            fail(
                f"leg {leg.key}: {leg.why}; the leg died before its parts, or"
                " its put failed, and a missing put is a red leg, never a"
                " smaller union"
            )
        if not leg.label:
            fail(
                f"leg {leg.key} names no label, so no record can supply what it"
                " skipped; the job runner sets WORKSHOP_LEG for every leg"
            )
        if leg.scope in ("verified", "nothing"):
            print(f"  coverage: leg {leg.label} ran {leg.scope!r}: no suite, no data")
        elif leg.scope == ABSENT:
            pass
        elif leg.scope in ("full", "affected", MEASURED):
            ran_scope = LegScope(leg.scope, leg.packages)
            expected = suites_that_ran(ran_scope, root, packages)
            missing = [unit.path for unit in expected if unit.path not in leg.units]
            if missing:
                fail(
                    f"leg {leg.label} ran {leg.scope!r} and its ref lacks"
                    f" {', '.join(missing)}: a leg that ran a suite puts its"
                    " parts, and a missing put is a red leg, never a smaller union"
                )
            ran += 1
            merged.extend(
                (unit, f"leg {leg.label}'s {path}")
                for path, unit in sorted(leg.units.items())
            )
        else:
            fail(
                f"leg {leg.label}: its ref names a scope the union does not read"
                f" ({leg.scope!r}); the check legs leave full, affected, nothing,"
                " verified, or measured"
            )
        pending = [unit for unit in units if unit.path not in leg.units]
        held: dict[str, Record] = {}
        carried: dict[str, tuple[str, Unit]] = {}
        for suite in pending:
            key = keys[suite.path]
            states: list[str] = []
            kept = ""
            for base in bases:
                record = _read_record(root, held, base, leg.label)
                row = record.at(suite.path, key)
                current = record.units.get(suite.path)
                if row is not None:
                    carried[suite.path] = (base, row)
                    if row is not current:
                        # The leg read the current row; a write since
                        # moved it on and kept it at this closure.
                        kept = ", a row kept after its record moved on"
                    break
                if leg.scope == ABSENT and current is not None:
                    # The leg did not run on this pull request, so no
                    # measurement of this closure exists on its
                    # platform: the record's last one stands in, and
                    # the merge point measures it afresh.
                    carried[suite.path] = (base, current)
                    kept = (
                        f", at closure {current.closure[:12]} since the leg did not run"
                    )
                    break
                states.append(
                    f"{base} only at {current.closure[:12]}"
                    if current is not None
                    else f"{base} none"
                )
            else:
                if leg.scope == ABSENT:
                    # The leg did not run and no record of any base has
                    # ever measured this suite on its platform: a package
                    # born on this branch. Nothing stands in, the legs that
                    # ran judge it, and the merge point measures it here.
                    print(
                        f"  coverage: leg {leg.label} has no record of"
                        f" {suite.path} yet; the merge point measures it"
                    )
                    continue
                fail(
                    f"leg {leg.label}: no record holds {suite.path} at closure"
                    f" {key[:12]} ({', '.join(states)}). A leg skips a suite"
                    " only when a record holds it at the suite's closure, and"
                    " a record keeps a row it moved on from for a day, so this"
                    " leg narrowed without the records, or the run outlived"
                    " that day; a run of the full gate measures it again."
                )
            base, row = carried[suite.path]
            merged.append((row, f"{base}'s record on {leg.label}, {suite.path}"))
            reused += 1
            print(
                f"  coverage: {suite.path} on {leg.label}: reused from run"
                f" {row.run} ({len(row.files)} files), {base}'s record{kept}"
            )
        if target:
            own = _read_record(root, held, target, leg.label)
            like = _numbered_like(root, held, bases, target, leg.label)
            writes.append((leg.label, dict(leg.units), carried, own, like))
    unput = [label for label in due if label not in present]
    if unput:
        which = f"leg {unput[0]}" if len(unput) == 1 else f"legs {', '.join(unput)}"
        fail(
            f"{which} left no parts on run {run.run_id}'s refs: every leg the run"
            f" runs puts its scope and its parts there ({RUN_FILE}) at its end, so"
            " a leg without them died before its write or its write was refused,"
            " and its own log says which; a missing put is a red leg, never a"
            " smaller union"
        )
    if not merged:
        print("  coverage: no unit to union; nothing judged")
        return ()
    _write_merged(root, merged)
    print(f"  coverage: the union of {ran} leg(s) and {reused} reused suite(s)")
    if not target:
        print(
            f"  coverage record: not written, a {run.event or 'local'} run"
            " that names no branch has no record of its own"
        )
        return tuple(packages)
    # Two runs of one branch overlap when a push follows a push: the
    # older run's gate job can finish last, and its rows would replace
    # the newer run's. Only the run of the branch's current head writes.
    moved, head = _moved_on(root, run, target)
    if moved:
        print(
            f"  coverage record: {target}'s head moved on to {moved[:12]} since"
            f" this run's {head[:12]}; not written, the newer run writes"
        )
        return tuple(packages)
    # The rows carried from another record are copied in; the target's
    # own carried rows already stand.
    puts = [
        (
            label,
            fresh,
            carried,
            own,
            like,
            {
                **fresh,
                **{
                    path: row for path, (base, row) in carried.items() if base != target
                },
            },
        )
        for label, fresh, carried, own, like in writes
    ]
    # Every leg's row of a unit numbers its tests alike: each write
    # appends every test the run's rows of the unit name on any leg.
    named = run_contexts([rows for *_, rows in puts])
    for label, fresh, carried, own, like, rows in puts:
        stale = own.stale(keys)
        # A stale row a new row replaces by name (a row of an older
        # shape, say) is replaced, not removed: only the rest go.
        removed = [name for name in stale if name not in {row_name(p) for p in rows}]
        if not rows and not stale:
            print(
                f"  coverage record: {target}/{label}: unchanged, {len(carried)}"
                " carried"
            )
            continue
        why = put_record(
            root,
            run,
            leg=label,
            fresh=rows,
            remove=stale,
            base=target,
            held=own,
            numbered_like=like,
            run_contexts=named,
        )
        if why:
            print(
                f"  coverage record: {target}/{label} not written ({why}); the"
                " next run reruns what it cannot reuse"
            )
            continue
        print(
            f"  coverage record: {target}/{label}: {len(fresh)} fresh,"
            f" {len(carried)} carried, {len(removed)} removed"
        )
    return tuple(packages)