Skip to content

livery.workshop._e2e_rows

livery.workshop._e2e_rows

What the newborn's store says a run did: the rows the loop's proofs read.

A proof asks the rows the legs wrote, never a job's log: a run's metrics row (each check leg's scope and the coverage section the gate job left), the verified series (the tree a run proved and what it rests on), a base's coverage record (which run measured each unit), the coverage marks, and a platform leg's durations series (the seconds per item a run's gate job recorded). A platform that ran as shards has one check row, the fold's, keyed by the platform leg with its shards listed under shard; the forge's rows of the shards stand beside it with their times alone, and no proof reads them as the leg. Every reader fails naming the series, the file and the run when the row it needs is not there, so a proof that fails says what the legs did not write.

The coverage rows are read by their ref and a few fields, since the loop lives in the base and the base imports no extension; their schema is the coverage extension's, and the loop leaves the base with phase 15 of the destination note.

Classes:

Name Description
ShardFacts

One shard of a folded check leg, as the fold listed it.

FoldFacts

The fold's record on a platform's check row: what its shards partitioned.

LegFacts

One job's half of a run's metrics row: what its gate ran.

RunFacts

A run's metrics row, read for a proof.

DurationsRow

What a platform leg's durations series holds for one run.

UnitRow

One unit of a base's coverage record: the run that measured it.

MarkRow

A package's current coverage mark: the newest row that set it.

Functions:

Name Description
run_facts

The metrics row of run_id, fetched fresh from the newborn's origin.

durations_row

The durations row run_id's gate job wrote on leg's platform, fetched fresh.

verified_by

The verified row run_id stamped; fails naming the series when none did.

verified_tree

The verified row of tree; fails naming the series when the record lacks it.

record_units

base's coverage record on leg: the current row of each unit, by path.

marks

The current mark per package, the newest row each; fails when unreadable.

Attributes:

Name Type Description
COVERAGE_RECORD
COVERAGE_MARKS

COVERAGE_RECORD module-attribute

COVERAGE_RECORD = git.Keyed('coverage', ('base', 'leg'))

COVERAGE_MARKS module-attribute

COVERAGE_MARKS = git.Series('coverage/marks', window=400, ci_only=False)

ShardFacts dataclass

ShardFacts(shard: str, leg: str, job: str, items: int)

One shard of a folded check leg, as the fold listed it.

Attributes:

Name Type Description
shard str

The shard, 1of2.

leg str

The shard's leg label (check-ubuntu-latest-3.14-1of2).

job str

The shard's job name, as the forge lists it.

items int

How many items the shard ran.

shard instance-attribute

shard: str

leg instance-attribute

leg: str

job instance-attribute

job: str

items instance-attribute

items: int

FoldFacts dataclass

FoldFacts(count: int, digest: str, rows: tuple[str, ...], shards: tuple[ShardFacts, ...])

The fold's record on a platform's check row: what its shards partitioned.

Attributes:

Name Type Description
count int

How many shards the platform ran as.

digest str

The digest of the set the shards partitioned.

rows tuple[str, ...]

The durations rows the shards partitioned by, newest first; empty when no row was there to read.

shards tuple[ShardFacts, ...]

Each shard, in order.

count instance-attribute

count: int

digest instance-attribute

digest: str

rows instance-attribute

rows: tuple[str, ...]

shards instance-attribute

shards: tuple[ShardFacts, ...]

LegFacts dataclass

LegFacts(job: str, scope: str, packages: tuple[str, ...], base_tree: str, leg: str, tasks: tuple[str, ...], shard: str = '', fold: FoldFacts | None = None)

One job's half of a run's metrics row: what its gate ran.

Attributes:

Name Type Description
job str

The job's name, as the forge lists it.

scope str

What the leg's gate ran: full, affected, nothing, verified, or unknown when the leg left no marker.

packages tuple[str, ...]

The suites a narrowed gate ran.

base_tree str

The tree a narrowed gate measured from; empty otherwise.

leg str

The leg's label, the key of its coverage record (check-ubuntu-latest-3.14).

tasks tuple[str, ...]

The tasks the job ran, by the names its trace recorded: the top-level tasks and each task's steps.

shard str

The shard whose own row this is (1of2), when the gate job's fold did not fold it into its platform's row; empty for a platform's row.

fold FoldFacts | None

What the fold recorded on a platform's row; None for a leg that ran as one shard of one.

job instance-attribute

job: str

scope instance-attribute

scope: str

packages instance-attribute

packages: tuple[str, ...]

base_tree instance-attribute

base_tree: str

leg instance-attribute

leg: str

tasks instance-attribute

tasks: tuple[str, ...]

shard class-attribute instance-attribute

shard: str = ''

fold class-attribute instance-attribute

fold: FoldFacts | None = None

RunFacts dataclass

RunFacts(run: str, jobs: dict[str, LegFacts], coverage: dict[str, float], shard_jobs: tuple[str, ...] = ())

A run's metrics row, read for a proof.

Attributes:

Name Type Description
run str

The run's id.

jobs dict[str, LegFacts]

Each job's facts, by the job's name; the forge's rows of a platform's shards are not among them.

coverage dict[str, float]

The coverage section the gate job left: the percent each unit was judged at, by the unit's path; empty when the gate job judged nothing.

shard_jobs tuple[str, ...]

The names of the forge's rows of a platform's shards, kept beside the fold's row with their times alone.

Methods:

Name Description
job

The one job whose name starts with prefix; fails naming the row.

run instance-attribute

run: str

jobs instance-attribute

jobs: dict[str, LegFacts]

coverage instance-attribute

coverage: dict[str, float]

shard_jobs class-attribute instance-attribute

shard_jobs: tuple[str, ...] = ()

job

job(prefix: str) -> LegFacts

The one job whose name starts with prefix; fails naming the row.

The answer is the leg's own row: for a platform that ran as shards, the fold's, never the forge's row of a shard. A row that is one shard's own, or shard rows with no fold's row among them, say the gate job's fold did not run, and the answer fails naming them.

Source code in packages/workshop/src/livery/workshop/_e2e_rows.py
def job(self, prefix: str) -> LegFacts:
    """The one job whose name starts with *prefix*; fails naming the row.

    The answer is the leg's own row: for a platform that ran as
    shards, the fold's, never the forge's row of a shard. A row
    that is one shard's own, or shard rows with no fold's row
    among them, say the gate job's fold did not run, and the
    answer fails naming them.
    """
    found = [facts for name, facts in self.jobs.items() if name.startswith(prefix)]
    shards = [name for name in self.shard_jobs if name.startswith(prefix)]
    if shards and not found:
        fail(
            f"run {self.run}: {_metrics.SERIES.ref}'s row names"
            f" {', '.join(shards)} and no folded row of their platform; the"
            " gate job's fold did not run"
        )
    own = [f"{facts.job} ({facts.shard})" for facts in found if facts.shard]
    if own:
        fail(
            f"run {self.run}: {_metrics.SERIES.ref}'s row holds"
            f" {', '.join(own)}, one shard's own row each, not the platform's;"
            " the gate job's fold did not run"
        )
    if len(found) != 1:
        fail(
            f"run {self.run}: {_metrics.SERIES.ref}'s row names"
            f" {len(found)} job(s) starting with {prefix!r}, expected one:"
            f" {', '.join(sorted(self.jobs))}"
        )
    return found[0]

DurationsRow dataclass

DurationsRow(ref: str, name: str, items: dict[str, float] | None)

What a platform leg's durations series holds for one run.

Attributes:

Name Type Description
ref str

The series' ref, durations/<platform leg> under the store's namespace.

name str

The row's file name, the run's.

items dict[str, float] | None

The seconds per item the row holds; None when the series holds no row of the run.

ref instance-attribute

ref: str

name instance-attribute

name: str

items instance-attribute

items: dict[str, float] | None

UnitRow dataclass

UnitRow(unit: str, run: str, sha: str, closure: str)

One unit of a base's coverage record: the run that measured it.

Attributes:

Name Type Description
unit str

The unit's path (packages/loop-echo, or tests).

run str

The run that measured it.

sha str

The commit that run checked out.

closure str

The closure identity the measurement is keyed by.

unit instance-attribute

unit: str

run instance-attribute

run: str

sha instance-attribute

sha: str

closure instance-attribute

closure: str

MarkRow dataclass

MarkRow(package: str, value: float, kind: str, by: str)

A package's current coverage mark: the newest row that set it.

Attributes:

Name Type Description
package str

The package path.

value float

The mark, in percent.

kind str

first, ratchet or accept.

by str

The run id or the person that wrote the row.

package instance-attribute

package: str

value instance-attribute

value: float

kind instance-attribute

kind: str

by instance-attribute

by: str

run_facts

run_facts(root: Path, run_id: int | str) -> RunFacts

The metrics row of run_id, fetched fresh from the newborn's origin.

Fails naming the series and the file when the row is not there: the gate job's collect did not write it.

Source code in packages/workshop/src/livery/workshop/_e2e_rows.py
def run_facts(root: Path, run_id: int | str) -> RunFacts:
    """The metrics row of *run_id*, fetched fresh from the newborn's origin.

    Fails naming the series and the file when the row is not there:
    the gate job's collect did not write it.
    """
    store = state_store(root)
    name = _metrics.run_file(str(run_id))
    with git.remote_snapshot(store, fetch=(_metrics.SERIES.name,)):
        found = _metrics.SERIES.rows(store, only=lambda file: file == name)
    if found.failed:
        fail(f"run {run_id}: {_metrics.SERIES.ref} could not be read: {found.reason}")
    row = next((item for item in found.rows if item.name == name), None)
    if row is None:
        fail(
            f"run {run_id}: no row {name} on {_metrics.SERIES.ref}; the gate job's"
            " collect did not write the run's row"
        )
    jobs: dict[str, LegFacts] = {}
    shard_jobs: list[str] = []
    for job, data in dict(row.data.get("jobs") or {}).items():
        if forge_shard_row(str(job), data):
            shard_jobs.append(str(job))
            continue
        scope = data.get("scope") or {}
        tasks = (*data.get("tasks", {}), *data.get("task_steps", {}))
        leg = str(scope.get("leg") or "")
        own, fold = _shard_facts(leg, data.get("shard"))
        jobs[str(job)] = LegFacts(
            job=str(job),
            scope=str(scope.get("scope", "unknown")),
            packages=tuple(str(p) for p in scope.get("packages") or ()),
            base_tree=str(scope.get("base_tree") or ""),
            leg=leg,
            tasks=tuple(str(t) for t in tasks),
            shard=own,
            fold=fold,
        )
    section = (row.data.get("sections") or {}).get("coverage") or {}
    coverage = {
        str(path): float(value)
        for path, value in dict(section.get("packages") or {}).items()
    }
    return RunFacts(str(run_id), jobs, coverage, tuple(shard_jobs))

durations_row

durations_row(root: Path, run_id: int | str, leg: str) -> DurationsRow

The durations row run_id's gate job wrote on leg's platform, fetched fresh.

The series is the platform leg's (livery.workshop.platform_leg), so a shard's label reads its platform's. Fails naming the series when it cannot be read; a series without the run's row answers items as None.

Source code in packages/workshop/src/livery/workshop/_e2e_rows.py
def durations_row(root: Path, run_id: int | str, leg: str) -> DurationsRow:
    """The durations row *run_id*'s gate job wrote on *leg*'s platform, fetched fresh.

    The series is the platform leg's ([livery.workshop.platform_leg][]),
    so a shard's label reads its platform's. Fails naming the series
    when it cannot be read; a series without the run's row answers
    ``items`` as ``None``.
    """
    series = _shard.DURATIONS.series(_shard.platform_leg(leg))
    name = _metrics.run_file(str(run_id))
    store = state_store(root)
    with git.remote_snapshot(store, fetch=(series.name,)):
        found = series.rows(store, only=name.__eq__)
    if found.failed:
        fail(f"run {run_id}: {series.ref} could not be read: {found.reason}")
    row = next((item for item in found.rows if item.name == name), None)
    if row is None:
        return DurationsRow(series.ref, name, None)
    items = {
        str(each): float(seconds)
        for each, seconds in dict(row.data.get("items") or {}).items()
        if isinstance(seconds, int | float) and not isinstance(seconds, bool)
    }
    return DurationsRow(series.ref, name, items)

verified_by

verified_by(root: Path, run_id: int | str) -> Verified

The verified row run_id stamped; fails naming the series when none did.

Source code in packages/workshop/src/livery/workshop/_e2e_rows.py
def verified_by(root: Path, run_id: int | str) -> _verified.Verified:
    """The verified row *run_id* stamped; fails naming the series when none did."""
    rows, why = _verified.proved(root)
    if why:
        fail(f"run {run_id}: {_verified.SERIES.ref} could not be read: {why}")
    for entry in rows.values():
        if entry.run == str(run_id):
            return entry
    fail(
        f"run {run_id}: no row on {_verified.SERIES.ref} names it; the gate job's"
        " stamp did not record the tree as proved"
    )

verified_tree

verified_tree(root: Path, tree: str) -> Verified

The verified row of tree; fails naming the series when the record lacks it.

Source code in packages/workshop/src/livery/workshop/_e2e_rows.py
def verified_tree(root: Path, tree: str) -> _verified.Verified:
    """The verified row of *tree*; fails naming the series when the record lacks it."""
    entry, why = _verified.record(root, tree)
    if why:
        fail(f"tree {tree[:12]}: {_verified.SERIES.ref} could not be read: {why}")
    if entry is None:
        fail(f"tree {tree[:12]}: no row on {_verified.SERIES.ref}; nothing proved it")
    return entry

record_units

record_units(root: Path, base: str, leg: str) -> dict[str, UnitRow]

base's coverage record on leg: the current row of each unit, by path.

A current row is named by its unit's slug alone; a kept row, named by its moment and closure as well, and a fixed-facts file under facts/ are not units of the record. The rows are read raw and four fields taken, so the loop carries neither the coverage extension's schema number nor its parser. Fails naming the ref when it cannot be read; an absent record is empty.

Source code in packages/workshop/src/livery/workshop/_e2e_rows.py
def record_units(root: Path, base: str, leg: str) -> dict[str, UnitRow]:
    """*base*'s coverage record on *leg*: the current row of each unit, by path.

    A current row is named by its unit's slug alone; a kept row, named
    by its moment and closure as well, and a fixed-facts file under
    ``facts/`` are not units of the record. The rows are read raw and
    four fields taken, so the loop carries neither the coverage
    extension's schema number nor its parser. Fails naming the ref when
    it cannot be read; an absent record is empty.
    """
    series = COVERAGE_RECORD.series(base, leg)
    store = state_store(root)

    def current(name: str) -> bool:
        return name.count(".") == 1 and "/" not in name

    with git.remote_snapshot(store, fetch=(series.name,)):
        found = git.read(store, series.ref, only=current)
    if found.failed:
        fail(f"{series.ref} could not be read: {found.reason}")
    units: dict[str, UnitRow] = {}
    for _name, text in sorted((found.files or {}).items()):
        try:
            data = json.loads(text)
        except ValueError:
            continue
        if not isinstance(data, dict):
            continue
        unit = str(data.get("unit") or "")
        if unit:
            units[unit] = UnitRow(
                unit,
                str(data.get("run") or ""),
                str(data.get("sha") or ""),
                str(data.get("closure") or ""),
            )
    return units

marks

marks(root: Path) -> dict[str, MarkRow]

The current mark per package, the newest row each; fails when unreadable.

Source code in packages/workshop/src/livery/workshop/_e2e_rows.py
def marks(root: Path) -> dict[str, MarkRow]:
    """The current mark per package, the newest row each; fails when unreadable."""
    store = state_store(root)
    with git.remote_snapshot(store, fetch=(COVERAGE_MARKS.name,)):
        found = COVERAGE_MARKS.rows(store)
    if found.failed:
        fail(f"{COVERAGE_MARKS.ref} could not be read: {found.reason}")
    current: dict[str, MarkRow] = {}
    for row in found.rows:
        data: dict[str, Any] = row.data
        package = str(data.get("package") or "")
        if not package or package in current:
            continue
        try:
            value = float(data["value"])
        except (KeyError, TypeError, ValueError):
            continue
        current[package] = MarkRow(
            package, value, str(data.get("kind") or ""), str(data.get("by") or "")
        )
    return current