Skip to content

livery.workshop._metrics

livery.workshop._metrics

Timing rows on the CI state store, and the reader that renders them.

One row per job per run, one file per run on the metrics series of the state store (livery.git), named by zero-padded run id so that name order is time order. A row is end to end, not gate-only: the queue wait, the job's wall, and every step's wall as the forge times them, beside the per-task durations and lane waits the leg's own trace recorded, and the test time per package summed from the trace's pytest slices.

Three writers, all inside CI. Each check leg ends with put_leg: it reads the trace the profiled gate wrote and puts the task half of its row on the leg's per-run ref, in one write with every file the leg's other entries left in its spool (LEG_SPOOL); a leg that ran as one shard of several carries the record it left of what it partitioned in the row, under shard; a write that fails is a red leg. The run's gate job first folds each platform's shards into one ref (livery.workshop._fold), then runs collect: it reads the per-run refs, asks the forge for the run's jobs, joins the two halves, folds in the sections the job's judges left (RUN_ROW), puts the run's file on the series under its window, and writes each check leg's seconds per item onto its platform leg's series of the livery.workshop.DURATIONS family; it fails open loudly, a printed reason and never a red job. The run's last green job runs drop_halves: the per-run refs go once every job that writes a half has completed green, so a re-run of a red gate job still finds what its legs put, and a point with no gate job, where every job puts a half and collects, keeps each job's half until the last one has folded it in. Local runs read: render is what fm ci.timings prints, the latest value, p50, and p90 per metric over the window, a platform that ran as shards as one section with a line per shard, the movers against a base window, and each section's trend.

Functions:

Name Description
run_ref

The per-run ref one leg of run writes.

leave_leg_file

Leave data in the leg's spool as name, for the leg's one write.

leg_files

The files the leg's entries left in its spool, by name; or why one cannot ride.

leave_section

Leave a section of the run's row, a number per package, for the gate job.

sections

The sections the gate job's entries left, by name; a malformed file is left out.

leg_row

The task half of job's row from its trace; (row, "") or (None, reason).

put_leg

Put job's task half and the leg's spool on its per-run ref, in one write.

run_file

The series file for run_id: zero-padded when numeric, so names sort by time.

collect

Join the legs' halves with the forge's times into the run's file; what happened.

drop_halves

Drop the run's per-run refs once every half is in and nothing is red.

gate_steps

The gate's steps of a leg's row by task key, in milliseconds.

tests_ms

The test time summed over packages, a row's packages, in milliseconds.

first_shard_head

The item seconds the first shard's unsplit steps stand for on row.

render

The timings, rendered: per job and metric the latest, p50, p90, and the movers.

Attributes:

Name Type Description
SCHEMA
SERIES
RUNS Keyed
ROW_FILE
LEG_SPOOL
RUN_ROW
SLOWEST
GATE_STEPS
TEST_PHASE

SCHEMA module-attribute

SCHEMA = 1

SERIES module-attribute

SERIES = git.Series('metrics', window=300, schema=SCHEMA)

RUNS module-attribute

RUNS: Keyed = git.Keyed('run', ('run', 'leg'), schema=SCHEMA, stale_after=timedelta(hours=6))

ROW_FILE module-attribute

ROW_FILE = 'row.json'

LEG_SPOOL module-attribute

LEG_SPOOL = '.workshop/leg'

RUN_ROW module-attribute

RUN_ROW = '.workshop/run-row'

SLOWEST module-attribute

SLOWEST = 20

GATE_STEPS module-attribute

GATE_STEPS = re.compile('^check/[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+$')

TEST_PHASE module-attribute

TEST_PHASE = 'check/test.'

run_ref

run_ref(run: RunContext, leg: str) -> str

The per-run ref one leg of run writes.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def run_ref(run: RunContext, leg: str) -> str:
    """The per-run ref one *leg* of *run* writes."""
    return RUNS.series(run.run_id, leg).ref

leave_leg_file

leave_leg_file(root: Path, name: str, data: Mapping[str, Any]) -> Path

Leave data in the leg's spool as name, for the leg's one write.

The leg's last entry, ci.leg, puts every file the spool holds on the leg's per-run ref, a series of livery.workshop.RUNS, beside its timing row and under its own name, where the gate job reads it. That write refuses a file named row.json, the timing row's own name. A file left twice under one name keeps the last data.

Returns:

Type Description
Path

The path of the file left under root.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def leave_leg_file(root: Path, name: str, data: Mapping[str, Any]) -> Path:
    """Leave *data* in the leg's spool as *name*, for the leg's one write.

    The leg's last entry, ``ci.leg``, puts every file the spool holds
    on the leg's per-run ref, a series of [livery.workshop.RUNS][],
    beside its timing row and under its own name, where the gate job
    reads it. That write refuses a file named ``row.json``, the timing
    row's own name. A file left twice under one name keeps the last
    *data*.

    Returns:
        The path of the file left under *root*.
    """
    spool = root / LEG_SPOOL
    spool.mkdir(parents=True, exist_ok=True)
    path = spool / name
    path.write_text(json.dumps(data, sort_keys=True), encoding="utf-8")
    return path

leg_files

leg_files(root: Path) -> tuple[dict[str, dict[str, Any]], str]

The files the leg's entries left in its spool, by name; or why one cannot ride.

A file that is not a JSON object, or one named as the leg's own row, refuses by name: the write would carry what no reader takes.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def leg_files(root: Path) -> tuple[dict[str, dict[str, Any]], str]:
    """The files the leg's entries left in its spool, by name; or why one cannot ride.

    A file that is not a JSON object, or one named as the leg's own
    row, refuses by name: the write would carry what no reader takes.
    """
    found: dict[str, dict[str, Any]] = {}
    spool = root / LEG_SPOOL
    if not spool.is_dir():
        return found, ""
    for path in sorted(spool.glob("*.json")):
        if path.name == ROW_FILE:
            return (
                {},
                f"{LEG_SPOOL}/{ROW_FILE} is the leg's own row; name the file otherwise",
            )
        try:
            loaded = json.loads(path.read_text("utf-8"))
        except (OSError, ValueError) as error:
            return {}, f"{LEG_SPOOL}/{path.name} does not parse ({error})"
        if not isinstance(loaded, dict):
            return {}, f"{LEG_SPOOL}/{path.name} is not a JSON object"
        found[path.name] = loaded
    return found, ""

leave_section

leave_section(root: Path, name: str, packages: Mapping[str, float], *, unit: str = '') -> str

Leave a section of the run's row, a number per package, for the gate job.

The gate job's ci.metrics.collect entry, before its verdict, folds every section left at root into the run's file on the metrics series, and fm ci.timings renders each as a trend, the numbers followed by unit. A section left twice under one name keeps the last numbers.

Returns:

Type Description
str

The line to print, naming the section and how many packages

str

it holds.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def leave_section(
    root: Path, name: str, packages: Mapping[str, float], *, unit: str = ""
) -> str:
    """Leave a section of the run's row, a number per package, for the gate job.

    The gate job's ``ci.metrics.collect`` entry, before its verdict,
    folds every section left at *root* into the run's file on the
    metrics series, and ``fm ci.timings`` renders each as a trend, the
    numbers followed by *unit*. A section left twice under one *name*
    keeps the last numbers.

    Returns:
        The line to print, naming the section and how many packages
        it holds.
    """
    directory = root / RUN_ROW
    directory.mkdir(parents=True, exist_ok=True)
    (directory / f"{name}.json").write_text(
        json.dumps({"unit": unit, "packages": dict(sorted(packages.items()))}),
        encoding="utf-8",
    )
    return f"{name} row: {len(packages)} package(s) for the run's record"

sections

sections(root: Path) -> dict[str, dict[str, Any]]

The sections the gate job's entries left, by name; a malformed file is left out.

A section is its unit and a number per package.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def sections(root: Path) -> dict[str, dict[str, Any]]:
    """The sections the gate job's entries left, by name; a malformed file is left out.

    A section is its unit and a number per package.
    """
    found: dict[str, dict[str, Any]] = {}
    directory = root / RUN_ROW
    if not directory.is_dir():
        return found
    for path in sorted(directory.glob("*.json")):
        try:
            loaded = json.loads(path.read_text("utf-8"))
        except (OSError, ValueError):
            continue
        section = _section(loaded)
        if section is not None:
            found[path.stem] = section
    return found

leg_row

leg_row(trace: Path, *, job: str) -> tuple[dict[str, Any] | None, str]

The task half of job's row from its trace; (row, "") or (None, reason).

The trace is the Chrome trace the profiled gate wrote: every task's slice (its request address and duration), the lane waits, each step inside its task, and each pytest phase of every test as a slice of its own. A step is a slice nested in its task, or an async pair (a begin and an end under one id) where it overlaps a sibling on the task's track, which is how footman draws the steps of one parallel() block; both read as the step's whole wall. The row's items is the seconds each item took, the test file of each test's node id, every phase of every test in it summed, to a tenth. A missing or unreadable trace is a reason, never a row.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def leg_row(trace: Path, *, job: str) -> tuple[dict[str, Any] | None, str]:
    """The task half of *job*'s row from its trace; ``(row, "")`` or ``(None, reason)``.

    The trace is the Chrome trace the profiled gate wrote: every
    task's slice (its request address and duration), the lane waits,
    each step inside its task, and each pytest phase of every test as
    a slice of its own. A step is a slice nested in its task, or an
    async pair (a begin and an end under one id) where it overlaps a
    sibling on the task's track, which is how footman draws the steps
    of one ``parallel()`` block; both read as the step's whole wall.
    The row's ``items`` is the seconds each item took, the test file
    of each test's node id, every phase of every test in it summed, to
    a tenth. A missing or unreadable trace is a reason, never a row.
    """
    if not trace.is_file():
        return None, f"no trace at {trace}: the gate did not run under --profile"
    try:
        loaded = json.loads(trace.read_text("utf-8"))
    except (OSError, ValueError) as error:
        return None, f"the trace at {trace} does not parse: {error}"
    events = loaded.get("traceEvents") if isinstance(loaded, dict) else loaded
    if not isinstance(events, list):
        return None, f"the trace at {trace} carries no event list"
    tasks: dict[str, float] = {}
    spans: list[tuple[str, int, float, float]] = []
    ran: list[tuple[str, int, float, float, float]] = []
    waits: dict[str, float] = defaultdict(float)
    packages: dict[str, dict[str, float]] = defaultdict(
        lambda: {"tests_ms": 0.0, "tests": 0}
    )
    # Every phase of every test, summed per item, the test file of its
    # node id: what a shard partitions the next run's items by.
    items: dict[str, float] = defaultdict(float)
    calls: list[tuple[float, str]] = []
    begins: list[float] = []
    ends: list[float] = []
    # A step that overlaps a sibling on its track is an async pair
    # rather than a slice, since the slices on one track must nest.
    # Its begin waits here for its end; the two are one span. A begin
    # the trace never ends is no span.
    begun: dict[tuple[Any, int, Any], dict[str, Any]] = {}
    for event in events:
        if not isinstance(event, dict):
            continue
        phase = event.get("ph")
        cat = str(event.get("cat", ""))
        if phase in ("b", "e"):
            if cat != "step":
                continue
            key = (event.get("pid"), _track(event), _pair_id(event))
            if phase == "b":
                begun[key] = event
            elif (opened := begun.pop(key, None)) is not None:
                start, end = _start(opened), _start(event)
                ms = round((end - start) / 1000.0, 1)
                name = str(opened.get("name", ""))
                ran.append((name, _track(event), start, end, ms))
            continue
        if phase != "X":
            continue
        name = str(event.get("name", ""))
        if event.get("pid") == 1:
            begins.append(float(event.get("ts", 0.0)))
            ends.append(float(event.get("ts", 0.0)) + float(event.get("dur", 0.0)))
        if cat == "task":
            tasks[name] = _ms(event)
            spans.append((name, _track(event), _start(event), _end(event)))
        elif cat == "step":
            ran.append((name, _track(event), _start(event), _end(event), _ms(event)))
        elif cat == "lane":
            waits[name.removeprefix("lane: ")] += _ms(event)
        elif cat.startswith("test."):
            items[name.split("::", 1)[0]] += _ms(event)
            package = _package_of(name)
            if package:
                packages[package]["tests_ms"] = round(
                    packages[package]["tests_ms"] + _ms(event), 1
                )
                if cat == "test.call":
                    packages[package]["tests"] += 1
                    calls.append((_ms(event), name))
    if not tasks:
        return None, f"the trace at {trace} records no task: nothing to row"
    steps = _steps_by_task(spans, ran)
    total = round((max(ends) - min(begins)) / 1000.0, 1) if begins else 0.0
    # The slowest tests by their call phase, for the speed judge's
    # warning to name; the per-package sums count every phase.
    calls.sort(reverse=True)
    return {
        "job": job,
        "total_ms": total,
        "tasks": dict(sorted(tasks.items())),
        "task_steps": dict(sorted(steps.items())),
        "waits_ms": {name: round(ms, 1) for name, ms in sorted(waits.items())},
        "packages": dict(sorted(packages.items())),
        "items": {item: round(ms / 1000.0, 1) for item, ms in sorted(items.items())},
        "slowest": [
            {"test": name, "s": round(ms / 1000.0, 2)} for ms, name in calls[:SLOWEST]
        ],
    }, ""

put_leg

put_leg(root: Path, run: RunContext, *, job: str, label: str, trace: Path) -> tuple[list[str], str]

Put job's task half and the leg's spool on its per-run ref, in one write.

Returns the lines to print and why the write failed, "" when it landed or there was nothing to write. The row is read from trace (livery.workshop._metrics.leg_row), with the scope the gate ran; a trace that cannot be read leaves the row out, says why, and the spool's files go alone. A leg that ran as one shard of several left what it partitioned in SHARD_FILE (livery.workshop.read_shard), and the row carries that record under shard for the gate job's fold; a leg that is one shard of one leaves no such file and its row carries no shard. Every file the leg's entries left in LEG_SPOOL rides beside the row under its own name (livery.workshop._metrics.leg_files), so the leg writes its ref once. A leg without a label, a shard file that is not a record, and a spool file no reader takes, fail the write by name.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def put_leg(
    root: Path, run: RunContext, *, job: str, label: str, trace: Path
) -> tuple[list[str], str]:
    """Put *job*'s task half and the leg's spool on its per-run ref, in one write.

    Returns the lines to print and why the write failed, ``""`` when
    it landed or there was nothing to write. The row is read from
    *trace* ([livery.workshop._metrics.leg_row][]), with the scope the
    gate ran; a trace that cannot be read leaves the row out, says
    why, and the spool's files go alone. A leg that ran as one shard
    of several left what it partitioned in `SHARD_FILE`
    ([livery.workshop.read_shard][]), and the row carries that record
    under ``shard`` for the gate job's fold; a leg that is one shard
    of one leaves no such file and its row carries no ``shard``. Every
    file the leg's entries left in `LEG_SPOOL` rides beside the row
    under its own name ([livery.workshop._metrics.leg_files][]), so
    the leg writes its ref once. A leg without a label, a shard file
    that is not a record, and a spool file no reader takes, fail the
    write by name.
    """
    lines: list[str] = []
    if not label:
        return lines, "the leg has no label, so its row has no key"
    files: dict[str, Mapping[str, Any]] = {}
    row, why = leg_row(trace, job=job)
    if row is None:
        lines.append(f"  {job}: {why}; the timing row stays unwritten")
    else:
        from dataclasses import asdict

        from livery.workshop._verified import leg_scope

        # The scope the gate ran, from the marker it left beside the
        # trace: the stamp after the verdict reads it back per leg.
        scope = leg_scope(trace.parent if trace.is_absolute() else root)
        row["scope"] = {**asdict(scope), "packages": list(scope.packages)}
        # What the shard partitioned, for the gate job's proof that the
        # platform's shards are one leg's work.
        marker, why = read_shard(root)
        if why:
            return lines, why
        if marker is not None:
            row["shard"] = marker_fields(marker)
        files[ROW_FILE] = row
    spooled, why = leg_files(root)
    if why:
        return lines, why
    files.update(spooled)
    if not files:
        return lines, ""
    why = RUNS.series(run.run_id, label).put(
        state_store(root), files, message=f"metrics: {job} of run {run.run_id}"
    )
    if why:
        return lines, why
    lines.append(f"  {label}: {', '.join(sorted(files))} on run {run.run_id}'s ref")
    return lines, ""

run_file

run_file(run_id: str) -> str

The series file for run_id: zero-padded when numeric, so names sort by time.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def run_file(run_id: str) -> str:
    """The series file for *run_id*: zero-padded when numeric, so names sort by time."""
    return f"{int(run_id):020d}.json" if run_id.isdigit() else f"{run_id}.json"

collect

collect(root: Path, repo: Repository, run: RunContext, *, sha: str) -> list[str]

Join the legs' halves with the forge's times into the run's file; what happened.

Every line names what was done or why not. Nothing is written when no leg left a row. The per-run refs stay: drop_halves takes them once every half is in, so a re-run of a red gate job still unions what its legs put, and a point whose every job collects folds each earlier job's half into its own write. A job the forge does not know keeps its trace half and is named; a row of another schema is skipped and named. The run is looked up under the head the runner's event names (_forge_jobs); sha is the checkout, the fallback and the row's own record. Every job the forge lists is recorded, the legs with their traces, the others with their forge times alone, and the run's own wall from its start to this collection. A platform that ran as shards has one row, folded under the platform's display name before the collect (livery.workshop._fold), a name the forge never ran: the row takes the forge times of the first shard's job and concludes as the shards' jobs did, and the forge's rows of the shards stand beside it with their times alone, for fm ci.timings. After the run's file lands, each check leg's seconds per item go onto its platform leg's series of the livery.workshop.DURATIONS family, one row per run, which the next run's shards partition by; a leg without them writes none, and a refused write is a line.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def collect(root: Path, repo: Repository, run: RunContext, *, sha: str) -> list[str]:
    """Join the legs' halves with the forge's times into the run's file; what happened.

    Every line names what was done or why not. Nothing is written
    when no leg left a row. The per-run refs stay: `drop_halves`
    takes them once every half is in, so a re-run of a red gate job
    still unions what its legs put, and a point whose every job
    collects folds each earlier job's half into its own write. A job
    the forge does not know keeps its trace half and is named; a row
    of another schema is skipped and named. The run is looked up
    under the head the runner's event names (`_forge_jobs`); *sha*
    is the checkout, the fallback and the row's own record. Every
    job the forge lists is recorded, the legs with their traces, the
    others with their forge times alone, and the run's own wall from
    its start to this collection. A platform that ran as shards has
    one row, folded under the platform's display name before the
    collect ([livery.workshop._fold][]), a name the forge never ran:
    the row takes the forge times of the first shard's job and
    concludes as the shards' jobs did, and the forge's rows of the
    shards stand beside it with their times alone, for
    ``fm ci.timings``. After the run's file lands, each check leg's
    seconds per item go onto its platform leg's series of the
    [livery.workshop.DURATIONS][] family, one row per run, which the
    next run's shards partition by; a leg without them writes none,
    and a refused write is a line.
    """
    lines: list[str] = []
    keys = RUNS.listed(state_store(root), run.run_id)
    if keys is None:
        return [
            f"  {RUNS.prefix}{run.run_id}/*: the remote could not be listed;"
            " nothing collected"
        ]
    halves: dict[str, dict[str, Any]] = {}
    for key in keys:
        series = RUNS.series(*key)
        found, why = series.row(state_store(root), ROW_FILE)
        if why:
            lines.append(f"  {series.ref}: {why}; skipped")
            continue
        if found is None:
            lines.append(f"  {series.ref}: empty; skipped")
            continue
        halves[str(found.data.get("job", key[-1]))] = found.data
    if not halves:
        lines.append(f"  run {run.run_id}: no leg left a row; nothing collected")
        return lines
    head = run.head_sha or sha
    forge_run, jobs, looked = _forge_jobs(repo, run, sha=sha)
    lines.extend(looked)
    if forge_run is None and not looked:
        lines.append(
            f"  run {run.run_id}: the forge lists no such run for {head[:12]};"
            " the rows carry their trace halves alone"
        )
    entry: dict[str, Any] = {
        "forge": run.forge,
        "run": run.run_id,
        "workflow": forge_run.workflow if forge_run else "",
        "event": run.event,
        "sha": head,
        "checkout": sha,
        "ref": run.ref,
        "created_at": forge_run.created_at if forge_run else "",
        "started_at": forge_run.started_at if forge_run else "",
        "collected_at": datetime.now(UTC).isoformat(timespec="seconds"),
        "jobs": {},
    }
    for name, half in sorted(halves.items()):
        job = jobs.get(name)
        row: dict[str, Any] = {
            "total_ms": half.get("total_ms"),
            "tasks": half.get("tasks", {}),
            "task_steps": half.get("task_steps", {}),
            "waits_ms": half.get("waits_ms", {}),
            "packages": half.get("packages", {}),
            "slowest": half.get("slowest", []),
            "scope": half.get("scope", {"scope": "unknown", "packages": []}),
        }
        shard = half.get("shard")
        if isinstance(shard, dict):
            row["shard"] = shard
        conclusion: str = job.conclusion if job is not None else ""
        if job is None and isinstance(shard, dict):
            job, conclusion, line = _folded_job(name, shard, jobs)
            if line:
                lines.append(line)
        if job is None:
            lines.append(f"  {name}: unknown to the forge; its trace half rides alone")
        else:
            row["conclusion"] = conclusion
            row.update(
                _times(job, created=entry["created_at"], now=entry["collected_at"])
            )
        entry["jobs"][name] = row
    for name, job in sorted(jobs.items()):
        if name not in entry["jobs"]:
            entry["jobs"][name] = _forge_row(
                job, created=entry["created_at"], now=entry["collected_at"]
            )
    entry["run_wall_ms"] = (
        _between(entry["started_at"], entry["collected_at"])
        if entry["started_at"]
        else None
    )
    left = sections(root)
    if left:
        entry["sections"] = left
        for name, section in left.items():
            lines.append(
                f"  {name}: {len(section['packages'])} package(s) recorded on the run"
            )
    why = SERIES.put(
        state_store(root),
        {run_file(run.run_id): entry},
        message=f"metrics: run {run.run_id} at {sha[:12]}",
    )
    if why:
        lines.append(f"  {SERIES.ref}: {why}; the per-run refs stay for a re-run")
        return lines
    lines.append(
        f"  {SERIES.ref}: run {run.run_id} recorded, {len(entry['jobs'])} job(s)"
    )
    lines.extend(_put_durations(root, run, halves))
    return lines

drop_halves

drop_halves(root: Path, repo: Repository, run: RunContext, *, sha: str, job: str, writers: tuple[str, ...]) -> list[str]

Drop the run's per-run refs once every half is in and nothing is red.

Runs as the last entry of a job that collected, after its own verdict, so its own job is green when it runs and is left out of the question. writers names the jobs that put a half, as the schedule spells them (check); a forge job answers to a writer by its display name's prefix (check (ubuntu-latest, 3.14)). The refs stay, and the lines say so, while a writer other than job has not completed green or any completed job is red: a re-run of a red job finds what the legs put, and the last of a point's collecting jobs is the one that drops. A job the forge does not list, or a run it does not know, keeps the refs too; the janitor's six-hour sweep is the net under every kept ref.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def drop_halves(
    root: Path,
    repo: Repository,
    run: RunContext,
    *,
    sha: str,
    job: str,
    writers: tuple[str, ...],
) -> list[str]:
    """Drop the run's per-run refs once every half is in and nothing is red.

    Runs as the last entry of a job that collected, after its own
    verdict, so its own job is green when it runs and is left out of
    the question. *writers* names the jobs that put a half, as the
    schedule spells them (``check``); a forge job answers to a writer
    by its display name's prefix (``check (ubuntu-latest, 3.14)``).
    The refs stay, and the lines say so, while a writer other than
    *job* has not completed green or any completed job is red: a
    re-run of a red job finds what the legs put, and the last of a
    point's collecting jobs is the one that drops. A job the forge
    does not list, or a run it does not know, keeps the refs too; the
    janitor's six-hour sweep is the net under every kept ref.
    """
    lines: list[str] = []
    keys = RUNS.listed(state_store(root), run.run_id)
    if keys is None:
        return [f"  {RUNS.prefix}{run.run_id}/*: the remote could not be listed; kept"]
    if not keys:
        return [f"  run {run.run_id}: no per-run ref to drop"]
    forge_run, jobs, looked = _forge_jobs(repo, run, sha=sha)
    lines.extend(looked)
    if forge_run is None:
        lines.append(f"  run {run.run_id}: the forge's jobs are unknown; the refs stay")
        return lines
    waiting = sorted(
        f"{name}: {found.conclusion or found.status}"
        for name, found in jobs.items()
        if name != job
        and any(name == writer or name.startswith(f"{writer} (") for writer in writers)
        and not _green(found)
    )
    red = sorted(
        f"{name}: {found.conclusion}"
        for name, found in jobs.items()
        if found.status == "completed" and not _green(found)
    )
    if waiting or red:
        lines.append(
            "  the per-run refs stay: " + ", ".join(dict.fromkeys([*red, *waiting]))
        )
        return lines
    for key in sorted(keys):
        ref = RUNS.series(*key).ref
        gone = git.drop(state_store(root), ref)
        lines.append(f"  {ref}: {gone or 'dropped'}")
    return lines

gate_steps

gate_steps(row: Mapping[str, Any]) -> dict[str, float]

The gate's steps of a leg's row by task key, in milliseconds.

The row's tasks and the steps its tasks ran (task_steps) whose key GATE_STEPS matches; a key in both is read from the tasks.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def gate_steps(row: Mapping[str, Any]) -> dict[str, float]:
    """The gate's steps of a leg's *row* by task key, in milliseconds.

    The row's tasks and the steps its tasks ran (``task_steps``) whose
    key `GATE_STEPS` matches; a key in both is read from the tasks.
    """
    found: dict[str, float] = {}
    for field in ("tasks", "task_steps"):
        values = row.get(field)
        if not isinstance(values, Mapping):
            continue
        for key, value in values.items():
            ms = _number(value)
            if GATE_STEPS.match(str(key)) and ms is not None:
                found.setdefault(str(key), ms)
    return found

tests_ms

tests_ms(packages: Any) -> float

The test time summed over packages, a row's packages, in milliseconds.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def tests_ms(packages: Any) -> float:
    """The test time summed over *packages*, a row's ``packages``, in milliseconds."""
    if not isinstance(packages, Mapping):
        return 0.0
    return sum(
        _number(package.get("tests_ms")) or 0.0
        for package in packages.values()
        if isinstance(package, Mapping)
    )

first_shard_head

first_shard_head(row: Mapping[str, Any]) -> float | None

The item seconds the first shard's unsplit steps stand for on row.

The unsplit steps are the gate's steps outside the test phase (TEST_PHASE); only a platform's first shard runs them, and the folded row of a platform's shards carries the first shard's steps. On the sharded runs the first shard's wall is about the summed wall W of those steps plus its slice's test wall, not the longer of the two: the steps hold the shard's cores while they run, and its tests wait. The partition packs item seconds, and a second of a shard's test wall holds p of them, p being the parallelism its tests reach, so the head is W times p. p is the median, over the shards after the first that the folded row lists under shard, of each one's summed test time over its wall; with no such shard, the row's own summed test time over its test steps' wall. When neither reads, the head is the longest unsplit step's wall, the least the first shard's tests wait.

Returns:

Type Description
float | None

The head, to a tenth of a second; None when the row names

float | None

no unsplit step.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def first_shard_head(row: Mapping[str, Any]) -> float | None:
    """The item seconds the first shard's unsplit steps stand for on *row*.

    The unsplit steps are the gate's steps outside the test phase
    (`TEST_PHASE`); only a platform's first shard runs them, and the
    folded row of a platform's shards carries the first shard's steps.
    On the sharded runs the first shard's wall is about the summed wall
    W of those steps plus its slice's test wall, not the longer of the
    two: the steps hold the shard's cores while they run, and its tests
    wait. The partition packs item seconds, and a second of a shard's
    test wall holds p of them, p being the parallelism its tests reach,
    so the head is W times p. p is the median, over the shards after
    the first that the folded row lists under ``shard``, of each one's
    summed test time over its wall; with no such shard, the row's own
    summed test time over its test steps' wall. When neither reads, the
    head is the longest unsplit step's wall, the least the first
    shard's tests wait.

    Returns:
        The head, to a tenth of a second; ``None`` when the row names
        no unsplit step.
    """
    steps = gate_steps(row)
    unsplit = [ms for key, ms in steps.items() if not key.startswith(TEST_PHASE)]
    if not unsplit:
        return None
    speed = _test_speed(row, steps)
    if speed is None:
        return round(max(unsplit) / 1000.0, 1)
    return round(sum(unsplit) / 1000.0 * speed, 1)

render

render(root: Path, *, since: int = 1, base: int = 20, movers: int = 10) -> list[str]

The timings, rendered: per job and metric the latest, p50, p90, and the movers.

The window is every run the series holds. The latest since runs are the recent side; the base runs before them are the base the movers compare against, by the change of the recent median over the base median. A platform that ran as shards is one section, its folded row, and under its metrics one line per shard with the shard's wall and queue times from the newest run, read from the forge's own rows of the shards; those rows are no sections of their own, and the movers read the platform legs alone. Nothing yet, an unreadable series, and a row of another schema each say so.

Source code in packages/workshop/src/livery/workshop/_metrics.py
def render(
    root: Path, *, since: int = 1, base: int = 20, movers: int = 10
) -> list[str]:
    """The timings, rendered: per job and metric the latest, p50, p90, and the movers.

    The window is every run the series holds. The latest *since* runs
    are the recent side; the *base* runs before them are the base the
    movers compare against, by the change of the recent median over
    the base median. A platform that ran as shards is one section, its
    folded row, and under its metrics one line per shard with the
    shard's wall and queue times from the newest run, read from the
    forge's own rows of the shards; those rows are no sections of their
    own, and the movers read the platform legs alone. Nothing yet, an
    unreadable series, and a row of another schema each say so.
    """
    found = SERIES.rows(state_store(root))
    if found.failed:
        return [f"  {found.reason}"]
    if not found.rows and not found.skipped:
        return ["  no timing rows yet: the gate job writes one per run"]
    lines = [f"  {line}" for line in found.skipped]
    # Oldest first: the trend reads the newest run at the end.
    entries = [row.data for row in reversed(found.rows)]
    if not entries:
        lines.append("  no readable timing rows")
        return lines
    newest = entries[-1]
    lines.append(
        f"  {len(entries)} run(s), latest {newest.get('run')}"
        f" at {str(newest.get('sha', ''))[:12]}"
    )
    recent, older = entries[-since:], entries[:-since]
    base_entries = older[-base:] if base else older
    series: dict[str, dict[str, list[float]]] = defaultdict(lambda: defaultdict(list))
    for entry in entries:
        for job, metrics in _metrics(entry).items():
            for metric, value in metrics.items():
                series[job][metric].append(value)
    latest = _metrics(entries[-1])
    for job in sorted(series):
        lines.append(f"  {job}")
        if newest.get("jobs", {}).get(job, {}).get("complete") is False:
            lines.append(
                "    still running when the gate job collected: measured to the"
                " collection"
            )
        lines.append(f"    {'metric':<44} {'latest':>9} {'p50':>9} {'p90':>9}")
        for metric, values in sorted(series[job].items()):
            now = latest.get(job, {}).get(metric)
            lines.append(
                f"    {metric[:44]:<44} {_seconds(now):>9}"
                f" {_percentile(values, 0.5) / 1000:8.1f}s"
                f" {_percentile(values, 0.9) / 1000:8.1f}s"
            )
        lines.extend(_shard_lines(newest, job))
    if base_entries and recent:
        deltas: list[tuple[float, str, str, float, float]] = []
        recent_series: dict[str, dict[str, list[float]]] = defaultdict(
            lambda: defaultdict(list)
        )
        base_series: dict[str, dict[str, list[float]]] = defaultdict(
            lambda: defaultdict(list)
        )
        for entry in recent:
            for job, metrics in _metrics(entry).items():
                for metric, value in metrics.items():
                    recent_series[job][metric].append(value)
        for entry in base_entries:
            for job, metrics in _metrics(entry).items():
                for metric, value in metrics.items():
                    base_series[job][metric].append(value)
        for job, per_metric in recent_series.items():
            for metric, recent_values in per_metric.items():
                before = base_series.get(job, {}).get(metric)
                if not before:
                    continue
                now, then = _percentile(recent_values, 0.5), _percentile(before, 0.5)
                deltas.append((now - then, job, metric, now, then))
        deltas.sort(key=lambda d: -abs(d[0]))
        lines.append(
            f"  movers: the last {len(recent)} run(s) against the"
            f" {len(base_entries)} before"
        )
        for delta, job, metric, now, then in deltas[:movers]:
            sign = "+" if delta >= 0 else "-"
            lines.append(
                f"    {sign}{abs(delta) / 1000:.1f}s  {job}: {metric}"
                f" ({then / 1000:.1f}s -> {now / 1000:.1f}s)"
            )
    lines.extend(_render_sections(entries))
    return lines