Skip to content

livery.workshop._scale

livery.workshop._scale

Build a workspace of many members and time the workshop's verbs over it.

The budgets of a sync and a gate are set from a measured baseline, and judged against the CI store. fm ci.scale builds the workspace that baseline is measured on, times fm sync, fm drift.check and fm check there, and prints the timings. Inside CI it also records them as one scale job on the run's file of the metrics series (livery.workshop._metrics), so fm ci.timings shows them beside the gate's own rows.

The workspace is a copy of this checkout's committed tree at HEAD, made a repository of its own with no remote, plus the generated members. The workshop under test is the copy's own member, so the timings measure the code being edited, and nothing is fetched from an index or a forge. The copy keeps this repository's own members and their source, so every checker reads them, but not their test suites: those test facts about this repository, which the generated members change.

Each verb is timed in three states:

  • cold: no environment in the copy and no gate record, the gate run whole. The machine's uv cache and tool store stay warm.
  • warm: again at once, nothing changed.
  • one changed: after an edit to the source of one python member no other member depends on.

The generated python members form a binary tree of runtime dependencies: member i depends on member (i - 1) // 2, declared as a [[depends]] edge and as a requirement in its pyproject.toml, so the graph the gate walks has depth as well as width.

This is development tooling of this repository, not part of the workshop a project uses.

Classes:

Name Description
Timing

One timed run of a verb in the fixture.

Functions:

Name Description
member_names

The generated members, as (directory name, template) pairs.

parent_of

The python member index depends on, or None for the root.

copy_tree

Copy source's committed tree at HEAD into destination as a new repository.

strip_own_suites

Remove the copy's own members' test modules and floors; the members.

generate

Add the generated members to the workspace at root; the members.

leaf

The python member the edit touches: the last, which no member depends on.

edit_member

Append one covered, formatted statement to name's module; the file edited.

run_verb

Run fm <verb> in root and time it; the timing, printed as it ends.

measure

Time every verb in every state in the fixture at root; the timings.

row

The scale job's row: the timings as tasks, exits and members.

record

Put job_row on the run's file of the metrics series; what happened.

ci_scale

Time sync, drift.check and check over a generated workspace.

run_fixture

Build the fixture at root from source, time it, record it; the timings.

Attributes:

Name Type Description
DEFAULT_COUNTS dict[str, int]
VERBS tuple[tuple[str, ...], ...]
STATES

DEFAULT_COUNTS module-attribute

DEFAULT_COUNTS: dict[str, int] = {'python-library': 240, 'cpp-package': 40, 'nanobind-module': 20}

VERBS module-attribute

VERBS: tuple[tuple[str, ...], ...] = (('sync',), ('drift.check',), ('check',))

STATES module-attribute

STATES = ('cold', 'warm', 'one changed')

Timing dataclass

Timing(name: str, ms: float, code: int)

One timed run of a verb in the fixture.

Attributes:

Name Type Description
name str

The metric's name, the verb then the state.

ms float

The wall time, in milliseconds.

code int

The verb's exit code.

name instance-attribute

name: str

ms instance-attribute

ms: float

code instance-attribute

code: int

member_names

member_names(counts: dict[str, int]) -> list[tuple[str, str]]

The generated members, as (directory name, template) pairs.

Names sort by template tag, then by a zero-padded index, so the python members' tree order is their name order.

Source code in packages/workshop/src/livery/workshop/_scale.py
def member_names(counts: dict[str, int]) -> list[tuple[str, str]]:
    """The generated members, as ``(directory name, template)`` pairs.

    Names sort by template tag, then by a zero-padded index, so the
    python members' tree order is their name order.
    """
    out: list[tuple[str, str]] = []
    for template, count in counts.items():
        tag = _TAGS.get(template)
        if tag is None:
            fail(
                f"no scale tag for template {template!r}: use one of {', '.join(_TAGS)}"
            )
        out.extend((f"scale-{tag}{index:03d}", template) for index in range(count))
    return out

parent_of

parent_of(index: int) -> int | None

The python member index depends on, or None for the root.

Source code in packages/workshop/src/livery/workshop/_scale.py
def parent_of(index: int) -> int | None:
    """The python member *index* depends on, or ``None`` for the root."""
    return None if index == 0 else (index - 1) // 2

copy_tree

copy_tree(source: Path, destination: Path) -> None

Copy source's committed tree at HEAD into destination as a new repository.

The copy has one commit and no remote. Its commits are unsigned: they are scratch, and a signer that waits for a person would stop an unattended run.

Source code in packages/workshop/src/livery/workshop/_scale.py
def copy_tree(source: Path, destination: Path) -> None:
    """Copy *source*'s committed tree at HEAD into *destination* as a new repository.

    The copy has one commit and no remote. Its commits are unsigned:
    they are scratch, and a signer that waits for a person would
    stop an unattended run.
    """
    destination.mkdir(parents=True, exist_ok=False)
    archive = destination.parent / f"{destination.name}.tar"
    _git(source, "archive", "--format=tar", f"--output={archive}", "HEAD")
    with tarfile.open(archive) as bundle:
        bundle.extractall(destination, filter="data")
    archive.unlink()
    _git(destination, "init", "--quiet", "--initial-branch=main")
    for key, value in (
        ("user.name", "scale fixture"),
        ("user.email", "scale@fixture.invalid"),
        ("commit.gpgsign", "false"),
        ("tag.gpgsign", "false"),
    ):
        _git(destination, "config", key, value)
    _commit(destination, "chore: the tree at HEAD")

strip_own_suites

strip_own_suites(root: Path) -> list[str]

Remove the copy's own members' test modules and floors; the members.

The fixture measures the verbs over the generated members. The suites of the members the copy started with test facts about the repository they come from, which the generated members change, and their time is not the fixture's. Each such member loses its test_*.py modules and keeps its helpers, since the rendered checker configuration names its tests directory; a directory left with no module gets one, named with the prefix the test layout rule wants of a helper: the package's path under packages/, a group directory's name included. Its [coverage] floor gives way to required = false, so nothing judges its coverage. The root tests directory is the project render's and stays as it is.

Source code in packages/workshop/src/livery/workshop/_scale.py
def strip_own_suites(root: Path) -> list[str]:
    """Remove the copy's own members' test modules and floors; the members.

    The fixture measures the verbs over the generated members. The
    suites of the members the copy started with test facts about the
    repository they come from, which the generated members change,
    and their time is not the fixture's. Each such member loses its
    ``test_*.py`` modules and keeps its helpers, since the rendered
    checker configuration names its ``tests`` directory; a directory
    left with no module gets one, named with the prefix the test
    layout rule wants of a helper: the package's path under
    ``packages/``, a group directory's name included. Its
    ``[coverage]`` floor gives way to ``required = false``, so nothing
    judges its coverage. The root ``tests`` directory is the project
    render's and stays as it is.
    """
    from livery.workshop._packages import discover_packages

    stripped: list[str] = []
    for package in discover_packages(root):
        tests = package.directory / "tests"
        if tests.is_dir():
            for module in tests.rglob("test_*.py"):
                module.unlink()
            if not any(tests.rglob("*.py")):
                name = package.member.replace("-", "_").replace("/", "_")
                (tests / f"{name}_copy.py").write_text(
                    '"""The scale fixture\'s copy keeps no suite of this member."""\n',
                    encoding="utf-8",
                )
        contract = package.directory / "workshop.toml"
        text = contract.read_text("utf-8")
        # A [coverage] floor is a number; a [[depends]] floor is a quoted version.
        contract.write_text(
            re.sub(r"(?m)^floor = [\d.]+$", "required = false", text),
            encoding="utf-8",
        )
        stripped.append(package.path)
    _commit(root, "chore: the copy's own suites removed")
    return stripped

generate

generate(root: Path, counts: dict[str, int]) -> list[tuple[str, str]]

Add the generated members to the workspace at root; the members.

The root's list takes in what the members' templates need first, written as its canonical list, since a creation installs nothing and judges the list: this checkout's environment, which runs the creation, installs each of them. Each member is rendered and added to the roster, then the project is rendered once and the result committed. Locking and installing are left to the first fm sync.

Source code in packages/workshop/src/livery/workshop/_scale.py
def generate(root: Path, counts: dict[str, int]) -> list[tuple[str, str]]:
    """Add the generated members to the workspace at *root*; the members.

    The root's list takes in what the members' templates need first,
    written as its canonical list, since a creation installs nothing
    and judges the list: this checkout's environment, which runs the
    creation, installs each of them. Each member is rendered and added
    to the roster, then the
    project is rendered once and the result committed. Locking and
    installing are left to the first `fm sync`.
    """
    from livery.workshop._extensions import list_in_root
    from livery.workshop._packages import declare_edge, discover_packages
    from livery.workshop._seeds import chain_needs, template_chain
    from livery.workshop._templates import apply_project, render_member

    members = member_names(counts)
    python = [name for name, template in members if template == "python-library"]
    list_in_root(
        root,
        (
            needed
            for template in dict.fromkeys(template for _name, template in members)
            for needed in chain_needs(template_chain(template))
        ),
    )
    for name, template in members:
        render_member(root, name, template)
    by_path = {package.path: package for package in discover_packages(root)}
    for index, name in enumerate(python):
        parent = parent_of(index)
        if parent is not None:
            declare_edge(
                by_path[f"packages/{name}"],
                by_path[f"packages/{python[parent]}"],
                kind="runtime",
                floor="0.0.0",
            )
    apply_project(root)
    _commit(root, f"chore: {len(members)} generated members")
    return members

leaf

leaf(members: list[tuple[str, str]]) -> str

The python member the edit touches: the last, which no member depends on.

Source code in packages/workshop/src/livery/workshop/_scale.py
def leaf(members: list[tuple[str, str]]) -> str:
    """The python member the edit touches: the last, which no member depends on."""
    python = [name for name, template in members if template == "python-library"]
    if not python:
        fail("the fixture has no python member to edit")
    return python[-1]

edit_member

edit_member(root: Path, name: str) -> Path

Append one covered, formatted statement to name's module; the file edited.

Source code in packages/workshop/src/livery/workshop/_scale.py
def edit_member(root: Path, name: str) -> Path:
    """Append one covered, formatted statement to *name*'s module; the file edited."""
    found = sorted((root / "packages" / name / "src").rglob("__init__.py"))
    if not found:
        fail(f"packages/{name}: no package module to edit")
    module = found[-1]
    with module.open("a", encoding="utf-8") as handle:
        # Two blank lines are formatted after any top-level statement.
        handle.write("\n\nSCALE_EDIT = 1\n")
    return module

run_verb

run_verb(root: Path, verb: tuple[str, ...], *, name: str) -> Timing

Run fm <verb> in root and time it; the timing, printed as it ends.

The child is this interpreter's footman, which enters the workspace's own environment. Its output is printed when it exits non-zero, and the timing records the exit code either way.

Source code in packages/workshop/src/livery/workshop/_scale.py
def run_verb(root: Path, verb: tuple[str, ...], *, name: str) -> Timing:
    """Run ``fm <verb>`` in *root* and time it; the timing, printed as it ends.

    The child is this interpreter's footman, which enters the
    workspace's own environment. Its output is printed when it
    exits non-zero, and the timing records the exit code either way.
    """
    start = time.perf_counter()
    result = footman.run(
        [sys.executable, "-m", "livery.footman", "--yes", *verb],
        cwd=root,
        env=_environment(root),
        nofail=True,
        timeout=7200.0,
    )
    ms = round((time.perf_counter() - start) * 1000.0, 1)
    print(f"  {name}: {ms / 1000:.1f}s, exit {result.code}")
    if result.code != 0:
        print(f"{result.stdout}{result.stderr}")
    return Timing(name=name, ms=ms, code=result.code)

measure

measure(root: Path, members: list[tuple[str, str]]) -> list[Timing]

Time every verb in every state in the fixture at root; the timings.

Source code in packages/workshop/src/livery/workshop/_scale.py
def measure(root: Path, members: list[tuple[str, str]]) -> list[Timing]:
    """Time every verb in every state in the fixture at *root*; the timings."""
    timings: list[Timing] = []
    shutil.rmtree(root / ".venv", ignore_errors=True)
    for state in STATES:
        if state == "one changed":
            edited = edit_member(root, leaf(members))
            print(f"  edited {edited.relative_to(root).as_posix()}")
        for verb in VERBS:
            spelled = " ".join(verb)
            args = (*verb, "--full") if verb == ("check",) and state == "cold" else verb
            timings.append(run_verb(root, args, name=f"{spelled} {state}"))
    return timings

row

row(timings: list[Timing], members: list[tuple[str, str]], *, job: str = 'scale') -> dict[str, Any]

The scale job's row: the timings as tasks, exits and members.

Source code in packages/workshop/src/livery/workshop/_scale.py
def row(
    timings: list[Timing], members: list[tuple[str, str]], *, job: str = "scale"
) -> dict[str, Any]:
    """The ``scale`` job's row: the timings as tasks, exits and members."""
    counts: dict[str, int] = {}
    for _name, template in members:
        counts[template] = counts.get(template, 0) + 1
    return {
        "job": job,
        "total_ms": round(sum(timing.ms for timing in timings), 1),
        "tasks": {timing.name: timing.ms for timing in timings},
        "codes": {timing.name: timing.code for timing in timings},
        "members": dict(sorted(counts.items())),
    }

record

record(root: Path, job_row: dict[str, Any], *, sha: str) -> str

Put job_row on the run's file of the metrics series; what happened.

Outside CI nothing is written and the line says so.

Source code in packages/workshop/src/livery/workshop/_scale.py
def record(root: Path, job_row: dict[str, Any], *, sha: str) -> str:
    """Put *job_row* on the run's file of the metrics series; what happened.

    Outside CI nothing is written and the line says so.
    """
    from livery.git import remote_snapshot
    from livery.workshop._ci_context import run_context, state_store
    from livery.workshop._metrics import SERIES, run_file

    run = run_context()
    if run is None:
        return "  not a CI run: the timings are recorded by CI only"
    entry: dict[str, Any] = {
        "forge": run.forge,
        "run": run.run_id,
        "event": run.event,
        "sha": run.head_sha or sha,
        "checkout": sha,
        "ref": run.ref,
        "collected_at": datetime.now(UTC).isoformat(timespec="seconds"),
        "jobs": {str(job_row["job"]): job_row},
    }
    with remote_snapshot(state_store(root), fetch=("metrics",)):
        why = SERIES.put(
            state_store(root),
            {run_file(run.run_id): entry},
            message=f"metrics: scale fixture of run {run.run_id} at {sha[:12]}",
        )
    return (
        f"  {SERIES.ref}: {why}"
        if why
        else f"  {SERIES.ref}: run {run.run_id} recorded"
    )

ci_scale

ci_scale(python: Annotated[int, doc('members born from python-library')] = DEFAULT_COUNTS['python-library'], cpp: Annotated[int, doc('members born from cpp-package')] = DEFAULT_COUNTS['cpp-package'], nanobind: Annotated[int, doc('members born from nanobind-module')] = DEFAULT_COUNTS['nanobind-module'], keep: Annotated[str, doc('build the fixture in this new directory and keep it')] = '') -> None

Time sync, drift.check and check over a generated workspace.

Copies the committed tree at HEAD, adds the generated members, and times each verb cold, warm with nothing changed, and after one member's source changed. Prints every timing. Inside CI the timings are also put on the metrics series as the run's scale job, which fm ci.timings shows. The fixture is removed afterwards unless keep names where to build it.

Source code in packages/workshop/src/livery/workshop/_scale.py
@ci.task(name="scale")
@git.unsigned
def ci_scale(
    python: Annotated[int, doc("members born from python-library")] = DEFAULT_COUNTS[
        "python-library"
    ],
    cpp: Annotated[int, doc("members born from cpp-package")] = DEFAULT_COUNTS[
        "cpp-package"
    ],
    nanobind: Annotated[int, doc("members born from nanobind-module")] = DEFAULT_COUNTS[
        "nanobind-module"
    ],
    keep: Annotated[
        str, doc("build the fixture in this new directory and keep it")
    ] = "",
) -> None:
    """Time sync, drift.check and check over a generated workspace.

    Copies the committed tree at HEAD, adds the generated members,
    and times each verb cold, warm with nothing changed, and after
    one member's source changed. Prints every timing. Inside CI the
    timings are also put on the metrics series as the run's
    ``scale`` job, which `fm ci.timings` shows. The fixture is
    removed afterwards unless *keep* names where to build it.
    """
    from livery.workshop._templates import _root  # pyright: ignore[reportPrivateUsage]

    source = _root()
    counts = _counts(python, cpp, nanobind)
    if keep:
        timings = run_fixture(source, Path(keep).resolve(), counts)
    else:
        with tempfile.TemporaryDirectory(
            prefix="fm-scale-", ignore_cleanup_errors=True
        ) as scratch:
            timings = run_fixture(source, Path(scratch) / "workspace", counts)
    failed = [timing.name for timing in timings if timing.code != 0]
    if failed:
        fail(f"the fixture's verbs exited non-zero: {', '.join(failed)}")

run_fixture

run_fixture(source: Path, root: Path, counts: dict[str, int]) -> list[Timing]

Build the fixture at root from source, time it, record it; the timings.

Source code in packages/workshop/src/livery/workshop/_scale.py
def run_fixture(source: Path, root: Path, counts: dict[str, int]) -> list[Timing]:
    """Build the fixture at *root* from *source*, time it, record it; the timings."""
    copy_tree(source, root)
    stripped = strip_own_suites(root)
    print(f"  {len(stripped)} member(s) of the copy without their suites")
    started = time.perf_counter()
    members = generate(root, counts)
    print(
        f"  generated {len(members)} member(s) in"
        f" {time.perf_counter() - started:.1f}s at {root}"
    )
    timings = measure(root, members)
    sha = _git(source, "rev-parse", "HEAD").strip()
    print(record(source, row(timings, members), sha=sha))
    return timings