Build a workspace of many members and time the workshop's verbs over it.
The budgets of a sync and a gate are set from a measured baseline,
and judged against the CI store. fm ci.scale builds the workspace
that baseline is measured on, times fm sync, fm drift.check
and fm check there, and prints the timings. Inside CI it also
records them as one scale job on the run's file of the
metrics series (livery.workshop._metrics), so fm ci.timings
shows them beside the gate's own rows.
The workspace is a copy of this checkout's committed tree at HEAD,
made a repository of its own with no remote, plus the generated
members. The workshop under test is the copy's own member, so the
timings measure the code being edited, and nothing is fetched from
an index or a forge. The copy keeps this repository's own members
and their source, so every checker reads them, but not their test
suites: those test facts about this repository, which the generated
members change.
Each verb is timed in three states:
cold: no environment in the copy and no gate record, the gate
run whole. The machine's uv cache and tool store stay warm.
warm: again at once, nothing changed.
one changed: after an edit to the source of one python member
no other member depends on.
The generated python members form a binary tree of runtime
dependencies: member i depends on member (i - 1) // 2,
declared as a [[depends]] edge and as a requirement in its
pyproject.toml, so the graph the gate walks has depth as well as
width.
This is development tooling of this repository, not part of the
workshop a project uses.
Classes:
| Name |
Description |
Timing |
One timed run of a verb in the fixture.
|
Functions:
| Name |
Description |
member_names |
The generated members, as (directory name, template) pairs.
|
parent_of |
The python member index depends on, or None for the root.
|
copy_tree |
Copy source's committed tree at HEAD into destination as a new repository.
|
strip_own_suites |
Remove the copy's own members' test modules and floors; the members.
|
generate |
Add the generated members to the workspace at root; the members.
|
leaf |
The python member the edit touches: the last, which no member depends on.
|
edit_member |
Append one covered, formatted statement to name's module; the file edited.
|
run_verb |
Run fm <verb> in root and time it; the timing, printed as it ends.
|
measure |
Time every verb in every state in the fixture at root; the timings.
|
row |
The scale job's row: the timings as tasks, exits and members.
|
record |
Put job_row on the run's file of the metrics series; what happened.
|
ci_scale |
Time sync, drift.check and check over a generated workspace.
|
run_fixture |
Build the fixture at root from source, time it, record it; the timings.
|
Attributes:
DEFAULT_COUNTS
module-attribute
DEFAULT_COUNTS: dict[str, int] = {'python-library': 240, 'cpp-package': 40, 'nanobind-module': 20}
VERBS
module-attribute
VERBS: tuple[tuple[str, ...], ...] = (('sync',), ('drift.check',), ('check',))
STATES
module-attribute
STATES = ('cold', 'warm', 'one changed')
Timing
dataclass
One timed run of a verb in the fixture.
Attributes:
| Name |
Type |
Description |
name |
str
|
The metric's name, the verb then the state.
|
ms |
float
|
The wall time, in milliseconds.
|
code |
int
|
|
member_names
The generated members, as (directory name, template) pairs.
Names sort by template tag, then by a zero-padded index, so the
python members' tree order is their name order.
Source code in packages/workshop/src/livery/workshop/_scale.py
| def member_names(counts: dict[str, int]) -> list[tuple[str, str]]:
"""The generated members, as ``(directory name, template)`` pairs.
Names sort by template tag, then by a zero-padded index, so the
python members' tree order is their name order.
"""
out: list[tuple[str, str]] = []
for template, count in counts.items():
tag = _TAGS.get(template)
if tag is None:
fail(
f"no scale tag for template {template!r}: use one of {', '.join(_TAGS)}"
)
out.extend((f"scale-{tag}{index:03d}", template) for index in range(count))
return out
|
parent_of
parent_of(index: int) -> int | None
The python member index depends on, or None for the root.
Source code in packages/workshop/src/livery/workshop/_scale.py
| def parent_of(index: int) -> int | None:
"""The python member *index* depends on, or ``None`` for the root."""
return None if index == 0 else (index - 1) // 2
|
copy_tree
copy_tree(source: Path, destination: Path) -> None
Copy source's committed tree at HEAD into destination as a new repository.
The copy has one commit and no remote. Its commits are unsigned:
they are scratch, and a signer that waits for a person would
stop an unattended run.
Source code in packages/workshop/src/livery/workshop/_scale.py
| def copy_tree(source: Path, destination: Path) -> None:
"""Copy *source*'s committed tree at HEAD into *destination* as a new repository.
The copy has one commit and no remote. Its commits are unsigned:
they are scratch, and a signer that waits for a person would
stop an unattended run.
"""
destination.mkdir(parents=True, exist_ok=False)
archive = destination.parent / f"{destination.name}.tar"
_git(source, "archive", "--format=tar", f"--output={archive}", "HEAD")
with tarfile.open(archive) as bundle:
bundle.extractall(destination, filter="data")
archive.unlink()
_git(destination, "init", "--quiet", "--initial-branch=main")
for key, value in (
("user.name", "scale fixture"),
("user.email", "scale@fixture.invalid"),
("commit.gpgsign", "false"),
("tag.gpgsign", "false"),
):
_git(destination, "config", key, value)
_commit(destination, "chore: the tree at HEAD")
|
strip_own_suites
Remove the copy's own members' test modules and floors; the members.
The fixture measures the verbs over the generated members. The
suites of the members the copy started with test facts about the
repository they come from, which the generated members change,
and their time is not the fixture's. Each such member loses its
test_*.py modules and keeps its helpers, since the rendered
checker configuration names its tests directory; a directory
left with no module gets one, named with the prefix the test
layout rule wants of a helper: the package's path under
packages/, a group directory's name included. Its
[coverage] floor gives way to required = false, so nothing
judges its coverage. The root tests directory is the project
render's and stays as it is.
Source code in packages/workshop/src/livery/workshop/_scale.py
| def strip_own_suites(root: Path) -> list[str]:
"""Remove the copy's own members' test modules and floors; the members.
The fixture measures the verbs over the generated members. The
suites of the members the copy started with test facts about the
repository they come from, which the generated members change,
and their time is not the fixture's. Each such member loses its
``test_*.py`` modules and keeps its helpers, since the rendered
checker configuration names its ``tests`` directory; a directory
left with no module gets one, named with the prefix the test
layout rule wants of a helper: the package's path under
``packages/``, a group directory's name included. Its
``[coverage]`` floor gives way to ``required = false``, so nothing
judges its coverage. The root ``tests`` directory is the project
render's and stays as it is.
"""
from livery.workshop._packages import discover_packages
stripped: list[str] = []
for package in discover_packages(root):
tests = package.directory / "tests"
if tests.is_dir():
for module in tests.rglob("test_*.py"):
module.unlink()
if not any(tests.rglob("*.py")):
name = package.member.replace("-", "_").replace("/", "_")
(tests / f"{name}_copy.py").write_text(
'"""The scale fixture\'s copy keeps no suite of this member."""\n',
encoding="utf-8",
)
contract = package.directory / "workshop.toml"
text = contract.read_text("utf-8")
# A [coverage] floor is a number; a [[depends]] floor is a quoted version.
contract.write_text(
re.sub(r"(?m)^floor = [\d.]+$", "required = false", text),
encoding="utf-8",
)
stripped.append(package.path)
_commit(root, "chore: the copy's own suites removed")
return stripped
|
generate
Add the generated members to the workspace at root; the members.
The root's list takes in what the members' templates need first,
written as its canonical list, since a creation installs nothing
and judges the list: this checkout's environment, which runs the
creation, installs each of them. Each member is rendered and added
to the roster, then the
project is rendered once and the result committed. Locking and
installing are left to the first fm sync.
Source code in packages/workshop/src/livery/workshop/_scale.py
| def generate(root: Path, counts: dict[str, int]) -> list[tuple[str, str]]:
"""Add the generated members to the workspace at *root*; the members.
The root's list takes in what the members' templates need first,
written as its canonical list, since a creation installs nothing
and judges the list: this checkout's environment, which runs the
creation, installs each of them. Each member is rendered and added
to the roster, then the
project is rendered once and the result committed. Locking and
installing are left to the first `fm sync`.
"""
from livery.workshop._extensions import list_in_root
from livery.workshop._packages import declare_edge, discover_packages
from livery.workshop._seeds import chain_needs, template_chain
from livery.workshop._templates import apply_project, render_member
members = member_names(counts)
python = [name for name, template in members if template == "python-library"]
list_in_root(
root,
(
needed
for template in dict.fromkeys(template for _name, template in members)
for needed in chain_needs(template_chain(template))
),
)
for name, template in members:
render_member(root, name, template)
by_path = {package.path: package for package in discover_packages(root)}
for index, name in enumerate(python):
parent = parent_of(index)
if parent is not None:
declare_edge(
by_path[f"packages/{name}"],
by_path[f"packages/{python[parent]}"],
kind="runtime",
floor="0.0.0",
)
apply_project(root)
_commit(root, f"chore: {len(members)} generated members")
return members
|
leaf
The python member the edit touches: the last, which no member depends on.
Source code in packages/workshop/src/livery/workshop/_scale.py
| def leaf(members: list[tuple[str, str]]) -> str:
"""The python member the edit touches: the last, which no member depends on."""
python = [name for name, template in members if template == "python-library"]
if not python:
fail("the fixture has no python member to edit")
return python[-1]
|
edit_member
Append one covered, formatted statement to name's module; the file edited.
Source code in packages/workshop/src/livery/workshop/_scale.py
| def edit_member(root: Path, name: str) -> Path:
"""Append one covered, formatted statement to *name*'s module; the file edited."""
found = sorted((root / "packages" / name / "src").rglob("__init__.py"))
if not found:
fail(f"packages/{name}: no package module to edit")
module = found[-1]
with module.open("a", encoding="utf-8") as handle:
# Two blank lines are formatted after any top-level statement.
handle.write("\n\nSCALE_EDIT = 1\n")
return module
|
run_verb
Run fm <verb> in root and time it; the timing, printed as it ends.
The child is this interpreter's footman, which enters the
workspace's own environment. Its output is printed when it
exits non-zero, and the timing records the exit code either way.
Source code in packages/workshop/src/livery/workshop/_scale.py
| def run_verb(root: Path, verb: tuple[str, ...], *, name: str) -> Timing:
"""Run ``fm <verb>`` in *root* and time it; the timing, printed as it ends.
The child is this interpreter's footman, which enters the
workspace's own environment. Its output is printed when it
exits non-zero, and the timing records the exit code either way.
"""
start = time.perf_counter()
result = footman.run(
[sys.executable, "-m", "livery.footman", "--yes", *verb],
cwd=root,
env=_environment(root),
nofail=True,
timeout=7200.0,
)
ms = round((time.perf_counter() - start) * 1000.0, 1)
print(f" {name}: {ms / 1000:.1f}s, exit {result.code}")
if result.code != 0:
print(f"{result.stdout}{result.stderr}")
return Timing(name=name, ms=ms, code=result.code)
|
measure
Time every verb in every state in the fixture at root; the timings.
Source code in packages/workshop/src/livery/workshop/_scale.py
| def measure(root: Path, members: list[tuple[str, str]]) -> list[Timing]:
"""Time every verb in every state in the fixture at *root*; the timings."""
timings: list[Timing] = []
shutil.rmtree(root / ".venv", ignore_errors=True)
for state in STATES:
if state == "one changed":
edited = edit_member(root, leaf(members))
print(f" edited {edited.relative_to(root).as_posix()}")
for verb in VERBS:
spelled = " ".join(verb)
args = (*verb, "--full") if verb == ("check",) and state == "cold" else verb
timings.append(run_verb(root, args, name=f"{spelled} {state}"))
return timings
|
row
The scale job's row: the timings as tasks, exits and members.
Source code in packages/workshop/src/livery/workshop/_scale.py
| def row(
timings: list[Timing], members: list[tuple[str, str]], *, job: str = "scale"
) -> dict[str, Any]:
"""The ``scale`` job's row: the timings as tasks, exits and members."""
counts: dict[str, int] = {}
for _name, template in members:
counts[template] = counts.get(template, 0) + 1
return {
"job": job,
"total_ms": round(sum(timing.ms for timing in timings), 1),
"tasks": {timing.name: timing.ms for timing in timings},
"codes": {timing.name: timing.code for timing in timings},
"members": dict(sorted(counts.items())),
}
|
record
Put job_row on the run's file of the metrics series; what happened.
Outside CI nothing is written and the line says so.
Source code in packages/workshop/src/livery/workshop/_scale.py
| def record(root: Path, job_row: dict[str, Any], *, sha: str) -> str:
"""Put *job_row* on the run's file of the metrics series; what happened.
Outside CI nothing is written and the line says so.
"""
from livery.git import remote_snapshot
from livery.workshop._ci_context import run_context, state_store
from livery.workshop._metrics import SERIES, run_file
run = run_context()
if run is None:
return " not a CI run: the timings are recorded by CI only"
entry: dict[str, Any] = {
"forge": run.forge,
"run": run.run_id,
"event": run.event,
"sha": run.head_sha or sha,
"checkout": sha,
"ref": run.ref,
"collected_at": datetime.now(UTC).isoformat(timespec="seconds"),
"jobs": {str(job_row["job"]): job_row},
}
with remote_snapshot(state_store(root), fetch=("metrics",)):
why = SERIES.put(
state_store(root),
{run_file(run.run_id): entry},
message=f"metrics: scale fixture of run {run.run_id} at {sha[:12]}",
)
return (
f" {SERIES.ref}: {why}"
if why
else f" {SERIES.ref}: run {run.run_id} recorded"
)
|
ci_scale
Time sync, drift.check and check over a generated workspace.
Copies the committed tree at HEAD, adds the generated members,
and times each verb cold, warm with nothing changed, and after
one member's source changed. Prints every timing. Inside CI the
timings are also put on the metrics series as the run's
scale job, which fm ci.timings shows. The fixture is
removed afterwards unless keep names where to build it.
Source code in packages/workshop/src/livery/workshop/_scale.py
| @ci.task(name="scale")
@git.unsigned
def ci_scale(
python: Annotated[int, doc("members born from python-library")] = DEFAULT_COUNTS[
"python-library"
],
cpp: Annotated[int, doc("members born from cpp-package")] = DEFAULT_COUNTS[
"cpp-package"
],
nanobind: Annotated[int, doc("members born from nanobind-module")] = DEFAULT_COUNTS[
"nanobind-module"
],
keep: Annotated[
str, doc("build the fixture in this new directory and keep it")
] = "",
) -> None:
"""Time sync, drift.check and check over a generated workspace.
Copies the committed tree at HEAD, adds the generated members,
and times each verb cold, warm with nothing changed, and after
one member's source changed. Prints every timing. Inside CI the
timings are also put on the metrics series as the run's
``scale`` job, which `fm ci.timings` shows. The fixture is
removed afterwards unless *keep* names where to build it.
"""
from livery.workshop._templates import _root # pyright: ignore[reportPrivateUsage]
source = _root()
counts = _counts(python, cpp, nanobind)
if keep:
timings = run_fixture(source, Path(keep).resolve(), counts)
else:
with tempfile.TemporaryDirectory(
prefix="fm-scale-", ignore_cleanup_errors=True
) as scratch:
timings = run_fixture(source, Path(scratch) / "workspace", counts)
failed = [timing.name for timing in timings if timing.code != 0]
if failed:
fail(f"the fixture's verbs exited non-zero: {', '.join(failed)}")
|
run_fixture
Build the fixture at root from source, time it, record it; the timings.
Source code in packages/workshop/src/livery/workshop/_scale.py
| def run_fixture(source: Path, root: Path, counts: dict[str, int]) -> list[Timing]:
"""Build the fixture at *root* from *source*, time it, record it; the timings."""
copy_tree(source, root)
stripped = strip_own_suites(root)
print(f" {len(stripped)} member(s) of the copy without their suites")
started = time.perf_counter()
members = generate(root, counts)
print(
f" generated {len(members)} member(s) in"
f" {time.perf_counter() - started:.1f}s at {root}"
)
timings = measure(root, members)
sha = _git(source, "rev-parse", "HEAD").strip()
print(record(source, row(timings, members), sha=sha))
return timings
|