livery.workshop._balance¶
livery.workshop._balance ¶
Predict each check platform's wall at one to several shards, and propose the counts.
The reader behind fm ci.balance. It reads the timing rows the
gate job writes (livery.workshop._metrics) per platform: the os
and Python of a check job's display name, check (macos-latest,
3.14); check (macos-latest, 3.14, 2of4) is one shard of that
platform, and a GitLab job's bare name reads its platform from the
row's own leg label. Per platform it takes the newest ten rows whose
jobs for it all ran the full gate and succeeded, folds the shards of
one platform into one, and takes the medians of four measures:
- F, the fixed setup: every step the forge timed except the
Checkcall, from the first shard, in seconds. - O, the summed wall of the gate's steps that are not test steps,
from the first shard. The gate's steps are the job's
check/<phase>.<step>tasks, and the test steps are the test phase's, livery.workshop._metrics.TEST_PHASE. - S, the summed test time over every package and every shard.
- p, the measured parallelism: a shard's S over its test steps' wall, the median over shards.
The wall at n shards is predicted as F + max(O, (S / p + O) / n). The
first shard runs the unsplit steps, and its tests wait about their
summed wall, so the partition starts the first shard's slice O p item
seconds in (livery.workshop._metrics.first_shard_head): the S + O p
item seconds split evenly over the n shards, and no count brings a
platform under F + O, the first shard with no tests left. The runner
seconds at n are about n F + S / p + O. The slowest platform, the one
with the largest wall at one shard, takes the largest count within
the cap whose every added shard saved at least F, so a shard that
costs more setup than it saves is not proposed. Every other platform
takes the smallest count whose wall is within
livery.workshop._balance.TOLERANCE of the slowest
platform's at its count; one that cannot get there at the cap takes
the cap and is named. The pull request counts apply the same rule
over a pull request's runners when they are fewer than [ci]
runners names. A platform with fewer than
livery.workshop._balance.ROWS_NEEDED readable rows keeps its count
and takes no part. The cap is [ci] shards-max, which the verb
reads and never writes.
The verb prints the table and the TOML the contract would hold. With
--write it rewrites [ci] shards, and [ci]
pull-request-shards when the pull request runners are fewer, in the
root workshop.toml, keeping every comment and every other key:
when a count differs from the contract's and the predicted wall at
the proposed counts differs from the wall at the contract's by more
than livery.workshop._balance.HYSTERESIS, so a count does not move
back and forth between two readings that predict about the same
wall. With --submit the change lands on the day's branch,
chore/balance-<date>, rendered, committed and submitted unarmed
for a person to merge; the nightly runs that once a day. The
constants here were chosen before the rows could set them, as the
speed marks' margins were (livery.workshop._speed).
Reach for render for what the verb prints, read for the same with
the counts a write judges, judge for the write's decision, write
and submit for the two acts, readings for the measures from rows
already in hand, wall and runner_seconds for the prediction, and
propose for the counts.
Classes:
| Name | Description |
|---|---|
Reading |
One platform's measures, medians over its readable rows. |
Proposal |
The counts a set of readings proposes. |
Counts |
One key's count per runner: the contract's, the proposal's, and both walls. |
Balance |
One reading of the record: the verb's lines and the counts a write judges. |
Verdict |
A write's decision on a |
Functions:
| Name | Description |
|---|---|
wall |
The leg's predicted wall at shards, in seconds: F + max(O, (S / p + O) / n). |
runner_seconds |
About what shards runners spend together, in seconds: n F + S / p + O. |
propose |
The counts for readings within cap; a platform short of rows takes no part. |
readings |
Each platform's reading from rows, newest first; the platforms in name order. |
spell |
table as the contract spells an inline table; |
report |
The lines the verb prints below the window line: the tables, then the proposal. |
counts |
One key's counts over runners: current, the proposal, and both walls. |
read |
Read the record and the contract: what |
render |
What |
judge |
Decide what a write puts in the contract for balance, or why nothing. |
holds_key |
Whether text's |
rewrite |
text with |
write |
Rewrite the contract's counts when |
on_default_branch |
Whether HEAD is on the history of origin's default branch. |
submit |
Land a due change on the day's branch and open its pull request, never armed. |
Attributes:
| Name | Type | Description |
|---|---|---|
CALL_STEP |
|
|
ROWS_READ |
|
|
ROWS_NEEDED |
|
|
TOLERANCE |
|
|
HYSTERESIS |
|
|
COMMENTS |
|
COMMENTS
module-attribute
¶
COMMENTS = {'shards': "# The shards each runner's check leg runs as; fm ci.balance --write sets them.", 'pull-request-shards': "# The shards a pull request's check legs run as; fm ci.balance --write sets them."}
Reading
dataclass
¶
Reading(os: str, python: str, seen: int, rows: int, fixed: float, other: float, tests: float, speed: float)
One platform's measures, medians over its readable rows.
Attributes:
| Name | Type | Description |
|---|---|---|
os |
str
|
The runner label ( |
python |
str
|
The leg's Python ( |
seen |
int
|
The full green rows the platform had among the newest livery.workshop._balance.ROWS_READ read. |
rows |
int
|
Those among them that carry the test steps' wall in every shard; the medians are over these. |
fixed |
float
|
F, the setup and teardown the forge timed around the call, in seconds. |
other |
float
|
O, the summed wall of the gate steps that are not test steps, in seconds; zero when the rows name none. |
tests |
float
|
S, the summed test time over every package and shard, in seconds. |
speed |
float
|
p, the measured parallelism: a shard's S over its test steps' wall, the median over shards. |
Proposal
dataclass
¶
Proposal(counts: dict[str, int], slowest: str = '', wall: float = 0.0, unsharded: float = 0.0, capped: tuple[str, ...] = ())
The counts a set of readings proposes.
Attributes:
| Name | Type | Description |
|---|---|---|
counts |
dict[str, int]
|
The count per runner label, one entry per runner with enough rows; a runner with two Pythons takes the larger. |
slowest |
str
|
The slowest platform, as the table names it; empty when no platform had enough rows. |
wall |
float
|
The slowest platform's predicted wall at its count, in seconds. |
unsharded |
float
|
Its predicted wall at one shard. |
capped |
tuple[str, ...]
|
The platforms that took the cap without reaching the tolerance, as the table names them. |
Counts
dataclass
¶
Counts(key: str, runners: tuple[str, ...], current: dict[str, int], proposed: dict[str, int], before: float, after: float)
One key's count per runner: the contract's, the proposal's, and both walls.
Attributes:
| Name | Type | Description |
|---|---|---|
key |
str
|
|
runners |
tuple[str, ...]
|
The runners the key covers, in the contract's order. |
current |
dict[str, int]
|
The count per runner the contract gives, one where it is silent. |
proposed |
dict[str, int]
|
The count per runner the proposal gives; a runner whose platform keeps its count has its current one. |
before |
float
|
The predicted wall of the slowest leg at the current counts, in seconds, over the platforms with enough rows. |
after |
float
|
The same at the proposed counts. |
table
property
¶
The proposed counts as the contract holds them: those above one, in order.
Balance
dataclass
¶
Balance(lines: list[str], shards: Counts | None = None, pull_request: Counts | None = None, reason: str = '')
One reading of the record: the verb's lines and the counts a write judges.
Attributes:
| Name | Type | Description |
|---|---|---|
lines |
list[str]
|
The window line, the tables and the proposal, as |
shards |
Counts | None
|
The counts of |
pull_request |
Counts | None
|
The counts of |
reason |
str
|
Why the record proposed nothing; empty when it did. |
Verdict
dataclass
¶
Verdict(tables: dict[str, dict[str, int]] | None, why: str = '', what: str = '', before: float = 0.0, after: float = 0.0)
A write's decision on a Balance: the tables for the contract, or why none.
Attributes:
| Name | Type | Description |
|---|---|---|
tables |
dict[str, dict[str, int]] | None
|
The table per key the write puts in the contract, an empty one removing the key; None when nothing is written. |
why |
str
|
Why nothing is written; empty when something is. |
what |
str
|
The wall the decision read, |
before |
float
|
That wall at the contract's counts, in seconds. |
after |
float
|
That wall at the proposed counts. |
wall ¶
The leg's predicted wall at shards, in seconds: F + max(O, (S / p + O) / n).
Source code in packages/workshop/src/livery/workshop/_balance.py
runner_seconds ¶
About what shards runners spend together, in seconds: n F + S / p + O.
propose ¶
The counts for readings within cap; a platform short of rows takes no part.
The slowest platform's count stops at the first added shard that
saves less than its setup. Every other platform takes the smallest
count within the tolerance of the slowest platform's wall, or the
cap, named in capped.
Source code in packages/workshop/src/livery/workshop/_balance.py
readings ¶
Each platform's reading from rows, newest first; the platforms in name order.
A row counts for a platform when every job of that platform in it ran the full gate and succeeded, and the newest livery.workshop._balance.ROWS_READ such rows are read. Among them, a row whose shard carries no test step's wall is seen and not read. A job that is not a check leg is skipped.
Source code in packages/workshop/src/livery/workshop/_balance.py
spell ¶
table as the contract spells an inline table; none when empty.
Source code in packages/workshop/src/livery/workshop/_balance.py
report ¶
report(readings: Sequence[Reading], *, cap: int, runners: Sequence[str], pull_request_runners: Sequence[str] | None = None) -> list[str]
The lines the verb prints below the window line: the tables, then the proposal.
Every reading gets its table or its reason; a runner in runners without a reading keeps its count and says so. The proposal is over the readings of runners; the pull request proposal over pull_request_runners when that list is shorter, else the counts are the same and say so.
Source code in packages/workshop/src/livery/workshop/_balance.py
counts ¶
counts(readings: Sequence[Reading], *, key: str, runners: Sequence[str], current: Mapping[str, int], cap: int) -> Counts
One key's counts over runners: current, the proposal, and both walls.
A runner whose platform has too few rows keeps its current count. The walls are the slowest leg's among the platforms with enough rows, at the current counts and at the proposed ones.
Source code in packages/workshop/src/livery/workshop/_balance.py
read ¶
Read the record and the contract: what fm ci.balance prints, and the counts.
Reads the metrics series and the root contract's [ci] runners,
[ci] pull-request-runners, [ci] shards, [ci]
pull-request-shards and [ci] shards-max. cap is --max,
zero for the contract's cap. An unreadable series prints its reason
and proposes nothing; an empty one says so.
Source code in packages/workshop/src/livery/workshop/_balance.py
render ¶
What fm ci.balance prints: the window, the tables, the proposal.
read with the counts left out; cap means what it means there.
Source code in packages/workshop/src/livery/workshop/_balance.py
judge ¶
Decide what a write puts in the contract for balance, or why nothing.
Nothing is written when the record proposed nothing, when every
count is the contract's, and when the predicted wall at the
proposed counts is within the hysteresis of the wall at the
contract's counts. The run wall decides, over [ci] runners;
when the run's counts are the contract's and only the pull
request's differ, the pull request wall decides. Otherwise the
tables are [ci] shards and, when the pull request runners are
fewer, [ci] pull-request-shards, both at the proposal.
Source code in packages/workshop/src/livery/workshop/_balance.py
holds_key ¶
Whether text's [ci] table spells key as one of its own keys.
Source code in packages/workshop/src/livery/workshop/_balance.py
rewrite ¶
text with [ci] <key> at table, every other line kept as written.
A key the table holds is rewritten in place as one inline table,
its trailing comment kept, however many lines it spanned; an
empty table removes the key with the comment lines directly
above it, since a comment on a key that is gone misleads. A key
the table lacks is added after its last key, under its own comment
(livery.workshop._balance.COMMENTS), or under the header when
the table has none; a contract without a [ci] table gains one
at its end. An absent key with an empty table changes nothing.
Source code in packages/workshop/src/livery/workshop/_balance.py
write ¶
Rewrite the contract's counts when judge says to; the tables written, or None.
Prints what each key becomes and the wall the decision read, or
why nothing is written. Refuses by name when a key the contract
holds is not spelled as a key of the [ci] table, as a
[ci.shards] table or a dotted key: the rewrite keeps the
contract's text and cannot reach it there.
Source code in packages/workshop/src/livery/workshop/_balance.py
on_default_branch ¶
Whether HEAD is on the history of origin's default branch.
Origin's default branch is fetched as HEAD, so no branch name
is needed and a checkout a merge moved on since still answers; a
detached HEAD at a commit of that history answers True.
Raises:
| Type | Description |
|---|---|
Failed
|
when origin cannot be fetched, in git's words. |
Source code in packages/workshop/src/livery/workshop/_balance.py
submit ¶
Land a due change on the day's branch and open its pull request, never armed.
The act, each step printed: off origin's default branch nothing is
read or written, the line names the branch, and the verb exits
green, so a nightly dispatched on another branch skips it. On it,
the record is read and judged as write judges it; nothing due
means no branch and no pull request. Otherwise origin is asked for
the day's branch, chore/balance-<date>, before anything in the
checkout changes: one holding these counts, from an earlier run of
the day, means the day's balance is already submitted, and one
holding other counts refuses by name, for a person to merge or
close its pull request first. Then, from a clean tree, the branch
is started in this checkout from a fetched origin/main
(livery.workshop._issue_tasks.start_here), the contract is
rewritten, the sync renders the workflows at the counts
(livery.workshop._sync.sync_checkout), the change is committed
as chore(ci): the check legs run as <counts>
(livery.workshop._commit.commit_flow) and submitted unarmed
(livery.workshop._submit.submit_flow), its gate run there and
not twice, and the checkout returns to where it was. on is the
day, today when None; cap is --max.
Raises:
| Type | Description |
|---|---|
Failed
|
naming what stopped it, with nothing written when it stops before the write. |
Source code in packages/workshop/src/livery/workshop/_balance.py
955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 | |