livery.workshop._metrics¶
livery.workshop._metrics ¶
Timing rows on the CI state store, and the reader that renders them.
One row per job per run, one file per run on the metrics series
of the state store (livery.git), named by zero-padded
run id so that name order is time order. A row is end to end, not
gate-only: the queue wait, the job's wall, and every step's wall as
the forge times them, beside the per-task durations and lane waits
the leg's own trace recorded, and the test time per package summed
from the trace's pytest slices.
Three writers, all inside CI. Each check leg ends with put_leg: it
reads the trace the profiled gate wrote and puts the task half of
its row on the leg's per-run ref, in one write with every file the
leg's other entries left in its spool (LEG_SPOOL); a leg that ran
as one shard of several carries the record it left of what it
partitioned in the row, under shard; a write that fails is a red
leg. The run's gate job first folds each platform's shards into one
ref (livery.workshop._fold), then runs collect: it reads the
per-run refs, asks the forge for the run's jobs, joins the two
halves, folds in the sections the job's judges left (RUN_ROW),
puts the run's file on the series under its window, and writes each
check leg's seconds per item onto its platform leg's series of
the livery.workshop.DURATIONS family; it fails open loudly, a
printed reason and never a red job. The run's last green job runs
drop_halves: the per-run refs go once every job that writes a half
has completed green, so a re-run of a red gate job still finds what
its legs put, and a point with no gate job, where every job puts a
half and collects, keeps each job's half until the last one has
folded it in. Local runs read: render is what fm ci.timings
prints, the latest value, p50, and p90 per metric over the window, a
platform that ran as shards as one section with a line per shard, the
movers against a base window, and each section's trend.
Functions:
| Name | Description |
|---|---|
run_ref |
The per-run ref one leg of run writes. |
leave_leg_file |
Leave data in the leg's spool as name, for the leg's one write. |
leg_files |
The files the leg's entries left in its spool, by name; or why one cannot ride. |
leave_section |
Leave a section of the run's row, a number per package, for the gate job. |
sections |
The sections the gate job's entries left, by name; a malformed file is left out. |
leg_row |
The task half of job's row from its trace; |
put_leg |
Put job's task half and the leg's spool on its per-run ref, in one write. |
run_file |
The series file for run_id: zero-padded when numeric, so names sort by time. |
collect |
Join the legs' halves with the forge's times into the run's file; what happened. |
drop_halves |
Drop the run's per-run refs once every half is in and nothing is red. |
gate_steps |
The gate's steps of a leg's row by task key, in milliseconds. |
tests_ms |
The test time summed over packages, a row's |
first_shard_head |
The item seconds the first shard's unsplit steps stand for on row. |
render |
The timings, rendered: per job and metric the latest, p50, p90, and the movers. |
Attributes:
| Name | Type | Description |
|---|---|---|
SCHEMA |
|
|
SERIES |
|
|
RUNS |
Keyed
|
|
ROW_FILE |
|
|
LEG_SPOOL |
|
|
RUN_ROW |
|
|
SLOWEST |
|
|
GATE_STEPS |
|
|
TEST_PHASE |
|
RUNS
module-attribute
¶
run_ref ¶
run_ref(run: RunContext, leg: str) -> str
leave_leg_file ¶
Leave data in the leg's spool as name, for the leg's one write.
The leg's last entry, ci.leg, puts every file the spool holds
on the leg's per-run ref, a series of livery.workshop.RUNS,
beside its timing row and under its own name, where the gate job
reads it. That write refuses a file named row.json, the timing
row's own name. A file left twice under one name keeps the last
data.
Returns:
| Type | Description |
|---|---|
Path
|
The path of the file left under root. |
Source code in packages/workshop/src/livery/workshop/_metrics.py
leg_files ¶
The files the leg's entries left in its spool, by name; or why one cannot ride.
A file that is not a JSON object, or one named as the leg's own row, refuses by name: the write would carry what no reader takes.
Source code in packages/workshop/src/livery/workshop/_metrics.py
leave_section ¶
Leave a section of the run's row, a number per package, for the gate job.
The gate job's ci.metrics.collect entry, before its verdict,
folds every section left at root into the run's file on the
metrics series, and fm ci.timings renders each as a trend, the
numbers followed by unit. A section left twice under one name
keeps the last numbers.
Returns:
| Type | Description |
|---|---|
str
|
The line to print, naming the section and how many packages |
str
|
it holds. |
Source code in packages/workshop/src/livery/workshop/_metrics.py
sections ¶
The sections the gate job's entries left, by name; a malformed file is left out.
A section is its unit and a number per package.
Source code in packages/workshop/src/livery/workshop/_metrics.py
leg_row ¶
The task half of job's row from its trace; (row, "") or (None, reason).
The trace is the Chrome trace the profiled gate wrote: every
task's slice (its request address and duration), the lane waits,
each step inside its task, and each pytest phase of every test as
a slice of its own. A step is a slice nested in its task, or an
async pair (a begin and an end under one id) where it overlaps a
sibling on the task's track, which is how footman draws the steps
of one parallel() block; both read as the step's whole wall.
The row's items is the seconds each item took, the test file
of each test's node id, every phase of every test in it summed, to
a tenth. A missing or unreadable trace is a reason, never a row.
Source code in packages/workshop/src/livery/workshop/_metrics.py
280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 | |
put_leg ¶
Put job's task half and the leg's spool on its per-run ref, in one write.
Returns the lines to print and why the write failed, "" when
it landed or there was nothing to write. The row is read from
trace (livery.workshop._metrics.leg_row), with the scope the
gate ran; a trace that cannot be read leaves the row out, says
why, and the spool's files go alone. A leg that ran as one shard
of several left what it partitioned in SHARD_FILE
(livery.workshop.read_shard), and the row carries that record
under shard for the gate job's fold; a leg that is one shard
of one leaves no such file and its row carries no shard. Every
file the leg's entries left in LEG_SPOOL rides beside the row
under its own name (livery.workshop._metrics.leg_files), so
the leg writes its ref once. A leg without a label, a shard file
that is not a record, and a spool file no reader takes, fail the
write by name.
Source code in packages/workshop/src/livery/workshop/_metrics.py
run_file ¶
The series file for run_id: zero-padded when numeric, so names sort by time.
collect ¶
collect(root: Path, repo: Repository, run: RunContext, *, sha: str) -> list[str]
Join the legs' halves with the forge's times into the run's file; what happened.
Every line names what was done or why not. Nothing is written
when no leg left a row. The per-run refs stay: drop_halves
takes them once every half is in, so a re-run of a red gate job
still unions what its legs put, and a point whose every job
collects folds each earlier job's half into its own write. A job
the forge does not know keeps its trace half and is named; a row
of another schema is skipped and named. The run is looked up
under the head the runner's event names (_forge_jobs); sha
is the checkout, the fallback and the row's own record. Every
job the forge lists is recorded, the legs with their traces, the
others with their forge times alone, and the run's own wall from
its start to this collection. A platform that ran as shards has
one row, folded under the platform's display name before the
collect (livery.workshop._fold), a name the forge never ran:
the row takes the forge times of the first shard's job and
concludes as the shards' jobs did, and the forge's rows of the
shards stand beside it with their times alone, for
fm ci.timings. After the run's file lands, each check leg's
seconds per item go onto its platform leg's series of the
livery.workshop.DURATIONS family, one row per run, which the
next run's shards partition by; a leg without them writes none,
and a refused write is a line.
Source code in packages/workshop/src/livery/workshop/_metrics.py
548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 | |
drop_halves ¶
drop_halves(root: Path, repo: Repository, run: RunContext, *, sha: str, job: str, writers: tuple[str, ...]) -> list[str]
Drop the run's per-run refs once every half is in and nothing is red.
Runs as the last entry of a job that collected, after its own
verdict, so its own job is green when it runs and is left out of
the question. writers names the jobs that put a half, as the
schedule spells them (check); a forge job answers to a writer
by its display name's prefix (check (ubuntu-latest, 3.14)).
The refs stay, and the lines say so, while a writer other than
job has not completed green or any completed job is red: a
re-run of a red job finds what the legs put, and the last of a
point's collecting jobs is the one that drops. A job the forge
does not list, or a run it does not know, keeps the refs too; the
janitor's six-hour sweep is the net under every kept ref.
Source code in packages/workshop/src/livery/workshop/_metrics.py
gate_steps ¶
The gate's steps of a leg's row by task key, in milliseconds.
The row's tasks and the steps its tasks ran (task_steps) whose
key GATE_STEPS matches; a key in both is read from the tasks.
Source code in packages/workshop/src/livery/workshop/_metrics.py
tests_ms ¶
The test time summed over packages, a row's packages, in milliseconds.
Source code in packages/workshop/src/livery/workshop/_metrics.py
first_shard_head ¶
The item seconds the first shard's unsplit steps stand for on row.
The unsplit steps are the gate's steps outside the test phase
(TEST_PHASE); only a platform's first shard runs them, and the
folded row of a platform's shards carries the first shard's steps.
On the sharded runs the first shard's wall is about the summed wall
W of those steps plus its slice's test wall, not the longer of the
two: the steps hold the shard's cores while they run, and its tests
wait. The partition packs item seconds, and a second of a shard's
test wall holds p of them, p being the parallelism its tests reach,
so the head is W times p. p is the median, over the shards after
the first that the folded row lists under shard, of each one's
summed test time over its wall; with no such shard, the row's own
summed test time over its test steps' wall. When neither reads, the
head is the longest unsplit step's wall, the least the first
shard's tests wait.
Returns:
| Type | Description |
|---|---|
float | None
|
The head, to a tenth of a second; |
float | None
|
no unsplit step. |
Source code in packages/workshop/src/livery/workshop/_metrics.py
render ¶
The timings, rendered: per job and metric the latest, p50, p90, and the movers.
The window is every run the series holds. The latest since runs are the recent side; the base runs before them are the base the movers compare against, by the change of the recent median over the base median. A platform that ran as shards is one section, its folded row, and under its metrics one line per shard with the shard's wall and queue times from the newest run, read from the forge's own rows of the shards; those rows are no sections of their own, and the movers read the platform legs alone. Nothing yet, an unreadable series, and a row of another schema each say so.
Source code in packages/workshop/src/livery/workshop/_metrics.py
962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 | |