1"""Predict each check platform's wall at one to several shards, and propose the counts.
3The reader behind ``fm ci.balance``. It reads the timing rows the
4gate job writes ([livery.workshop._metrics][]) per platform: the os
5and Python of a check job's display name, ``check (macos-latest,
63.14)``; ``check (macos-latest, 3.14, 2of4)`` is one shard of that
7platform, and a GitLab job's bare name reads its platform from the
8row's own leg label. Per platform it takes the newest ten rows whose
9jobs for it all ran the full gate and succeeded, folds the shards of
10one platform into one, and takes the medians of four measures:
12- F, the fixed setup: every step the forge timed except the ``Check``
13 call, from the first shard, in seconds.
14- O, the summed wall of the gate's steps that are not test steps,
15 from the first shard. The gate's steps are the job's
16 ``check/<phase>.<step>`` tasks, and the test steps are the test
17 phase's, [livery.workshop._metrics.TEST_PHASE][].
18- S, the summed test time over every package and every shard.
19- p, the measured parallelism: a shard's S over its test steps' wall,
20 the median over shards.
22The wall at n shards is predicted as F + max(O, (S / p + O) / n). The
23first shard runs the unsplit steps, and its tests wait about their
24summed wall, so the partition starts the first shard's slice O p item
25seconds in ([livery.workshop._metrics.first_shard_head][]): the S + O p
26item seconds split evenly over the n shards, and no count brings a
27platform under F + O, the first shard with no tests left. The runner
28seconds at n are about n F + S / p + O. The slowest platform, the one
29with the largest wall at one shard, takes the largest count within
30the cap whose every added shard saved at least F, so a shard that
31costs more setup than it saves is not proposed. Every other platform
32takes the smallest count whose wall is within
33[livery.workshop._balance.TOLERANCE][] of the slowest
34platform's at its count; one that cannot get there at the cap takes
35the cap and is named. The pull request counts apply the same rule
36over a pull request's runners when they are fewer than ``[ci]
37runners`` names. A platform with fewer than
38[livery.workshop._balance.ROWS_NEEDED][] readable rows keeps its count
39and takes no part. The cap is ``[ci] shards-max``, which the verb
40reads and never writes.
42The verb prints the table and the TOML the contract would hold. With
43``--write`` it rewrites ``[ci] shards``, and ``[ci]
44pull-request-shards`` when the pull request runners are fewer, in the
45root ``workshop.toml``, keeping every comment and every other key:
46when a count differs from the contract's and the predicted wall at
47the proposed counts differs from the wall at the contract's by more
48than [livery.workshop._balance.HYSTERESIS][], so a count does not move
49back and forth between two readings that predict about the same
50wall. With ``--submit`` the change lands on the day's branch,
51``chore/balance-<date>``, rendered, committed and submitted unarmed
52for a person to merge; the nightly runs that once a day. The
53constants here were chosen before the rows could set them, as the
54speed marks' margins were ([livery.workshop._speed][]).
56Reach for `render` for what the verb prints, `read` for the same with
57the counts a write judges, `judge` for the write's decision, `write`
58and `submit` for the two acts, `readings` for the measures from rows
59already in hand, `wall` and `runner_seconds` for the prediction, and
60`propose` for the counts.
61"""
83#: The step the forge times for the gate's own call. Every other step
84#: the forge times is the job's setup and teardown, F.
87#: The newest full green rows read per platform.
90#: A platform with fewer readable rows than this keeps its count: a
91#: median of two says little. Chosen before the rows could set it.
94#: Another platform's wall within this fraction of the slowest
95#: platform's counts as level with it. Chosen before the rows could
96#: set it; a reading of a month of rows may move it.
99#: The write happens only when the predicted wall at the proposed
100#: counts differs from the wall at the contract's counts by more than
101#: this fraction of the latter, so two readings that predict about the
102#: same wall do not move a count back and forth. Chosen before the rows
103#: could set it.
106#: The comment written above a key the balancer adds to the contract.
108 "shards": (
109 "# The shards each runner's check leg runs as; fm ci.balance --write sets them."
110 ),
111 "pull-request-shards": (
112 "# The shards a pull request's check legs run as; fm ci.balance --write"
113 " sets them."
114 ),
115}
118 r"^check \((?P<os>[^,()]+), (?P<version>[^,()]+)(?:, (?P<shard>\d+)of\d+)?\)$"
119)
121 r"^check-(?P<os>.+?)-(?P<version>\d+(?:\.\d+)*)(?:-(?P<shard>\d+)of\d+)?$"
122)
127 """One platform's measures, medians over its readable rows.
129 Attributes:
130 os: The runner label (``macos-latest``).
131 python: The leg's Python (``3.14``).
132 seen: The full green rows the platform had among the newest
133 [livery.workshop._balance.ROWS_READ][] read.
134 rows: Those among them that carry the test steps' wall in
135 every shard; the medians are over these.
136 fixed: F, the setup and teardown the forge timed around the
137 call, in seconds.
138 other: O, the summed wall of the gate steps that are not test
139 steps, in seconds; zero when the rows name none.
140 tests: S, the summed test time over every package and shard,
141 in seconds.
142 speed: p, the measured parallelism: a shard's S over its test
143 steps' wall, the median over shards.
144 """
157 """The platform as the table names it: ``macos-latest, 3.14``."""
158 return f"{self.os}, {self.python}"
162 """Whether the platform has the rows a proposal needs."""
163 return self.rows >= ROWS_NEEDED
168 """The counts a set of readings proposes.
170 Attributes:
171 counts: The count per runner label, one entry per runner with
172 enough rows; a runner with two Pythons takes the larger.
173 slowest: The slowest platform, as the table names it; empty
174 when no platform had enough rows.
175 wall: The slowest platform's predicted wall at its count, in
176 seconds.
177 unsharded: Its predicted wall at one shard.
178 capped: The platforms that took the cap without reaching the
179 tolerance, as the table names them.
180 """
190 """The leg's predicted wall at *shards*, in seconds: F + max(O, (S / p + O) / n)."""
191 spread = (reading.tests / reading.speed + reading.other) / shards
192 return reading.fixed + max(reading.other, spread)
196 """About what *shards* runners spend together, in seconds: n F + S / p + O."""
197 return shards * reading.fixed + reading.tests / reading.speed + reading.other
201 return wall(reading, 1)
205 """The counts for *readings* within *cap*; a platform short of rows takes no part.
207 The slowest platform's count stops at the first added shard that
208 saves less than its setup. Every other platform takes the smallest
209 count within the tolerance of the slowest platform's wall, or the
210 cap, named in ``capped``.
211 """
212 cap = max(1, cap)
213 able = [reading for reading in readings if reading.enough]
214 if not able:
215 return Proposal({})
216 slowest = max(able, key=_unsharded)
217 count = 1
218 while (
219 count < cap and wall(slowest, count) - wall(slowest, count + 1) >= slowest.fixed
220 ):
221 count += 1
222 target = wall(slowest, count) * (1 + TOLERANCE)
223 counts: dict[str, int] = {}
224 capped: list[str] = []
225 for reading in able:
226 if reading is slowest:
227 chosen = count
228 else:
229 chosen = next(
230 (n for n in range(1, cap + 1) if wall(reading, n) <= target), cap
231 )
232 if wall(reading, chosen) > target:
233 capped.append(reading.platform)
234 counts[reading.os] = max(counts.get(reading.os, 0), chosen)
235 return Proposal(
236 counts, slowest.platform, wall(slowest, count), wall(slowest, 1), tuple(capped)
237 )
241 if isinstance(value, bool) or not isinstance(value, int | float):
242 return None
243 return float(value)
247 """The os, Python and shard index of a check job; None for another job.
249 The display name first (``check (ubuntu-latest, 3.14, 2of4)``),
250 then the leg label the job's row carries under ``scope.leg``,
251 which is how a GitLab job, displayed as the bare ``check``, names
252 its runner. An unsharded job is shard 1.
253 """
254 match = _DISPLAY.match(name)
255 if match is None:
256 scope = job.get("scope")
257 leg = str(scope.get("leg") or "") if isinstance(scope, Mapping) else ""
258 match = _LABEL.match(leg)
259 if match is None:
260 return None
261 return match["os"], match["version"], int(match["shard"] or 1)
265 scope = job.get("scope")
266 return (
267 job.get("conclusion") == "success"
268 and isinstance(scope, Mapping)
269 and scope.get("scope") == "full"
270 )
274 """The forge's steps of *job* except the call, summed; an untimed step is 0."""
275 total = 0.0
276 for step in job.get("steps") or []:
277 if isinstance(step, Mapping) and step.get("name") != CALL_STEP:
278 total += _number(step.get("ms")) or 0.0
279 return total
291 """One row's measures for a platform from its shards' jobs, the first shard first.
293 None when a shard carries no test step's wall or no test time: p
294 cannot be read from it. A shard's test wall is its test steps'
295 summed.
296 """
297 speeds: list[float] = []
298 total = 0.0
299 for job in shards:
300 steps = gate_steps(job)
301 test_ms = sum(ms for key, ms in steps.items() if key.startswith(TEST_PHASE))
302 tests = tests_ms(job.get("packages"))
303 if test_ms <= 0 or tests <= 0:
304 return None
305 speeds.append(tests / test_ms)
306 total += tests
307 steps = gate_steps(shards[0])
308 other = sum(ms for key, ms in steps.items() if not key.startswith(TEST_PHASE))
309 return _Measures(
310 _fixed_ms(shards[0]) / 1000.0,
311 other / 1000.0,
312 total / 1000.0,
313 statistics.median(speeds),
314 )
318 """Each platform's reading from *rows*, newest first; the platforms in name order.
320 A row counts for a platform when every job of that platform in it
321 ran the full gate and succeeded, and the newest
322 [livery.workshop._balance.ROWS_READ][] such rows are read. Among
323 them, a row whose shard carries no test step's wall is seen and
324 not read. A job that is not a check leg is skipped.
325 """
326 seen: dict[tuple[str, str], int] = defaultdict(int)
327 measures: dict[tuple[str, str], list[_Measures]] = defaultdict(list)
328 for row in rows:
329 jobs = row.get("jobs")
330 if not isinstance(jobs, Mapping):330 ↛ 331line 330 didn't jump to line 331 because the condition on line 330 was never true
331 continue
332 by_platform: dict[tuple[str, str], list[tuple[int, Mapping[str, Any]]]] = (
333 defaultdict(list)
334 )
335 for name, job in jobs.items():
336 if not isinstance(job, Mapping):336 ↛ 337line 336 didn't jump to line 337 because the condition on line 336 was never true
337 continue
338 found = _platform(str(name), job)
339 if found is not None:
340 by_platform[found[:2]].append((found[2], job))
341 for key, shards in by_platform.items():
342 if seen[key] >= ROWS_READ:
343 continue
344 shards.sort(key=itemgetter(0))
345 if not all(_full_green(job) for _, job in shards):
346 continue
347 seen[key] += 1
348 measured = _measure([job for _, job in shards])
349 if measured is not None:
350 measures[key].append(measured)
351 out: list[Reading] = []
352 for key in sorted(seen):
353 taken = measures.get(key, [])
354 out.append(
355 Reading(
356 key[0],
357 key[1],
358 seen[key],
359 len(taken),
360 fixed=statistics.median([m.fixed for m in taken]) if taken else 0.0,
361 other=statistics.median([m.other for m in taken]) if taken else 0.0,
362 tests=statistics.median([m.tests for m in taken]) if taken else 0.0,
363 speed=statistics.median([m.speed for m in taken]) if taken else 0.0,
364 )
365 )
366 return out
370 """The table of one platform, or the line saying why it keeps its count."""
371 if not reading.enough:
372 if reading.rows < reading.seen:
373 return [
374 f" {reading.platform}: keeps its count: {reading.seen} full green"
375 f" row(s), {reading.rows} with the test steps' wall ({TEST_PHASE}*),"
376 f" {ROWS_NEEDED} needed"
377 ]
378 return [
379 f" {reading.platform}: keeps its count: {reading.seen} full green"
380 f" row(s), {ROWS_NEEDED} needed"
381 ]
382 lines = [
383 f" {reading.platform}: {reading.rows} row(s): F {reading.fixed:.0f} s,"
384 f" O {reading.other:.0f} s, S {reading.tests:.0f} s, p {reading.speed:.1f}",
385 f" {'shards':>6} {'wall':>8} {'runner s':>10}",
386 ]
387 lines.extend(
388 f" {n:>6} {wall(reading, n):>6.0f} s {runner_seconds(reading, n):>8.0f} s"
389 for n in range(1, cap + 1)
390 )
391 return lines
395 """The TOML the contract would hold for *key*, or why there is none."""
396 if not proposal.counts:
397 return (
398 f" nothing proposed: no platform has {ROWS_NEEDED} full green rows"
399 " with the test steps' wall"
400 )
401 sharded = _ordered(proposal.counts, runners)
402 if not sharded:
403 return f" [ci] {key}: none, every platform runs as one shard"
404 return f" [ci] {key} = {spell(sharded)}"
408 """The counts above one, in the order of *runners*, as the contract holds them."""
409 return {runner: counts[runner] for runner in runners if counts.get(runner, 1) > 1}
413 """*table* as the contract spells an inline table; ``none`` when empty."""
414 if not table:
415 return "none"
416 return "{ " + ", ".join(f'"{runner}" = {n}' for runner, n in table.items()) + " }"
420 if not proposal.counts:
421 return []
422 return [
423 f" predicted {what} wall: {proposal.wall:.0f} s for the slowest leg,"
424 f" from {proposal.unsharded:.0f} s"
425 ]
429 readings: Sequence[Reading],
430 *,
431 cap: int,
432 runners: Sequence[str],
433 pull_request_runners: Sequence[str] | None = None,
434) -> list[str]:
435 """The lines the verb prints below the window line: the tables, then the proposal.
437 Every reading gets its table or its reason; a runner in *runners*
438 without a reading keeps its count and says so. The proposal is
439 over the readings of *runners*; the pull request proposal over
440 *pull_request_runners* when that list is shorter, else the counts
441 are the same and say so.
442 """
443 cap = max(1, cap)
444 lines: list[str] = []
445 for reading in readings:
446 lines.extend(_platform_lines(reading, cap))
447 present = {reading.os for reading in readings}
448 lines.extend(
449 f" {runner}: keeps its count: 0 full green row(s), {ROWS_NEEDED} needed"
450 for runner in runners
451 if runner not in present
452 )
453 full = propose([r for r in readings if r.os in runners], cap=cap)
454 lines.extend(_named_capped(readings, full, cap))
455 lines.append(_toml_line("shards", full, runners))
456 narrower = pull_request_runners is not None and len(pull_request_runners) < len(
457 runners
458 )
459 pulls = (
460 propose([r for r in readings if r.os in (pull_request_runners or ())], cap=cap)
461 if narrower
462 else None
463 )
464 if pulls is not None:
465 lines.extend(_named_capped(readings, pulls, cap))
466 lines.append(
467 _toml_line("pull-request-shards", pulls, pull_request_runners or ())
468 )
469 elif full.counts:
470 lines.append(" [ci] pull-request-shards: the same")
471 lines.extend(_wall_line("run", full))
472 if pulls is not None:
473 lines.extend(_wall_line("pull request", pulls))
474 return lines
478 readings: Sequence[Reading], proposal: Proposal, cap: int
479) -> list[str]:
480 """One line per platform that took the cap without reaching the tolerance."""
481 by_platform = {reading.platform: reading for reading in readings}
482 percent = round(TOLERANCE * 100)
483 return [
484 f" {platform} takes the cap, {cap}: {wall(by_platform[platform], cap):.0f} s"
485 f" is not within {percent}% of {proposal.wall:.0f} s"
486 for platform in proposal.capped
487 ]
491 """The day a row's run was created, else the day the store stamped it."""
492 stamp = str(entry.get("created_at") or entry.get("when") or "")
493 return stamp[:10] or "an unknown day"
498 """One key's count per runner: the contract's, the proposal's, and both walls.
500 Attributes:
501 key: ``shards`` or ``pull-request-shards``.
502 runners: The runners the key covers, in the contract's order.
503 current: The count per runner the contract gives, one where it
504 is silent.
505 proposed: The count per runner the proposal gives; a runner
506 whose platform keeps its count has its current one.
507 before: The predicted wall of the slowest leg at the current
508 counts, in seconds, over the platforms with enough rows.
509 after: The same at the proposed counts.
510 """
521 """Whether a count differs between the proposal and the contract."""
522 return self.proposed != self.current
526 """Whether the wall moves by more than the hysteresis between the two."""
527 return abs(self.after - self.before) > HYSTERESIS * self.before
531 """The proposed counts as the contract holds them: those above one, in order."""
532 return _ordered(self.proposed, self.runners)
536 """The current counts as the contract holds them."""
537 return _ordered(self.current, self.runners)
542 """One reading of the record: the verb's lines and the counts a write judges.
544 Attributes:
545 lines: The window line, the tables and the proposal, as `render`
546 prints them.
547 shards: The counts of ``[ci] shards`` over ``[ci] runners``;
548 None when the record proposed nothing, and ``reason`` says
549 why.
550 pull_request: The counts of ``[ci] pull-request-shards`` over
551 ``[ci] pull-request-runners`` when that list is shorter than
552 ``[ci] runners``; None otherwise, and a write leaves the key
553 as it is.
554 reason: Why the record proposed nothing; empty when it did.
555 """
565 """A write's decision on a `Balance`: the tables for the contract, or why none.
567 Attributes:
568 tables: The table per key the write puts in the contract, an
569 empty one removing the key; None when nothing is written.
570 why: Why nothing is written; empty when something is.
571 what: The wall the decision read, ``run`` or ``pull request``.
572 before: That wall at the contract's counts, in seconds.
573 after: That wall at the proposed counts.
574 """
584 readings: Sequence[Reading],
585 *,
586 key: str,
587 runners: Sequence[str],
588 current: Mapping[str, int],
589 cap: int,
590) -> Counts:
591 """One key's counts over *runners*: *current*, the proposal, and both walls.
593 A runner whose platform has too few rows keeps its current count.
594 The walls are the slowest leg's among the platforms with enough
595 rows, at the current counts and at the proposed ones.
596 """
597 proposal = propose([r for r in readings if r.os in runners], cap=cap)
598 proposed = {
599 runner: proposal.counts.get(runner, current[runner]) for runner in runners
600 }
601 able = [r for r in readings if r.os in runners and r.enough]
602 return Counts(
603 key,
604 tuple(runners),
605 dict(current),
606 proposed,
607 max((wall(r, current[r.os]) for r in able), default=0.0),
608 max((wall(r, proposed[r.os]) for r in able), default=0.0),
609 )
613 """The cap the proposal stays within: ``[ci] shards-max``, or *cap* when under it.
615 Zero reads the contract's. A *cap* above the contract's refuses by
616 name: a count above ``[ci] shards-max`` is one the contract refuses
617 at its next load.
618 """
619 most = shards_max(ci)
620 if cap <= 0:
621 return most
622 if cap > most:
623 fail(
624 f"--max={cap} is above [ci] shards-max, {most}: raise the cap in the"
625 " contract, or lower --max"
626 )
627 return cap
631 """Read the record and the contract: what ``fm ci.balance`` prints, and the counts.
633 Reads the metrics series and the root contract's ``[ci] runners``,
634 ``[ci] pull-request-runners``, ``[ci] shards``, ``[ci]
635 pull-request-shards`` and ``[ci] shards-max``. *cap* is ``--max``,
636 zero for the contract's cap. An unreadable series prints its reason
637 and proposes nothing; an empty one says so.
638 """
639 found = SERIES.rows(state_store(root))
640 if found.failed:
641 return Balance(
642 [f" {found.reason}", " nothing proposed: the record could not be read"],
643 reason="the record could not be read",
644 )
645 entries = [row.data for row in found.rows]
646 if not entries:
647 return Balance(
648 [
649 " no timing rows yet: the gate job writes one per run",
650 " nothing proposed: the record holds no row",
651 ],
652 reason="the record holds no row",
653 )
654 lines = [f" {line}" for line in found.skipped]
655 lines.append(
656 f" {len(entries)} run(s) in the window, {_day(entries[-1])} to"
657 f" {_day(entries[0])}; the newest {ROWS_READ} full green row(s) per platform"
658 " are read"
659 )
660 ci = load_contract(root / "workshop.toml").get("ci") or {}
661 runners = [str(runner) for runner in ci.get("runners") or ["ubuntu-latest"]]
662 pulls = pull_request_runners(ci)
663 cap = _cap(ci, cap)
664 measured = readings(entries)
665 lines.extend(report(measured, cap=cap, runners=runners, pull_request_runners=pulls))
666 if not any(r.enough and r.os in runners for r in measured):
667 return Balance(
668 lines,
669 reason=(
670 f"no platform has {ROWS_NEEDED} full green rows with the test"
671 " steps' wall"
672 ),
673 )
674 current = shard_counts(ci, key="shards", runners=runners, listed="[ci] runners")
675 shards = counts(measured, key="shards", runners=runners, current=current, cap=cap)
676 pull_request = None
677 if len(pulls) < len(runners):
678 if ci.get("pull-request-shards") is None:
679 held = {runner: current[runner] for runner in pulls}
680 else:
681 held = shard_counts(
682 ci,
683 key="pull-request-shards",
684 runners=pulls,
685 listed="[ci] pull-request-runners",
686 )
687 pull_request = counts(
688 measured, key="pull-request-shards", runners=pulls, current=held, cap=cap
689 )
690 return Balance(lines, shards, pull_request)
694 """What ``fm ci.balance`` prints: the window, the tables, the proposal.
696 `read` with the counts left out; *cap* means what it means there.
697 """
698 return read(root, cap=cap).lines
702 """Decide what a write puts in the contract for *balance*, or why nothing.
704 Nothing is written when the record proposed nothing, when every
705 count is the contract's, and when the predicted wall at the
706 proposed counts is within the hysteresis of the wall at the
707 contract's counts. The run wall decides, over ``[ci] runners``;
708 when the run's counts are the contract's and only the pull
709 request's differ, the pull request wall decides. Otherwise the
710 tables are ``[ci] shards`` and, when the pull request runners are
711 fewer, ``[ci] pull-request-shards``, both at the proposal.
712 """
713 shards = balance.shards
714 if shards is None:
715 return Verdict(None, f"nothing written: {balance.reason}")
716 pulls = balance.pull_request
717 if shards.changed:
718 judged, what = shards, "run"
719 elif pulls is not None and pulls.changed:
720 judged, what = pulls, "pull request"
721 else:
722 return Verdict(None, "nothing written: every count is the contract's")
723 if not judged.moved:
724 return Verdict(
725 None,
726 f"nothing written: the predicted {what} wall moves from"
727 f" {judged.before:.0f} s to {judged.after:.0f} s, within"
728 f" {round(HYSTERESIS * 100)}%",
729 )
730 tables = {"shards": shards.table}
731 if pulls is not None:
732 tables["pull-request-shards"] = pulls.table
733 return Verdict(tables, "", what, judged.before, judged.after)
736#: The ``[ci]`` header, on a line of its own, a comment after it allowed.
739#: Any table header: the line's first character after its indent is ``[``.
742#: A key's line, bare or quoted, up to its ``=``.
747 lines: Sequence[str], first: int, end: int, column: int
748) -> tuple[int, str]:
749 """Where the value at *column* of line *first* ends, and its trailing comment.
751 The value ends on the first line that closes every bracket it
752 opened, brackets inside strings and comments not counted; the
753 comment is the one after the value on that line, empty without
754 one.
755 """
756 depth = 0
757 quote = ""
758 line = first
759 at = column
760 while line < end:760 ↛ 790line 760 didn't jump to line 790 because the condition on line 760 was always true
761 text = lines[line]
762 comment = ""
763 while at < len(text):
764 char = text[at]
765 if quote:
766 if char == "\\" and quote.startswith('"'):766 ↛ 767line 766 didn't jump to line 767 because the condition on line 766 was never true
767 at += 2
768 elif text.startswith(quote, at):
769 at += len(quote)
770 quote = ""
771 else:
772 at += 1
773 continue
774 if char == "#":
775 comment = text[at:].rstrip()
776 break
777 if char in "\"'":
778 quote = char * 3 if text.startswith(char * 3, at) else char
779 at += len(quote)
780 continue
781 if char in "[{":
782 depth += 1
783 elif char in "]}":
784 depth -= 1
785 at += 1
786 if depth <= 0 and not quote:
787 return line, comment
788 line += 1
789 at = 0
790 return end - 1, ""
795 """One key of the ``[ci]`` table as written.
797 Attributes:
798 name: The key, unquoted.
799 first: The line its name stands on.
800 last: The line its value ends on, the same for a one-line value.
801 comment: The comment after the value on its last line; empty
802 without one.
803 indent: What stands before the name on its line.
804 """
814 """The keys of the lines *start* to *end*, in order."""
815 found: list[_Key] = []
816 line = start
817 while line < end:
818 match = _KEY_LINE.match(lines[line])
819 if match is None:
820 line += 1
821 continue
822 last, comment = _value_end(lines, line, end, match.end())
823 found.append(
824 _Key(match.group(2) or match.group(3), line, last, comment, match.group(1))
825 )
826 line = last + 1
827 return found
831 """The ``[ci]`` header's line and the line after its last; None without one."""
832 header = next((i for i, line in enumerate(lines) if _CI_HEADER.match(line)), None)
833 if header is None:
834 return None
835 end = next(
836 (i for i in range(header + 1, len(lines)) if _HEADER.match(lines[i])),
837 len(lines),
838 )
839 return header, end
843 """Whether *text*'s ``[ci]`` table spells *key* as one of its own keys."""
844 lines = text.split("\n")
845 table = _ci_table(lines)
846 if table is None:846 ↛ 847line 846 didn't jump to line 847 because the condition on line 846 was never true
847 return False
848 header, end = table
849 return any(found.name == key for found in _keys(lines, header + 1, end))
853 """*text* with ``[ci] <key>`` at *table*, every other line kept as written.
855 A key the table holds is rewritten in place as one inline table,
856 its trailing comment kept, however many lines it spanned; an
857 empty *table* removes the key with the comment lines directly
858 above it, since a comment on a key that is gone misleads. A key
859 the table lacks is added after its last key, under its own comment
860 ([livery.workshop._balance.COMMENTS][]), or under the header when
861 the table has none; a contract without a ``[ci]`` table gains one
862 at its end. An absent key with an empty *table* changes nothing.
863 """
864 lines = text.split("\n")
865 spelled = f"{key} = {spell(table)}"
866 found = _ci_table(lines)
867 if found is None:
868 if not table:
869 return text
870 base = text.rstrip("\n")
871 return (base + "\n\n" if base else "") + f"[ci]\n{COMMENTS[key]}\n{spelled}\n"
872 header, end = found
873 keys = _keys(lines, header + 1, end)
874 for held in keys:
875 if held.name != key:
876 continue
877 if table:
878 after = f" {held.comment}" if held.comment else ""
879 lines[held.first : held.last + 1] = [held.indent + spelled + after]
880 else:
881 above = held.first
882 while above > header + 1 and lines[above - 1].strip().startswith("#"):
883 above -= 1
884 del lines[above : held.last + 1]
885 return "\n".join(lines)
886 if not table:
887 return text
888 if keys:
889 indent, at = keys[-1].indent, keys[-1].last + 1
890 else:
891 indent, at = "", header + 1
892 lines[at:at] = [indent + COMMENTS[key], indent + spelled]
893 return "\n".join(lines)
897 """Rewrite the contract's counts when `judge` says to; the tables written, or None.
899 Prints what each key becomes and the wall the decision read, or
900 why nothing is written. Refuses by name when a key the contract
901 holds is not spelled as a key of the ``[ci]`` table, as a
902 ``[ci.shards]`` table or a dotted key: the rewrite keeps the
903 contract's text and cannot reach it there.
904 """
905 verdict = judge(balance)
906 if verdict.tables is None:
907 print(f" {verdict.why}")
908 return None
909 path = root / "workshop.toml"
910 text = path.read_text(encoding="utf-8")
911 ci = load_contract(path).get("ci") or {}
912 for key, table in verdict.tables.items():
913 if ci.get(key) is not None and not holds_key(text, key):
914 fail(
915 f"[ci] {key} is not a key of the [ci] table in {path.name} (a"
916 f" [ci.{key}] table, or a dotted key): spell it `{key} = {{ ... }}`"
917 " under [ci] for the balancer to rewrite it"
918 )
919 text = rewrite(text, key, table)
920 path.write_text(text, encoding="utf-8")
921 for found in (balance.shards, balance.pull_request):
922 if found is not None and found.key in verdict.tables:
923 print(
924 f" [ci] {found.key}: {spell(found.held)} becomes"
925 f" {spell(verdict.tables[found.key])}"
926 )
927 print(
928 f" the predicted {verdict.what} wall moves from {verdict.before:.0f} s to"
929 f" {verdict.after:.0f} s"
930 )
931 return verdict.tables
935 """Whether HEAD is on the history of origin's default branch.
937 Origin's default branch is fetched as ``HEAD``, so no branch name
938 is needed and a checkout a merge moved on since still answers; a
939 detached HEAD at a commit of that history answers True.
941 Raises:
942 Failed: when origin cannot be fetched, in git's words.
943 """
944 try:
945 git.fetch(root, "HEAD")
946 except git.GitError as error:
947 fail(f"origin could not be fetched: {error}")
948 answer = git.call(root, "merge-base", "--is-ancestor", "HEAD", "FETCH_HEAD")
949 if answer.code not in (0, 1):949 ↛ 950line 949 didn't jump to line 950 because the condition on line 949 was never true
950 said = (answer.stderr or answer.stdout).strip()
951 fail(f"git merge-base exited {answer.code}: {said}")
952 return answer.code == 0
956 """Land a due change on the day's branch and open its pull request, never armed.
958 The act, each step printed: off origin's default branch nothing is
959 read or written, the line names the branch, and the verb exits
960 green, so a nightly dispatched on another branch skips it. On it,
961 the record is read and judged as `write` judges it; nothing due
962 means no branch and no pull request. Otherwise origin is asked for
963 the day's branch, ``chore/balance-<date>``, before anything in the
964 checkout changes: one holding these counts, from an earlier run of
965 the day, means the day's balance is already submitted, and one
966 holding other counts refuses by name, for a person to merge or
967 close its pull request first. Then, from a clean tree, the branch
968 is started in this checkout from a fetched ``origin/main``
969 ([livery.workshop._issue_tasks.start_here][]), the contract is
970 rewritten, the sync renders the workflows at the counts
971 ([livery.workshop._sync.sync_checkout][]), the change is committed
972 as ``chore(ci): the check legs run as <counts>``
973 ([livery.workshop._commit.commit_flow][]) and submitted unarmed
974 ([livery.workshop._submit.submit_flow][]), its gate run there and
975 not twice, and the checkout returns to where it was. *on* is the
976 day, today when None; *cap* is ``--max``.
978 Raises:
979 Failed: naming what stopped it, with nothing written when it
980 stops before the write.
981 """
982 from livery.git import remote_snapshot
984 if not on_default_branch(root):
985 where = git.current_branch(root) or "(detached)"
986 print(
987 f" not on origin's default branch (on {where}): nothing is written or"
988 " submitted; the balancer submits from the default branch alone"
989 )
990 return
991 with remote_snapshot(state_store(root), fetch=("metrics",)):
992 balance = read(root, cap=cap)
993 for line in balance.lines:
994 print(line)
995 verdict = judge(balance)
996 if verdict.tables is None:
997 print(f" {verdict.why}")
998 print(" no branch, no pull request")
999 return
1000 day = on or date.today()
1001 branch = f"chore/balance-{day:%Y%m%d}"
1002 if _origin_holds(root, branch):
1003 held = _held(root, branch, verdict.tables)
1004 if held == verdict.tables:
1005 print(
1006 f" the day's balance is already submitted: {branch} on origin holds"
1007 " these counts; nothing is written"
1008 )
1009 return
1010 fail(
1011 f"{branch} on origin holds other counts ({_spell_tables(held)}) than"
1012 f" this reading proposes ({_spell_tables(verdict.tables)}): merge or"
1013 " close its pull request, then run the balancer again; nothing is"
1014 " written"
1015 )
1016 if not git.is_clean(root):
1017 fail(
1018 "the working tree has uncommitted changes: commit or stash them, then"
1019 " run the balancer again; nothing is written"
1020 )
1021 from livery.workshop._commit import commit_flow
1022 from livery.workshop._forge_lane import this_repository
1023 from livery.workshop._issue_tasks import fetch_base, start_here
1024 from livery.workshop._submit import forge_failures_named, submit_flow
1025 from livery.workshop._sync import sync_checkout
1027 came_from = git.current_branch(root) or git.head_sha(root)
1028 repo = this_repository(root)
1029 fetch_base(root)
1030 start_here(root, repo, branch)
1031 write(root, balance)
1032 sync_checkout(root)
1033 subject = _subject(balance, verdict.tables)
1034 # The submit's gate judges this tree next; a gate at the commit
1035 # would judge the same tree twice.
1036 commit_flow(
1037 root,
1038 "chore",
1039 subject,
1040 scope="ci",
1041 body=(
1042 f"the predicted {verdict.what} wall moves from {verdict.before:.0f} s"
1043 f" to {verdict.after:.0f} s"
1044 ),
1045 check=False,
1046 )
1047 with forge_failures_named():
1048 number = submit_flow(
1049 repo,
1050 root,
1051 title=f"chore(ci): {subject}",
1052 armed=False,
1053 armed_reason="the balancer, which never arms",
1054 follow_to_verdict=False,
1055 )
1056 print(f" submitted unarmed: PR #{number} waits for a person")
1057 if git.local_branch_exists(root, came_from):1057 ↛ 1061line 1057 didn't jump to line 1061 because the condition on line 1057 was always true
1058 git.switch(root, came_from)
1059 print(f" back on {came_from}")
1060 else:
1061 git.run(root, "checkout", "--detach", came_from)
1062 print(f" back at {came_from[:12]}, detached as before")
1066 """The commit subject's words: the counts, and a pull request's that differ."""
1067 shards = tables["shards"]
1068 words = f"the check legs run as {_spoken(shards)}"
1069 pulls = balance.pull_request
1070 if pulls is not None:1070 ↛ 1074line 1070 didn't jump to line 1074 because the condition on line 1070 was always true
1071 same = {r: n for r, n in shards.items() if r in pulls.runners}
1072 if tables["pull-request-shards"] != same:1072 ↛ 1073line 1072 didn't jump to line 1073 because the condition on line 1072 was never true
1073 words += f", pull requests as {_spoken(tables['pull-request-shards'])}"
1074 return words
1078 return ", ".join(f"{runner} {n}" for runner, n in table.items()) or "one shard each"
1082 return "; ".join(f"[ci] {key} {spell(table)}" for key, table in tables.items())
1086 """Whether origin holds *branch*, asked of origin itself; a refusal names it."""
1087 try:
1088 return bool(git.ls_remote(root, f"refs/heads/{branch}"))
1089 except git.GitError as error:
1090 fail(f"could not ask origin for {branch}: {error}\n nothing is written")
1094 root: Path, branch: str, tables: Mapping[str, Mapping[str, int]]
1095) -> dict[str, dict[str, int]]:
1096 """The counts origin's *branch* holds for the keys of *tables*, as written."""
1097 git.fetch(root, f"+refs/heads/{branch}:refs/remotes/origin/{branch}")
1098 text = git.file_at(root, f"refs/remotes/origin/{branch}", "workshop.toml")
1099 loaded: dict[str, Any] = tomllib.loads(text) if text else {}
1100 ci = _table_or_empty(loaded.get("ci"))
1101 return {key: _table_of(ci.get(key)) for key in tables}
1105 """*value* when it is a table, else an empty one."""
1106 if isinstance(value, dict):
1107 table: dict[str, Any] = value
1108 return table
1109 return {}
1113 """The counts above one in *value*, a table of runner to count; none otherwise."""
1114 return {
1115 runner: n
1116 for runner, n in _table_or_empty(value).items()
1117 if isinstance(n, int) and not isinstance(n, bool) and n > 1
1118 }