1"""CI as points: what each point is, what it runs, and the verb that runs it.
3A consumer never thinks about CI. They reason about a named point in
4the process and attach a task to it. The four points are ``gate``
5(a pull request, or a dispatch by hand), ``merge`` (main after a
6merge), ``nightly`` (the clock), and ``release`` (dispatched by the
7merge point). A point is a declaration, [livery.workshop._points.Point][]:
8its workflow file, its events, and its jobs, each job stating what it
9needs in the workshop's words ([livery.workshop._points.Job][]); each
10forge's renderer says those in its own. The emitted workflow is a
11static shell, one ``fm ci.run --point=<p> --job=<j>`` per job, with
12``--shard=<i>of<n>`` on a check leg the contract splits, and what a
13job does is data: the builtin schedule below, plus the
14``[[ci.schedule]]`` entries a workspace declares in ``workshop.toml``.
15The matrix shape belongs to a point's job, never to a task.
17Reach for [livery.workshop._points.run_point][] to run a job's
18entries, and `schedule` to see them. Every entry runs as a child of
19the runner's own command, so a scheduled task is any task the
20workspace mounts, and the child's exit is the entry's.
22The merge point inherits the gate's jobs: a push to main runs the
23same checks, plus the jobs only a merge has (the docs deploy, the
24governance reconcile). A shell spells ``--point=gate`` on the shared
25jobs and the verb promotes the point on a push, so the YAML carries
26no decision.
28A package contributes a point of its own through ``[[ci.point]]`` in
29its ``workshop.toml``: a scheduled point with a dispatch entry, one
30job on the runners and Pythons it names, running the task it names
31with the job token and nothing more. `points` is every point a
32workspace has, the builtin four first; every reader takes that set.
33"""
50 LEG_VARIABLE,
51 POINT_VARIABLE,
52 run_context,
53 state_store,
54)
59 from livery.workshop._influence import Inputs
61#: The events a point may run on, in the forges' words.
65#: The nightly's dispatch input naming the scheduled tasks a run takes,
66#: comma-separated; empty runs the whole schedule. A rehearsal names the
67#: loop alone, ``fm ci.dispatch --point=nightly --only=ci.e2e``.
73 """One input a dispatched point takes, as the forge prompts for it.
75 Attributes:
76 name: The input's name, as the dispatch passes it.
77 description: What the input is, for the person dispatching.
78 required: Whether a dispatch without it is refused.
79 default: The value an absent optional input takes.
80 """
90 """One job of a point's shell, in the workshop's words.
92 Each forge's renderer says these in that forge's words: a grant, a
93 step, an action, an image. Nothing here is forge-shaped.
95 Attributes:
96 name: The job's name, and the ``--job`` its one call passes.
97 matrix: The legs the job fans out to: ``legs`` for one per
98 runner and gate Python, ``pythons`` for one per Python of
99 the whole matrix on the first runner, ``runners`` for one
100 per runner label the job's ``labels`` function names, with
101 no Python of its own, ``declared`` for one per runner and
102 Python the job itself names, ``""`` for one job on one
103 runner.
104 runners: The runners a ``declared`` matrix fans out to.
105 pythons: The Pythons a ``declared`` matrix fans out to.
106 labels: The function naming the runner labels a ``runners``
107 matrix fans out to, given the workspace's root; None for a
108 job of any other matrix. A ``runners`` job whose function
109 names no label renders no job, and a need on it drops.
110 needs: The jobs of the point this one waits for.
111 always: Whether the job runs when a job it needs was red, the
112 verdict job's shape.
113 fetch: How deep the checkout is: ``full`` for every commit,
114 ``tags`` for the tags as well, ``2`` for the commit and its
115 parent, ``""`` for the commit alone.
116 token: The credential the verb sees as ``FORGE_TOKEN``: ``job``
117 for the run's own token, ``repository`` for the repository's
118 secret when present and the job's otherwise, ``secret`` for
119 the repository's secret alone, ``admin`` for the admin
120 secret as ``FORGE_ADMIN_TOKEN``, ``""`` for none.
121 writes: Whether the job pushes to the repository (the state
122 store, a receipt tag), which needs the write grant.
123 pushes: Whether the job pushes through git itself, so the
124 checkout carries the credential the push needs.
125 publishes_index: Whether the job uploads to a package index,
126 so the index token reaches it as ``UV_PUBLISH_TOKEN``.
127 installs: The function naming the system packages the job
128 installs before it enters, given the workspace's root;
129 None for none. The extension that contributes the job
130 names it, so the render installs them without knowing
131 what they are for.
132 conan_cache: Whether the job builds native packages, so the
133 lane restores and saves conan's home around it.
134 deploy: The function naming the seam the job publishes a site
135 through after the call, given the workspace's root; None
136 for a job that publishes none. A ``pages`` seam is the
137 forge's own hosting: GitHub's grant and upload, GitLab's
138 ``pages`` job.
139 publishes: The artifact the job uploads, ``""`` for none.
140 paths: The paths the uploaded artifact carries, as the forge
141 globs them; ``()`` for a job that publishes none.
142 collects: The artifact the job downloads first, ``""`` for none.
143 environment: The named deployment environment the job runs in,
144 ``""`` for none.
145 dispatches: Whether the job starts another workflow through the
146 forge's API, which GitHub's workflow token may do only with
147 the ``actions: write`` grant.
148 inputs: The files the job's entries read, declared as a step
149 declares its own: a pull request's run that changed none of
150 them skips the entries. None for a job that always runs.
151 step: What the forge's run page calls the one call; the job's
152 name capitalised when empty.
153 note: The comment the rendered shell prints above the job.
154 joins: Whether the job is a contribution to a job the point
155 already has, declared or inherited. It renders as no job of
156 its own: its entries follow that job's, and its plumbing,
157 what it collects and publishes, the seam it deploys through
158 and the packages it installs, renders on that job under the
159 contributing point's own event condition.
160 """
190 """One point: its workflow file, its events, and its jobs.
192 Attributes:
193 name: The point's name, and the ``--point`` its jobs pass.
194 workflow: The workflow file the point's shell is, as GitHub and
195 Gitea address a dispatch and name a run. Two points may
196 share one, the gate and the merge point do, when their
197 events do not overlap. GitLab has one pipeline document and
198 names a dispatched pipeline after this.
199 events: The events that start the point's runs, from
200 `EVENT_NAMES`.
201 jobs: The point's own jobs, in order.
202 inherits: A point whose jobs this one runs first, the gate's
203 for the merge point; ``""`` for none.
204 inputs: The inputs a dispatch takes. A point with a required
205 one is dispatched by a verb that supplies it, never by
206 hand. Every job of a point that takes a ``workshop`` input
207 installs the released workshop it names, over the
208 checkout's own, before its one call.
209 ref_input: The input naming the commit the jobs check out,
210 ``""`` for the run's own commit.
211 cron: When the clock starts the point, for a point whose
212 events include ``schedule``, in cron's five fields, UTC.
213 GitHub and Gitea read it from the workflow file; on
214 GitLab the clock is a pipeline schedule the governance
215 reconcile creates from it.
216 note: The comment the rendered shell prints under its name.
217 """
230#: When the clock starts a scheduled point, UTC: the nightly's hour,
231#: and every contributed point's, whose cadence is judged on the
232#: runner by `due` rather than by a cron of its own.
235#: The four builtin points, in the order a change meets them.
237 Point(
238 "gate",
239 "ci.yml",
240 ("pull_request", "workflow_dispatch"),
241 note=(
242 "A pull request, a push to main, and a dispatch by hand. A"
243 " dispatched run pays the full gate: the check verb reads the"
244 " event and narrows on a pull request alone, so every event"
245 " spells the same call."
246 ),
247 jobs=(
248 Job(
249 "check",
250 matrix="legs",
251 fetch="full",
252 writes=True,
253 note=(
254 "The check legs: each runs the gate, and its row rides"
255 " its per-run ref on the state store, in one write with"
256 " the files its other entries left, for the gate job."
257 " The scoped gate measures from"
258 " the proved tree nearest the checkout's, an earlier"
259 " push's merge rebuilt from its two commits among them,"
260 " or from the merge base with the pull request's base"
261 " branch; each needs history a shallow clone lacks."
262 ),
263 ),
264 Job(
265 "gate",
266 needs=("check",),
267 always=True,
268 fetch="full",
269 token="job",
270 writes=True,
271 dispatches=True,
272 step="Verdict",
273 note=(
274 "The one required context. Branch protection points"
275 " here, so the matrix can grow or shrink without touching"
276 " repository settings. Its entries run the judges the"
277 " extensions add, collect the run's timing rows, ask the"
278 " forge for the jobs it needs, and stamp the tree a green"
279 " run proved; a red run is judged too. The stamp composes"
280 " a narrowed run with the record of the tree its legs"
281 " measured from, and names the two commits a merge"
282 " checkout joined. On a push to main the same job then"
283 " dispatches the release wave, runs what the extensions"
284 " add to the merge point, and runs the janitor."
285 ),
286 ),
287 ),
288 ),
289 Point(
290 "merge",
291 "ci.yml",
292 ("push",),
293 inherits="gate",
294 jobs=(
295 Job(
296 "govern",
297 fetch="2",
298 token="admin",
299 note=(
300 "The repository settings reconciled when the merge"
301 " changed a contract or the owners file. The commit's own"
302 " file list decides whether anything is governed; a depth"
303 " of one would read a squash as a root."
304 ),
305 ),
306 ),
307 ),
308 Point(
309 "nightly",
310 "nightly.yml",
311 ("schedule", "workflow_dispatch"),
312 inputs=(
313 Input(
314 ONLY,
315 "the scheduled tasks to run, comma-separated; empty runs the"
316 " whole schedule",
317 ),
318 ),
319 cron=CLOCK,
320 note=(
321 "The nightly point: the clock and a dispatch entry, one call per"
322 " Python. What runs is the workshop's builtin schedule plus"
323 " [[ci.schedule]] in workshop.toml, and the tests that declare"
324 " the nightly point are selected in."
325 ),
326 jobs=(
327 Job(
328 "nightly",
329 matrix="pythons",
330 fetch="full",
331 token="repository",
332 writes=True,
333 note=(
334 "A verification checks the tree out at a release tag. A pull"
335 " request a scheduled task opens with the job token starts"
336 " no workflow, so the repository's token carries the"
337 " nightly where there is one. The job's profile trace"
338 " rides the state store at its end, which needs the write"
339 " grant as the check legs' puts do."
340 ),
341 ),
342 ),
343 ),
344 Point(
345 "release",
346 "release.yml",
347 ("workflow_dispatch",),
348 inputs=(
349 Input("ref", "the release squash to publish", required=True),
350 Input(
351 "workshop",
352 "a released livery-workshop version to drive the wave; empty"
353 " runs the squash's own",
354 ),
355 ),
356 ref_input="ref",
357 note=(
358 "The train: a workflow.release PR merges, this publishes its"
359 " squash, and the receipt tags are cut only after the index"
360 " confirms each member. A tag is a receipt, never a trigger: the"
361 " merge point's dispatch job starts the wave at the release"
362 " squash, and a hand dispatch with --ref is the recovery entry"
363 " when a publish died mid-wave; --workshop names a released"
364 " driver for a wave whose own workshop was the fault."
365 ),
366 jobs=(
367 Job(
368 "publish",
369 fetch="full",
370 token="repository",
371 writes=True,
372 pushes=True,
373 publishes_index=True,
374 environment="pypi",
375 step="Publish the wave",
376 note=(
377 "The wave: publish the ref, cut the receipt tags after"
378 " the index confirms each member. The receipt push"
379 " carries the repository's token where there is one: the"
380 " job token may not push a ref whose commit carries a"
381 " workflow file that differs from the tip's."
382 ),
383 ),
384 ),
385 ),
386)
390 """Refuse a set of declarations a shell cannot be rendered from.
392 A point name declared twice, an event outside `EVENT_NAMES`, a
393 point inheriting one that is not declared, two points sharing a
394 workflow file whose events overlap (a run would belong to both),
395 a job name declared twice in one point, a job needing a job the
396 point does not have, a job collecting an artifact no job of the
397 point publishes, and a job collecting an artifact from jobs it
398 does not wait for each refuse naming the point and the fault. A
399 job starts once the jobs it needs end, so a job it does not wait
400 for may not have uploaded the artifact yet. A join waits for what
401 the job it joins waits for.
402 """
403 by_name: dict[str, Point] = {}
404 for point in points:
405 if point.name in by_name:
406 fail(f"the {point.name} point is declared twice")
407 by_name[point.name] = point
408 for point in points:
409 for event in point.events:
410 if event not in EVENT_NAMES:
411 fail(
412 f"the {point.name} point runs on {event!r}, which is not an"
413 f" event; the events are {', '.join(EVENT_NAMES)}"
414 )
415 if point.inherits and point.inherits not in by_name:
416 fail(
417 f"the {point.name} point inherits {point.inherits!r}, which is"
418 " not a declared point"
419 )
420 if ("schedule" in point.events) != bool(point.cron):420 ↛ 421line 420 didn't jump to line 421 because the condition on line 420 was never true
421 fail(
422 f"the {point.name} point "
423 + (
424 "runs on the clock and names no cron"
425 if "schedule" in point.events
426 else "names a cron and does not run on the clock"
427 )
428 )
429 for other in points:
430 if other.name >= point.name or other.workflow != point.workflow:
431 continue
432 shared = sorted(set(point.events) & set(other.events))
433 if shared:
434 fail(
435 f"the {other.name} and {point.name} points share"
436 f" {point.workflow} and both run on {', '.join(shared)}:"
437 " a run would belong to both"
438 )
439 inherited = by_name[point.inherits].jobs if point.inherits else ()
440 names: list[str] = [job.name for job in inherited if not job.joins]
441 for job in point.jobs:
442 if job.joins:
443 continue
444 if job.name in names:
445 fail(f"the {point.name} point declares the job {job.name!r} twice")
446 names.append(job.name)
447 lineage = (*inherited, *point.jobs)
448 # A join renders on the job it joins: it waits for what that job
449 # waits for, and its own needs never render.
450 waits = {job.name: job.needs for job in lineage if not job.joins}
451 for job in point.jobs:
452 for need in job.needs:
453 if need not in names:
454 fail(
455 f"the {point.name} point's {job.name} job needs"
456 f" {need!r}, which the point does not have; its jobs"
457 f" are {', '.join(names)}"
458 )
459 if not job.collects:
460 continue
461 publishers = list(
462 dict.fromkeys(
463 other.name for other in lineage if other.publishes == job.collects
464 )
465 )
466 if not publishers:
467 fail(
468 f"the {point.name} point's {job.name} job collects"
469 f" {job.collects!r}, which no job of the point publishes"
470 )
471 if not set(publishers) & set(waits.get(job.name, job.needs)):
472 fail(
473 f"the {point.name} point's {job.name} job collects"
474 f" {job.collects!r} and waits for no job that publishes it"
475 f" ({', '.join(publishers)}); a job collects only from the"
476 " jobs it needs"
477 )
482#: The builtin points by name.
485#: The builtin points, in the order a change meets them. A workspace
486#: may have more: `points` adds the ones its packages contribute.
489#: What a contributed point's name may look like: a workflow file
490#: name and a job name at once.
494#: The runner's merged task tree, kept by the workshop's ``pre_tasks``
495#: hook for the span of one invocation: discovery merges every extension's
496#: tree into the invocation and resets the module-level root, so a
497#: check against what this runner mounts reads it here.
502 """Whether the runner mounts the task at the dotted address *task*.
504 The invocation's merged tree when a run kept one. A process that
505 kept no tree, a test reading the repository's own contract, holds
506 only what it happened to import, never the workspace's mounted
507 set, so it cannot answer and defers: the runner checks at the
508 point's dispatch and at every load inside a run.
509 """
510 if MOUNTED is None:510 ↛ 512line 510 didn't jump to line 512 because the condition on line 510 was always true
511 return True
512 return MOUNTED.get(task) is not None
517 """A point a package contributes, and the entry that runs it.
519 Attributes:
520 point: The point, a scheduled one with a dispatch entry and
521 one job named after it.
522 entry: The task the job runs, with its cadence.
523 """
530 root: Path, *, mounted: Callable[[str], bool] | None = None
531) -> tuple[Contribution, ...]:
532 """The ``[[ci.point]]`` declarations of *root*'s packages, refusing bad ones.
534 Each table names a ``name`` (the point's, a workflow file name and
535 a job name at once), a ``task``, and optionally ``args``,
536 ``every`` (a cadence from `CADENCES`), ``runners`` (the root
537 contract's when absent) and ``pythons`` (the newest gate Python
538 when absent). A name that is a builtin point or that two packages
539 claim, a name `POINT_NAME` refuses, a cadence that is not one, a
540 task *mounted* does not know (the runner's own registry when
541 absent), and a permission, secret or environment key each refuse
542 at load, naming the package and the point.
543 """
544 from livery.workshop._packages import discover_packages
545 from livery.workshop._pythons import gate_pythons
547 mounted = mounted or _mounted
548 contract = load_contract(root / "workshop.toml")
549 ci_table = contract.get("ci") or {}
550 default_runners = tuple(
551 str(r) for r in (ci_table.get("runners") or ["ubuntu-latest"])
552 )
553 default_pythons: tuple[str, ...] | None = None
554 found: dict[str, str] = {}
555 contributions: list[Contribution] = []
556 for package in discover_packages(root):
557 package_contract = load_contract(package.directory / "workshop.toml")
558 raw = (package_contract.get("ci") or {}).get("point") or []
559 if not isinstance(raw, list):559 ↛ 560line 559 didn't jump to line 560 because the condition on line 559 was never true
560 fail(f"{package.path}: [ci] point must be a list of [[ci.point]] tables")
561 for index, item in enumerate(raw, start=1):
562 where = f"{package.path} [[ci.point]] entry {index}"
563 if not isinstance(item, dict):563 ↛ 564line 563 didn't jump to line 564 because the condition on line 563 was never true
564 fail(f"{where} is not a table")
565 name = str(item.get("name", ""))
566 task = str(item.get("task", ""))
567 if not name:
568 fail(f"{where}: names no point")
569 if name in POINT_BY_NAME:
570 fail(
571 f"{where}: {name!r} is a builtin point; a package contributes"
572 " a point of its own and adds no job to the gate"
573 )
574 if not POINT_NAME.match(name):
575 fail(
576 f"{where}: {name!r} is not a point name; a name is a workflow"
577 " file's, lower-case letters, digits and dashes, starting"
578 " with a letter"
579 )
580 if name in found:
581 fail(
582 f"{where}: {name!r} is already the point {found[name]}"
583 " declares; two packages cannot share one"
584 )
585 if not task:
586 fail(f"{where} ({name}): names no task")
587 if not mounted(task):
588 fail(
589 f"{where} ({name}): the runner mounts no task {task!r}; a"
590 " point runs a task some extension mounts"
591 )
592 args = item.get("args", [])
593 every = str(item.get("every", ""))
594 runners = item.get("runners", list(default_runners))
595 pythons = item.get("pythons")
596 if pythons is None:
597 if default_pythons is None:597 ↛ 599line 597 didn't jump to line 599 because the condition on line 597 was always true
598 default_pythons = (gate_pythons(root)[-1],)
599 pythons = list(default_pythons)
600 for label, values in (("runners", runners), ("pythons", pythons)):
601 if not values or not all(values):
602 fail(
603 f"{where} ({name}): {label} must be a non-empty list of strings"
604 )
605 found[name] = package.path
606 point = Point(
607 name,
608 f"{name}.yml",
609 ("schedule", "workflow_dispatch"),
610 cron=CLOCK,
611 note=(
612 f"The {name} point, contributed by {package.path}: the clock"
613 " and a dispatch entry, one call per runner and Python it"
614 " names, with the job token and nothing more."
615 ),
616 jobs=(
617 Job(
618 name,
619 matrix="declared",
620 runners=tuple(str(r) for r in runners),
621 pythons=tuple(str(v) for v in pythons),
622 token="job",
623 ),
624 ),
625 )
626 entry = Entry(
627 name, name, task, tuple(args), source=package.path, every=every
628 )
629 contributions.append(Contribution(point, entry))
630 return tuple(contributions)
635 """One job a mounted extension adds to a builtin point, with its entries.
637 Attributes:
638 point: The builtin point the job joins.
639 job: The job, as the point would declare it.
640 entries: The tasks the job runs, each naming the point and the job.
641 gates: Whether the point's verdict waits for the job.
642 extension: The extension that contributed it, for a refusal.
643 """
654#: The jobs the extensions joined, by point, job and extension: several
655#: extensions may join one job.
660 """Whether the builtin *point* has the job *name*, declared, inherited or added."""
661 by_name = {declared.name: declared for declared in DECLARED}
662 if point not in by_name:662 ↛ 663line 662 didn't jump to line 663 because the condition on line 662 was never true
663 return False
664 lineage = [point]
665 if by_name[point].inherits:
666 lineage.append(by_name[point].inherits)
667 for name_ in lineage:
668 if name in {job.name for job in by_name[name_].jobs}:
669 return True
670 if any(item.job.name == name for item in contributed_jobs(name_)):
671 return True
672 return False
676 """Each builtin point that declares a verdict job of its own, with the job's name.
678 A verdict job runs whatever the jobs it needs did; the judges an
679 extension adds join one of these.
680 """
681 return tuple(
682 (point.name, job.name) for point in DECLARED for job in point.jobs if job.always
683 )
687 """Why judges may not join *job* of *point*; empty when they may.
689 Judges join a point's own verdict job alone: they run before it
690 collects the run's rows, and a red one stops it before the verdict
691 and the stamp.
692 """
693 known = verdict_jobs()
694 if (point, job) in known:
695 return ""
696 listed = ", ".join(f"the {name} point's {verdict} job" for name, verdict in known)
697 return (
698 f"names judges on the {point} point's {job} job, which is not a verdict"
699 f" job a point declares; judges join {listed}, before it collects the"
700 " run's rows"
701 )
705 point: str,
706 job: Job,
707 *,
708 entries: tuple[Entry, ...] = (),
709 gates: bool = False,
710 extension: str,
711) -> None:
712 """Add *job* to the builtin *point* on behalf of *extension*, or join one.
714 The job sits before the first job of the point that needs it, else
715 before the point's verdict job, else last (`composed_points`), so a
716 contributed job renders before the jobs that wait for it and where
717 a declared one would; a gating job joins the verdict's needs. A
718 *job* that ``joins`` names a job the point already has
719 (`point_has_job`): its entries follow that job's own, before any
720 the job runs last, its plumbing renders on that job, and its needs
721 join the needs of a job the point has of its own. Refuses a point
722 that is not builtin, a job name the point already has when the job
723 does not join, a join of a job the point does not have, a join that
724 gates, a name another extension contributed, an entry naming
725 another point or job, and a judge on a job that is not the point's
726 own verdict job (`judges_refusal`).
727 """
728 by_name = {declared.name: declared for declared in DECLARED}
729 if point not in by_name:
730 fail(
731 f"{extension} contributes the job {job.name!r} to {point!r}, which is not a"
732 f" builtin point; the points are {', '.join(by_name)}"
733 )
734 for entry in entries:
735 if entry.point != point or entry.job != job.name:
736 fail(
737 f"{extension} contributes the job {job.name!r} to the {point} point"
738 f" with an entry for {entry.point}/{entry.job}; an entry names"
739 " the job it runs in"
740 )
741 if any(entry.judges for entry in entries):
742 why = judges_refusal(point, job.name)
743 if why:
744 fail(f"{extension} {why}")
745 if job.joins:
746 if not point_has_job(point, job.name):
747 fail(
748 f"{extension} joins the job {job.name!r} of the {point} point,"
749 " which has no such job"
750 )
751 if gates:
752 fail(
753 f"{extension} joins the job {job.name!r} of the {point} point and"
754 " gates; a join adds entries and plumbing, and the verdict's"
755 " needs stay the job's own"
756 )
757 _JOINED_JOBS[(point, job.name, extension)] = JobContribution(
758 point, job, tuple(entries), False, extension
759 )
760 return
761 declared_names = [declared.name for declared in by_name[point].jobs]
762 if job.name in declared_names:
763 fail(
764 f"{extension} contributes the job {job.name!r} to the {point} point, which"
765 " declares it already"
766 )
767 other = _CONTRIBUTED_JOBS.get((point, job.name))
768 if other is not None and other.extension != extension:
769 fail(
770 f"{extension} contributes the job {job.name!r} to the {point} point, which"
771 f" {other.extension} contributed already"
772 )
773 _CONTRIBUTED_JOBS[(point, job.name)] = JobContribution(
774 point, job, tuple(entries), gates, extension
775 )
779 *, initial: bool = False
780) -> tuple[
781 dict[tuple[str, str], JobContribution],
782 dict[tuple[str, str, str], JobContribution],
783 Tasks | None,
784]:
785 """The jobs the extensions added and joined, and the kept task tree, for restore.
787 Judges ride their joins' entries. [livery.workshop._points.restore][]
788 puts all three back. With *initial*, the state a process starts with
789 instead: no job added or joined, and no task tree kept.
790 """
791 if initial:
792 return {}, {}, None
793 return dict(_CONTRIBUTED_JOBS), dict(_JOINED_JOBS), MOUNTED
797 state: tuple[
798 dict[tuple[str, str], JobContribution],
799 dict[tuple[str, str, str], JobContribution],
800 Tasks | None,
801 ],
802) -> None:
803 """Put the added jobs, the joins and the kept task tree back to *state*."""
804 global MOUNTED
805 added, joined, tree = state
806 _CONTRIBUTED_JOBS.clear()
807 _CONTRIBUTED_JOBS.update(added)
808 _JOINED_JOBS.clear()
809 _JOINED_JOBS.update(joined)
810 MOUNTED = tree
814 """The jobs the extensions contributed to *point*, in contribution order."""
815 return tuple(item for item in _CONTRIBUTED_JOBS.values() if item.point == point)
819 """The joins the extensions made to *point*'s jobs, in contribution order."""
820 return tuple(item for item in _JOINED_JOBS.values() if item.point == point)
824 """The extension that contributed *job* to *point*; the base for its own jobs."""
825 item = _CONTRIBUTED_JOBS.get((point, job))
826 return item.extension if item is not None else "livery.workshop"
830 """What *point*'s verdict waits for: its declared needs, then the gating jobs."""
831 declared = {item.name: item for item in DECLARED}
832 verdict = next((job for job in declared[point].jobs if job.always), None)
833 needs = tuple(verdict.needs) if verdict is not None else ()
834 return needs + tuple(
835 item.job.name for item in contributed_jobs(point) if item.gates
836 )
840 """The builtin points with the extensions' contributed jobs in place.
842 A contributed job sits before the first job of its point that needs
843 it, else before the point's verdict job, else last: a job renders
844 before the jobs that wait for it, and where a declared one would.
845 The verdict waits for what `verdict_needs` names. A join's needs
846 join the needs of the job it joins when the point has that job of
847 its own, declared or contributed; the read of a declaration refuses
848 needs on a join of a job the point inherits, whose needs are the
849 point's it inherits from.
850 """
851 composed: list[Point] = []
852 for point in DECLARED:
853 added = [item.job for item in contributed_jobs(point.name)]
854 joined = [item.job for item in joined_jobs(point.name)]
855 if not added and not joined:
856 composed.append(point)
857 continue
858 jobs = [
859 _waiting(
860 replace(job, needs=verdict_needs(point.name)) if job.always else job,
861 joined,
862 )
863 for job in point.jobs
864 ]
865 for job in added:
866 needing = [i for i, other in enumerate(jobs) if job.name in other.needs]
867 verdict = [i for i, other in enumerate(jobs) if other.always]
868 jobs.insert((needing or verdict or [len(jobs)])[0], _waiting(job, joined))
869 # A join renders as no job: it rides the point's jobs so every
870 # reader of what a job installs, deploys and publishes sees it.
871 jobs.extend(joined)
872 composed.append(replace(point, jobs=tuple(jobs)))
873 return tuple(composed)
877 """*job* with the needs of each of *joins* that joins it added to its own."""
878 extra = [need for join in joins if join.name == job.name for need in join.needs]
879 if not extra:
880 return job
881 return replace(job, needs=tuple(dict.fromkeys((*job.needs, *extra))))
885 """Every point *root* has: the builtin four, then the ones its packages contribute.
887 ``None`` is a process outside any workspace, which has the builtin
888 four alone, with the jobs the mounted extensions contributed in place.
889 The whole set is verified as one: a contributed point cannot
890 share a file or a name with another.
891 """
892 builtin = composed_points()
893 if root is None:
894 verify_points(builtin)
895 return builtin
896 everything = builtin + tuple(item.point for item in contributed(root))
897 verify_points(everything)
898 return everything
902 """`points`, plus a job for each name only the contract's entries give.
904 A ``[[ci.schedule]]`` entry in the root contract may name a job its
905 point does not declare (`jobs_of` lists it); the shell needs that
906 job to call it. The job runs on one runner, the newest gate
907 Python, with the checkout depth, token and write grant of the
908 point's first job: the root contract is where a grant is decided,
909 so this widens nothing a package could reach.
910 """
911 everything = points(root)
912 entries = declared(root)
913 out: list[Point] = []
914 for point in everything:
915 names = {job.name for job in point.jobs} | {point.name}
916 names |= {job.name for job in inherited_jobs(point, everything)}
917 model = point.jobs[0] if point.jobs else Job(point.name)
918 added: list[Job] = []
919 for entry in entries:
920 if entry.point != point.name or entry.job in names:
921 continue
922 names.add(entry.job)
923 added.append(
924 Job(
925 entry.job,
926 fetch=model.fetch,
927 token=model.token,
928 writes=model.writes,
929 note=(
930 f"The {entry.job} job: the [[ci.schedule]] entries"
931 " that name it, on one runner."
932 ),
933 )
934 )
935 out.append(replace(point, jobs=(*point.jobs, *added)) if added else point)
936 return tuple(out)
940 """The jobs *point* runs before its own, from the point it inherits."""
941 if not point.inherits:
942 return ()
943 return next(other for other in everything if other.name == point.inherits).jobs
947 """`points`, by name."""
948 return {point.name: point for point in points(root)}
952 """The jobs whose declared inputs read *path*, each ``<point>/<job>``."""
953 from livery.workshop._influence import reads
955 return tuple(
956 f"{point.name}/{job.name}"
957 for point in points(root)
958 for job in point.jobs
959 if job.inputs is not None and reads(job.inputs, root, path)
960 )
964 """The line that skips *job* of *point* when nothing it reads changed, or empty.
966 Only a job that declares ``inputs`` skips, and only on a pull
967 request's run in a workspace declaring ``[ci] affected-legs``, the
968 terms the check legs narrow on: a push, a dispatch, the clock and a
969 person's own run always run every entry. The changes are the run's
970 own ([livery.workshop.ci_changes][]), read the way the check legs
971 read them, and the inputs select as a step's do, so a change to
972 the sources of the extension that contributed the job runs it.
973 """
974 from livery.workshop._extensions import extension_provider
975 from livery.workshop._influence import select
976 from livery.workshop._packages import discover_packages
977 from livery.workshop._quality import ci_changes
979 declared = next(
980 (item for item in point_by_name(root)[point].jobs if item.name == job), None
981 )
982 if declared is None or declared.inputs is None:
983 return ""
984 changes = ci_changes(root)
985 if changes is None:
986 return ""
987 packages = discover_packages(root) if (root / "packages").is_dir() else ()
988 provider = extension_provider(contributor_of(point, job), packages)
989 if select(declared.inputs, changes, provider=provider).runs:
990 return ""
991 return (
992 f" {point}/{job}: nothing the job reads changed in this run"
993 f" ({len(changes.paths)} path(s) changed); its entries are skipped"
994 )
998 root: Path, everything: tuple[Point, ...] | None = None
999) -> dict[tuple[str, str], tuple[str, ...]]:
1000 """What each job installs before it enters, by its point and its name.
1002 Only a job that names an ``installs`` function has an entry. Its
1003 packages are sorted and named once, so the rendered file is the
1004 same on every run. *everything* is the points to read; *root*'s
1005 when absent.
1006 """
1007 everything = everything if everything is not None else points(root)
1008 return {
1009 (point.name, job.name): tuple(sorted(set(job.installs(root))))
1010 for point in everything
1011 for job in point.jobs
1012 if job.installs is not None
1013 }
1017 root: Path, everything: tuple[Point, ...] | None = None
1018) -> dict[tuple[str, str], str]:
1019 """The seam each deploying job publishes through, by its point and its name.
1021 Only a job that names a ``deploy`` function has an entry.
1022 *everything* is the points to read; *root*'s when absent.
1023 """
1024 everything = everything if everything is not None else points(root)
1025 return {
1026 (point.name, job.name): job.deploy(root)
1027 for point in everything
1028 for job in point.jobs
1029 if job.deploy is not None
1030 }
1034 root: Path, everything: tuple[Point, ...] | None = None
1035) -> dict[tuple[str, str], tuple[str, ...]]:
1036 """The runner labels each job's ``labels`` function names, by point and job.
1038 Only a job that names a ``labels`` function has an entry: the
1039 labels the function names for *root*, in its order and each once,
1040 so the rendered file is the same on every run. An empty entry is a
1041 ``runners`` job this workspace renders no job for. *everything* is
1042 the points to read; *root*'s when absent.
1043 """
1044 everything = everything if everything is not None else points(root)
1045 return {
1046 (point.name, job.name): tuple(dict.fromkeys(job.labels(root)))
1047 for point in everything
1048 for job in point.jobs
1049 if job.labels is not None
1050 }
1054 """The events that start *point*'s runs; refuses a name that is not a point."""
1055 by_name = point_by_name(root)
1056 if point not in by_name:1056 ↛ 1057line 1056 didn't jump to line 1057 because the condition on line 1056 was never true
1057 fail(f"{point!r} is not a point; the points are {', '.join(by_name)}")
1058 return by_name[point].events
1062 """The points a person starts by hand: a dispatch entry and no input to supply.
1064 An optional input, the nightly's ``only``, leaves a point
1065 dispatchable; a required one, the release's ``ref``, does not.
1066 """
1067 return tuple(
1068 point.name
1069 for point in points(root)
1070 if "workflow_dispatch" in point.events
1071 and not any(item.required for item in point.inputs)
1072 )
1075#: The events that trigger each point's runs, in the forges' words.
1078#: The points a person starts by hand through ``ci.dispatch``: a
1079#: dispatch entry and no input to supply. The merge point runs on a
1080#: push alone, and the release wave takes a required input, so the
1081#: merge point dispatches it through ``workflow.release.dispatch``.
1083 point.name
1084 for point in DECLARED
1085 if "workflow_dispatch" in point.events
1086 and not any(item.required for item in point.inputs)
1087)
1089#: A point whose jobs include another point's: the merge point runs
1090#: the gate's jobs and its own.
1095 """Whether the jobs run profiled, for their timing rows (``[ci] timing``)."""
1096 from livery.workshop._traces import policy
1098 kept, _why = policy(root)
1099 return kept.timing
1103 """Whether a job pushes its trace to the trace refs (``[ci] profile``)."""
1104 from livery.workshop._traces import policy
1106 kept, _why = policy(root)
1107 return kept.profile
1111 """The job's drop box, where every entry leaves its own timeline.
1113 The plugin owns the directory and its sweeping; this names it to the
1114 entries through the environment the runner already builds for them,
1115 which is why nothing here writes this process's own.
1116 """
1117 from livery.footman.profile import box
1119 return box()
1123 """Record the commit this run ran on, where the forge files it elsewhere.
1125 A line and never a verdict, like the trace beside it. Written per
1126 job because any job of the run can write it and the write merges;
1127 a run whose own head is the commit it checked out writes nothing.
1128 """
1129 from livery.workshop._traces import about
1131 try:
1132 line = about(root)
1133 except Exception as error:
1134 line = f"profile: what this run ran on was not recorded ({error})"
1135 if line:
1136 print(f" {line}")
1140 """Write the job's own trace from what its entries left, and push it.
1142 *leg* is the label this job gave the entries it spawned. This
1143 process is not one of them, so its own environment never carried
1144 the label and the push has to be told.
1146 Every line is printed and none is a verdict: a job's timeline is
1147 something noticed about the work, never part of deciding it.
1148 """
1149 from livery.footman.profile import as_trace, swept
1150 from livery.workshop._traces import push
1152 try:
1153 events, origin = swept(drop)
1154 if not events:1154 ↛ 1155line 1154 didn't jump to line 1155 because the condition on line 1154 was never true
1155 return
1156 written = drop.parent / f"{drop.name}.json"
1157 written.write_text(as_trace(events, origin=origin), encoding="utf-8")
1158 line = push(root, written, job=job, leg=leg)
1159 written.unlink(missing_ok=True)
1160 except Exception as error:
1161 line = f"profile: the job's trace was not kept ({error})"
1162 if line:1162 ↛ exitline 1162 didn't return from function '_push_job' because the condition on line 1162 was always true
1163 print(f" {line}")
1167 """The file an entry's own trace is written to, named after the task.
1169 One file per entry rather than one per job: the leg's timing row reads
1170 the gate's own file, and a person reading a runner's working directory
1171 can tell which entry a trace belongs to.
1172 """
1173 from livery.git import slug
1175 return f"fm-profile-{slug(task)}.json"
1178#: The gate's own trace, which the leg's timing row reads.
1184 """One scheduled task: at which point and job, which task, which arguments.
1186 Attributes:
1187 point: One of `POINTS`.
1188 job: The job of the point the entry runs in.
1189 task: The task's dotted address, as the workspace mounts it.
1190 args: The arguments, each formatted with the run's facts:
1191 ``{display}`` the job's display name, ``{label}`` the
1192 leg's label, ``{os}`` and ``{python}`` the matrix values,
1193 ``{shard}`` the leg's shard spelled ``<i>of<n>`` (``1of1``
1194 on a leg that is not split), and each of the point's
1195 dispatch inputs by name (``{ref}`` on the release), empty
1196 when the run was not dispatched with it.
1197 source: Where the entry was declared.
1198 every: The cadence, ``""`` for every run of the point, ``1w``
1199 for one run a week, ``2w`` for one run every two weeks;
1200 a run on any other day skips the entry and names the day
1201 it runs next.
1202 once: Whether the entry runs on a matrix job's first leg alone,
1203 the first runner and the first Python, and on that leg's
1204 first shard alone; every leg and shard otherwise.
1205 sharded: Whether the entry takes the leg's shard,
1206 ``--shard=<i>of<n>`` after its arguments, when the leg
1207 runs as one of several shards; on a leg that runs as one
1208 shard of one the entry takes nothing more.
1209 last: Whether the entry runs after every entry an extension
1210 adds to the job, the janitor's shape; the added entries go
1211 before the first such entry.
1212 judges: Whether the entry judges the run before the verdict:
1213 a judge an extension adds to a point's verdict job runs
1214 before the job collects the run's rows, and a red one stops
1215 the job there, as a red builtin entry does.
1216 """
1230#: The cadences an entry may declare, in days.
1236A variable, not a function, so a test fixes the day instead of
1237waiting for a Monday.
1238"""
1242 """Whether a cadence runs on *on*, and the day it runs next.
1244 A weekly entry runs on Mondays; a two-weekly entry on the Monday
1245 of an even ISO week, so every workspace agrees on the day without
1246 a record of the last run. An empty cadence is due on every run.
1247 """
1248 if not every:
1249 return True, on
1250 monday = on - timedelta(days=on.weekday())
1251 if every == "1w":
1252 return on == monday, (monday if on == monday else monday + timedelta(days=7))
1253 even = monday.isocalendar().week % 2 == 0
1254 if on == monday and even:
1255 return True, on
1256 following = monday + timedelta(days=7)
1257 if following.isocalendar().week % 2 != 0:
1258 following += timedelta(days=7)
1259 return False, following
1262#: The task that collects the run's rows in the verdict job: the
1263#: judges the extensions add run before it, so what they leave for
1264#: the run's row is folded in.
1267#: The task that drops the run's per-run refs, the last entry of a job
1268#: that collected: after its verdict, and only once every job that
1269#: puts a half has completed green.
1272#: The task that ends every job of a point with no verdict job: it
1273#: puts the job's half from the traces its entries left, collects the
1274#: run's row, and drops the refs when it is the run's last green job.
1277#: The workshop's own schedule. Every entry runs profiled, so a job's
1278#: own timeline is the union of the work it did; the gate's check job
1279#: also records the leg's timing row, and its verdict job collects the
1280#: run's rows, then judges the jobs it needs. The merge
1281#: point adds the deploy, the governance reconcile, which classifies
1282#: its own commit and exits fast when no contract path changed, and
1283#: the release dispatch, green unless a merged release is unpublished.
1285 # The walk takes the leg's shard when the leg is one of several, so
1286 # the steps that split run their share; an unsplit leg calls it bare.
1287 Entry("gate", "check", "check", sharded=True),
1288 # The leg's one write, after whatever the extensions add to the
1289 # job: its timing row and every file their entries left go on its
1290 # per-run ref together.
1291 Entry("gate", "check", "ci.leg", ("--job={display}",), last=True),
1292 # The title check first: it reads the pull request's title from
1293 # the event payload, is green off a release branch, and refuses a
1294 # release title the changelogs do not match before anything else
1295 # is judged. It runs here so the legs never wait for a job of
1296 # its own. The judges the extensions add run after it, before the
1297 # collect, so what they leave rides the run's row.
1298 Entry("gate", "gate", "workflow.release.check-title"),
1299 # The fold next, before any judge reads the run's refs: a platform
1300 # that ran as shards has one ref per shard, and the judges, the
1301 # collect and the stamp read one leg per platform.
1302 Entry("gate", "gate", "ci.fold"),
1303 Entry("gate", "gate", COLLECT),
1304 # The speed judge reads the run's row the collect just put: a
1305 # suite over its mark for the second run in a row is red here,
1306 # before the verdict, so the record never names a slow tree green.
1307 Entry("gate", "gate", "speed.judge"),
1308 # The verdict waits for the check and for every contributed job that
1309 # gates; `schedule` writes the composed list into this entry.
1310 Entry("gate", "gate", "ci.verdict", ("--needs=check",)),
1311 # After a green verdict only: a red verdict fails the job before
1312 # this entry, so the record never names a tree a run proved red.
1313 Entry("gate", "gate", "ci.verified.stamp"),
1314 # After the stamp, so the forge's merge follows it as closely as
1315 # it can: a release pull request whose base moved under its set is
1316 # refused before the merge, where the publish would refuse it
1317 # after. Green off a release branch. The stamp before it holds
1318 # either way: it records the tree, and the tree is green.
1319 Entry("gate", "gate", "workflow.release.check-fresh"),
1320 # Last of all, after a green verdict and whatever the extensions
1321 # add: the legs' per-run refs go only once this job has nothing
1322 # left to read from them, so a red attempt leaves them for its
1323 # re-run, which runs this job again without its legs.
1324 Entry("gate", "gate", DROP_TASK, ("--point=gate", "--job={display}"), last=True),
1325 Entry("merge", "govern", "workflow.configure", ("--if-changed",)),
1326 # The wave is dispatched from the gate job itself, right after
1327 # main's own verdict: the verb reads the manifest at HEAD and the
1328 # receipts on the remote, and is green unless a merged release is
1329 # unpublished. A job of its own cost a runner's entry for a second
1330 # of work.
1331 Entry("merge", "gate", "workflow.release.dispatch"),
1332 # The janitor after the stamp, on the merge point alone, and after
1333 # whatever the extensions add to the job: every merge tidies the
1334 # remote store, and a pull request's run writes nothing it does
1335 # not own.
1336 Entry("merge", "gate", "janitor", last=True),
1337 # The clock's point: the doctor first, so the night's record opens
1338 # with the runner's and the forge's state, mending what footman
1339 # can (a runner that keeps its caches across jobs holds the scratch
1340 # of processes that died there; a finding that stands after the
1341 # mend is still red), then the whole check with the tests that
1342 # declare the nightly point selected in, on every python of the
1343 # matrix.
1344 Entry("nightly", "nightly", "doctor", ("--fix",)),
1345 Entry("nightly", "nightly", "check"),
1346 # The wave, at the squash the dispatch names: the publish that cuts
1347 # the receipts. The verb decides for itself what the ref holds, and
1348 # whether a job of the run already built its wheels.
1349 Entry("release", "publish", "workflow.release.publish", ("--ref={ref}",)),
1350)
1354 """The ``[[ci.schedule]]`` entries of *root*'s contract, refusing bad ones.
1356 Each entry names a ``point`` (one of `points`), a ``task``, and
1357 optionally a ``job`` (the point's own name when absent), ``args``
1358 and ``every``, a cadence from `CADENCES`. An unknown point, a
1359 missing task, arguments that are not strings, or a cadence that
1360 is not one refuse at load, naming the entry.
1361 """
1362 contract = load_contract(root / "workshop.toml")
1363 raw = (contract.get("ci") or {}).get("schedule") or []
1364 if not isinstance(raw, list):1364 ↛ 1365line 1364 didn't jump to line 1365 because the condition on line 1364 was never true
1365 fail("[ci] schedule must be a list of [[ci.schedule]] tables")
1366 entries: list[Entry] = []
1367 names = tuple(point.name for point in points(root))
1368 for index, item in enumerate(raw, start=1):
1369 if not isinstance(item, dict):1369 ↛ 1370line 1369 didn't jump to line 1370 because the condition on line 1369 was never true
1370 fail(f"[[ci.schedule]] entry {index} is not a table")
1371 point = str(item.get("point", ""))
1372 task = str(item.get("task", ""))
1373 if point not in names:
1374 fail(
1375 f"[[ci.schedule]] entry {index}: point {point!r} is not a"
1376 f" point; the points are {', '.join(names)}"
1377 )
1378 if not task:
1379 fail(f"[[ci.schedule]] entry {index} ({point}): names no task")
1380 # The contract's judge holds the arguments to strings and the
1381 # cadence to one of `CADENCES`.
1382 args = item.get("args", [])
1383 every = str(item.get("every", ""))
1384 once = bool(item.get("once", False))
1385 entries.append(
1386 Entry(
1387 point,
1388 str(item.get("job", point)),
1389 task,
1390 tuple(args),
1391 source="workshop.toml",
1392 every=every,
1393 once=once,
1394 )
1395 )
1396 return tuple(entries)
1400 """The builtin entries with the extensions' contributed jobs' entries in place.
1402 A contributed job's entries follow the declared entries of its
1403 point's jobs before the verdict, and the verdict's ``--needs``
1404 names the composed list.
1405 """
1406 from dataclasses import replace
1408 composed: list[Entry] = []
1409 for entry in BUILTIN:
1410 if entry.task == "ci.verdict":
1411 for item in contributed_jobs(entry.point):
1412 composed.extend(item.entries)
1413 composed.append(
1414 replace(
1415 entry, args=(f"--needs={','.join(verdict_needs(entry.point))}",)
1416 )
1417 )
1418 continue
1419 composed.append(entry)
1420 seen = {(entry.point, entry.job, entry.task) for entry in composed}
1421 for point in DECLARED:
1422 if any(job.always for job in point.jobs):
1423 continue
1424 for item in contributed_jobs(point.name):
1425 composed.extend(
1426 entry
1427 for entry in item.entries
1428 if (entry.point, entry.job, entry.task) not in seen
1429 )
1430 for point in DECLARED:
1431 for item in joined_jobs(point.name):
1432 _join_entries(composed, item)
1433 return tuple(composed)
1437 """Put *item*'s entries into *composed* where its job runs what it adds.
1439 Before the first entry of the job that runs last, else after the
1440 job's last entry, else at the end: the join's entries run after
1441 the job's own and before its sweep. A judge goes before the job's
1442 collect (`COLLECT`), so what it leaves for the run's row is folded
1443 in, and the verdict and the stamp come after it.
1444 """
1445 key = (item.point, item.job.name)
1446 owned = [i for i, e in enumerate(composed) if (e.point, e.job) == key]
1447 judges = [entry for entry in item.entries if entry.judges]
1448 if judges:
1449 at = next((i for i in owned if composed[i].task == COLLECT), None)
1450 if at is None:1450 ↛ 1451line 1450 didn't jump to line 1451 because the condition on line 1450 was never true
1451 at = owned[-1] + 1 if owned else len(composed)
1452 composed[at:at] = judges
1453 owned = [i for i, e in enumerate(composed) if (e.point, e.job) == key]
1454 at = next((i for i in owned if composed[i].last), None)
1455 if at is None:
1456 at = owned[-1] + 1 if owned else len(composed)
1457 composed[at:at] = [entry for entry in item.entries if not entry.judges]
1461 """Every entry: the builtin ones, the packages' contributed ones, the contract's.
1463 Then the record each job of a point without a verdict job ends
1464 with (`recording`), after every entry the contract gave the job.
1465 """
1466 scheduled = (
1467 builtin_schedule()
1468 + tuple(item.entry for item in contributed(root))
1469 + declared(root)
1470 )
1471 return scheduled + recording(root, scheduled)
1475 """Whether *point*, or the point it inherits, declares a verdict job."""
1476 verdicts = {name for name, _job in verdict_jobs()}
1477 return point.name in verdicts or point.inherits in verdicts
1481 """The record ending every job of each point of *root* with no verdict job.
1483 A verdict job collects what its point's legs put; a point without
1484 one (the nightly, the release, every point a package contributes)
1485 would leave no row of its run, so each of its jobs ends with
1486 `RECORD_TASK`, which puts the job's own half, collects the run's
1487 row and drops the per-run refs when it is the run's last green
1488 job. The jobs are the point's declared ones and any a
1489 ``[[ci.schedule]]`` entry in *scheduled* names.
1490 """
1491 entries: list[Entry] = []
1492 for point in points(root):
1493 if _has_verdict_job(point):
1494 continue
1495 jobs = [job.name for job in point.jobs]
1496 for entry in scheduled:
1497 if entry.point == point.name and entry.job not in jobs:
1498 jobs.append(entry.job)
1499 entries.extend(
1500 Entry(
1501 point.name,
1502 job,
1503 RECORD_TASK,
1504 (f"--point={point.name}", "--job={display}"),
1505 last=True,
1506 )
1507 for job in jobs
1508 )
1509 return tuple(entries)
1513 """The jobs of *point* that put a half of the run's row, by their schedule names.
1515 The check legs of a point with a verdict job (`ci.leg`), and every
1516 job of a point without one (`RECORD_TASK`). A forge lists a matrix
1517 job's legs under the job's name with the leg in brackets, so a
1518 reader matches by prefix.
1519 """
1520 return tuple(
1521 dict.fromkeys(
1522 entry.job
1523 for entry in schedule(root)
1524 if entry.task in ("ci.leg", RECORD_TASK)
1525 and entry.point in (INHERITS.get(point), point)
1526 )
1527 )
1531 """The workflow file *point*'s shell is; refuses a name that is not a point."""
1532 by_name = point_by_name(root)
1533 if point not in by_name:
1534 fail(f"{point!r} is not a point; the points are {', '.join(by_name)}")
1535 return by_name[point].workflow
1539 """The jobs *point* has: declared ones, inherited first, then any an entry names.
1541 A ``[[ci.schedule]]`` entry may name a job no declaration has; it
1542 is listed after the declared jobs, in schedule order.
1543 """
1544 by_name = point_by_name(root)
1545 if point not in by_name:
1546 fail(f"{point!r} is not a point; the points are {', '.join(by_name)}")
1547 declared_point = by_name[point]
1548 inherited = by_name[declared_point.inherits].jobs if declared_point.inherits else ()
1549 names: list[str] = []
1550 for job in (*inherited, *declared_point.jobs):
1551 if job.name not in names:
1552 names.append(job.name)
1553 for entry in schedule(root):
1554 if entry.point in (INHERITS.get(point), point) and entry.job not in names:
1555 names.append(entry.job)
1556 return tuple(names)
1560 """The first leg of *job* at *point*: its first runner and its first Python.
1562 An axis the job does not have is empty. The check legs fan out
1563 over ``[ci] runners`` and the gate's Pythons, the nightly over every
1564 Python on the first runner, a declared job over its own lists. A
1565 ``runners`` job's call names no leg and a job with no matrix has
1566 none, so neither has a leg to name.
1567 """
1568 from livery.workshop._pythons import gate_pythons, python_matrix
1570 found = next(
1571 (item for item in point_by_name(root)[point].jobs if item.name == job), None
1572 )
1573 if found is None or not found.matrix:
1574 return "", ""
1575 ci = load_contract(root / "workshop.toml").get("ci") or {}
1576 runners = [str(runner) for runner in ci.get("runners") or ["ubuntu-latest"]]
1577 if found.matrix == "legs":1577 ↛ 1578line 1577 didn't jump to line 1578 because the condition on line 1577 was never true
1578 return runners[0], gate_pythons(root)[0]
1579 if found.matrix == "pythons":1579 ↛ 1581line 1579 didn't jump to line 1581 because the condition on line 1579 was always true
1580 return "", python_matrix(root)[0]
1581 if found.matrix == "declared":
1582 return (
1583 found.runners[0] if found.runners else "",
1584 found.pythons[0] if found.pythons else "",
1585 )
1586 return "", ""
1590 """The entries *job* of *point* runs, in order; refuses an unknown job.
1592 A point's own name is always a job of it, with nothing scheduled
1593 until an entry attaches: the nightly shell exists before its
1594 first entry, and runs green and empty until then.
1595 """
1596 jobs = jobs_of(root, point)
1597 if job not in jobs and job != point:
1598 fail(
1599 f"the {point} point has no job {job!r}; its jobs are"
1600 f" {', '.join(jobs) or 'none'}"
1601 )
1602 return tuple(
1603 entry
1604 for entry in schedule(root)
1605 if entry.point in (INHERITS.get(point), point) and entry.job == job
1606 )
1610 """Every task *point* runs, across its jobs, in schedule order, once each."""
1611 found: dict[str, None] = {}
1612 for job in jobs_of(root, point):
1613 for entry in entries_for(root, point, job):
1614 found.setdefault(entry.task, None)
1615 return tuple(found)
1619 """The tasks a dispatch's *only* names, judged against what *point* runs.
1621 Empty for an empty *only*, which runs the whole schedule.
1623 Raises:
1624 Failed: when *only* names a task the point does not run, naming
1625 the ones it does.
1626 """
1627 asked = tuple(
1628 dict.fromkeys(name.strip() for name in only.split(",") if name.strip())
1629 )
1630 if not asked:
1631 return ()
1632 runs = scheduled_tasks(root, point)
1633 unknown = [name for name in asked if name not in runs]
1634 if unknown:
1635 fail(
1636 f"the {point} point runs {', '.join(runs) or 'nothing'}; --only names"
1637 f" {', '.join(unknown)}, which it does not run"
1638 )
1639 return asked
1642#: The label a pull request carries so the gate runs every runner for
1643#: it, where its check legs run on fewer. `fm submit
1644#: --every-runner` sets it through the forge, and a person sets it on
1645#: the forge by hand; the gate workflow runs on ``labeled``, its legs
1646#: job picks the full matrix while the label is there, and the coverage
1647#: union expects every leg. The repository configuration creates the
1648#: label (`fm workflow.configure`).
1653 """The runners a pull request's check legs fan out to.
1655 ``[ci] pull-request-runners`` when the contract declares it, else
1656 the first runner ``[ci] runners`` names (``ubuntu-latest`` without
1657 that key). A one-runner pull request carries the other legs'
1658 coverage rows from main's record, so a package whose floor needs
1659 the platform legs' fresh rows on a pull request lists every runner.
1660 A label the runners list does not name refuses naming both lists,
1661 since a pull request's legs run on runners every event runs on; an
1662 empty list refuses too.
1663 """
1664 runners = [str(runner) for runner in ci.get("runners") or ["ubuntu-latest"]]
1665 declared: Any = ci.get("pull-request-runners")
1666 if declared is None:
1667 return runners[:1]
1668 chosen = [str(runner) for runner in declared]
1669 strangers = [runner for runner in chosen if runner not in runners]
1670 if strangers:
1671 fail(
1672 f"[ci] pull-request-runners names {', '.join(strangers)}, which [ci]"
1673 f" runners does not list ({', '.join(runners)}); a pull request's"
1674 " legs run on runners every event runs on"
1675 )
1676 if not chosen:
1677 fail(
1678 "[ci] pull-request-runners is empty; name a runner [ci] runners"
1679 " lists, or remove the key so the first runner runs"
1680 )
1681 return chosen
1685 """Name the list a pull request's runners come from, for a shard refusal.
1687 ``[ci] pull-request-runners`` when the contract declares it. When
1688 the key is absent the refusal names it and the first runner
1689 ``[ci] runners`` names, which stands in for it.
1690 """
1691 if ci.get("pull-request-runners") is not None:
1692 return "[ci] pull-request-runners"
1693 return "[ci] pull-request-runners (absent, so the first of [ci] runners)"
1697 """Name the check legs a pull request's run produces, ``check-<os>-<python>`` each.
1699 One per runner the root contract's ``[ci] pull-request-runners``
1700 names and Python the gate runs; the first runner's legs among those
1701 [livery.workshop.check_legs][] names when the key is absent. The
1702 task fails, naming both lists, when the key names a runner
1703 ``[ci] runners`` does not list, and when it is empty.
1704 """
1705 from livery.workshop._pythons import gate_pythons
1707 contract = load_contract(root / "workshop.toml")
1708 ci = contract.get("ci") or {}
1709 pythons = gate_pythons(root)
1710 return [
1711 f"check-{runner}-{python}"
1712 for runner in pull_request_runners(ci)
1713 for python in pythons
1714 ]
1718 """Name the check legs the gate produces, ``check-<os>-<python>`` each.
1720 One per runner the root contract names (``[ci] runners``,
1721 ``ubuntu-latest`` without it) and Python the gate runs: the
1722 contract's ``[ci] python-versions``, or the newest Python of the
1723 derived matrix. Each label is spelled as the job runner names the
1724 leg it runs, the key of the leg's rows on the state store.
1725 """
1726 from livery.workshop._pythons import gate_pythons
1728 contract = load_contract(root / "workshop.toml")
1729 runners = list((contract.get("ci") or {}).get("runners") or ["ubuntu-latest"])
1730 pythons = gate_pythons(root)
1731 return [f"check-{runner}-{python}" for runner in runners for python in pythons]
1734#: The most shards a runner may take while ``[ci] shards-max`` is absent.
1739 """The most shards a runner may take: ``[ci] shards-max``, `SHARDS_MAX` when absent.
1741 A value that is not a whole number, or is below 1, fails naming it.
1742 """
1743 declared: Any = ci.get("shards-max")
1744 if declared is None:
1745 return SHARDS_MAX
1746 if isinstance(declared, bool) or not isinstance(declared, int):
1747 fail(
1748 f"[ci] shards-max is {declared!r}, which is not a whole number; it is"
1749 " the most shards a runner may take"
1750 )
1751 if declared < 1:
1752 fail(f"[ci] shards-max is {declared}; the cap is at least 1")
1753 return declared
1757 ci: dict[str, Any], *, key: str, runners: Sequence[str], listed: str
1758) -> dict[str, int]:
1759 """The shards per runner ``[ci] <key>`` gives, one for a runner it does not name.
1761 *runners* are the labels the legs fan out to and *listed* names the
1762 key they came from, for the refusal. The task fails, naming the key
1763 and the runner, when the table names a runner *runners* does not
1764 hold, gives a count below 1 or above `shards_max`, or gives one
1765 that is not a whole number.
1766 """
1767 cap = shards_max(ci)
1768 counts = dict.fromkeys(runners, 1)
1769 declared: Any = ci.get(key)
1770 if declared is None:
1771 return counts
1772 if not isinstance(declared, dict):
1773 fail(f"[ci] {key} is {declared!r}; it is a table of runner label to count")
1774 table: dict[str, Any] = declared
1775 strangers = [runner for runner in table if runner not in runners]
1776 if strangers:
1777 fail(
1778 f"[ci] {key} names {', '.join(strangers)}, which {listed} does not list"
1779 f" ({', '.join(runners)}); a count belongs to a runner the legs run on"
1780 )
1781 for runner, count in table.items():
1782 if isinstance(count, bool) or not isinstance(count, int):
1783 fail(
1784 f"[ci] {key} gives {runner} {count!r}, which is not a whole number;"
1785 f" a count is from 1 to {cap}"
1786 )
1787 if count < 1:
1788 fail(f"[ci] {key} gives {runner} {count}; a count is at least 1")
1789 if count > cap:
1790 fail(
1791 f"[ci] {key} gives {runner} {count}, above the cap of {cap}"
1792 " ([ci] shards-max); lower the count or raise the cap"
1793 )
1794 counts[runner] = count
1795 return counts
1799 runners: Sequence[str], pythons: Sequence[str], counts: Mapping[str, int]
1800) -> list[str]:
1801 """One leg per runner and Python, split into shards where the count is above 1."""
1802 legs: list[str] = []
1803 for runner in runners:
1804 count = counts.get(runner, 1)
1805 for python in pythons:
1806 leg = f"check-{runner}-{python}"
1807 if count == 1:
1808 legs.append(leg)
1809 else:
1810 legs.extend(
1811 f"{leg}-{Shard(index, count)}" for index in range(1, count + 1)
1812 )
1813 return legs
1817 """Name the shard legs the gate produces, split as ``[ci] shards`` says.
1819 One leg per runner and Python [livery.workshop.check_legs][] names,
1820 spelled as that leg, ``check-<os>-<python>``, for a count of one,
1821 the count of every runner the table does not name; and
1822 ``check-<os>-<python>-<i>of<n>`` for each of the ``n`` shards
1823 otherwise, in order. ``[ci] shards-max`` caps a count, 4 when
1824 absent. The task fails, naming the key and the runner, when the
1825 table names a runner ``[ci] runners`` does not list, gives a count
1826 below 1 or above the cap, or gives one that is not a whole number.
1827 """
1828 from livery.workshop._pythons import gate_pythons
1830 contract = load_contract(root / "workshop.toml")
1831 ci = contract.get("ci") or {}
1832 runners = [str(runner) for runner in ci.get("runners") or ["ubuntu-latest"]]
1833 counts = shard_counts(ci, key="shards", runners=runners, listed="[ci] runners")
1834 return _shard_legs(runners, gate_pythons(root), counts)
1838 """Name the shard legs a pull request's run produces, split as the contract says.
1840 One leg per runner and Python [livery.workshop.pull_request_legs][]
1841 names, split as [livery.workshop.check_shards][] splits the gate's:
1842 by ``[ci] pull-request-shards``, judged against the pull request's
1843 runners, or by ``[ci] shards`` when that key is absent. The task
1844 fails as `check_shards` fails, and as `pull_request_legs` fails.
1845 """
1846 from livery.workshop._pythons import gate_pythons
1848 contract = load_contract(root / "workshop.toml")
1849 ci = contract.get("ci") or {}
1850 runners = pull_request_runners(ci)
1851 if ci.get("pull-request-shards") is None:
1852 every = [str(runner) for runner in ci.get("runners") or ["ubuntu-latest"]]
1853 counts = shard_counts(ci, key="shards", runners=every, listed="[ci] runners")
1854 else:
1855 counts = shard_counts(
1856 ci,
1857 key="pull-request-shards",
1858 runners=runners,
1859 listed=pull_request_listed(ci),
1860 )
1861 return _shard_legs(runners, gate_pythons(root), counts)
1865 """The platform legs the contract splits into shards, on any event.
1867 Each platform leg once, in matrix order: one ``[ci] shards`` or
1868 ``[ci] pull-request-shards`` gives above one
1869 ([livery.workshop.check_shards][],
1870 [livery.workshop.pull_request_shards][]). Empty without a root
1871 contract. The task fails as those fail, naming the key and the
1872 runner.
1873 """
1874 if not (root / "workshop.toml").is_file():
1875 return []
1876 found: dict[str, None] = {}
1877 for leg in (*check_shards(root), *pull_request_shards(root)):
1878 if shard_of(leg) != ONE:
1879 found.setdefault(platform_leg(leg), None)
1880 return list(found)
1884 """The point a shell's ``gate`` means on a push: the merge point.
1886 The shared jobs spell ``--point=gate``; inside CI, a push event
1887 is the merge point, so the promotion happens here and never in
1888 the YAML. Outside CI, and on every other event, the point stands.
1889 """
1890 run = run_context()
1891 if point == "gate" and run is not None and run.event == "push":
1892 return "merge"
1893 return point
1897 """Run one entry as a child of the runner's own command; its exit code.
1899 Streamed, never captured: the job's log is the run's evidence,
1900 and what each verb said belongs there in order, green or red.
1901 *env* is the child's whole environment.
1902 """
1903 from livery.footman import run
1905 return run(argv, nofail=True, capture=False, env=env).code
1909 root: Path,
1910 point: str,
1911 job: str,
1912 *,
1913 os_label: str = "",
1914 python: str = "",
1915 shard: str = "",
1916 spawn: Callable[[list[str], dict[str, str]], int] = _spawn,
1917) -> None:
1918 """Run every entry of *job* at *point*, in order; a red entry fails the job.
1920 A red builtin entry stops the job at once, since the entries after
1921 it read its verdict, and so does a red judge an extension adds to
1922 the verdict job; a red entry of the contract's schedule stands
1923 on its own, so the scheduled entries after it still run, and the
1924 job fails at the end naming each red one. A dispatch that names
1925 tasks in the nightly's ``only`` input runs those alone, builtin or
1926 scheduled, and says what it skipped.
1928 *os_label* and *python* are the matrix facts the shell passes for
1929 a matrix job; they format the entries' arguments and name the
1930 leg. *shard* is the leg's shard, ``2of4``, on a check leg the
1931 contract splits; empty runs as one shard of one. A count above
1932 one ends the label and the display name
1933 (``check-windows-latest-3.14-2of4``, ``check (windows-latest,
1934 3.14, 2of4)``, and on GitLab ``check: [2of4]``, as it lists a job
1935 run as ``parallel: matrix``), and an entry declared ``sharded``
1936 takes ``--shard=2of4``; one shard of one leaves the label, the
1937 display name and every call as the unsplit leg's. A spelling outside
1938 ``<i>of<n>`` fails naming it, before any entry runs. An entry
1939 with ``once`` runs on the first leg's first shard alone, and the
1940 skip line names the shard when the leg is the first. An entry
1941 with a cadence runs only on its day, judged by
1942 `today`, and says when it runs next otherwise. Every entry runs
1943 under ``--profile``, writing its own trace beside the others, and
1944 the job's box collects them into one timeline, which the runner
1945 pushes where ``[ci] profile`` asks; ``[ci] timing = false`` does
1946 none of that. Every child's
1947 environment names the leg in
1948 `livery.workshop._ci_context.LEG_VARIABLE`, the key of the leg's rows
1949 and stamps, and the resolved point in
1950 [livery.workshop.POINT_VARIABLE][], which selects the tests a point
1951 runs. A job that declares ``inputs`` runs no entry on
1952 a pull request that changed nothing it reads (`unread_by_the_job`).
1953 *spawn* runs one entry's command in that environment and returns
1954 its exit code; the default is the runner's own child.
1955 """
1956 from livery.git import remote_snapshot
1957 from livery.workshop._ci_context import SNAPSHOT_VARIABLE
1959 try:
1960 split = Shard.parse(shard) if shard else ONE
1961 except ValueError as error:
1962 fail(str(error))
1963 resolved = effective_point(point)
1964 entries = entries_for(root, resolved, job)
1965 if resolved != point:
1966 print(f" point: {point} on a push is the {resolved} point")
1967 skip = unread_by_the_job(root, resolved, job)
1968 if skip:1968 ↛ 1969line 1968 didn't jump to line 1969 because the condition on line 1968 was never true
1969 print(skip)
1970 return
1971 # The display name is the job's name as the forge lists it, since
1972 # the collect step joins the leg's row with the forge's job by it:
1973 # GitHub and Gitea show a matrix job as ``check (ubuntu-latest,
1974 # 3.14)``; GitLab names the job by its key alone, the matrix
1975 # riding its variables and never its name, except that a job run
1976 # as ``parallel: matrix`` is listed as ``check: [2of4]``.
1977 run = run_context()
1978 # Only the dimensions the matrix actually has: the nightly point runs a
1979 # matrix of pythons and no runners, and the forge lists that job as
1980 # ``nightly (3.14)``. Spelling an absent dimension as an empty one gave
1981 # ``nightly (, 3.14)``, which matched no job the forge had.
1982 parts = [part for part in (os_label, python) if part]
1983 # The shard ends the label and the display name only where the leg
1984 # is one of several: one shard of one is the unsplit leg, spelled
1985 # as the platform leg every record is keyed by.
1986 if split.count > 1:
1987 parts.append(str(split))
1988 gitlab = run is not None and run.forge == "gitlab"
1989 if gitlab:
1990 display = f"{job}: [{split}]" if split.count > 1 else job
1991 else:
1992 display = f"{job} ({', '.join(parts)})" if parts else job
1993 label = "-".join([job, *parts])
1994 facts = {
1995 "display": display,
1996 "label": label,
1997 "os": os_label,
1998 "python": python,
1999 "shard": str(split),
2000 }
2001 # A dispatched point's inputs, as the run received them, by name:
2002 # the release's ``{ref}`` names the squash the wave publishes.
2003 from livery.workshop._ci_context import dispatch_inputs
2005 facts.update(
2006 dispatch_inputs(
2007 tuple(item.name for item in point_by_name(root)[resolved].inputs)
2008 )
2009 )
2010 asked = asked_tasks(root, resolved, facts.get(ONLY, ""))
2011 prog = footman.prog()
2012 # One listing of the state store's namespace for the whole job:
2013 # every entry reads through it and records what it writes for the
2014 # entries after it, so the job lists once, not once per entry.
2015 with contextlib.ExitStack() as scope:
2016 # Every entry writes its own trace and drops a copy in the job's
2017 # box, so the job's own timeline is the union of the work it did,
2018 # and a check leg's timing row is read from the gate's. `[ci]
2019 # timing = false` opens no box, passes no flag, and pushes
2020 # nothing: off costs nothing rather than less. `[ci] profile`
2021 # decides whether the box's union leaves the runner at the end;
2022 # off, nothing is assembled from the box at all.
2023 drop = scope.enter_context(profile_box()) if _tracing(root) else None
2024 pushing = drop is not None and _pushing(root)
2025 published = scope.enter_context(
2026 remote_snapshot(state_store(root), publish=True)
2027 )
2028 env = {**os.environ, LEG_VARIABLE: label, POINT_VARIABLE: resolved}
2029 if published:
2030 env[SNAPSHOT_VARIABLE] = published
2031 if drop is not None:
2032 # Named here and nowhere else: a workspace that keeps no traces
2033 # never imports the plugin that writes them.
2034 from livery.footman.profile import DROP
2036 env[DROP] = str(drop)
2037 red: list[str] = []
2038 try:
2039 for entry in entries:
2040 if asked and entry.task not in asked:
2041 print(
2042 f" {resolved}/{job}: {entry.task} ({entry.source}) is not"
2043 " among the tasks this dispatch asked for; skipped"
2044 )
2045 continue
2046 if entry.once:
2047 first_os, first_python = first_leg(root, resolved, job)
2048 leg_is_first = (not os_label or os_label == first_os) and (
2049 not python or python == first_python
2050 )
2051 if not leg_is_first:
2052 first = ", ".join(
2053 part for part in (first_os, first_python) if part
2054 )
2055 print(
2056 f" {resolved}/{job}: {entry.task} ({entry.source}) runs"
2057 f" once, on the first leg ({first}); skipped"
2058 )
2059 continue
2060 if not split.first:
2061 print(
2062 f" {resolved}/{job}: {entry.task} ({entry.source}) runs"
2063 f" once, on the first shard ({Shard(1, split.count)});"
2064 " skipped"
2065 )
2066 continue
2067 if entry.every:
2068 is_due, when = due(entry.every, today())
2069 if not is_due:
2070 print(
2071 f" {resolved}/{job}: {entry.task} ({entry.source}) runs"
2072 f" every {entry.every}; next on {when:%Y-%m-%d}, skipped"
2073 )
2074 continue
2075 argv = [prog]
2076 if drop is not None:
2077 argv.append(f"--profile={trace_file(entry.task)}")
2078 argv.append(entry.task)
2079 argv.extend(arg.format(**facts) for arg in entry.args)
2080 if entry.sharded and split.count > 1:
2081 argv.append(f"--shard={split}")
2082 print(f" {resolved}/{job}: {entry.task} ({entry.source})")
2083 # The record lists the store for itself: a sibling job's
2084 # half may have landed after this job's listing was
2085 # taken, and the job's listing never learns a new ref.
2086 handed = env
2087 if entry.task == RECORD_TASK:
2088 handed = {k: v for k, v in env.items() if k != SNAPSHOT_VARIABLE}
2089 code = spawn(argv, handed)
2090 if code != 0:
2091 reason = f"{resolved}/{job}: {entry.task} exited {code}"
2092 if entry.source == "builtin" or entry.judges:
2093 fail(reason)
2094 print(f" {reason}; the scheduled entries after it still run")
2095 red.append(reason)
2096 finally:
2097 # A red job's timeline is the one most worth having, so the
2098 # push happens whatever the entries did. What the run ran on
2099 # is a row the merge point reads, so every job records it,
2100 # traced or not.
2101 if drop is not None and pushing:
2102 _push_job(root, drop, job=display, leg=label)
2103 _push_about(root)
2104 if red:
2105 fail("; ".join(red))
2106 if not entries:2106 ↛ 2107line 2106 didn't jump to line 2107 because the condition on line 2106 was never true
2107 print(f" {resolved}/{job}: nothing scheduled")