1"""The legs' traces: what a run cost, kept where no sync brings it.
3A profiled job leaves two records, and the workspace contract prices them
4apart under ``[ci]``. ``timing``, on unless the contract says otherwise,
5runs every entry of every job profiled: each check leg writes a Chrome
6trace of every task, step and test it ran, and its timing row, kilobytes
7that `fm ci.timings`, the speed marks and `fm ci.balance` read, comes from
8that trace. ``profile``, off unless asked, pushes the trace itself,
9megabytes a leg that only `fm ci.profile` reads, to a namespace of its own,
10outside the one refspec a sync mirrors. ``profile-window`` says how many
11runs the pushed channel keeps, ``profile-into`` where an assembled file
12lands, and ``profile-keep`` how many of those files stay there. A pushed
13trace is the one a profiled run writes, so ``profile = true`` beside
14``timing = false`` refuses.
16Off is zero cost rather than less. ``timing = false`` runs nothing
17profiled, writes no row and pushes nothing. With ``profile`` off the trace
18stays on the runner and goes with it, so a checkout pays nothing for traces
19nobody asked for, and the gate reads none of them. The file is a Chrome
20trace of that leg: every task, step and test it ran, and the wall-clock
21origin of its own clock, so one leg's timeline can be laid beside another's.
23The push is observational. A trace that is not there and a push origin
24refuses are each one printed line, and none of them decides a job.
26[livery.workshop._traces.assemble][] reads the other way: one trace for a
27whole run, built from the forge's own job times and whatever traces the
28channel holds. The forge is asked for the skeleton and nothing else, so
29nothing here knows a forge.
30"""
45 LEG_VARIABLE,
46 RunContext,
47 run_context,
48 state_store,
49)
52 from livery.forge import Job, Repository
54#: The one file a leg puts on its trace ref: the trace as its writer
55#: wrote it, byte for byte, so nothing re-serialises a megabyte to
56#: store it and two legs of one run delta against each other in the
57#: pack.
60#: Beside it, the job the forge lists this leg as. The ref is keyed by
61#: the leg label the runner sets, and a matrix job's forge name is
62#: spelled another way entirely (``check (ubuntu-latest, 3.14)``
63#: against ``check-ubuntu-latest-3.14``), so the name the assembler
64#: joins on is written down rather than derived.
67#: The contract's keys, under ``[ci]``.
74#: What the contract answers when it says nothing. The timing rows
75#: are on: they are kilobytes a leg, and the timings, the speed marks
76#: and the balance read them. The push is off: a trace is megabytes a
77#: leg, every run, and only an assembled run reads it, so a workspace
78#: asks for it while investigating. A hundred runs is about a working
79#: week on a repository pushing twenty a day, which is what the channel
80#: is for: a Monday can still open Friday's run. Consecutive runs of
81#: one shape cost about 10 KiB each on the channel, because they delta
82#: against each other, so the bound is retention and not size. The
83#: checkout keeps fewer and for a different reason: a file there is
84#: plain on disk and deltas against nothing, so fifty bounds a disk
85#: rather than a ref.
90#: Beside the checkout's other records of what it did to itself: the
91#: receipts of the tools it installed, the entry points it linked, the
92#: state it keeps. An assembled trace is the same species, and the
93#: whole directory is already ignored, so the workspace root gains
94#: nothing from profiling.
100 """What the contract says about the legs' traces.
102 Attributes:
103 timing: Whether every job runs its entries profiled. On, each
104 check leg writes a trace of every task, step and test it
105 ran, and its timing row is read from that trace. Off means
106 zero cost: no entry runs profiled, nothing is written on a
107 runner, no row, no push.
108 profile: Whether a job pushes the trace it wrote to the trace
109 refs, for a run to be assembled from. Off, the trace stays
110 on the runner and goes with it, and the timing row is
111 unaffected. On needs ``timing`` on, and a contract saying
112 otherwise refuses. A run from a period when it was off
113 still assembles at the job level, because that shape comes
114 from the forge.
115 window: How many runs' traces the pushed channel keeps. This
116 one governs what a later assembly can still read, so it
117 decides how far back a chain can be built.
118 into: Where an assembled file lands, relative to the
119 workspace root unless it is absolute.
120 keep: How many assembled files stay in that directory. This
121 one governs this checkout's own disk and nothing else: the
122 files here are output, never input, so sweeping them
123 strands no chain.
124 """
134 """The contract's ``[ci]`` trace keys for *root*, and what it got wrong.
136 The contract's judge refuses a key of the wrong type. A count
137 below zero or an empty path is named with what it must be, and the
138 default stands for it.
140 Raises:
141 Failed: when ``profile`` is true beside ``timing = false``,
142 naming both keys. A pushed trace is the one a profiled run
143 writes, so the contradiction refuses here, where every
144 reader of the policy meets it once, rather than as a line
145 one reader prints and another never sees.
146 """
147 from livery.workshop._contract import load_contract
149 contract = load_contract(root / "workshop.toml")
150 found: dict[str, Any] = contract.get("ci") or {}
151 timing = _flag(found, TIMING_KEY, TIMING_DEFAULT)
152 profile = _flag(found, PROFILE_KEY, PROFILE_DEFAULT)
153 if profile and not timing:
154 fail(
155 f"[ci] {PROFILE_KEY} is true and [ci] {TIMING_KEY} is false; the trace"
156 f" pushed is the one a profiled run writes, so {PROFILE_KEY} = true"
157 f" implies {TIMING_KEY} = true: drop {TIMING_KEY} = false or set"
158 f" {PROFILE_KEY} = false"
159 )
160 window, why_window = _count(found, WINDOW_KEY, WINDOW_DEFAULT)
161 into, why_into = _text(found, INTO_KEY, INTO_DEFAULT)
162 keep, why_keep = _count(found, KEEP_KEY, KEEP_DEFAULT)
163 return Policy(timing, profile, window, into, keep), "; ".join(
164 why for why in (why_window, why_into, why_keep) if why
165 )
169 """Keep the newest *keep* assembled files under *at*; what went.
171 Runs after each write, so a checkout that profiles often does not
172 accumulate. Newest by modification time, because the name carries a
173 run id or a commit and neither orders by age. A *keep* of zero
174 keeps everything: the count is a ceiling, and no ceiling means no
175 sweeping rather than sweeping all of it.
177 Every trace in the directory is a candidate, whichever writer made
178 it. The directory belongs to the mechanism, so a new writer gets
179 housekeeping and an ignore rule without either being taught about
180 it; the cost is that a file put there by hand is swept like the
181 rest, and a trace to keep belongs somewhere else.
183 Returns one line per file removed, for the caller to report beside
184 whatever it wrote. A file that cannot be removed is named and the
185 sweep goes on: this is housekeeping and never a verdict.
186 """
187 if keep <= 0:
188 return []
189 found = sorted(
190 (path for path in at.glob("*.json") if path.is_file()),
191 key=lambda path: path.stat().st_mtime,
192 reverse=True,
193 )
194 lines = []
195 for path in found[keep:]:
196 try:
197 path.unlink()
198 except OSError as error:
199 lines.append(f"profile: {path.name} stayed ({error})")
200 continue
201 lines.append(f"profile: swept {path.name}")
202 return lines
206 value = found.get(key)
207 return fallback if value is None else bool(value)
211 value = found.get(key)
212 if value is None:
213 return fallback, ""
214 if int(value) >= 0:
215 return int(value), ""
216 return fallback, f"[ci] {key} is {value!r}; it is a whole number of runs"
220 value = found.get(key)
221 if value is None:
222 return fallback, ""
223 if value:
224 return str(value), ""
225 return fallback, f"[ci] {key} is {value!r}; it is a path"
229 """How many runs of traces the contract keeps, for the janitor."""
230 kept, _why = policy(root)
231 return kept.window
234#: One ref per leg of a run, keyed as the per-run metrics are, so the
235#: two halves of one leg answer to the same key. The files are not
236#: rows, so the count of runs is the whole bound the janitor applies.
238 # Under the traces' own namespace, so the ref reads
239 # ``refs/workshop-trace/run/<run>/<leg>``, the per-run metrics'
240 # shape one namespace over.
241 "run",
242 ("run", "leg"),
243 namespace=git.TRACE_NAMESPACE,
244 holds_rows=False,
245 keep=_kept_runs,
246)
249#: What a run ran on, when the forge files it under another commit: one
250#: ref per commit, a row per run. A run dispatched on a branch is filed
251#: under that branch's tip, which a merge landing first moves past the
252#: commit the dispatch named, so the run writes down the commit it
253#: actually checked out. Read beside the forge's own listing, never
254#: instead of it, and kept by age because a sha ranks by nothing.
256 "about",
257 ("commit",),
258 namespace=git.TRACE_NAMESPACE,
259 stale_after=timedelta(days=90),
260)
264 """The leg this process belongs to, as the job runner named it."""
265 return os.environ.get(LEG_VARIABLE, "")
269 root: Path,
270 trace: Path,
271 *,
272 job: str = "",
273 leg: str = "",
274 run: RunContext | None = None,
275) -> str:
276 """Put *trace* on this leg's ref of the run; the line to print.
278 *job* is the job's name as the forge lists it, written down beside
279 the trace: it is what an assembler joins the leg to its job by, and
280 a matrix job's forge name cannot be derived from the leg's label.
282 *leg* is that label. The job runner gives it to the entries it
283 spawns and pushes from its own process, which never had it in its
284 environment, so the caller that knows the label says it;
285 `livery.workshop._traces.this_leg` answers for a caller running
286 inside a leg, where the environment is the only source.
288 Empty when the contract asks for no pushed traces (``[ci] profile``,
289 off unless asked), so a workspace that wants none neither pushes nor
290 says anything. Every other answer is a line: a trace that was never
291 written, a leg nothing named, a run this is not, or origin's own
292 refusal.
293 """
294 kept, why = policy(root)
295 if not kept.profile:
296 return ""
297 if why:
298 return f"profile: {why}"
299 found = run or run_context()
300 if found is None:
301 return "profile: not a CI run; the trace stays on this machine"
302 named = leg or this_leg()
303 if not named:
304 return f"profile: {LEG_VARIABLE} names no leg; the trace is not pushed"
305 if not trace.is_file():
306 return f"profile: no trace at {trace}; nothing to push"
307 series = TRACES.series(found.run_id, named)
308 files = {TRACE_FILE: trace.read_text(encoding="utf-8")}
309 if job:
310 files[JOB_FILE] = json.dumps({"job": job}, sort_keys=True)
311 refused = git.put(
312 state_store(root),
313 series.ref,
314 files,
315 message=f"trace: {named} of run {found.run_id}",
316 ci_only=True,
317 )
318 if refused:
319 return f"profile: the trace was not pushed ({refused})"
320 size = trace.stat().st_size
321 return f"profile: {size // 1024} KiB pushed to {series.ref}"
325 """Record the commit this run ran on when the forge files it elsewhere; the line.
327 A push run and a pull request run are filed under the commit they
328 ran on, so nothing needs writing down. A dispatched run is filed
329 under the tip of the ref it was dispatched on, and another merge
330 landing between the dispatch and the run moves that tip past the
331 commit the dispatch named. Then the forge's listing for the commit
332 the run is about does not hold it, and only the run itself knows.
334 Written whatever the contract says about traces. The record is one
335 row, and more than the assembler reads it: the merge point asks
336 [livery.workshop._traces.runs_about][] for the wave of a squash the
337 forge filed under a tip that moved on, so a workspace that keeps no
338 trace still keeps this. Empty outside a run, and on every run whose
339 own head is the commit it checked out, which is most of them. Every
340 other answer is a line.
341 """
342 found = run or run_context()
343 if found is None:
344 return ""
345 head = git.head_sha(root)
346 if not head or head == found.head_sha:
347 return "" # the forge already files it here
348 refused = ABOUT.series(head).put(
349 state_store(root),
350 {found.run_id: {"run": found.run_id, "event": found.event, "ref": found.ref}},
351 message=f"about: run {found.run_id} ran on {head[:12]}",
352 )
353 if refused:353 ↛ 354line 353 didn't jump to line 354 because the condition on line 353 was never true
354 return f"profile: what this run ran on was not recorded ({refused})"
355 return f"profile: run {found.run_id} recorded as the run of {head[:12]}"
360 """One run about a commit.
362 Attributes:
363 run_id: The forge's own id for the run.
364 event: What started it, as the forge or the run's own record
365 spells it.
366 recorded: Whether the run said so itself, because the forge
367 files it under another commit.
368 """
376 root: Path, repo: Repository, commit: str
377) -> tuple[tuple[Caused, ...], list[str]]:
378 """Every run about *commit*, and a line per thing worth saying.
380 Two sources in one answer. The forge's listing for the commit holds
381 the push run, the pull request runs and every dispatched run it
382 filed under it. Beside that, the runs that recorded themselves,
383 because the forge filed them under a commit that had moved on. A run
384 both sources name appears once, from the forge.
385 """
386 found = [
387 Caused(str(run.id), run.event) for run in repo.checks.runs(head_sha=commit)
388 ]
389 known = {caused.run_id for caused in found}
390 with git.remote_snapshot(state_store(root), fetch=(ABOUT.series(commit).ref,)):
391 rows = ABOUT.series(commit).rows(state_store(root))
392 lines = [f"{commit[:12]}: {rows.reason}"] if rows.reason else []
393 for row in rows.rows:
394 run_id = str(row.data.get("run") or "")
395 if run_id and run_id not in known:395 ↛ 393line 395 didn't jump to line 393 because the condition on line 395 was always true
396 found.append(Caused(run_id, str(row.data.get("event") or ""), True))
397 known.add(run_id)
398 return tuple(found), lines
403 """One commit of a chain, and the runs about it.
405 Attributes:
406 commit: The commit.
407 runs: Every run about it that assembled to something, in the
408 order they were walked.
409 """
417 """A chain of runs as one timeline.
419 Attributes:
420 events: Every run's events on one clock, in epoch microseconds,
421 which is what a drop box takes and what a file records its
422 zero against.
423 nodes: The commits walked, in order.
424 lines: A line per thing worth saying about the walking.
425 """
433 root: Path,
434 repo: Repository,
435 commit: str,
436 *,
437 depth: int = 2,
438 now: datetime | None = None,
439) -> Walked:
440 """Every run about *commit*, and the runs of what its merge produced.
442 Three recorded facts and no guesses: what the forge lists for a
443 commit, what a run wrote down about the commit it ran on, and the
444 merge commit a pull request names. So a chain followed while it
445 happens and one walked from the same commit weeks later are the same
446 tree, which is the property this exists for.
448 A release is two commits and one hop. The branch commit carries the
449 pull request's run; the merge commit carries the base's own run and
450 the wave dispatched at it, which is filed under the base's tip and
451 found through the commit it recorded rather than through a time.
453 Args:
454 root: The workspace root, whose channel the traces are read from.
455 repo: The repository the runs belong to.
456 commit: Where the walk starts.
457 depth: How many merges to follow. Zero walks the one commit.
458 now: The moment a job still running is measured to.
460 Returns:
461 The events, the commits walked with the runs about each, and a
462 line per thing worth saying. A commit with no run and no merge
463 is a node with nothing under it, never a guess.
464 """
465 moment = now or datetime.now(UTC)
466 events: list[dict[str, Any]] = []
467 lines: list[str] = []
468 nodes: list[Node] = []
469 walked: set[str] = set()
470 taken: set[str] = set()
471 ceiling = 0
472 ahead = [(commit, depth)]
473 while ahead:
474 at, left = ahead.pop(0)
475 if not at or at in walked:
476 continue
477 walked.add(at)
478 caused, why = runs_about(root, repo, at)
479 lines += why
480 kept: list[str] = []
481 said: list[str] = []
482 for run in caused:
483 if run.run_id in taken:483 ↛ 484line 483 didn't jump to line 484 because the condition on line 483 was never true
484 continue
485 taken.add(run.run_id)
486 made = assemble(root, repo, run.run_id, head_sha=at, clock=0.0, now=moment)
487 lines += [f"run {run.run_id}: {line}" for line in made.lines]
488 if not made.events:488 ↛ 489line 488 didn't jump to line 489 because the condition on line 488 was never true
489 continue
490 moved, ceiling = _renumbered(made.events, by=ceiling, about=at, run=run)
491 events += moved
492 kept.append(run.run_id)
493 said.append(
494 f"run {run.run_id} ({run.event}"
495 + (", recorded)" if run.recorded else ")")
496 )
497 nodes.append(Node(at, tuple(kept)))
498 lines.append(f"{at[:12]}: {', '.join(said) if said else 'no run assembled'}")
499 if left <= 0:
500 continue
501 found = repo.pr.find_by_head_sha(at)
502 if found is None or not found.merged_sha:
503 continue
504 lines.append(f"{at[:12]} merged as {found.merged_sha[:12]} by #{found.number}")
505 ahead.append((found.merged_sha, left - 1))
506 return Walked(events, tuple(nodes), lines)
510 events: list[dict[str, Any]], *, by: int, about: str, run: Caused
511) -> tuple[list[dict[str, Any]], int]:
512 """*events* with every process shifted by *by*; them and the highest used.
514 Every run's own process carries the same number, and its legs'
515 groups are numbered from the same place, so two runs laid side by
516 side would read as one process. Shifting by what the chain has used
517 so far keeps them apart however many runs it walks. The run's own
518 process also takes the commit it is about into its name, because
519 two runs of one chain are told apart by that and not by their ids.
520 """
521 highest = by
522 moved: list[dict[str, Any]] = []
523 for event in events:
524 own = event.get("pid")
525 if not isinstance(own, int):525 ↛ 526line 525 didn't jump to line 526 because the condition on line 525 was never true
526 moved.append(event)
527 continue
528 shifted = {**event, "pid": own + by}
529 highest = max(highest, own + by)
530 if (
531 own == RUN_PID
532 and event.get("ph") == "M"
533 and event.get("name") == "process_name"
534 ):
535 shifted["args"] = {
536 "name": f"run {run.run_id} of {about[:12]} ({run.event})"
537 }
538 moved.append(shifted)
539 return moved, highest
544 """One run as a timeline, and where that timeline's zero sits.
546 Attributes:
547 events: The trace's events, every stamp relative to ``origin``.
548 origin: That zero, in wall-clock microseconds, which is what lets
549 another timeline be laid on this one.
550 lines: A line per thing worth saying about the assembling.
551 """
558#: The assembled trace's own process: the run, whose tracks are its
559#: jobs. A leg's own trace keeps process groups of its own, numbered
560#: from `LEG_PIDS` upward so two legs of one run never share one.
566 root: Path,
567 repo: Repository,
568 run_id: str,
569 *,
570 head_sha: str = "",
571 clock: float | None = None,
572 now: datetime | None = None,
573) -> Assembled:
574 """One trace of a whole run: its jobs, their steps, and the legs' traces.
576 The forge answers the skeleton: every job with its status, its
577 conclusion, its start and end, and its own steps. Each job is a
578 track of the run's process, so jobs that ran at once read as the
579 parallel work they were. Where the channel holds that job's trace,
580 its events are laid on the run's clock through the origin the
581 trace recorded, in process groups of their own: the alignment is
582 what nests them, and it comes from two clocks that were written
583 down independently rather than from the order anything arrived in.
585 A job with no trace keeps its slice and its steps. A job the forge
586 gives no times at all, a skip, is an instant event carrying its
587 conclusion, because a thing with no duration is an event and not a
588 span. A job still running is a slice up to the moment of assembly.
590 Args:
591 root: The workspace root, which the traces are read through.
592 repo: The repository whose run this is.
593 run_id: The forge's own id for the run.
594 head_sha: The commit the run checked, when the caller knows
595 it. With it the run's own row is read too, so the wait
596 before each job can be drawn; without it the jobs' own
597 times are the whole skeleton.
598 clock: The wall-clock moment, in microseconds, every stamp is
599 measured from. The run's own first moment by default, which
600 is a file that starts at zero; ``0.0`` leaves the stamps on
601 the wall clock itself, which is what a drop box takes.
602 now: The moment a running job's slice ends; the present by
603 default.
605 Returns:
606 The events, the wall-clock moment they are measured from, and a line
607 per thing worth saying. No events at all means the forge listed no
608 job for the run.
609 """
610 # The plugin lays one trace on another's clock, and is optional: an
611 # ordinary run of any verb must not import it, and importing this
612 # module is how the submit reaches `drop_run`.
613 from livery.footman.profile import laid_on
615 moment = now or datetime.now(UTC)
616 lines: list[str] = []
617 jobs = repo.checks.jobs(int(run_id)) if run_id.isdigit() else ()
618 if not run_id.isdigit():
619 lines.append(f"run {run_id}: not a number, so the forge cannot be asked")
620 return Assembled([], 0.0, lines)
621 if not jobs:
622 lines.append(f"run {run_id}: the forge lists no job for it")
623 return Assembled([], 0.0, lines)
624 created = _run_created(repo, run_id, head_sha, lines) if head_sha else None
625 zero = _zero_of(jobs, created, moment) if clock is None else clock
626 events: list[dict[str, Any]] = [
627 {
628 "ph": "M",
629 "name": "process_name",
630 "pid": RUN_PID,
631 "args": {"name": f"run {run_id}"},
632 }
633 ]
634 # One listing and one fetch for every leg's trace of the run.
635 with git.remote_snapshot(state_store(root), fetch=(f"{TRACES.prefix}{run_id}/",)):
636 traced, refusals = _legs_of(root, run_id)
637 lines += refusals
638 pids = itertools.count(LEG_PIDS)
639 ordered = sorted(jobs, key=_job_order)
640 for tid, job in enumerate(ordered, start=1):
641 events += _job_events(job, tid=tid, zero=zero, created=created, now=moment)
642 text = traced.get(job.name) or traced.get(git.slug(job.name))
643 if text is None:
644 continue
645 found, why = laid_on(text, zero=zero, pids=pids, label=job.name)
646 if why:
647 lines.append(f"{job.name}: {why}")
648 continue
649 events += found
650 drawn, said = _residue(
651 job, found, tid=tid, total=len(ordered), zero=zero, now=moment
652 )
653 events += drawn
654 own = f"{job.name}: {len(found)} event(s) of its own"
655 lines.append(f"{own}, {said}" if said else own)
656 return Assembled(events, zero, lines)
660 """*stamp* as wall-clock microseconds, or ``None`` when it says nothing.
662 A stamp with no offset is read as UTC, because that is what a forge
663 means by one. Read as local time instead, which is what the standard
664 library does with a naive stamp, a leg's trace would land as far from
665 its job as the reader's own offset: an hour in London, two in
666 Amsterdam, and silently.
667 """
668 if not stamp:
669 return None
670 try:
671 read = datetime.fromisoformat(stamp.replace("Z", "+00:00"))
672 except ValueError:
673 return None
674 if read.tzinfo is None:
675 read = read.replace(tzinfo=UTC)
676 return read.timestamp() * 1e6
680 """Jobs in the order they started, the ones that never did last."""
681 began = _epoch_us(job.started_at)
682 return (float("inf") if began is None else began, job.name)
686 repo: Repository, run_id: str, head_sha: str, lines: list[str]
687) -> float | None:
688 """When the forge accepted the run, so the wait before a job can be drawn."""
689 found = next(
690 (r for r in repo.checks.runs(head_sha=head_sha) if str(r.id) == run_id), None
691 )
692 if found is None:
693 lines.append(
694 f"run {run_id}: not among the forge's runs for {head_sha[:12]};"
695 " the waits before its jobs are not drawn"
696 )
697 return None
698 return _epoch_us(found.created_at)
702 """The assembled trace's zero: the first thing that happened in the run."""
703 moments = [began for job in jobs if (began := _epoch_us(job.started_at))]
704 if created is not None:
705 moments.append(created)
706 return min(moments, default=now.timestamp() * 1e6)
710 job: Job, *, tid: int, zero: float, created: float | None, now: datetime
711) -> list[dict[str, Any]]:
712 """One job as a track: the wait before it, its own span, and its steps."""
713 args: dict[str, Any] = {"status": job.status, "conclusion": job.conclusion}
714 events: list[dict[str, Any]] = [
715 {
716 "ph": "M",
717 "name": "thread_name",
718 "pid": RUN_PID,
719 "tid": tid,
720 "args": {"name": job.name},
721 },
722 # Ordered explicitly: a job's residue track is numbered past
723 # every job's own, and this puts it under its job rather than
724 # with the other residues at the bottom.
725 {
726 "ph": "M",
727 "name": "thread_sort_index",
728 "pid": RUN_PID,
729 "tid": tid,
730 "args": {"sort_index": tid * 2},
731 },
732 ]
733 began = _epoch_us(job.started_at)
734 ended = _epoch_us(job.completed_at) if began is not None else None
735 if began is not None and ended is None:
736 ended = now.timestamp() * 1e6
737 args["running"] = True # the slice ends where the reading was taken
738 own: dict[str, Any] = {
739 "cat": "job",
740 "name": job.name,
741 "pid": RUN_PID,
742 "tid": tid,
743 "ts": 0.0 if began is None else round(began - zero, 1),
744 "args": args,
745 }
746 if began is None or ended is None or ended <= began:
747 # Nothing ran, or nothing lasted: an event, not a span of no length.
748 # A job the forge skipped is here because it happened, and a forge
749 # times in whole seconds, which is how a skip came back ending a
750 # second before it began. Its steps are still drawn below: they have
751 # nothing to sit inside, and they happened too.
752 events.append({**own, "ph": "i", "s": "t"})
753 else:
754 if created is not None and began > created:
755 events.append(
756 {
757 "ph": "X",
758 "cat": "queue",
759 "name": f"{job.name}: queued",
760 "pid": RUN_PID,
761 "tid": tid,
762 "ts": round(created - zero, 1),
763 "dur": round(began - created, 1),
764 }
765 )
766 events.append({**own, "ph": "X", "dur": round(ended - began, 1)})
767 for step in job.steps:
768 start, stop = _epoch_us(step.started_at), _epoch_us(step.completed_at)
769 if start is None or stop is None:
770 continue # a step the forge did not time is not drawn
771 drawn: dict[str, Any] = {
772 "cat": "step",
773 "name": step.name,
774 "pid": RUN_PID,
775 "tid": tid,
776 "ts": round(start - zero, 1),
777 "args": {"conclusion": step.conclusion},
778 }
779 # The same rule as a job's: what had no extent is an event. A
780 # forge times in whole seconds, so a step inside one is common.
781 events.append(
782 {**drawn, "ph": "X", "dur": round(stop - start, 1)}
783 if stop > start
784 else {**drawn, "ph": "i", "s": "t"}
785 )
786 return events
790 job: Job,
791 found: list[dict[str, Any]],
792 *,
793 tid: int,
794 total: int,
795 zero: float,
796 now: datetime,
797) -> tuple[list[dict[str, Any]], str]:
798 """The job's span less its own trace, drawn; the events and what to say.
800 A job's wall clock is the forge's. Under it sit the timelines of the
801 entries that job ran, and the difference is the runner's own work:
802 the checkout, the caches, the tool store, the post-job save. Drawn,
803 so a reader adds a job up and finds nothing unexplained, and drawn
804 from the entries' own stamps, so the arithmetic is the same on a
805 forge that reports steps and one that reports none.
807 Nothing to draw, and nothing said, when the forge gave the job no
808 start, when the trace carries no stamp, or when the trace fills the
809 span. A trace that reaches outside the span is said and not drawn:
810 the two clocks disagree about the job, and a span of negative
811 length would hide that.
813 Args:
814 job: The job the forge described.
815 found: That job's own trace, already on the run's clock.
816 tid: The job's own track.
817 total: How many jobs the run has, which is what the residue
818 tracks are numbered past.
819 zero: Where the run's clock starts, in wall-clock microseconds.
820 now: The moment a job still running is measured to.
822 Returns:
823 The events, and the part of the job's line that describes them.
824 """
825 began = _epoch_us(job.started_at)
826 if began is None:
827 return [], ""
828 ended = _epoch_us(job.completed_at)
829 if ended is None:829 ↛ 830line 829 didn't jump to line 830 because the condition on line 829 was never true
830 ended = now.timestamp() * 1e6
831 stamped = [event for event in found if isinstance(event.get("ts"), (int, float))]
832 if not stamped:
833 return [], ""
834 first = min(float(event["ts"]) for event in stamped)
835 last = max(float(event["ts"]) + float(event.get("dur") or 0.0) for event in stamped)
836 opened = began - zero
837 closed = ended - zero
838 track = total + tid
839 events: list[dict[str, Any]] = []
840 spans = (("setup", opened, first - opened), ("teardown", last, closed - last))
841 outside = [name for name, _at, span in spans if span < 0]
842 if outside:
843 return [], f"its trace reaches past the job's own span ({', '.join(outside)})"
844 for name, at, span in spans:
845 if span <= 0:
846 continue # the trace starts or ends where the job does
847 events.append(
848 {
849 "ph": "X",
850 "cat": "runner",
851 "name": name,
852 "pid": RUN_PID,
853 "tid": track,
854 "ts": round(at, 1),
855 "dur": round(span, 1),
856 }
857 )
858 if not events:858 ↛ 859line 858 didn't jump to line 859 because the condition on line 858 was never true
859 return [], ""
860 return [
861 {
862 "ph": "M",
863 "name": "thread_name",
864 "pid": RUN_PID,
865 "tid": track,
866 "args": {"name": f"{job.name}: runner"},
867 },
868 {
869 "ph": "M",
870 "name": "thread_sort_index",
871 "pid": RUN_PID,
872 "tid": track,
873 "args": {"sort_index": tid * 2 + 1},
874 },
875 *events,
876 ], ", ".join(f"{event['name']} {event['dur'] / 1e6:.1f}s" for event in events)
880 """Every leg's trace for *run_id*, by the job it belongs to; and the lines.
882 One read of each ref, which carries the trace and the name of the
883 job the forge lists the leg as. A leg that recorded no job name is
884 answered for by its own key, which is what a job whose name needs
885 no spelling looks like.
886 """
887 keys = TRACES.listed(state_store(root), run_id)
888 if keys is None:
889 return {}, [f"{TRACES.prefix}{run_id}/*: could not be listed; no leg's own"]
890 lines: list[str] = []
891 found: dict[str, str] = {}
892 for key in keys:
893 series = TRACES.at(*key)
894 read = git.read(state_store(root), series.ref)
895 if read.files is None:
896 lines.append(f"{key[-1]}: {read.reason or 'its ref could not be read'}")
897 continue
898 text = read.files.get(TRACE_FILE)
899 if text is None:
900 lines.append(f"{key[-1]}: {series.ref} carries no {TRACE_FILE}")
901 continue
902 found[_job_named(read.files.get(JOB_FILE)) or key[-1]] = text
903 return found, lines
907 """The job name a leg wrote down, or empty when it wrote none."""
908 if not text:
909 return ""
910 try:
911 data: Any = json.loads(text)
912 except ValueError:
913 return ""
914 name = data.get("job") if isinstance(data, dict) else None
915 return name if isinstance(name, str) else ""
919 """The forge's newest run for *head_sha*, or empty when it lists none."""
920 runs = repo.checks.runs(head_sha=head_sha)
921 return str(runs[0].id) if runs else ""
925 """Put the run of *git*'s head in this command's own drop box; the line.
927 For a verb that followed a run to its end. With a trace being kept, the
928 run it watched goes into the same box its own children drop into, so one
929 file holds the local command, the run it caused, every job of that run,
930 and every leg's tasks and tests underneath. The stamps are the wall
931 clock's, which is what a box takes, and this run's writer lays the whole
932 thing on its own clock.
934 Empty when no trace is being kept, which is the ordinary case: the verb
935 calls this whatever its line asked for and pays one environment read.
937 Never raises. A trace is something noticed about a run, so nothing here
938 may change what the run decided: every failure, git's and the forge's
939 alike, comes back as the line to print. A CI leg proved the point by
940 running its own tests under a profile, where a rig with no repository
941 turned a watched merge into a failure.
942 """
943 from livery.footman.profile import dropped, keeping
945 # Assembling a run costs forge calls, so the cheap question comes first.
946 if not keeping():
947 return ""
948 try:
949 head_sha = git.head_sha(root)
950 found = run_id or newest_run(repo, head_sha)
951 if not found:951 ↛ 952line 951 didn't jump to line 952 because the condition on line 951 was never true
952 return f"profile: the forge lists no run for {head_sha[:12]}"
953 made = assemble(root, repo, found, head_sha=head_sha, clock=0.0)
954 if not made.events:954 ↛ 955line 954 didn't jump to line 955 because the condition on line 954 was never true
955 why = made.lines[0] if made.lines else f"run {found} assembled to nothing"
956 return f"profile: {why}"
957 if dropped(made.events) is None:957 ↛ 958line 957 didn't jump to line 958 because the condition on line 957 was never true
958 return ""
959 except Exception as error:
960 return f"profile: the run was not traced ({error})"
961 jobs = len({e["tid"] for e in made.events if e.get("cat") == "job"})
962 return f"profile: run {found} joins this trace, {jobs} job(s)"
966 repo: Repository, root: Path, *, commit: str = "", depth: int = 2
967) -> str:
968 """Put the chain from *commit* in this command's own drop box; the line.
970 For a verb that followed its work to the end of everything it
971 caused: a release, whose pull request run, base run and wave are one
972 story. The stamps are the wall clock's, which is what a box takes,
973 and this box's writer lays the whole thing on its own clock.
975 ``HEAD`` by default. Empty when no trace is being kept, which is the
976 ordinary case, so the verb pays one environment read.
978 Never raises. A trace is something noticed about work that is
979 already done, so nothing here may change what that work decided:
980 every failure, git's and the forge's alike, comes back as the line
981 to print.
982 """
983 from livery.footman.profile import dropped, keeping
985 if not keeping():985 ↛ 987line 985 didn't jump to line 987 because the condition on line 985 was always true
986 return ""
987 try:
988 at = commit or git.head_sha(root)
989 walked = chain(root, repo, at, depth=depth, now=None)
990 if not walked.events:
991 why = walked.lines[-1] if walked.lines else f"nothing about {at[:12]}"
992 return f"profile: {why}"
993 if dropped(walked.events) is None:
994 return ""
995 except Exception as error:
996 return f"profile: the chain was not traced ({error})"
997 runs = sum(len(node.runs) for node in walked.nodes)
998 return f"profile: {runs} run(s) of {len(walked.nodes)} commit(s) join this trace"
1002 root: Path,
1003 repo: Repository,
1004 commit: str,
1005 *,
1006 depth: int = 2,
1007 into: Path | None = None,
1008) -> tuple[Path | None, list[str]]:
1009 """Write the chain from *commit* where a person can open it; and the lines.
1011 The same walk the release command drops into its own trace, asked
1012 for from a commit instead. The directory is the contract's
1013 ``[ci] profile-into`` unless the caller names another.
1014 """
1015 from livery.footman.profile import as_trace
1017 kept, why = policy(root)
1018 if why:1018 ↛ 1019line 1018 didn't jump to line 1019 because the condition on line 1018 was never true
1019 return None, [f"profile: {why}"]
1020 walked = chain(root, repo, commit, depth=depth)
1021 if not walked.events:
1022 return None, walked.lines
1023 where = into or Path(kept.into)
1024 at = where if where.is_absolute() else root / where
1025 at.mkdir(parents=True, exist_ok=True)
1026 path = at / f"chain-{commit[:12]}.json"
1027 stamps = [
1028 float(event["ts"])
1029 for event in walked.events
1030 if isinstance(event.get("ts"), (int, float))
1031 ]
1032 origin = min(stamps) if stamps else 0.0
1033 shifted = [
1034 {**event, "ts": round(float(event["ts"]) - origin, 1)}
1035 if isinstance(event.get("ts"), (int, float))
1036 else event
1037 for event in walked.events
1038 ]
1039 path.write_text(as_trace(shifted, origin=origin), encoding="utf-8")
1040 return path, [*walked.lines, *sweep(at, kept.keep)]
1044 root: Path,
1045 repo: Repository,
1046 *,
1047 run_id: str = "",
1048 head_sha: str = "",
1049 into: Path | None = None,
1050) -> tuple[Path | None, list[str]]:
1051 """Write a run's assembled trace where a person can open it; and the lines.
1053 For investigating a run that has already ended, which is what the traces
1054 are kept for. Without a *run_id* the forge's newest run for *head_sha* is
1055 the one. The directory is the contract's ``[ci] profile-into`` unless the
1056 caller names another.
1057 """
1058 from livery.footman.profile import as_trace
1060 kept, why = policy(root)
1061 if why:
1062 return None, [f"profile: {why}"]
1063 found = run_id or (newest_run(repo, head_sha) if head_sha else "")
1064 if not found:
1065 return None, [f"profile: the forge lists no run for {head_sha[:12]}"]
1066 made = assemble(root, repo, found, head_sha=head_sha)
1067 if not made.events:1067 ↛ 1068line 1067 didn't jump to line 1068 because the condition on line 1067 was never true
1068 return None, made.lines
1069 where = into or Path(kept.into)
1070 at = where if where.is_absolute() else root / where
1071 at.mkdir(parents=True, exist_ok=True)
1072 path = at / f"run-{found}.json"
1073 # The stamps are relative to the run's own first moment, so the file says
1074 # where that sits: another timeline can be laid on this one.
1075 path.write_text(as_trace(made.events, origin=made.origin), encoding="utf-8")
1076 return path, [*made.lines, *sweep(at, kept.keep)]