livery.workshop._shard¶
livery.workshop._shard ¶
A check leg's shard: its spelling, its share of the items, and the durations.
A platform's check leg may run as several shards, each on a runner of
its own, each running a share of the items a test step partitions;
pytest's items are test files. A shard is spelled <i>of<n> at the
end of the leg's label, check-<os>-<python>-2of4;
the label without it is the platform leg, the key every record below
the gate job's fold is kept by. livery.workshop.Shard parses and
spells one, livery.workshop.platform_leg and
livery.workshop.shard_of split a label.
The share is a function of its inputs: livery.workshop.partition
packs the items longest first into the shard with the least summed
seconds, the first shard's sum starting at the head, so every shard of
a run computes the same partition from the same set, the same
durations and the same head, and every item lands in exactly one
shard. The durations and the head are the medians
livery.workshop.read_durations takes over the newest rows of the
platform leg's series in the livery.workshop.DURATIONS family,
bounded by the checkout's committer time
(livery.workshop.checkout_time) so a row written while the run is
in flight reaches no shard. A shard leaves what it partitioned in
SHARD_FILE, under the checkout's .workshop/ directory
(livery.workshop.leave_shard), and the gate job reads it back
(livery.workshop.read_shard) to prove the shards of a platform
partitioned one set from one row.
Classes:
| Name | Description |
|---|---|
Shard |
One shard of a leg: which of how many, spelled |
ShardMarker |
What a shard left in |
Functions:
| Name | Description |
|---|---|
platform_leg |
The leg label without its shard; a label without one is returned as it is. |
shard_of |
The shard the leg label ends in; |
partition |
Pack items into count shards by their durations, every item in exactly one. |
files_digest |
A hex digest of the sorted names in files. |
read_durations |
The median seconds per item, and the head, of leg's rows before before. |
checkout_time |
The committer time of HEAD's commit in the checkout at root, timezone-aware. |
marker_fields |
marker as |
parse_marker |
The record loaded holds, as |
leave_shard |
Leave the shard's record in |
read_shard |
Read the record a shard left in |
Attributes:
| Name | Type | Description |
|---|---|---|
SCHEMA |
|
|
WINDOW |
|
|
NEWEST |
|
|
SHARD_FILE |
|
|
ONE |
|
|
DURATIONS |
Keyed
|
|
DURATIONS
module-attribute
¶
DURATIONS: Keyed = git.Keyed('durations', ('leg',), window=WINDOW, schema=SCHEMA, current=_current_legs)
Shard
dataclass
¶
One shard of a leg: which of how many, spelled <i>of<n>.
Attributes:
| Name | Type | Description |
|---|---|---|
index |
int
|
The shard's number, from 1. |
count |
int
|
How many shards the leg runs as. |
Raises:
| Type | Description |
|---|---|
ValueError
|
When the count is below 1, the index below 1, or the index above the count. |
Methods:
| Name | Description |
|---|---|
__post_init__ |
|
parse |
The shard text spells, |
__str__ |
|
__post_init__ ¶
parse
classmethod
¶
The shard text spells, 2of4.
Raises:
| Type | Description |
|---|---|
ValueError
|
When text is not of the form, naming it. |
Source code in packages/workshop/src/livery/workshop/_shard.py
ShardMarker
dataclass
¶
ShardMarker(shard: Shard, digest: str, items: tuple[str, ...], rows: tuple[str, ...], before: datetime, suites: tuple[str, ...] = (), attempt: str = '')
What a shard left in SHARD_FILE: what it ran and what it partitioned from.
Attributes:
| Name | Type | Description |
|---|---|---|
shard |
Shard
|
The shard the leg ran as. |
digest |
str
|
The digest of the set it partitioned, livery.workshop.files_digest of every item of every shard. |
items |
tuple[str, ...]
|
The items this shard ran, in packing order; pytest's are test files. |
rows |
tuple[str, ...]
|
The durations rows it read, newest first. |
before |
datetime
|
The bound it read them under, the checkout's committer time. |
suites |
tuple[str, ...]
|
The units whose items were in the set, by path. A unit the record does not name ran whole on the first shard, with every step that does not read the shard, so the first shard leaves its parts. A record written without the key names none. |
attempt |
str
|
The attempt of the run the shard ran in, as the runner
counts them (livery.workshop.RunContext's |
platform_leg ¶
The leg label without its shard; a label without one is returned as it is.
check-windows-latest-3.14-2of4 gives check-windows-latest-3.14.
Source code in packages/workshop/src/livery/workshop/_shard.py
shard_of ¶
The shard the leg label ends in; ONE when it ends in none.
Raises:
| Type | Description |
|---|---|
ValueError
|
When the label ends in a shard spelling outside the
form, such as |
Source code in packages/workshop/src/livery/workshop/_shard.py
partition ¶
partition(items: Iterable[str], durations: Mapping[str, float], count: int, *, head: float = 0.0) -> tuple[tuple[str, ...], ...]
Pack items into count shards by their durations, every item in exactly one.
The items go longest first, ties by name, each into the shard with the least summed seconds so far, ties to the lowest shard. The first shard's sum starts at head seconds, the time of the steps it runs that the other shards leave to it, so its share of the items is lighter by about that much; a head above every other shard's final sum leaves the first shard no item. An item durations does not name weighs the median of the named durations, or one second when none are named. The result is a function of the set, the durations and the head alone: the same whatever order items comes in, and an item given twice is one item. With more shards than items the surplus shards are empty.
Returns:
| Type | Description |
|---|---|
tuple[str, ...]
|
One tuple of items per shard, in the order the items were |
...
|
packed. |
Raises:
| Type | Description |
|---|---|
ValueError
|
When count is below 1, or head is below 0. |
Source code in packages/workshop/src/livery/workshop/_shard.py
files_digest ¶
A hex digest of the sorted names in files.
Two shards that partitioned one set have one digest, so the gate job can prove they did.
Source code in packages/workshop/src/livery/workshop/_shard.py
read_durations ¶
read_durations(root: Path, leg: str, *, before: datetime) -> tuple[dict[str, float], float, tuple[str, ...], str]
The median seconds per item, and the head, of leg's rows before before.
The rows are the newest NEWEST of the series durations/<platform
leg> whose store stamp is before before; a row stamped at or
after it, or not stamped, is not read, so every shard of a run
reads the same rows whatever lands while the run is in flight. The
median of an item is over the rows that name it with a number
under items. The head is the median of every row read's
head, a row without a number there counting as 0, and is what
livery.workshop.partition starts the first shard's sum at. A
naive before is read as UTC.
Returns:
| Type | Description |
|---|---|
dict[str, float]
|
The seconds per item, the head in seconds, the names of the |
float
|
rows read, newest first, and why the series could not be read. |
tuple[str, ...]
|
A series that cannot be read is |
str
|
never a raise: the caller runs as one shard of one and says |
tuple[dict[str, float], float, tuple[str, ...], str]
|
so. An absent series is |
Source code in packages/workshop/src/livery/workshop/_shard.py
checkout_time ¶
The committer time of HEAD's commit in the checkout at root, timezone-aware.
Every shard of a run checks out the same commit, so this bounds the durations rows every shard reads without a call to the forge.
Raises:
| Type | Description |
|---|---|
GitError
|
When HEAD names no commit. |
Source code in packages/workshop/src/livery/workshop/_shard.py
marker_fields ¶
marker_fields(marker: ShardMarker) -> dict[str, Any]
marker as SHARD_FILE holds it, and a leg's row under shard.
Source code in packages/workshop/src/livery/workshop/_shard.py
parse_marker ¶
parse_marker(loaded: Any, *, source: str) -> tuple[ShardMarker | None, str]
The record loaded holds, as marker_fields wrote it; or why it is none.
source names where it came from, for the reason.
Returns:
| Type | Description |
|---|---|
ShardMarker | None
|
|
str
|
is not a record. |
Source code in packages/workshop/src/livery/workshop/_shard.py
leave_shard ¶
leave_shard(root: Path, *, shard: Shard, digest: str, items: Iterable[str], rows: Iterable[str], before: datetime, suites: Iterable[str] = (), attempt: str = '') -> Path
Leave the shard's record in SHARD_FILE at root, for the gate job's proof.
suites names the units whose items are in the set; a unit
left out ran whole on the first shard. attempt is the run's
attempt the shard ran in, the run context's; empty when unknown.
The leg's one write copies the record into its timing row under
shard, where the gate job's fold reads it.
Returns:
| Type | Description |
|---|---|
Path
|
The path of the file left. |
Source code in packages/workshop/src/livery/workshop/_shard.py
read_shard ¶
read_shard(root: Path) -> tuple[ShardMarker | None, str]
Read the record a shard left in SHARD_FILE at root.
Returns:
| Type | Description |
|---|---|
ShardMarker | None
|
|
str
|
file; |