Skip to content

livery.workshop._shard

livery.workshop._shard

A check leg's shard: its spelling, its share of the items, and the durations.

A platform's check leg may run as several shards, each on a runner of its own, each running a share of the items a test step partitions; pytest's items are test files. A shard is spelled <i>of<n> at the end of the leg's label, check-<os>-<python>-2of4; the label without it is the platform leg, the key every record below the gate job's fold is kept by. livery.workshop.Shard parses and spells one, livery.workshop.platform_leg and livery.workshop.shard_of split a label.

The share is a function of its inputs: livery.workshop.partition packs the items longest first into the shard with the least summed seconds, the first shard's sum starting at the head, so every shard of a run computes the same partition from the same set, the same durations and the same head, and every item lands in exactly one shard. The durations and the head are the medians livery.workshop.read_durations takes over the newest rows of the platform leg's series in the livery.workshop.DURATIONS family, bounded by the checkout's committer time (livery.workshop.checkout_time) so a row written while the run is in flight reaches no shard. A shard leaves what it partitioned in SHARD_FILE, under the checkout's .workshop/ directory (livery.workshop.leave_shard), and the gate job reads it back (livery.workshop.read_shard) to prove the shards of a platform partitioned one set from one row.

Classes:

Name Description
Shard

One shard of a leg: which of how many, spelled <i>of<n>.

ShardMarker

What a shard left in SHARD_FILE: what it ran and what it partitioned from.

Functions:

Name Description
platform_leg

The leg label without its shard; a label without one is returned as it is.

shard_of

The shard the leg label ends in; ONE when it ends in none.

partition

Pack items into count shards by their durations, every item in exactly one.

files_digest

A hex digest of the sorted names in files.

read_durations

The median seconds per item, and the head, of leg's rows before before.

checkout_time

The committer time of HEAD's commit in the checkout at root, timezone-aware.

marker_fields

marker as SHARD_FILE holds it, and a leg's row under shard.

parse_marker

The record loaded holds, as marker_fields wrote it; or why it is none.

leave_shard

Leave the shard's record in SHARD_FILE at root, for the gate job's proof.

read_shard

Read the record a shard left in SHARD_FILE at root.

Attributes:

Name Type Description
SCHEMA
WINDOW
NEWEST
SHARD_FILE
ONE
DURATIONS Keyed

SCHEMA module-attribute

SCHEMA = 1

WINDOW module-attribute

WINDOW = 30

NEWEST module-attribute

NEWEST = 5

SHARD_FILE module-attribute

SHARD_FILE = '.workshop/fm-shard.json'

ONE module-attribute

ONE = Shard(1, 1)

DURATIONS module-attribute

DURATIONS: Keyed = git.Keyed('durations', ('leg',), window=WINDOW, schema=SCHEMA, current=_current_legs)

Shard dataclass

Shard(index: int, count: int)

One shard of a leg: which of how many, spelled <i>of<n>.

Attributes:

Name Type Description
index int

The shard's number, from 1.

count int

How many shards the leg runs as.

Raises:

Type Description
ValueError

When the count is below 1, the index below 1, or the index above the count.

Methods:

Name Description
__post_init__
parse

The shard text spells, 2of4.

__str__

index instance-attribute

index: int

count instance-attribute

count: int

first property

first: bool

Whether this is the first shard, the one that runs what is not split.

__post_init__

__post_init__() -> None
Source code in packages/workshop/src/livery/workshop/_shard.py
def __post_init__(self) -> None:
    why = _unsound(self.index, self.count)
    if why:
        raise ValueError(f"shard {self.index} of {self.count}: {why}")

parse classmethod

parse(text: str) -> Shard

The shard text spells, 2of4.

Raises:

Type Description
ValueError

When text is not of the form, naming it.

Source code in packages/workshop/src/livery/workshop/_shard.py
@classmethod
def parse(cls, text: str) -> Shard:
    """The shard *text* spells, ``2of4``.

    Raises:
        ValueError: When *text* is not of the form, naming it.
    """
    matched = _SPELLING.match(text)
    if matched is None:
        raise ValueError(
            f"{text!r} is not a shard; a shard is spelled <i>of<n>, such as 2of4"
        )
    index, count = int(matched.group(1)), int(matched.group(2))
    why = _unsound(index, count)
    if why:
        raise ValueError(f"{text!r} is not a shard: {why}")
    return cls(index, count)

__str__

__str__() -> str
Source code in packages/workshop/src/livery/workshop/_shard.py
def __str__(self) -> str:
    return f"{self.index}of{self.count}"

ShardMarker dataclass

ShardMarker(shard: Shard, digest: str, items: tuple[str, ...], rows: tuple[str, ...], before: datetime, suites: tuple[str, ...] = (), attempt: str = '')

What a shard left in SHARD_FILE: what it ran and what it partitioned from.

Attributes:

Name Type Description
shard Shard

The shard the leg ran as.

digest str

The digest of the set it partitioned, livery.workshop.files_digest of every item of every shard.

items tuple[str, ...]

The items this shard ran, in packing order; pytest's are test files.

rows tuple[str, ...]

The durations rows it read, newest first.

before datetime

The bound it read them under, the checkout's committer time.

suites tuple[str, ...]

The units whose items were in the set, by path. A unit the record does not name ran whole on the first shard, with every step that does not read the shard, so the first shard leaves its parts. A record written without the key names none.

attempt str

The attempt of the run the shard ran in, as the runner counts them (livery.workshop.RunContext's attempt); empty when the runner did not say, and for a record written without the key. Two shards of one platform from two attempts partitioned from the record as it was each time, which the fold names when their partitions differ.

shard instance-attribute

shard: Shard

digest instance-attribute

digest: str

items instance-attribute

items: tuple[str, ...]

rows instance-attribute

rows: tuple[str, ...]

before instance-attribute

before: datetime

suites class-attribute instance-attribute

suites: tuple[str, ...] = ()

attempt class-attribute instance-attribute

attempt: str = ''

platform_leg

platform_leg(label: str) -> str

The leg label without its shard; a label without one is returned as it is.

check-windows-latest-3.14-2of4 gives check-windows-latest-3.14.

Source code in packages/workshop/src/livery/workshop/_shard.py
def platform_leg(label: str) -> str:
    """The leg *label* without its shard; a label without one is returned as it is.

    ``check-windows-latest-3.14-2of4`` gives ``check-windows-latest-3.14``.
    """
    matched = _TAIL.match(label)
    return matched.group("leg") if matched else label

shard_of

shard_of(label: str) -> Shard

The shard the leg label ends in; ONE when it ends in none.

Raises:

Type Description
ValueError

When the label ends in a shard spelling outside the form, such as 0of2, naming it.

Source code in packages/workshop/src/livery/workshop/_shard.py
def shard_of(label: str) -> Shard:
    """The shard the leg *label* ends in; `ONE` when it ends in none.

    Raises:
        ValueError: When the label ends in a shard spelling outside the
            form, such as ``0of2``, naming it.
    """
    matched = _TAIL.match(label)
    return Shard.parse(matched.group("shard")) if matched else ONE

partition

partition(items: Iterable[str], durations: Mapping[str, float], count: int, *, head: float = 0.0) -> tuple[tuple[str, ...], ...]

Pack items into count shards by their durations, every item in exactly one.

The items go longest first, ties by name, each into the shard with the least summed seconds so far, ties to the lowest shard. The first shard's sum starts at head seconds, the time of the steps it runs that the other shards leave to it, so its share of the items is lighter by about that much; a head above every other shard's final sum leaves the first shard no item. An item durations does not name weighs the median of the named durations, or one second when none are named. The result is a function of the set, the durations and the head alone: the same whatever order items comes in, and an item given twice is one item. With more shards than items the surplus shards are empty.

Returns:

Type Description
tuple[str, ...]

One tuple of items per shard, in the order the items were

...

packed.

Raises:

Type Description
ValueError

When count is below 1, or head is below 0.

Source code in packages/workshop/src/livery/workshop/_shard.py
def partition(
    items: Iterable[str],
    durations: Mapping[str, float],
    count: int,
    *,
    head: float = 0.0,
) -> tuple[tuple[str, ...], ...]:
    """Pack *items* into *count* shards by their *durations*, every item in exactly one.

    The items go longest first, ties by name, each into the shard with
    the least summed seconds so far, ties to the lowest shard. The
    first shard's sum starts at *head* seconds, the time of the steps
    it runs that the other shards leave to it, so its share of the
    items is lighter by about that much; a head above every other
    shard's final sum leaves the first shard no item. An item
    *durations* does not name weighs the median of the named
    durations, or one second when none are named. The result is a
    function of the set, the durations and the head alone: the same
    whatever order *items* comes in, and an item given twice is one
    item. With more shards than items the surplus shards are empty.

    Returns:
        One tuple of items per shard, in the order the items were
        packed.

    Raises:
        ValueError: When *count* is below 1, or *head* is below 0.
    """
    if count < 1:
        raise ValueError(f"count is {count}; a partition has at least one shard")
    if head < 0:
        raise ValueError(f"head is {head}; a head is at least 0 seconds")
    named = [float(seconds) for seconds in durations.values()]
    fill = statistics.median(named) if named else 1.0

    def weight(name: str) -> float:
        seconds = durations.get(name)
        return float(seconds) if seconds is not None else fill

    ordered = sorted(set(items), key=lambda name: (-weight(name), name))
    shards: list[list[str]] = [[] for _ in range(count)]
    sums = [0.0] * count
    sums[0] = float(head)
    for name in ordered:
        at = min(range(count), key=lambda index: (sums[index], index))
        shards[at].append(name)
        sums[at] += weight(name)
    return tuple(tuple(shard) for shard in shards)

files_digest

files_digest(files: Iterable[str]) -> str

A hex digest of the sorted names in files.

Two shards that partitioned one set have one digest, so the gate job can prove they did.

Source code in packages/workshop/src/livery/workshop/_shard.py
def files_digest(files: Iterable[str]) -> str:
    """A hex digest of the sorted names in *files*.

    Two shards that partitioned one set have one digest, so the gate
    job can prove they did.
    """
    text = "\n".join(sorted(set(files)))
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

read_durations

read_durations(root: Path, leg: str, *, before: datetime) -> tuple[dict[str, float], float, tuple[str, ...], str]

The median seconds per item, and the head, of leg's rows before before.

The rows are the newest NEWEST of the series durations/<platform leg> whose store stamp is before before; a row stamped at or after it, or not stamped, is not read, so every shard of a run reads the same rows whatever lands while the run is in flight. The median of an item is over the rows that name it with a number under items. The head is the median of every row read's head, a row without a number there counting as 0, and is what livery.workshop.partition starts the first shard's sum at. A naive before is read as UTC.

Returns:

Type Description
dict[str, float]

The seconds per item, the head in seconds, the names of the

float

rows read, newest first, and why the series could not be read.

tuple[str, ...]

A series that cannot be read is ({}, 0.0, (), reason),

str

never a raise: the caller runs as one shard of one and says

tuple[dict[str, float], float, tuple[str, ...], str]

so. An absent series is ({}, 0.0, (), "").

Source code in packages/workshop/src/livery/workshop/_shard.py
def read_durations(
    root: Path, leg: str, *, before: datetime
) -> tuple[dict[str, float], float, tuple[str, ...], str]:
    """The median seconds per item, and the head, of *leg*'s rows before *before*.

    The rows are the newest `NEWEST` of the series ``durations/<platform
    leg>`` whose store stamp is before *before*; a row stamped at or
    after it, or not stamped, is not read, so every shard of a run
    reads the same rows whatever lands while the run is in flight. The
    median of an item is over the rows that name it with a number
    under ``items``. The head is the median of every row read's
    ``head``, a row without a number there counting as 0, and is what
    [livery.workshop.partition][] starts the first shard's sum at. A
    naive *before* is read as UTC.

    Returns:
        The seconds per item, the head in seconds, the names of the
        rows read, newest first, and why the series could not be read.
        A series that cannot be read is ``({}, 0.0, (), reason)``,
        never a raise: the caller runs as one shard of one and says
        so. An absent series is ``({}, 0.0, (), "")``.
    """
    bound = before if before.tzinfo is not None else before.replace(tzinfo=UTC)
    series = DURATIONS.series(platform_leg(leg))
    found = series.rows(state_store(root))
    if found.failed:
        return {}, 0.0, (), found.reason
    chosen: list[git.Row] = []
    for row in found.rows:
        when = _moment(row.when)
        if when is None or when >= bound:
            continue
        chosen.append(row)
        if len(chosen) == NEWEST:
            break
    seen: dict[str, list[float]] = defaultdict(list)
    for row in chosen:
        held: Any = row.data.get("items")
        if not isinstance(held, dict):
            continue
        for name, seconds in held.items():
            if isinstance(seconds, int | float) and not isinstance(seconds, bool):
                seen[str(name)].append(float(seconds))
    medians = {
        name: round(statistics.median(values), 1)
        for name, values in sorted(seen.items())
    }
    heads = [_seconds(row.data.get("head")) for row in chosen]
    head = round(statistics.median(heads), 1) if heads else 0.0
    return medians, head, tuple(row.name for row in chosen), ""

checkout_time

checkout_time(root: Path) -> datetime

The committer time of HEAD's commit in the checkout at root, timezone-aware.

Every shard of a run checks out the same commit, so this bounds the durations rows every shard reads without a call to the forge.

Raises:

Type Description
GitError

When HEAD names no commit.

Source code in packages/workshop/src/livery/workshop/_shard.py
def checkout_time(root: Path) -> datetime:
    """The committer time of HEAD's commit in the checkout at *root*, timezone-aware.

    Every shard of a run checks out the same commit, so this bounds
    the durations rows every shard reads without a call to the forge.

    Raises:
        GitError: When HEAD names no commit.
    """
    spelled = git.run(root, "log", "-1", "--format=%cI", "HEAD").strip()
    return datetime.fromisoformat(spelled)

marker_fields

marker_fields(marker: ShardMarker) -> dict[str, Any]

marker as SHARD_FILE holds it, and a leg's row under shard.

Source code in packages/workshop/src/livery/workshop/_shard.py
def marker_fields(marker: ShardMarker) -> dict[str, Any]:
    """*marker* as `SHARD_FILE` holds it, and a leg's row under ``shard``."""
    return {
        "shard": str(marker.shard),
        "digest": marker.digest,
        "items": list(marker.items),
        "rows": list(marker.rows),
        "before": marker.before.isoformat(),
        "suites": list(marker.suites),
        "attempt": marker.attempt,
    }

parse_marker

parse_marker(loaded: Any, *, source: str) -> tuple[ShardMarker | None, str]

The record loaded holds, as marker_fields wrote it; or why it is none.

source names where it came from, for the reason.

Returns:

Type Description
ShardMarker | None

(record, "") when it reads; (None, reason) when loaded

str

is not a record.

Source code in packages/workshop/src/livery/workshop/_shard.py
def parse_marker(loaded: Any, *, source: str) -> tuple[ShardMarker | None, str]:
    """The record *loaded* holds, as `marker_fields` wrote it; or why it is none.

    *source* names where it came from, for the reason.

    Returns:
        ``(record, "")`` when it reads; ``(None, reason)`` when *loaded*
        is not a record.
    """
    if not isinstance(loaded, dict):
        return None, f"{source} is not a JSON object"
    try:
        shard = Shard.parse(str(loaded.get("shard", "")))
    except ValueError as error:
        return None, f"{source}: {error}"
    stamp = str(loaded.get("before", "") or "")
    before = _moment(stamp)
    if before is None:
        return None, f"{source}: {stamp!r} is not a time"
    return ShardMarker(
        shard=shard,
        digest=str(loaded.get("digest", "") or ""),
        items=tuple(str(name) for name in loaded.get("items", []) or []),
        rows=tuple(str(name) for name in loaded.get("rows", []) or []),
        before=before,
        suites=tuple(str(name) for name in loaded.get("suites", []) or []),
        attempt=str(loaded.get("attempt", "") or ""),
    ), ""

leave_shard

leave_shard(root: Path, *, shard: Shard, digest: str, items: Iterable[str], rows: Iterable[str], before: datetime, suites: Iterable[str] = (), attempt: str = '') -> Path

Leave the shard's record in SHARD_FILE at root, for the gate job's proof.

suites names the units whose items are in the set; a unit left out ran whole on the first shard. attempt is the run's attempt the shard ran in, the run context's; empty when unknown. The leg's one write copies the record into its timing row under shard, where the gate job's fold reads it.

Returns:

Type Description
Path

The path of the file left.

Source code in packages/workshop/src/livery/workshop/_shard.py
def leave_shard(
    root: Path,
    *,
    shard: Shard,
    digest: str,
    items: Iterable[str],
    rows: Iterable[str],
    before: datetime,
    suites: Iterable[str] = (),
    attempt: str = "",
) -> Path:
    """Leave the shard's record in `SHARD_FILE` at *root*, for the gate job's proof.

    *suites* names the units whose items are in the set; a unit
    left out ran whole on the first shard. *attempt* is the run's
    attempt the shard ran in, the run context's; empty when unknown.
    The leg's one write copies the record into its timing row under
    ``shard``, where the gate job's fold reads it.

    Returns:
        The path of the file left.
    """
    path = root / SHARD_FILE
    path.parent.mkdir(parents=True, exist_ok=True)
    marker = ShardMarker(
        shard, digest, tuple(items), tuple(rows), before, tuple(suites), attempt
    )
    path.write_text(json.dumps(marker_fields(marker), sort_keys=True), encoding="utf-8")
    return path

read_shard

read_shard(root: Path) -> tuple[ShardMarker | None, str]

Read the record a shard left in SHARD_FILE at root.

Returns:

Type Description
ShardMarker | None

(record, "") when it reads; (None, "") when there is no

str

file; (None, reason) when the file is not a record.

Source code in packages/workshop/src/livery/workshop/_shard.py
def read_shard(root: Path) -> tuple[ShardMarker | None, str]:
    """Read the record a shard left in `SHARD_FILE` at *root*.

    Returns:
        ``(record, "")`` when it reads; ``(None, "")`` when there is no
        file; ``(None, reason)`` when the file is not a record.
    """
    path = root / SHARD_FILE
    if not path.is_file():
        return None, ""
    try:
        loaded: Any = json.loads(path.read_text("utf-8"))
    except (OSError, ValueError) as error:
        return None, f"{SHARD_FILE} does not parse ({error})"
    return parse_marker(loaded, source=SHARD_FILE)