Skip to content

livery.toolroom.store._lock

livery.toolroom.store._lock

The lock: one version per tool for the repository, resolved against the catalogue.

A requirement is a tool's name with a floor, declared at one of the sites, and may name the hosts it applies to; the sites' requirements union, and the lock takes for each tool the newest version the catalogue lists that satisfies every floor and resolves on every host the tool is required on. A version of a downloaded kind resolves on a host when it has that host's artifact; a delegated kind resolves everywhere its installer does. A requirement that cannot be satisfied refuses naming the tool, each floor and its site, and for a host that no eligible version has, the first version that has it. A tool required on some of the locked hosts alone is locked on those, and the entry says which; one whose scope names no locked host is not locked at all.

The lock is a file in the repository, toolroom.lock, so every checkout and every runner installs the same version; it moves only through a lock or an upgrade, never on its own.

Reach for livery.toolroom.store.Requirement, livery.toolroom.store.resolve_lock and livery.toolroom.store.Lock.

Classes:

Name Description
LockError

A requirement the catalogue cannot satisfy; the message names why.

Requirement

One site's requirement of a tool: a floor, or any version, on some hosts or all.

Graph

The resolved graph of a delegated tool's version, as the lock names it.

Locked

One tool as the lock holds it.

Lock

The repository's lock: the hosts it locks for and one version per tool.

Functions:

Name Description
resolve_lock

The lock for requirements against catalogue, on hosts.

Attributes:

Name Type Description
LOCK_FILE

The lock's name at the repository root.

LOCK_SCHEMA

The lock's shape. Bumped when a reader must know.

GRAPHS

The directory beside the lock holding one resolved graph per delegated tool.

LOCK_FILE module-attribute

LOCK_FILE = 'toolroom.lock'

The lock's name at the repository root.

LOCK_SCHEMA module-attribute

LOCK_SCHEMA = 1

The lock's shape. Bumped when a reader must know.

GRAPHS module-attribute

GRAPHS = 'toolroom.graphs'

The directory beside the lock holding one resolved graph per delegated tool.

LockError

Bases: ValueError

A requirement the catalogue cannot satisfy; the message names why.

Requirement dataclass

Requirement(name: str, floor: str = '', site: str = '', hosts: tuple[str, ...] = (), optional: bool = False, exclude: tuple[str, ...] = ())

One site's requirement of a tool: a floor, or any version, on some hosts or all.

Attributes:

Name Type Description
name str

The tool's name.

floor str

The lowest version that satisfies; empty for any.

site str

Where the requirement was declared, for a refusal.

hosts tuple[str, ...]

The hosts the requirement applies to, each a platform (windows, every locked host of that platform) or a host key (windows-x64); empty for every locked host.

optional bool

Whether ? marks it optional: locked where the catalogue can serve it, never refused.

exclude tuple[str, ...]

The platforms and host keys ! removes from its hosts.

Methods:

Name Description
parse

A requirement from its spelling, name?>=floor@scope.

__str__
satisfied_by

Whether version is at or above the floor.

on

The hosts of locked this requirement applies to, in their order.

name instance-attribute

name: str

floor class-attribute instance-attribute

floor: str = ''

site class-attribute instance-attribute

site: str = ''

hosts class-attribute instance-attribute

hosts: tuple[str, ...] = ()

optional class-attribute instance-attribute

optional: bool = False

exclude class-attribute instance-attribute

exclude: tuple[str, ...] = ()

parse classmethod

parse(text: str, *, site: str = '') -> Requirement

A requirement from its spelling, name?>=floor@scope.

Every part after the name is optional.

? marks it optional. The scope after @ is comma-separated platforms or host keys, each excluded with a leading !: dotnet_coverage@windows, tea>=1.1@linux,macos-arm, docker?@!windows-arm. The grammar is livery.toolroom.store.Spec's; a tool requirement takes no options.

Raises:

Type Description
LockError

for a spelling that is none of these, or a scope naming no host or a token that is neither a platform nor a host key.

Source code in packages/toolroom-store/src/livery/toolroom/store/_lock.py
@classmethod
def parse(cls, text: str, *, site: str = "") -> Requirement:
    """A requirement from its spelling, ``name?>=floor@scope``.

    Every part after the name is optional.

    ``?`` marks it optional. The scope after ``@`` is comma-separated
    platforms or host keys, each excluded with a leading ``!``:
    ``dotnet_coverage@windows``, ``tea>=1.1@linux,macos-arm``,
    ``docker?@!windows-arm``. The grammar is
    [livery.toolroom.store.Spec][]'s; a tool requirement takes
    no options.

    Raises:
        LockError: for a spelling that is none of these, or a
            scope naming no host or a token that is neither a
            platform nor a host key.
    """
    where = site or "a requirement"
    try:
        spec = Spec.parse(text, where=where)
    except SpecError as error:
        raise LockError(str(error)) from None
    if spec.options:
        raise LockError(
            f"{where}: {text!r} names options; a tool requirement takes"
            " none, spell it `name?>=floor@scope`"
        )
    return cls(
        spec.name,
        spec.floor,
        site,
        spec.scope.include,
        spec.optional,
        spec.scope.exclude,
    )

__str__

__str__() -> str
Source code in packages/toolroom-store/src/livery/toolroom/store/_lock.py
def __str__(self) -> str:
    return str(
        Spec(
            self.name,
            optional=self.optional,
            floor=self.floor,
            scope=Scope(self.hosts, self.exclude),
        )
    )

satisfied_by

satisfied_by(version: str) -> bool

Whether version is at or above the floor.

Source code in packages/toolroom-store/src/livery/toolroom/store/_lock.py
def satisfied_by(self, version: str) -> bool:
    """Whether *version* is at or above the floor."""
    return not self.floor or version_tuple(version) >= version_tuple(self.floor)

on

on(locked: Iterable[str]) -> tuple[str, ...]

The hosts of locked this requirement applies to, in their order.

Source code in packages/toolroom-store/src/livery/toolroom/store/_lock.py
def on(self, locked: Iterable[str]) -> tuple[str, ...]:
    """The hosts of *locked* this requirement applies to, in their order."""
    return Scope(self.hosts, self.exclude).hosts(locked)

Graph dataclass

Graph(file: str, digest: Digest, by: str = '')

The resolved graph of a delegated tool's version, as the lock names it.

A delegated kind installs a graph, not a file: the package named is one of many its installer resolves, and the transitive part moves between resolves at one version. The graph pins all of it, in the installer's own format, in a file beside the lock; the lock carries its name and its digest, so the file that installs is the file that was resolved.

Attributes:

Name Type Description
file str

The file's name under GRAPHS, beside the lock.

digest Digest

The file's bytes, so a graph edited by hand is caught.

by str

What resolved it, name and version, for the day an install refuses and someone must know what wrote it.

Methods:

Name Description
to_json

The graph as a JSON object.

from_json

The graph in value.

file instance-attribute

file: str

digest instance-attribute

digest: Digest

by class-attribute instance-attribute

by: str = ''

to_json

to_json() -> dict[str, Any]

The graph as a JSON object.

Source code in packages/toolroom-store/src/livery/toolroom/store/_lock.py
def to_json(self) -> dict[str, Any]:
    """The graph as a JSON object."""
    return {"file": self.file, "digest": str(self.digest), "by": self.by}

from_json classmethod

from_json(value: Any, *, where: str) -> Graph

The graph in value.

Raises:

Type Description
LockError

when it is not one, naming where.

Source code in packages/toolroom-store/src/livery/toolroom/store/_lock.py
@classmethod
def from_json(cls, value: Any, *, where: str) -> Graph:
    """The graph in *value*.

    Raises:
        LockError: when it is not one, naming *where*.
    """
    if (
        not isinstance(value, dict)
        or not isinstance(value.get("file"), str)
        or not value["file"]
        or not isinstance(value.get("digest"), str)
    ):
        raise LockError(f"{where}: the graph is not one")
    try:
        digest = Digest.parse(value["digest"])
    except ValueError as error:
        raise LockError(f"{where}: {error}") from None
    return cls(value["file"], digest, str(value.get("by", "")))

Locked dataclass

Locked(version: str, hosts: dict[str, Digest] = dict(), graph: Graph | None = None, on: tuple[str, ...] = (), allow_host: bool = False, optional: bool = False)

One tool as the lock holds it.

Attributes:

Name Type Description
version str

The version locked.

hosts dict[str, Digest]

Per host the tool is locked on, the deployment's digest; empty for a delegated kind, whose installer resolves the host.

graph Graph | None

The resolved graph of a delegated kind's version, or None for a downloaded kind and for a delegated one locked before a graph was written for it.

on tuple[str, ...]

The locked hosts the tool is required on, when they are fewer than the lock's; empty when it is required on every one.

allow_host bool

Whether a copy of the tool already on the machine may serve instead of the locked version, when it satisfies the requirement's floor; the workspace's host-allowed list says so, and the entry carries allow-host so every checkout and CI agree on which tools may vary.

optional bool

Whether every site that requires the tool marks it optional, so a host it is not locked on lacks it by design.

Methods:

Name Description
applies

Whether the tool is locked for host.

to_json

The entry as a JSON object.

version instance-attribute

version: str

hosts class-attribute instance-attribute

hosts: dict[str, Digest] = field(default_factory=dict)

graph class-attribute instance-attribute

graph: Graph | None = None

on class-attribute instance-attribute

on: tuple[str, ...] = ()

allow_host class-attribute instance-attribute

allow_host: bool = False

optional class-attribute instance-attribute

optional: bool = False

applies

applies(host: str) -> bool

Whether the tool is locked for host.

Source code in packages/toolroom-store/src/livery/toolroom/store/_lock.py
def applies(self, host: str) -> bool:
    """Whether the tool is locked for *host*."""
    return not self.on or host in self.on

to_json

to_json() -> dict[str, Any]

The entry as a JSON object.

Source code in packages/toolroom-store/src/livery/toolroom/store/_lock.py
def to_json(self) -> dict[str, Any]:
    """The entry as a JSON object."""
    out: dict[str, Any] = {
        "version": self.version,
        "hosts": {host: str(digest) for host, digest in sorted(self.hosts.items())},
    }
    if self.graph is not None:
        out["graph"] = self.graph.to_json()
    if self.on:
        out["on"] = list(self.on)
    if self.allow_host:
        out["allow-host"] = True
    if self.optional:
        out["optional"] = True
    return out

Lock dataclass

Lock(hosts: tuple[str, ...], tools: dict[str, Locked], notes: tuple[str, ...] = ())

The repository's lock: the hosts it locks for and one version per tool.

Attributes:

Name Type Description
hosts tuple[str, ...]

The host keys every locked download resolves on.

tools dict[str, Locked]

Tool name to its locked version and deployments.

notes tuple[str, ...]

What the resolution left out and why: an optional tool the catalogue could not serve. Not written to the file.

Methods:

Name Description
on_host

The tools locked for host, in name order.

to_json

The lock as a JSON object, tools in name order.

save

Write the lock to path, indented, with a trailing newline.

load

The lock at path.

hosts instance-attribute

hosts: tuple[str, ...]

tools instance-attribute

tools: dict[str, Locked]

notes class-attribute instance-attribute

notes: tuple[str, ...] = field(default=(), compare=False)

on_host

on_host(host: str) -> tuple[str, ...]

The tools locked for host, in name order.

Source code in packages/toolroom-store/src/livery/toolroom/store/_lock.py
def on_host(self, host: str) -> tuple[str, ...]:
    """The tools locked for *host*, in name order."""
    return tuple(
        name for name in sorted(self.tools) if self.tools[name].applies(host)
    )

to_json

to_json() -> dict[str, Any]

The lock as a JSON object, tools in name order.

Source code in packages/toolroom-store/src/livery/toolroom/store/_lock.py
def to_json(self) -> dict[str, Any]:
    """The lock as a JSON object, tools in name order."""
    return {
        "schema": LOCK_SCHEMA,
        "hosts": list(self.hosts),
        "tools": {name: self.tools[name].to_json() for name in sorted(self.tools)},
    }

save

save(path: Path) -> None

Write the lock to path, indented, with a trailing newline.

Source code in packages/toolroom-store/src/livery/toolroom/store/_lock.py
def save(self, path: Path) -> None:
    """Write the lock to *path*, indented, with a trailing newline."""
    path.write_text(json.dumps(self.to_json(), indent=2) + "\n", encoding="utf-8")

load classmethod

load(path: Path) -> Lock

The lock at path.

Raises:

Type Description
LockError

when the file is not a lock of this schema, naming the file.

Source code in packages/toolroom-store/src/livery/toolroom/store/_lock.py
@classmethod
def load(cls, path: Path) -> Lock:
    """The lock at *path*.

    Raises:
        LockError: when the file is not a lock of this schema,
            naming the file.
    """
    try:
        data = json.loads(path.read_text(encoding="utf-8"))
    except (OSError, ValueError) as error:
        raise LockError(f"{path}: not a lock ({error})") from None
    if not isinstance(data, dict) or data.get("schema") != LOCK_SCHEMA:
        raise LockError(f"{path}: not a lock of schema {LOCK_SCHEMA}")
    hosts = data.get("hosts")
    tools = data.get("tools")
    if (
        not isinstance(hosts, list)
        or not all(isinstance(h, str) and h in HOSTS for h in hosts)
        or not isinstance(tools, dict)
    ):
        raise LockError(
            f"{path}: the lock names hosts outside the six, or no tools"
        )
    locked: dict[str, Locked] = {}
    for name, entry in tools.items():
        if (
            not isinstance(entry, dict)
            or not isinstance(entry.get("version"), str)
            or not isinstance(entry.get("hosts"), dict)
        ):
            raise LockError(f"{path}: the entry for {name} is not one")
        try:
            digests = {
                str(host): Digest.parse(str(digest))
                for host, digest in entry["hosts"].items()
            }
        except ValueError as error:
            raise LockError(f"{path}: {name}: {error}") from None
        raw = entry.get("graph")
        graph = (
            None if raw is None else Graph.from_json(raw, where=f"{path}: {name}")
        )
        scope = entry.get("on", [])
        if not isinstance(scope, list) or not all(
            isinstance(h, str) and h in hosts for h in scope
        ):
            raise LockError(
                f"{path}: {name}: `on` names a host outside the lock's"
                f" ({', '.join(hosts)})"
            )
        allowed = entry.get("allow-host", False)
        if not isinstance(allowed, bool):
            raise LockError(
                f"{path}: {name}: `allow-host` is {allowed!r}, not true or false"
            )
        optional = entry.get("optional", False)
        if not isinstance(optional, bool):
            raise LockError(
                f"{path}: {name}: `optional` is {optional!r}, not true or false"
            )
        locked[str(name)] = Locked(
            entry["version"], digests, graph, tuple(scope), allowed, optional
        )
    return cls(tuple(hosts), locked)

resolve_lock

resolve_lock(catalogue: Catalogue, requirements: Iterable[Requirement], *, hosts: Iterable[str], keep: Lock | None = None, upgrade: Iterable[str] = (), host_allowed: Iterable[str] = ()) -> Lock

The lock for requirements against catalogue, on hosts.

Each tool takes the newest version that satisfies every floor declared for it and resolves on every host it is required on: the union of its requirements' scopes, the whole of hosts for one with no scope. A tool required on fewer hosts than the lock's is locked on those and its entry names them; one whose scopes reach no locked host is left out. A tool keep already locks stays at its version when that version still satisfies and resolves, unless it is named in upgrade; a tool no requirement names any more leaves the lock. A tool named in host_allowed carries the allowance in its entry, whatever keep said of it: the allowance is the sites' current word, never a kept one.

An optional requirement (?) is never refused. Its hosts join a tool's entry where the version the required sites chose has an artifact; a tool every site marks optional takes the newest version satisfying its floors that resolves somewhere, is locked where it resolves, and its entry says optional. What is left out, a host or a whole tool, is named in the lock's notes.

Raises:

Type Description
LockError

naming the first requirement that cannot be met: a tool the catalogue does not list, a floor above every version, or a host no eligible version has.

Source code in packages/toolroom-store/src/livery/toolroom/store/_lock.py
def resolve_lock(
    catalogue: Catalogue,
    requirements: Iterable[Requirement],
    *,
    hosts: Iterable[str],
    keep: Lock | None = None,
    upgrade: Iterable[str] = (),
    host_allowed: Iterable[str] = (),
) -> Lock:
    """The lock for *requirements* against *catalogue*, on *hosts*.

    Each tool takes the newest version that satisfies every floor
    declared for it and resolves on every host it is required on: the
    union of its requirements' scopes, the whole of *hosts* for one
    with no scope. A tool required on fewer hosts than the lock's is
    locked on those and its entry names them; one whose scopes reach
    no locked host is left out. A tool *keep* already locks stays at
    its version when that version still satisfies and resolves,
    unless it is named in *upgrade*; a tool no requirement names any
    more leaves the lock. A tool named in *host_allowed* carries the
    allowance in its entry, whatever *keep* said of it: the allowance
    is the sites' current word, never a kept one.

    An optional requirement (``?``) is never refused. Its hosts join a
    tool's entry where the version the required sites chose has an
    artifact; a tool every site marks optional takes the newest
    version satisfying its floors that resolves somewhere, is locked
    where it resolves, and its entry says ``optional``. What is left
    out, a host or a whole tool, is named in the lock's ``notes``.

    Raises:
        LockError: naming the first requirement that cannot be met:
            a tool the catalogue does not list, a floor above every
            version, or a host no eligible version has.
    """
    locked_hosts = tuple(hosts)
    for host in locked_hosts:
        if host not in HOSTS:
            raise LockError(f"host {host!r} is not one of {', '.join(HOSTS)}")
    by_tool: dict[str, list[Requirement]] = {}
    for requirement in requirements:
        by_tool.setdefault(requirement.name, []).append(requirement)
    moving = set(upgrade)
    allowed = set(host_allowed)
    tools: dict[str, Locked] = {}
    notes: list[str] = []
    for name in sorted(by_tool):
        wants = by_tool[name]
        required = [want for want in wants if not want.optional]
        must = tuple(
            host
            for host in locked_hosts
            if any(host in want.on(locked_hosts) for want in required)
        )
        may = tuple(
            host
            for host in locked_hosts
            if host not in must
            and any(host in want.on(locked_hosts) for want in wants if want.optional)
        )
        if not must and not may:
            continue
        sites = ", ".join(sorted({w.site for w in wants if w.site})) or "a requirement"
        try:
            listed = catalogue.listed(name)
        except CatalogueError as error:
            if not required:
                notes.append(f"{name}: optional and not catalogued ({error}); left out")
                continue
            raise LockError(f"{error}; required by {sites}") from None
        held = keep.tools.get(name) if keep and name not in moving else None
        if required:
            version = (
                held.version
                if held is not None and _eligible(listed, held.version, wants, must)
                else _newest(listed, wants, must)
            )
        else:
            found = _newest_anywhere(listed, wants, may, held)
            if found is None:
                notes.append(
                    f"{name}: optional, and no version satisfying"
                    f" {', '.join(str(w) for w in wants)} resolves on"
                    f" {', '.join(may)}; left out"
                )
                continue
            version = found
        reach = (*must, *(host for host in may if _resolves(listed, version, (host,))))
        on = tuple(host for host in locked_hosts if host in reach)
        absent = [host for host in may if host not in on]
        if absent:
            notes.append(
                f"{name}: optional, {version} has no artifact for"
                f" {', '.join(absent)}; left out there"
            )
        scope = on if on != locked_hosts else ()
        tools[name] = replace(
            _locked(listed, version, on, scope),
            allow_host=name in allowed,
            optional=not required,
        )
    return Lock(locked_hosts, tools, tuple(notes))