Skip to content

livery.toolroom.store._catalogue

livery.toolroom.store._catalogue

The catalogue: what a consumer resolves against, from the index or the records.

A consumer never replays a record. It reads the index, one tree per tool with the versions in order, each host's deployment and each version's stub, and resolves against that; the authoring site, which holds the records that build the index, reads them directly and gets the same catalogue. An index is read on demand: the pointer first, a tool's tree when the tool is asked for, a version's hosts when a lock needs them, so listing eleven locked tools costs eleven trees and never every version of every tool. Either way a deployment's digest is the digest of its canonical JSON, so a lock written against the records names the same deployment a consumer fetches from the index.

Reach for livery.toolroom.store.Catalogue.of_records and livery.toolroom.store.Catalogue.of_index.

Classes:

Name Description
CatalogueError

An index or a records directory that cannot be read as a catalogue.

Listed

One tool as the catalogue lists it.

Catalogue

Every tool a consumer may require, with what each version resolves to.

Functions:

Name Description
is_records

Whether source is a directory of records rather than an index.

index_source

Where a workspace reads its catalogue, as its [toolroom] index names it.

read_pointer

The pointer document at source, a directory or an HTTP base URL.

build_current

Whether the index at index stands as its build record fingerprints it.

Attributes:

Name Type Description
POINTER

The pointer document at an index's root, naming each tool's tree.

POINTER_SCHEMA

The pointer document's shape this reader understands.

BUILD_FILE

The build's record beside the pointer: what it fingerprinted, so a gate can ask.

POINTER module-attribute

POINTER = 'pointer.json'

The pointer document at an index's root, naming each tool's tree.

POINTER_SCHEMA module-attribute

POINTER_SCHEMA = 1

The pointer document's shape this reader understands.

BUILD_FILE module-attribute

BUILD_FILE = 'build.json'

The build's record beside the pointer: what it fingerprinted, so a gate can ask.

CatalogueError

Bases: ValueError

An index or a records directory that cannot be read as a catalogue.

Listed dataclass

Listed(name: str, kind: str, description: str, versions: tuple[str, ...], hosts: Mapping[str, Mapping[str, Digest]], package: str = '', mode: str = '', min_version: str = '', runtime: str = '')

One tool as the catalogue lists it.

Attributes:

Name Type Description
name str

The tool's name.

kind str

The installer kind, one of the record's kinds.

description str

One line on what the tool is.

package str

What a delegated kind's installer installs, when it differs from the name.

mode str

The record's materialisation mode; empty for the kind's default.

runtime str

What an npm tool runs on; empty for node.

versions tuple[str, ...]

Every version tracked, oldest first, as the record orders them.

hosts Mapping[str, Mapping[str, Digest]]

Per version, the host keys the version has an artifact for, with the digest of each host's deployment.

name instance-attribute

name: str

kind instance-attribute

kind: str

description instance-attribute

description: str

versions instance-attribute

versions: tuple[str, ...]

hosts instance-attribute

package class-attribute instance-attribute

package: str = ''

mode class-attribute instance-attribute

mode: str = ''

min_version class-attribute instance-attribute

min_version: str = ''

runtime class-attribute instance-attribute

runtime: str = ''

Catalogue dataclass

Catalogue(tools: Mapping[str, Listed], _deployments: dict[tuple[str, str, str], Deployment] = dict(), _store: Store | None = None, _records: dict[str, Record] = dict())

Every tool a consumer may require, with what each version resolves to.

Attributes:

Name Type Description
tools Mapping[str, Listed]

Tool name to its listing.

Methods:

Name Description
listed

The listing of name.

deployment

The deployment of name at version on host.

stub

The stub of name at version, rendered from the version's surface.

at

The catalogue at source, whichever it holds: records, or an index.

of_records

The catalogue of the records under directory, the authoring site's.

of_index

The catalogue of the index at source, a directory or an HTTP base URL.

tools instance-attribute

tools: Mapping[str, Listed]

listed

listed(name: str) -> Listed

The listing of name.

Raises:

Type Description
CatalogueError

when the catalogue has no such tool, naming what it has.

Source code in packages/toolroom-store/src/livery/toolroom/store/_catalogue.py
def listed(self, name: str) -> Listed:
    """The listing of *name*.

    Raises:
        CatalogueError: when the catalogue has no such tool, naming
            what it has.
    """
    found = self.tools.get(name)
    if found is None:
        known = ", ".join(sorted(self.tools)) or "nothing"
        raise CatalogueError(f"no record of {name}; the catalogue lists {known}")
    return found

deployment

deployment(name: str, version: str, host: str) -> Deployment

The deployment of name at version on host.

From the records it is resolved; from the index it is the blob the tool's tree names, fetched through the sources.

Raises:

Type Description
CatalogueError

when the version has no such host, or the blob cannot be read.

Source code in packages/toolroom-store/src/livery/toolroom/store/_catalogue.py
def deployment(self, name: str, version: str, host: str) -> Deployment:
    """The deployment of *name* at *version* on *host*.

    From the records it is resolved; from the index it is the blob
    the tool's tree names, fetched through the sources.

    Raises:
        CatalogueError: when the version has no such host, or the
            blob cannot be read.
    """
    listed = self.listed(name)
    digest = listed.hosts.get(version, {}).get(host)
    if digest is None:
        has = ", ".join(listed.hosts.get(version, {})) or "none"
        raise CatalogueError(
            f"{name} {version}: no host {host}; the version has {has}"
        )
    held = self._deployments.get((name, version, host))
    if held is not None:
        return held
    if self._store is None:  # pragma: no cover - records fill every deployment
        raise CatalogueError(f"{name} {version} {host}: no deployment was read")
    try:
        data = json.loads(self._store.fetch(digest).read_bytes())
    except Exception as error:
        raise CatalogueError(
            f"{name} {version} {host}: the deployment {digest} cannot be read:"
            f" {error}"
        ) from error
    found = Deployment.from_json(data, where=f"{name} {version} {host}")
    self._deployments[(name, version, host)] = found
    return found

stub

stub(name: str, version: str) -> str

The stub of name at version, rendered from the version's surface.

From the records the surface is the version's resolved observation; from the index it is the version's observation and surface blobs. Either way the text is the same rendering, the version's own verbs and options, exactly what a workspace locking that version types against.

Raises:

Type Description
CatalogueError

when the version was never read, or its blobs cannot be read.

Source code in packages/toolroom-store/src/livery/toolroom/store/_catalogue.py
def stub(self, name: str, version: str) -> str:
    """The stub of *name* at *version*, rendered from the version's surface.

    From the records the surface is the version's resolved
    observation; from the index it is the version's `observation`
    and `surface` blobs. Either way the text is the same rendering,
    the version's own verbs and options, exactly what a workspace
    locking that version types against.

    Raises:
        CatalogueError: when the version was never read, or its
            blobs cannot be read.
    """
    self.listed(name)
    record = self._records.get(name)
    if record is not None:
        found = surface_at(record, version) if version in record.versions else None
        if found is None:
            read = ", ".join(o.version for o in observations(record)) or "none"
            raise CatalogueError(
                f"{name} {version}: never read; the versions read are {read}"
            )
        return render_observation(name, found)
    tools = self.tools
    if not isinstance(tools, _LazyTools):  # pragma: no cover - records fill it
        raise CatalogueError(f"{name} {version}: no surface was read")
    return render_observation(name, tools.observation(name, version))

at classmethod

at(source: str, *, home: Home, offline: bool = False) -> Catalogue

The catalogue at source, whichever it holds: records, or an index.

A directory with no pointer file and record files in it is the authoring site's records; anything else, a directory with a pointer or an HTTP base, is an index read through home's store. One decision, made here for every reader.

Source code in packages/toolroom-store/src/livery/toolroom/store/_catalogue.py
@classmethod
def at(cls, source: str, *, home: Home, offline: bool = False) -> Catalogue:
    """The catalogue at *source*, whichever it holds: records, or an index.

    A directory with no pointer file and record files in it is the
    authoring site's records; anything else, a directory with a
    pointer or an HTTP base, is an index read through *home*'s
    store. One decision, made here for every reader.
    """
    if is_records(source):
        return cls.of_records(Path(source))
    return cls.of_index(source, home=home, offline=offline)

of_records classmethod

of_records(directory: Path) -> Catalogue

The catalogue of the records under directory, the authoring site's.

Raises:

Type Description
RecordError

for a record that does not validate.

Source code in packages/toolroom-store/src/livery/toolroom/store/_catalogue.py
@classmethod
def of_records(cls, directory: Path) -> Catalogue:
    """The catalogue of the records under *directory*, the authoring site's.

    Raises:
        RecordError: for a record that does not validate.
    """
    tools: dict[str, Listed] = {}
    deployments: dict[tuple[str, str, str], Deployment] = {}
    records: dict[str, Record] = {}
    for path in records_in(directory):
        record = Record.load(path)
        records[record.name] = record
        hosts: dict[str, dict[str, Digest]] = {}
        for delta in record.deltas:
            hosts[delta.version] = {}
            for host in delta.hosts:
                deployment = resolve(record, delta.version, host)
                deployments[(record.name, delta.version, host)] = deployment
                hosts[delta.version][host] = deployment.digest()
        tools[record.name] = Listed(
            record.name,
            record.kind,
            record.description,
            record.versions,
            hosts,
            package=record.package,
            mode=record.mode,
            min_version=record.min_version,
            runtime=record.runtime,
        )
    return cls(tools, deployments, None, records)

of_index classmethod

of_index(source: str, *, home: Home, offline: bool = False) -> Catalogue

The catalogue of the index at source, a directory or an HTTP base URL.

The pointer is read from the source; a tool's tree is fetched into the home's store through the source when the tool is first asked for, and a version's hosts when they are, so a second read answers from the machine and a read touches only what it needs.

Raises:

Type Description
CatalogueError

when the pointer cannot be read or is not one; a tree the pointer names and cannot be fetched refuses when its tool is asked for.

Source code in packages/toolroom-store/src/livery/toolroom/store/_catalogue.py
@classmethod
def of_index(cls, source: str, *, home: Home, offline: bool = False) -> Catalogue:
    """The catalogue of the index at *source*, a directory or an HTTP base URL.

    The pointer is read from the source; a tool's tree is fetched
    into the home's store through the source when the tool is first
    asked for, and a version's hosts when they are, so a second read
    answers from the machine and a read touches only what it needs.

    Raises:
        CatalogueError: when the pointer cannot be read or is not
            one; a tree the pointer names and cannot be fetched
            refuses when its tool is asked for.
    """
    pointer = _read_pointer(source)
    entries: dict[str, dict[str, Any]] = {}
    for name, entry in sorted(pointer["tools"].items()):
        if not isinstance(entry, dict) or not isinstance(entry.get("tree"), str):
            raise CatalogueError(
                f"{source}: the pointer's entry for {name} is not one"
            )
        entries[str(name)] = entry
    store = home.open_store(sources=(_source_of(source),), offline=offline)
    return cls(_LazyTools(store, entries, source), {}, store)

is_records

is_records(source: str) -> bool

Whether source is a directory of records rather than an index.

A directory with no pointer file and record files in it; a URL never.

Source code in packages/toolroom-store/src/livery/toolroom/store/_catalogue.py
def is_records(source: str) -> bool:
    """Whether *source* is a directory of records rather than an index.

    A directory with no pointer file and record files in it; a URL never.
    """
    if "://" in source:
        return False
    directory = Path(source)
    return not (directory / POINTER).is_file() and bool(records_in(directory))

index_source

index_source(table: Mapping[str, object], root: Path) -> str

Where a workspace reads its catalogue, as its [toolroom] index names it.

A URL is the published index; a path, relative to root, is a directory holding an index or the records that build one.

Raises:

Type Description
CatalogueError

when the table names no source.

Source code in packages/toolroom-store/src/livery/toolroom/store/_catalogue.py
def index_source(table: Mapping[str, object], root: Path) -> str:
    """Where a workspace reads its catalogue, as its ``[toolroom] index`` names it.

    A URL is the published index; a path, relative to *root*, is a
    directory holding an index or the records that build one.

    Raises:
        CatalogueError: when the table names no source.
    """
    declared = table.get("index")
    if not isinstance(declared, str) or not declared:
        raise CatalogueError(
            "workshop.toml: [toolroom] index names no source; set it to the"
            " published index's URL, or to a directory holding an index or the"
            " records"
        )
    if "://" in declared:
        return declared
    return str(root / declared)

read_pointer

read_pointer(source: str) -> dict[str, Any]

The pointer document at source, a directory or an HTTP base URL.

Raises:

Type Description
CatalogueError

when no pointer can be read there, or what is read is not one.

Source code in packages/toolroom-store/src/livery/toolroom/store/_catalogue.py
def read_pointer(source: str) -> dict[str, Any]:
    """The pointer document at *source*, a directory or an HTTP base URL.

    Raises:
        CatalogueError: when no pointer can be read there, or what is
            read is not one.
    """
    return _read_pointer(source)

build_current

build_current(index: Path) -> bool

Whether the index at index stands as its build record fingerprints it.

The build writes build.json beside the pointer, naming the records directory with a stat fingerprint. Current means the pointer is there, the record reads, and the fingerprint stands; anything else, an absent record included, is not current. Stats alone are read, so the answer costs milliseconds and a caller can skip a build, or the spawn of one, without reading a record.

Source code in packages/toolroom-store/src/livery/toolroom/store/_catalogue.py
def build_current(index: Path) -> bool:
    """Whether the index at *index* stands as its build record fingerprints it.

    The build writes `build.json` beside the pointer, naming the records
    directory with a stat fingerprint. Current means the pointer is
    there, the record reads, and the fingerprint stands; anything else,
    an absent record included, is not current. Stats alone are read, so the answer costs
    milliseconds and a caller can skip a build, or the spawn of one,
    without reading a record.
    """
    try:
        held = json.loads((index / BUILD_FILE).read_text(encoding="utf-8"))
    except (OSError, ValueError):
        return False
    if not isinstance(held, dict) or not (index / POINTER).is_file():
        return False
    records = held.get("records")
    if not isinstance(records, dict) or not isinstance(records.get("path"), str):
        return False
    return bool(tree_fingerprint([records["path"]]) == records.get("fingerprint"))