Skip to content

livery.toolroom.bench._index

livery.toolroom.bench._index

The published index: every record materialised into a store served as static files.

A client never replays a record. The build resolves every version of every tool and lands what it resolves to as content-addressed objects, one tree per tool:

tool                        the tool axis, the record's first line as canonical JSON
versions                    every version tracked, oldest first
<version>/observation       who read the version; its help, extractor, absences
<version>/hosts/<host>      the deployment resolved for that host
<version>/surface           the version's verbs whole, each with its options

Adjacent versions share every unchanged blob by content address, and any version is addressable without replay. Refs are local, so beside the store the build writes one pointer document naming each tool's current tree and the digest of the record it was built from. Everything under a tree is immutable and cacheable for as long as anyone likes; only the pointer needs a short lifetime. The index holds no stubs: a consumer renders the stub of the version it locks from that version's surface, through the store.

The build replays to identical digests: two builds from genesis land equal objects and equal pointers. A build into a directory holding an earlier build reads that pointer and reuses every tool whose record digest did not move, and a build whose records directory is unmoved since the last, by the stat fingerprint the build keeps in build.json beside the pointer, reads no record at all and answers from the pointer, an optimisation and never authority, since --from-genesis rebuilds every tool from the records alone.

Classes:

Name Description
Built

What one build did, as data.

Functions:

Name Description
record_digest

The digest of the record file path as authored: its name and its bytes.

open_index

The index store at into, created on first use.

read_pointer

The pointer document at into, or None when there is none.

load_records

Every record under records, by name.

build

Materialise every record under records into the index at into.

materialise

Land record whole into store and return its tree's digest.

Attributes:

Name Type Description
INDEX

The namespace of the index store: index/<tool> names the tool's tree; volatile.

POINTER

The pointer document, beside the store's layout at the index root.

SCHEMA

The pointer document's shape. Bumped when a reader must know.

BY

Who moves the index refs: the build, named by its verb.

INDEX module-attribute

INDEX = 'index'

The namespace of the index store: index/<tool> names the tool's tree; volatile.

POINTER module-attribute

POINTER = 'pointer.json'

The pointer document, beside the store's layout at the index root.

SCHEMA module-attribute

SCHEMA = 1

The pointer document's shape. Bumped when a reader must know.

BY module-attribute

BY = Subject('call', 'toolroom.index.build')

Who moves the index refs: the build, named by its verb.

Built dataclass

Built(into: str, tools: dict[str, str], rebuilt: tuple[str, ...], reused: tuple[str, ...], dropped: tuple[str, ...])

What one build did, as data.

Attributes:

Name Type Description
into str

The index root, holding the store's layout and the pointer.

tools dict[str, str]

Tool name to the digest of its current tree, every tool.

rebuilt tuple[str, ...]

The tools materialised on this build.

reused tuple[str, ...]

The tools whose earlier tree stood, their record unmoved.

dropped tuple[str, ...]

The tools the earlier pointer named and no record has.

into instance-attribute

into: str

tools instance-attribute

tools: dict[str, str]

rebuilt instance-attribute

rebuilt: tuple[str, ...]

reused instance-attribute

reused: tuple[str, ...]

dropped instance-attribute

dropped: tuple[str, ...]

record_digest

record_digest(path: Path) -> Digest

The digest of the record file path as authored: its name and its bytes.

Hashed as written, never re-serialised: a record moved anywhere digests the same, one edited anywhere does not, and the cost is a read of the bytes rather than a parse and a canonical dump.

Source code in packages/toolroom-bench/src/livery/toolroom/bench/_index.py
def record_digest(path: Path) -> Digest:
    """The digest of the record file *path* as authored: its name and its bytes.

    Hashed as written, never re-serialised: a record moved anywhere
    digests the same, one edited anywhere does not, and the cost is a
    read of the bytes rather than a parse and a canonical dump.
    """
    from livery.strongroom import digest_of

    return digest_of(f"{path.name}\0".encode() + path.read_bytes() + b"\0")

open_index

open_index(into: Path) -> Store

The index store at into, created on first use.

Raises:

Type Description
ManifestError

when into holds a store of another layout.

Source code in packages/toolroom-bench/src/livery/toolroom/bench/_index.py
def open_index(into: Path) -> Store:
    """The index store at *into*, created on first use.

    Raises:
        ManifestError: when *into* holds a store of another layout.
    """
    from livery.strongroom import Namespace, Store

    namespaces = (Namespace(INDEX, "volatile"),)
    if (into / "strongroom.json").is_file():
        return Store.open(into, namespaces=namespaces)
    return Store.create(into, namespaces=namespaces)

read_pointer

read_pointer(into: Path) -> dict[str, Any] | None

The pointer document at into, or None when there is none.

Raises:

Type Description
ValueError

when the file is there and is not a pointer this build understands, naming the file.

Source code in packages/toolroom-bench/src/livery/toolroom/bench/_index.py
def read_pointer(into: Path) -> dict[str, Any] | None:
    """The pointer document at *into*, or `None` when there is none.

    Raises:
        ValueError: when the file is there and is not a pointer this
            build understands, naming the file.
    """
    path = into / POINTER
    if not path.is_file():
        return None
    try:
        found = json.loads(path.read_text(encoding="utf-8"))
    except ValueError as error:
        raise ValueError(f"{path}: not JSON ({error})") from None
    if not isinstance(found, dict) or found.get("schema") != SCHEMA:
        raise ValueError(f"{path}: not a pointer document of schema {SCHEMA}")
    tools = found.get("tools")
    if not isinstance(tools, dict):
        raise ValueError(f"{path}: the pointer names no tools object")
    return found

load_records

load_records(records: Path) -> list[Record]

Every record under records, by name.

Raises:

Type Description
RecordError

for a record that does not validate, as the store refuses it.

Source code in packages/toolroom-bench/src/livery/toolroom/bench/_index.py
def load_records(records: Path) -> list[Record]:
    """Every record under *records*, by name.

    Raises:
        RecordError: for a record that does not validate, as the store
            refuses it.
    """
    from livery.toolroom.store import Record, records_in

    return [Record.load(path) for path in records_in(records)]

build

build(records: Path, into: Path, *, from_genesis: bool = False) -> Built

Materialise every record under records into the index at into.

A tool whose record digest matches the pointer's, and whose tree the store still names, is reused; every other tool is materialised and its ref moved. A tool the pointer names and no record has is dropped. from_genesis ignores the pointer and materialises every tool.

Before any record is read, the build asks whether anything moved: build.json beside the pointer holds a stat fingerprint of the records directory from the last build, and when it stands and every tree and ref the pointer names is present, the build answers from the pointer alone.

Returns:

Type Description
Built

What the build did.

Raises:

Type Description
RecordError

for a record that does not validate.

ValueError

for a pointer that is not one.

Source code in packages/toolroom-bench/src/livery/toolroom/bench/_index.py
def build(records: Path, into: Path, *, from_genesis: bool = False) -> Built:
    """Materialise every record under *records* into the index at *into*.

    A tool whose record digest matches the pointer's, and whose tree the
    store still names, is reused; every other tool is materialised and
    its ref moved. A tool the pointer names and no record has is
    dropped. *from_genesis* ignores the pointer and materialises every
    tool.

    Before any record is read, the build asks whether anything moved:
    `build.json` beside the pointer holds a stat fingerprint of the
    records directory from the last build, and when it stands and every
    tree and ref the pointer names is present, the build answers from
    the pointer alone.

    Returns:
        What the build did.

    Raises:
        RecordError: for a record that does not validate.
        ValueError: for a pointer that is not one.
    """
    from livery.toolroom.store import (
        BUILD_FILE,
        RECORD_SUFFIX,
        build_current,
        tree_fingerprint,
    )

    fingerprint = tree_fingerprint([str(records.resolve())])
    if not from_genesis and build_current(into):
        answer = _standing(into)
        if answer is not None:
            return answer
    loaded = load_records(records)
    digests = {
        record.name: str(record_digest(records / f"{record.name}{RECORD_SUFFIX}"))
        for record in loaded
    }
    store = open_index(into)
    previous: dict[str, Any] = {}
    if not from_genesis:
        pointer = read_pointer(into)
        previous = dict(pointer["tools"]) if pointer else {}
    tools: dict[str, str] = {}
    rebuilt: list[str] = []
    reused: list[str] = []
    for record in loaded:
        current = store.ref(INDEX, record.name)
        held = previous.get(record.name)
        stood = (
            isinstance(held, dict)
            and held.get("record") == digests[record.name]
            and current is not None
            and str(current) == held.get("tree")
            and store.state(current) == "present"
        )
        if stood and current is not None:
            tools[record.name] = str(current)
            reused.append(record.name)
        else:
            tree = materialise(store, record)
            if current != tree:
                store.set_ref(INDEX, record.name, tree, previous=current, by=BY)
            tools[record.name] = str(tree)
            rebuilt.append(record.name)
    dropped: list[str] = []
    named = {record.name for record in loaded}
    for stale in store.refs(INDEX):
        if stale in named:
            continue
        current = store.ref(INDEX, stale)
        if current is not None:
            store.drop_ref(INDEX, stale, previous=current)
        dropped.append(stale)
    document = {
        "schema": SCHEMA,
        "algorithm": store.algorithm.name,
        "tools": {
            record.name: {"tree": tools[record.name], "record": digests[record.name]}
            for record in loaded
        },
    }
    (into / POINTER).write_text(
        json.dumps(document, indent=2, sort_keys=True) + "\n", encoding="utf-8"
    )
    built_from = {
        "schema": SCHEMA,
        "records": {"path": str(records.resolve()), "fingerprint": fingerprint},
    }
    (into / BUILD_FILE).write_text(
        json.dumps(built_from, indent=2, sort_keys=True) + "\n", encoding="utf-8"
    )
    return Built(str(into), tools, tuple(rebuilt), tuple(reused), tuple(dropped))

materialise

materialise(store: Store, record: Record) -> Digest

Land record whole into store and return its tree's digest.

Source code in packages/toolroom-bench/src/livery/toolroom/bench/_index.py
def materialise(store: Store, record: Record) -> Digest:
    """Land *record* whole into *store* and return its tree's digest."""
    from livery.toolroom.store import observations

    entries: list[Entry] = [
        _blob(store, "tool", record.to_json()),
        _blob(store, "versions", list(record.versions)),
    ]
    seen = {found.version: found for found in observations(record)}
    for delta in record.deltas:
        version: list[Entry] = []
        if delta.hosts:
            version.append(
                _tree(
                    store,
                    "hosts",
                    [
                        _blob(store, host, _deployment(record, delta.version, host))
                        for host in delta.hosts
                    ],
                )
            )
        found = seen.get(delta.version)
        if found is not None:
            version.append(
                _blob(
                    store,
                    "observation",
                    {
                        "date": found.date,
                        "help": found.help,
                        "platforms": list(found.platforms),
                        "extractor": found.extractor,
                        "absent": {
                            verb: {option: list(who) for option, who in options.items()}
                            for verb, options in found.absent.items()
                        },
                    },
                )
            )
            version.append(_blob(store, "surface", found.verbs))
        entries.append(_tree(store, delta.version, version))
    return _tree(store, record.name, entries).digest