Skip to content

livery.strongroom._views

livery.strongroom._views

The materialiser: the only route from a digest to a path.

view fills a directory from a tree, choosing the cheapest safe rung per entry and recording which one it used; collect reads declared outputs back into a tree, learning each blob's executable bit from the caller, the file mode, the view's record, or the platform, in that order; drop_view removes only what the record lists. A view is a root while it lives, and a link never leaves it: an entry whose target escapes is parked, and a collected link that escapes is refused. prefetch warms everything one digest reaches, and shed evicts local copies a named source holds, except what a live view depends on.

The functions here are the bodies of the livery.strongroom.Store methods of the same names; reach for the methods.

Classes:

Name Description
ViewEntry

One path a view created, and how.

ViewRecord

A view: local state in the index, and a root while its directory exists.

DropReport

What dropping a view removed and what it left.

ShedReport

What a shed did.

Functions:

Name Description
new_view_id

A fresh view id; the seam tests fix for a known name.

view

Fill at from the tree, cheapest safe rung per entry, and record it.

views

Every recorded view, by id.

view_record

The record of view_id, or None.

retire_record

Forget a view: its record goes, its paths are not touched.

drop_view

Remove what the record lists and nothing else; then forget the view.

live_roots

Every live view's tree; a view whose directory is gone is retired.

collect

Read the declared paths under at back into a tree, landing every object.

prefetch

Fetch digest and everything it reaches; the objects fetched, in order.

shed

Evict every local copy source holds, except what a live view depends on.

source_holds

Whether source has the object, by existence and never by trust.

Attributes:

Name Type Description
PATH_BUDGET

The most UTF-8 bytes a path inside a view may take, separators included.

EntryRung

How one view entry was made: a ladder rung, a real symlink, or not at all.

ENTRY_RUNGS tuple[EntryRung, ...]

PATH_BUDGET module-attribute

PATH_BUDGET = 1024

The most UTF-8 bytes a path inside a view may take, separators included.

EntryRung module-attribute

EntryRung = Literal['clone', 'hardlink', 'link', 'copy', 'symlink', 'parked']

How one view entry was made: a ladder rung, a real symlink, or not at all.

ENTRY_RUNGS module-attribute

ENTRY_RUNGS: tuple[EntryRung, ...] = ('clone', 'hardlink', 'link', 'copy', 'symlink', 'parked')

ViewEntry dataclass

ViewEntry(path: str, rung: EntryRung, digest: Digest | None = None, note: str = '', executable: bool = False)

One path a view created, and how.

Attributes:

Name Type Description
path str

relative to the view's root, forward-slashed.

rung EntryRung

how it was made; parked means it was refused and the note says why.

digest Digest | None

the blob it presents, or None for a symlink entry.

note str

why a symlink became a copy or was parked; empty otherwise.

executable bool

whether the tree marked the blob executable; what collect answers for this path where the file mode cannot.

Methods:

Name Description
to_json

The entry as a JSON object.

from_json

Decode an entry, refusing a rung outside the ladder.

path instance-attribute

path: str

rung instance-attribute

rung: EntryRung

digest class-attribute instance-attribute

digest: Digest | None = None

note class-attribute instance-attribute

note: str = ''

executable class-attribute instance-attribute

executable: bool = False

to_json

to_json() -> dict[str, Value]

The entry as a JSON object.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def to_json(self) -> dict[str, Value]:
    """The entry as a JSON object."""
    return {
        "path": self.path,
        "rung": self.rung,
        "digest": None if self.digest is None else str(self.digest),
        "note": self.note,
        "executable": self.executable,
    }

from_json classmethod

from_json(value: Value) -> ViewEntry

Decode an entry, refusing a rung outside the ladder.

Source code in packages/strongroom/src/livery/strongroom/_views.py
@classmethod
def from_json(cls, value: Value) -> ViewEntry:
    """Decode an entry, refusing a rung outside the ladder."""
    fields = expect_object(
        value, ("path", "rung", "digest", "note", "executable"), where="entry"
    )
    digest = fields["digest"]
    return cls(
        expect_str(fields["path"], where="entry.path"),
        _entry_rung(expect_str(fields["rung"], where="entry.rung")),
        None
        if digest is None
        else Digest.parse(expect_str(digest, where="entry.digest")),
        expect_str(fields["note"], where="entry.note"),
        expect_bool(fields["executable"], where="entry.executable"),
    )

ViewRecord dataclass

ViewRecord(id: str, tree: Digest, at: str, writable: bool, created: str, entries: tuple[ViewEntry, ...] = tuple(), directories: tuple[str, ...] = tuple())

A view: local state in the index, and a root while its directory exists.

Attributes:

Name Type Description
id str

the record's name under index/views/.

tree Digest

the tree the view presents.

at str

the view's root directory.

writable bool

whether the caller said it would write.

created str

when, an instant.

entries tuple[ViewEntry, ...]

every path created, in creation order.

directories tuple[str, ...]

every directory created, shallowest first.

Methods:

Name Description
encode

The record's canonical JSON bytes.

decode

Decode a record.

depends_on

The objects this view reads through the store's own files.

id instance-attribute

id: str

tree instance-attribute

tree: Digest

at instance-attribute

at: str

writable instance-attribute

writable: bool

created instance-attribute

created: str

entries class-attribute instance-attribute

entries: tuple[ViewEntry, ...] = field(default_factory=tuple)

directories class-attribute instance-attribute

directories: tuple[str, ...] = field(default_factory=tuple)

root property

root: Path

The view's root as a path.

encode

encode() -> bytes

The record's canonical JSON bytes.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def encode(self) -> bytes:
    """The record's canonical JSON bytes."""
    return canonical(
        {
            "id": self.id,
            "tree": str(self.tree),
            "at": self.at,
            "writable": self.writable,
            "created": self.created,
            "entries": [entry.to_json() for entry in self.entries],
            "directories": list(self.directories),
        }
    )

decode classmethod

decode(data: bytes) -> ViewRecord

Decode a record.

Raises:

Type Description
FormatError

when the bytes are not a record.

Source code in packages/strongroom/src/livery/strongroom/_views.py
@classmethod
def decode(cls, data: bytes) -> ViewRecord:
    """Decode a record.

    Raises:
        FormatError: when the bytes are not a record.
    """
    try:
        value: Value = json.loads(data)
    except ValueError as error:
        raise FormatError(f"view record bytes are not JSON: {error}") from None
    fields = expect_object(
        value,
        ("id", "tree", "at", "writable", "created", "entries", "directories"),
        where="view",
    )
    return cls(
        expect_str(fields["id"], where="view.id"),
        Digest.parse(expect_str(fields["tree"], where="view.tree")),
        expect_str(fields["at"], where="view.at"),
        expect_bool(fields["writable"], where="view.writable"),
        expect_str(fields["created"], where="view.created"),
        tuple(
            ViewEntry.from_json(item)
            for item in expect_list(fields["entries"], where="view.entries")
        ),
        tuple(
            expect_str(item, where="view.directories[]")
            for item in expect_list(fields["directories"], where="view.directories")
        ),
    )

depends_on

depends_on() -> set[Digest]

The objects this view reads through the store's own files.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def depends_on(self) -> set[Digest]:
    """The objects this view reads through the store's own files."""
    return {
        entry.digest
        for entry in self.entries
        if entry.rung in ("hardlink", "link") and entry.digest is not None
    }

DropReport dataclass

DropReport(removed: tuple[str, ...] = tuple(), left: tuple[str, ...] = tuple())

What dropping a view removed and what it left.

Attributes:

Name Type Description
removed tuple[str, ...]

every path removed, files first, then directories deepest first, then the root as ..

left tuple[str, ...]

every path found that the record did not list, with the reason; never touched.

removed class-attribute instance-attribute

removed: tuple[str, ...] = field(default_factory=tuple)

left class-attribute instance-attribute

left: tuple[str, ...] = field(default_factory=tuple)

ShedReport dataclass

ShedReport(source: str, shed: tuple[Digest, ...] = tuple(), kept: tuple[Digest, ...] = tuple())

What a shed did.

Attributes:

Name Type Description
source str

the source relied on, described.

shed tuple[Digest, ...]

every local copy evicted because the source holds it.

kept tuple[Digest, ...]

every object kept because a live view depends on it.

source instance-attribute

source: str

shed class-attribute instance-attribute

shed: tuple[Digest, ...] = field(default_factory=tuple)

kept class-attribute instance-attribute

kept: tuple[Digest, ...] = field(default_factory=tuple)

new_view_id

new_view_id() -> str

A fresh view id; the seam tests fix for a known name.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def new_view_id() -> str:
    """A fresh view id; the seam tests fix for a known name."""
    return secrets.token_hex(8)

view

view(store: Store, tree_digest: Digest, at: Path, *, writable: bool, allow_hardlink: bool) -> ViewRecord

Fill at from the tree, cheapest safe rung per entry, and record it.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def view(
    store: Store, tree_digest: Digest, at: Path, *, writable: bool, allow_hardlink: bool
) -> ViewRecord:
    """Fill *at* from the tree, cheapest safe rung per entry, and record it."""
    if at.exists() and (not at.is_dir() or any(at.iterdir())):
        raise FileExistsError(f"{at} exists and is not an empty directory")
    tree = Tree.decode(store.fetch(tree_digest).read_bytes())
    plan = _Plan()
    _plan(store, tree, "", plan)
    at.mkdir(parents=True, exist_ok=True)
    for directory in plan.directories:
        (at / directory).mkdir()
    ladder = _Ladder(writable=writable, allow_hardlink=allow_hardlink)
    entries: list[ViewEntry] = []
    for path, entry in plan.blobs:
        try:
            source = store.fetch(entry.digest)
        except (ErasedObject, MissingObject) as error:
            raise type(error)(f"view entry {path!r}: {error}") from None
        destination = at / path
        rung = ladder.make(source, destination, executable=entry.executable)
        _set_mode(destination, rung, executable=entry.executable, writable=writable)
        entries.append(ViewEntry(path, rung, entry.digest, executable=entry.executable))
    for path, link in plan.links:
        entries.append(_make_link(at, path, link))
    record = ViewRecord(
        new_view_id(),
        tree_digest,
        str(at),
        writable,
        store.clock(),
        tuple(entries),
        tuple(plan.directories),
    )
    _write_record(store, record)
    return record

views

views(store: Store) -> list[ViewRecord]

Every recorded view, by id.

A record another process retires between the listing and the read is left out: that view is forgotten already.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def views(store: Store) -> list[ViewRecord]:
    """Every recorded view, by id.

    A record another process retires between the listing and the read
    is left out: that view is forgotten already.
    """
    found: list[ViewRecord] = []
    for path in sorted(_records_dir(store).glob("*.json")):
        record = _read_record(path)
        if record is not None:
            found.append(record)
    return found

view_record

view_record(store: Store, view_id: str) -> ViewRecord | None

The record of view_id, or None.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def view_record(store: Store, view_id: str) -> ViewRecord | None:
    """The record of *view_id*, or None."""
    return _read_record(_records_dir(store) / f"{view_id}.json")

retire_record

retire_record(store: Store, view_id: str) -> None

Forget a view: its record goes, its paths are not touched.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def retire_record(store: Store, view_id: str) -> None:
    """Forget a view: its record goes, its paths are not touched."""
    (_records_dir(store) / f"{view_id}.json").unlink(missing_ok=True)

drop_view

drop_view(store: Store, view_id: str) -> DropReport

Remove what the record lists and nothing else; then forget the view.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def drop_view(store: Store, view_id: str) -> DropReport:
    """Remove what the record lists and nothing else; then forget the view."""
    record = view_record(store, view_id)
    if record is None:
        raise FileNotFoundError(f"view {view_id} is not recorded at {store.root}")
    root = record.root
    if not root.exists():
        retire_record(store, view_id)
        return DropReport((), (f"{root}: the view's directory is already gone",))
    removed: list[str] = []
    left: list[str] = []
    for entry in record.entries:
        path = root / entry.path
        if entry.rung == "parked" or not _lexists(path):
            continue
        if path.is_dir() and not path.is_symlink():
            rungs.remove_tree(path)
        else:
            rungs.remove(path)
        removed.append(entry.path)
    for directory in [*reversed(record.directories), "."]:
        path = root / directory
        if not path.is_dir():
            # Damage the view has met since it was made: a directory
            # something else removed is gone already, which is what
            # dropping it wanted.
            removed.append(directory)
            continue
        strays = sorted(child.name for child in path.iterdir())
        if strays:
            left.append(f"{directory}: not created by the view: {', '.join(strays)}")
            continue
        path.rmdir()
        removed.append(directory)
    retire_record(store, view_id)
    return DropReport(tuple(removed), tuple(left))

live_roots

live_roots(store: Store) -> list[Digest]

Every live view's tree; a view whose directory is gone is retired.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def live_roots(store: Store) -> list[Digest]:
    """Every live view's tree; a view whose directory is gone is retired."""
    roots: list[Digest] = []
    for record in views(store):
        if record.root.exists():
            roots.append(record.tree)
        else:
            retire_record(store, record.id)
    return roots

collect

collect(store: Store, at: Path, declared: Iterable[str], executable: Iterable[str] = ()) -> Tree

Read the declared paths under at back into a tree, landing every object.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def collect(
    store: Store, at: Path, declared: Iterable[str], executable: Iterable[str] = ()
) -> Tree:
    """Read the declared paths under *at* back into a tree, landing every object."""
    wanted = frozenset(executable)
    for relative in sorted(wanted):
        if not _lexists(at.joinpath(*relative.split("/"))):
            raise FileNotFoundError(
                f"declared executable {relative!r} is absent under {at}"
            )
    collecting = _Collecting(at, wanted, _made_under(store, at))
    root = _Node()
    for relative in declared:
        parts = relative.split("/")
        path = at.joinpath(*parts)
        if not _lexists(path):
            raise FileNotFoundError(
                f"declared output {relative!r} is absent under {at}"
            )
        node = root
        for part in parts[:-1]:
            node = node.dirs.setdefault(part, _Node())
        collected = _collect_path(store, collecting, path, relative)
        if isinstance(collected, _Node):
            node.dirs[parts[-1]] = collected
        else:
            node.files[parts[-1]] = collected
    return _land_tree(store, root)

prefetch

prefetch(store: Store, digest: Digest) -> list[Digest]

Fetch digest and everything it reaches; the objects fetched, in order.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def prefetch(store: Store, digest: Digest) -> list[Digest]:
    """Fetch *digest* and everything it reaches; the objects fetched, in order."""
    fetched: list[Digest] = []
    seen: set[Digest] = set()
    stack = [digest]
    while stack:
        current = stack.pop()
        if current in seen:
            continue
        seen.add(current)
        store.fetch(current)
        fetched.append(current)
        stack.extend(children(store.read(current)))
    return fetched

shed

shed(store: Store, source: Source) -> ShedReport

Evict every local copy source holds, except what a live view depends on.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def shed(store: Store, source: Source) -> ShedReport:
    """Evict every local copy *source* holds, except what a live view depends on."""
    needed: set[Digest] = set()
    for record in views(store):
        if record.root.exists():
            needed |= record.depends_on()
    shed_out: list[Digest] = []
    kept: list[Digest] = []
    for digest in list(store.objects()):
        if digest in needed:
            kept.append(digest)
        elif source_holds(store, source, digest):
            store.evict(digest)
            shed_out.append(digest)
    return ShedReport(source.describe(), tuple(shed_out), tuple(kept))

source_holds

source_holds(store: Store, source: Source, digest: Digest) -> bool

Whether source has the object, by existence and never by trust.

Source code in packages/strongroom/src/livery/strongroom/_views.py
def source_holds(store: Store, source: Source, digest: Digest) -> bool:
    """Whether *source* has the object, by existence and never by trust."""
    algorithm, fanout, rest = digest.path_parts
    if isinstance(source, FolderSource):
        return (source.path / "objects" / algorithm / fanout / rest).exists()
    if isinstance(source, HttpSource):
        return store.head(source, f"objects/{algorithm}/{fanout}/{rest}")
    return source.digest == digest