Skip to content

livery.strongroom._store

livery.strongroom._store

The local store: objects landed by digest, refs moved by compare-and-swap.

A store is a directory with a root manifest, an objects/ tree, a refs/ tree and a local index. Objects are immutable and idempotent to land; refs are the only mutable thing, and every move writes a record beside the ref under a per-ref lock. The layout is the one in spec/layout.md; the index under index/ is this implementation's own and not a format.

Reach for livery.strongroom.Store: create or open, then land and path for objects, ref and set_ref for names.

Classes:

Name Description
Manifest

The root manifest: which layout and which algorithm this store speaks.

Namespace

A ref namespace and its mutation class.

Landed

What a landing produced.

ScrubReport

What a scrub found.

Store

A local store: objects/ by digest, refs/ by name.

Functions:

Name Description
now

The current instant, RFC 3339 in UTC to the second.

silent

The default progress sink: report nothing.

Attributes:

Name Type Description
MANIFEST_NAME

The root manifest's file name.

LAYOUT_VERSION

The layout this implementation speaks.

MutationClass

How a namespace's refs may move; see spec/namespaces.md.

ObjectState

The three states an object name can be in.

MUTATION_CLASSES tuple[MutationClass, ...]
Clock

Returns the current instant as an RFC 3339 UTC string.

OWNED tuple[Namespace, ...]

The two namespaces the store declares itself; see spec/namespaces.md.

MANIFEST_NAME module-attribute

MANIFEST_NAME = 'strongroom.json'

The root manifest's file name.

LAYOUT_VERSION module-attribute

LAYOUT_VERSION = 1

The layout this implementation speaks.

MutationClass module-attribute

MutationClass = Literal['write-once', 'monotone', 'volatile']

How a namespace's refs may move; see spec/namespaces.md.

ObjectState module-attribute

ObjectState = Literal['present', 'absent', 'erased']

The three states an object name can be in.

MUTATION_CLASSES module-attribute

MUTATION_CLASSES: tuple[MutationClass, ...] = ('write-once', 'monotone', 'volatile')

Clock module-attribute

Clock = Callable[[], str]

Returns the current instant as an RFC 3339 UTC string.

OWNED module-attribute

OWNED: tuple[Namespace, ...] = (Namespace('pins', 'volatile'), Namespace('pending', 'volatile'))

The two namespaces the store declares itself; see spec/namespaces.md.

Manifest dataclass

Manifest(layout: int, algorithm: str)

The root manifest: which layout and which algorithm this store speaks.

Attributes:

Name Type Description
layout int

the layout version.

algorithm str

the one algorithm of this address space.

Methods:

Name Description
encode

The manifest's canonical JSON bytes.

decode

Decode a manifest, refusing bytes that are not one.

layout instance-attribute

layout: int

algorithm instance-attribute

algorithm: str

encode

encode() -> bytes

The manifest's canonical JSON bytes.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def encode(self) -> bytes:
    """The manifest's canonical JSON bytes."""
    return canonical({"layout": self.layout, "algorithm": self.algorithm})

decode classmethod

decode(data: bytes) -> Manifest

Decode a manifest, refusing bytes that are not one.

Parameters:

Name Type Description Default
data bytes

the manifest file's bytes.

required

Returns:

Type Description
Manifest

The manifest.

Raises:

Type Description
ManifestError

when the bytes are not a manifest.

Source code in packages/strongroom/src/livery/strongroom/_store.py
@classmethod
def decode(cls, data: bytes) -> Manifest:
    """Decode a manifest, refusing bytes that are not one.

    Args:
        data: the manifest file's bytes.

    Returns:
        The manifest.

    Raises:
        ManifestError: when the bytes are not a manifest.
    """
    try:
        value: Value = json.loads(data)
        fields = expect_object(value, ("layout", "algorithm"), where="manifest")
        return cls(
            expect_int(fields["layout"], where="manifest.layout"),
            expect_str(fields["algorithm"], where="manifest.algorithm"),
        )
    except ValueError as error:
        raise ManifestError(f"{MANIFEST_NAME} is not a manifest: {error}") from None

Namespace dataclass

Namespace(name: str, mutation: MutationClass)

A ref namespace and its mutation class.

Attributes:

Name Type Description
name str

the namespace, lowercase letters, digits and dashes.

mutation MutationClass

how its refs may move.

Methods:

Name Description
__post_init__

name instance-attribute

name: str

mutation instance-attribute

mutation: MutationClass

__post_init__

__post_init__() -> None
Source code in packages/strongroom/src/livery/strongroom/_store.py
def __post_init__(self) -> None:
    if not self.name or self.name.strip("abcdefghijklmnopqrstuvwxyz0123456789-"):
        raise ValueError(
            f"namespace {self.name!r} is not lowercase letters, digits and dashes"
        )
    if self.mutation not in MUTATION_CLASSES:
        known = ", ".join(MUTATION_CLASSES)
        raise ValueError(
            f"namespace {self.name!r} mutation {self.mutation!r} is not"
            f" one of {known}"
        )

Landed dataclass

Landed(digest: Digest, size: int, written: bool)

What a landing produced.

Attributes:

Name Type Description
digest Digest

the object's name.

size int

its byte count.

written bool

whether the bytes were written now; false when the object already existed, which is success without a write.

digest instance-attribute

digest: Digest

size instance-attribute

size: int

written instance-attribute

written: bool

ScrubReport dataclass

ScrubReport(verified: tuple[Digest, ...] = tuple(), corrupt: tuple[Digest, ...] = tuple())

What a scrub found.

Attributes:

Name Type Description
verified tuple[Digest, ...]

every object whose bytes matched its name.

corrupt tuple[Digest, ...]

every object that did not; each is removed with its mark, so the next fill lands it again.

verified class-attribute instance-attribute

verified: tuple[Digest, ...] = field(default_factory=tuple)

corrupt class-attribute instance-attribute

corrupt: tuple[Digest, ...] = field(default_factory=tuple)

Store

Store(root: Path, algorithm: Algorithm, namespaces: Iterable[Namespace], *, clock: Clock, lock_stale: float, lock_timeout: float, sources: Iterable[Source], offline: bool, progress: Progress)

A local store: objects/ by digest, refs/ by name.

Open one with livery.strongroom.Store.create or livery.strongroom.Store.open. Every namespace a caller will write is declared at open with its mutation class; pins/ and pending/ are declared by the store.

Attributes:

Name Type Description
root Path

the store's directory.

algorithm Algorithm

the address space's algorithm.

namespaces dict[str, Namespace]

every declared namespace by name.

sources tuple[Source, ...]

the sources consulted in order for an object the local directory lacks.

offline bool

whether origin hints are never consulted.

Methods:

Name Description
create

Create a store at root and open it.

open

Open the store at root.

object_path

The path an object of digest lives at, present or not.

state

Whether digest is present, absent, or erased here.

land

Land bytes as an object, verifying them on the way in.

put

Land data and return its name.

path

A read-only path to the object, verified on first access.

read

The object's bytes, through livery.strongroom.Store.path.

verify

Hash the object in full and mark it verified.

verified_size

The size this store verified the object at, or None if never.

objects

Every object present here, in no particular order.

scrub

Hash every object and remove the ones that do not match.

fetch

A path to the object, consulting the sources when it is not here.

fill

Land digests into the folder tier at target.

tombstone

The tombstone under the object's path, or None when there is none.

write_tombstone

Write stone under its object's path, in this tier.

evict

Remove the object's bytes and its marks; the name is untouched.

clock

The current instant, from the clock the store was opened with.

publish_begin

Begin a publish: root target under pending/<id> first.

publish_commit

Commit a publish: move the real ref, then drop the pending one.

retire

Drop a pending ref deliberately, for a publish that will not commit.

pendings

Every pending publish's id, sorted; groups included.

begin

Open a group of ref moves: an empty journal under pending/<id>.

add

Append a move to the group; its manifest roots the target from now on.

commit

Apply the group's moves, every one or none, and drop the pending ref.

group

The group under pending/<group_id>.

groups

Every group that has begun and not committed, sorted by id.

transaction

A group under a with: record moves, and the exit commits or retires.

drop_ref

Remove a volatile ref and its record by compare-and-swap.

pin

Root digest under pins/<name>, replacing an earlier pin of that name.

unpin

Drop pins/<name> under the maintenance lease, never beside a sweep.

erase

Erase the object's bytes and keep the fact as a tombstone.

reachable

Every object reachable from digest, itself included.

roots

Every root: what every ref on disk names, and every live view's tree.

sweep

Remove every object no root reaches, and old scratch.

view

Fill at from the tree by the cheapest safe rung per entry.

views

Every recorded view, live or not, by id.

view_record

The record of one view, or None.

drop_view

Remove what the view's record lists, and nothing else.

collect

Read the declared paths under at back into a tree, landing every object.

prefetch

Fetch digest and everything it reaches, for offline use.

shed

Evict every local copy source holds, except what a live view needs.

head

Whether the HTTP source answers a HEAD for relative.

ref_path

The file a ref lives at, present or not.

ref

What the ref names, or None when it does not exist.

record

The record beside the ref, or None when there is none.

refs

Every ref path in the namespace, sorted.

set_ref

Move a ref by compare-and-swap, writing its record.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def __init__(
    self,
    root: Path,
    algorithm: Algorithm,
    namespaces: Iterable[Namespace],
    *,
    clock: Clock,
    lock_stale: float,
    lock_timeout: float,
    sources: Iterable[Source],
    offline: bool,
    progress: Progress,
) -> None:
    self.root = root
    self.algorithm = algorithm
    self.sources = tuple(sources)
    self.offline = offline
    self._progress = progress
    for source in self.sources:
        if isinstance(source, FolderSource):
            self._check_folder(source)
    self.namespaces = {owned.name: owned for owned in OWNED}
    for namespace in namespaces:
        declared = self.namespaces.get(namespace.name)
        if declared is not None and declared != namespace:
            raise ValueError(
                f"namespace {namespace.name!r} is already declared"
                f" {declared.mutation}, not {namespace.mutation}"
            )
        self.namespaces[namespace.name] = namespace
    self._clock = clock
    self._lock_stale = lock_stale
    self._lock_timeout = lock_timeout
    self._checked_http: set[str] = set()

root instance-attribute

root: Path = root

algorithm instance-attribute

algorithm: Algorithm = algorithm

sources instance-attribute

sources: tuple[Source, ...] = tuple(sources)

offline instance-attribute

offline: bool = offline

namespaces instance-attribute

namespaces: dict[str, Namespace] = {owned.name: owned for owned in OWNED}

create classmethod

create(root: Path, *, algorithm: str = 'sha256', namespaces: Iterable[Namespace] = (), clock: Clock = now, lock_stale: float = 600.0, lock_timeout: float = 30.0, sources: Iterable[Source] = (), offline: bool = False, progress: Progress = silent) -> Store

Create a store at root and open it.

Parameters:

Name Type Description Default
root Path

the directory; created when missing.

required
algorithm str

a registry entry's name.

'sha256'
namespaces Iterable[Namespace]

the caller's namespaces, each with its class.

()
clock Clock

the instant source for records.

now
lock_stale float

seconds after which a lock is broken even when its holder looks alive: the pid of a holder that died can belong to another process by then.

600.0
lock_timeout float

seconds to wait for a live lock.

30.0
sources Iterable[Source]

where to look for an object the store lacks, in order. A folder source is checked now; an HTTP source on first use.

()
offline bool

never consult an origin hint.

False
progress Progress

where a fetch reports a skipped or refused source; silent by default.

silent

Returns:

Type Description
Store

The open store.

Raises:

Type Description
ManifestError

when root already holds a manifest.

FormatError

when the algorithm is not registered.

Source code in packages/strongroom/src/livery/strongroom/_store.py
@classmethod
def create(
    cls,
    root: Path,
    *,
    algorithm: str = "sha256",
    namespaces: Iterable[Namespace] = (),
    clock: Clock = now,
    lock_stale: float = 600.0,
    lock_timeout: float = 30.0,
    sources: Iterable[Source] = (),
    offline: bool = False,
    progress: Progress = silent,
) -> Store:
    """Create a store at *root* and open it.

    Args:
        root: the directory; created when missing.
        algorithm: a registry entry's name.
        namespaces: the caller's namespaces, each with its class.
        clock: the instant source for records.
        lock_stale: seconds after which a lock is broken even when
            its holder looks alive: the pid of a holder that died
            can belong to another process by then.
        lock_timeout: seconds to wait for a live lock.
        sources: where to look for an object the store lacks, in
            order. A folder source is checked now; an HTTP source
            on first use.
        offline: never consult an origin hint.
        progress: where a fetch reports a skipped or refused
            source; silent by default.

    Returns:
        The open store.

    Raises:
        ManifestError: when *root* already holds a manifest.
        FormatError: when the algorithm is not registered.
    """
    entry = registered(algorithm)
    root.mkdir(parents=True, exist_ok=True)
    manifest = root / MANIFEST_NAME
    if manifest.exists():
        raise ManifestError(f"{root} is already a store: {MANIFEST_NAME} exists")
    _write_atomically(manifest, Manifest(LAYOUT_VERSION, entry.name).encode())
    for child in ("objects", "refs", "index"):
        (root / child).mkdir(exist_ok=True)
    return cls.open(
        root,
        namespaces=namespaces,
        clock=clock,
        lock_stale=lock_stale,
        lock_timeout=lock_timeout,
        sources=sources,
        offline=offline,
        progress=progress,
    )

open classmethod

open(root: Path, *, namespaces: Iterable[Namespace] = (), clock: Clock = now, lock_stale: float = 600.0, lock_timeout: float = 30.0, sources: Iterable[Source] = (), offline: bool = False, progress: Progress = silent) -> Store

Open the store at root.

Parameters:

Name Type Description Default
root Path

the store's directory.

required
namespaces Iterable[Namespace]

the caller's namespaces, each with its class.

()
clock Clock

the instant source for records.

now
lock_stale float

seconds after which a lock is broken even when its holder looks alive: the pid of a holder that died can belong to another process by then.

600.0
lock_timeout float

seconds to wait for a live lock.

30.0
sources Iterable[Source]

where to look for an object the store lacks, in order. A folder source is checked now; an HTTP source on first use.

()
offline bool

never consult an origin hint.

False
progress Progress

where a fetch reports a skipped or refused source; silent by default.

silent

Returns:

Type Description
Store

The store.

Raises:

Type Description
ManifestError

when the manifest is missing or malformed, or names a layout or an algorithm this implementation does not speak; or when a folder source's manifest is missing or of another store.

ValueError

when a namespace is declared twice with different classes.

Source code in packages/strongroom/src/livery/strongroom/_store.py
@classmethod
def open(
    cls,
    root: Path,
    *,
    namespaces: Iterable[Namespace] = (),
    clock: Clock = now,
    lock_stale: float = 600.0,
    lock_timeout: float = 30.0,
    sources: Iterable[Source] = (),
    offline: bool = False,
    progress: Progress = silent,
) -> Store:
    """Open the store at *root*.

    Args:
        root: the store's directory.
        namespaces: the caller's namespaces, each with its class.
        clock: the instant source for records.
        lock_stale: seconds after which a lock is broken even when
            its holder looks alive: the pid of a holder that died
            can belong to another process by then.
        lock_timeout: seconds to wait for a live lock.
        sources: where to look for an object the store lacks, in
            order. A folder source is checked now; an HTTP source
            on first use.
        offline: never consult an origin hint.
        progress: where a fetch reports a skipped or refused
            source; silent by default.

    Returns:
        The store.

    Raises:
        ManifestError: when the manifest is missing or malformed,
            or names a layout or an algorithm this implementation
            does not speak; or when a folder source's manifest is
            missing or of another store.
        ValueError: when a namespace is declared twice with
            different classes.
    """
    path = root / MANIFEST_NAME
    try:
        manifest = Manifest.decode(path.read_bytes())
    except FileNotFoundError:
        raise ManifestError(f"{root} is not a store: no {MANIFEST_NAME}") from None
    if manifest.layout != LAYOUT_VERSION:
        raise ManifestError(
            f"{root} is layout {manifest.layout}; this implementation"
            f" speaks layout {LAYOUT_VERSION}"
        )
    try:
        entry = registered(manifest.algorithm)
    except FormatError as error:
        raise ManifestError(f"{root}: {error}") from None
    return cls(
        root,
        entry,
        namespaces,
        clock=clock,
        lock_stale=lock_stale,
        lock_timeout=lock_timeout,
        sources=sources,
        offline=offline,
        progress=progress,
    )

object_path

object_path(digest: Digest) -> Path

The path an object of digest lives at, present or not.

Raises:

Type Description
IntegrityError

when the digest is of another algorithm than this store's; one address space, one algorithm.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def object_path(self, digest: Digest) -> Path:
    """The path an object of *digest* lives at, present or not.

    Raises:
        IntegrityError: when the digest is of another algorithm
            than this store's; one address space, one algorithm.
    """
    if digest.algorithm != self.algorithm.name:
        raise IntegrityError(
            f"{digest} is a {digest.algorithm} name; this store's"
            f" address space is {self.algorithm.name}"
        )
    algorithm, fanout, rest = digest.path_parts
    return self.root / "objects" / algorithm / fanout / rest

state

state(digest: Digest) -> ObjectState

Whether digest is present, absent, or erased here.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def state(self, digest: Digest) -> ObjectState:
    """Whether *digest* is present, absent, or erased here."""
    path = self.object_path(digest)
    if path.with_name(path.name + _TOMBSTONE).exists():
        return "erased"
    if path.exists():
        return "present"
    return "absent"

land

land(source: IO[bytes] | bytes, *, expected: Digest | None = None) -> Landed

Land bytes as an object, verifying them on the way in.

The bytes stream through the digest into a .part scratch file and are moved into place atomically. Landing is idempotent: an object that already exists is success without a write. Two writers of the same digest land identical bytes, so no lock is taken.

Parameters:

Name Type Description Default
source IO[bytes] | bytes

the bytes, or a binary stream read to its end.

required
expected Digest | None

the digest the bytes must have, when known. A mismatch removes the scratch and raises.

None

Returns:

Type Description
Landed

The landing.

Raises:

Type Description
IntegrityError

when expected is given and the bytes do not match it.

ErasedObject

when the object was erased here; its tombstone stands until a sweep decides otherwise.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def land(
    self, source: IO[bytes] | bytes, *, expected: Digest | None = None
) -> Landed:
    """Land bytes as an object, verifying them on the way in.

    The bytes stream through the digest into a `.part` scratch file
    and are moved into place atomically. Landing is idempotent: an
    object that already exists is success without a write. Two
    writers of the same digest land identical bytes, so no lock is
    taken.

    Args:
        source: the bytes, or a binary stream read to its end.
        expected: the digest the bytes must have, when known. A
            mismatch removes the scratch and raises.

    Returns:
        The landing.

    Raises:
        IntegrityError: when *expected* is given and the bytes do
            not match it.
        ErasedObject: when the object was erased here; its
            tombstone stands until a sweep decides otherwise.
    """
    stream = BytesIO(source) if isinstance(source, bytes) else source
    if expected is not None:
        destination = self.object_path(expected)
        scratch = _scratch_beside(destination)
    else:
        scratch = _scratch_beside(
            self.root / "objects" / self.algorithm.name / "incoming"
        )
    scratch.parent.mkdir(parents=True, exist_ok=True)
    hasher = self.algorithm.constructor(b"")
    size = 0
    with scratch.open("wb") as handle:
        while chunk := stream.read(_CHUNK):
            hasher.update(chunk)
            size += len(chunk)
            handle.write(chunk)
    digest = Digest(self.algorithm.name, hasher.hexdigest())
    if expected is not None and digest != expected:
        scratch.unlink()
        raise IntegrityError(
            f"landing expected {expected} and the bytes are {digest};"
            " nothing was kept"
        )
    destination = self.object_path(digest)
    state = self.state(digest)
    if state == "erased":
        scratch.unlink()
        raise self._erased(digest)
    if state == "present":
        scratch.unlink()
        return Landed(digest, size, written=False)
    destination.parent.mkdir(parents=True, exist_ok=True)
    try:
        _replace(scratch, destination)
    except PermissionError:
        # Another process holds the destination (Windows): a reader,
        # or the twin landing the same bytes, still placing them.
        # The bytes are the same by name, so the landing succeeded
        # once the destination verifies; the wait is for the twin
        # to finish, and a destination that never verifies keeps
        # the refusal.
        scratch.unlink()
        if not _settled(
            lambda: destination.exists() and self._hash_file(destination) == digest
        ):
            raise
        self._write_mark(digest, size)
        return Landed(digest, size, written=False)
    self._write_mark(digest, size)
    return Landed(digest, size, written=True)

put

put(data: bytes) -> Digest

Land data and return its name.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def put(self, data: bytes) -> Digest:
    """Land *data* and return its name."""
    return self.land(data).digest

path

path(digest: Digest) -> Path

A read-only path to the object, verified on first access.

Size is checked against the verified mark on every access; an object without a mark (landed by copy, not by this store) is hashed in full and marked.

Raises:

Type Description
MissingObject

when the object is absent.

ErasedObject

when the object was erased.

IntegrityError

when the bytes do not match the name; the object and its mark are removed so a fill lands it again.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def path(self, digest: Digest) -> Path:
    """A read-only path to the object, verified on first access.

    Size is checked against the verified mark on every access; an
    object without a mark (landed by copy, not by this store) is
    hashed in full and marked.

    Raises:
        MissingObject: when the object is absent.
        ErasedObject: when the object was erased.
        IntegrityError: when the bytes do not match the name; the
            object and its mark are removed so a fill lands it
            again.
    """
    state = self.state(digest)
    path = self.object_path(digest)
    if state == "erased":
        raise self._erased(digest)
    if state == "absent":
        raise MissingObject(f"{digest} is not in the store at {self.root}")
    size = self.verified_size(digest)
    if size is None:
        self.verify(digest)
    else:
        actual = path.stat().st_size
        if actual != size:
            self.evict(digest)
            raise IntegrityError(
                f"{digest} is {actual} bytes on disk and was verified at"
                f" {size}; the object is removed"
            )
    return path

read

read(digest: Digest) -> bytes

The object's bytes, through livery.strongroom.Store.path.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def read(self, digest: Digest) -> bytes:
    """The object's bytes, through [livery.strongroom.Store.path][]."""
    return self.path(digest).read_bytes()

verify

verify(digest: Digest) -> int

Hash the object in full and mark it verified.

Returns:

Type Description
int

The object's size.

Raises:

Type Description
MissingObject

when the object is absent.

IntegrityError

when the bytes do not match; the object and its mark are removed.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def verify(self, digest: Digest) -> int:
    """Hash the object in full and mark it verified.

    Returns:
        The object's size.

    Raises:
        MissingObject: when the object is absent.
        IntegrityError: when the bytes do not match; the object and
            its mark are removed.
    """
    path = self.object_path(digest)
    if not path.exists():
        raise MissingObject(f"{digest} is not in the store at {self.root}")
    actual = self._hash_file(path)
    if actual != digest:
        self.evict(digest)
        raise IntegrityError(
            f"{digest} holds bytes named {actual}; the object is removed"
        )
    size = path.stat().st_size
    self._write_mark(digest, size)
    return size

verified_size

verified_size(digest: Digest) -> int | None

The size this store verified the object at, or None if never.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def verified_size(self, digest: Digest) -> int | None:
    """The size this store verified the object at, or None if never."""
    try:
        return int(self._mark(digest).read_text("ascii"))
    except FileNotFoundError:
        return None

objects

objects() -> Iterator[Digest]

Every object present here, in no particular order.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def objects(self) -> Iterator[Digest]:
    """Every object present here, in no particular order."""
    objects = self.root / "objects" / self.algorithm.name
    for fanout in sorted(objects.glob("??")):
        for path in sorted(fanout.iterdir()):
            if path.name.endswith(_RESERVED_SUFFIXES):
                continue
            yield Digest(self.algorithm.name, fanout.name + path.name)

scrub

scrub() -> ScrubReport

Hash every object and remove the ones that do not match.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def scrub(self) -> ScrubReport:
    """Hash every object and remove the ones that do not match."""
    verified: list[Digest] = []
    corrupt: list[Digest] = []
    for digest in list(self.objects()):
        try:
            self.verify(digest)
        except IntegrityError:
            corrupt.append(digest)
        else:
            verified.append(digest)
    return ScrubReport(tuple(verified), tuple(corrupt))

fetch

fetch(digest: Digest) -> Path

A path to the object, consulting the sources when it is not here.

Sources are consulted in order. A hit is verified on arrival: a copy-policy folder, an HTTP tier and an origin hint land the bytes here; a reference-policy folder answers with a path into the folder and lands nothing. An unreachable source is skipped and reported; a source serving wrong bytes is refused and reported, and the next one consulted. An origin hint is consulted only for its own digest and never when offline.

Returns:

Type Description
Path

A read-only path: this store's object, or the folder's

Path

under a reference policy.

Raises:

Type Description
MissingObject

when no source has it. The message names the digest, and when an origin hint was passed over because the store is offline, the URL that would have satisfied it.

ErasedObject

when the object was erased here.

ManifestError

when an HTTP source's manifest is of another store.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def fetch(self, digest: Digest) -> Path:
    """A path to the object, consulting the sources when it is not here.

    Sources are consulted in order. A hit is verified on arrival:
    a copy-policy folder, an HTTP tier and an origin hint land the
    bytes here; a reference-policy folder answers with a path into
    the folder and lands nothing. An unreachable source is skipped
    and reported; a source serving wrong bytes is refused and
    reported, and the next one consulted. An origin hint is
    consulted only for its own digest and never when offline.

    Returns:
        A read-only path: this store's object, or the folder's
        under a reference policy.

    Raises:
        MissingObject: when no source has it. The message names
            the digest, and when an origin hint was passed over
            because the store is offline, the URL that would have
            satisfied it.
        ErasedObject: when the object was erased here.
        ManifestError: when an HTTP source's manifest is of
            another store.
    """
    state = self.state(digest)
    if state == "erased":
        raise self._erased(digest)
    if state == "present":
        return self.path(digest)
    passed_over: OriginHint | None = None
    for source in self.sources:
        if isinstance(source, OriginHint):
            if source.digest != digest:
                continue
            if self.offline:
                passed_over = source
                continue
        try:
            found = self._fetch_from(source, digest)
        except Unreachable as error:
            self._progress(f"skipped {source.describe()}: {error}")
            continue
        except IntegrityError as error:
            self._progress(f"refused {source.describe()}: {error}")
            continue
        if found is not None:
            return found
    offline = (
        ""
        if passed_over is None
        else (
            f"; the store is offline and {passed_over.url} would have satisfied it"
        )
    )
    raise MissingObject(
        f"{digest} is not in the store at {self.root} nor at any of its"
        f" {len(self.sources)} source(s){offline}"
    )

fill

fill(target: Path, digests: Iterable[Digest]) -> list[Landed]

Land digests into the folder tier at target.

The folder becomes a store of this store's algorithm when it is not one already, and afterwards opens as a livery.strongroom.FolderSource that serves every filled digest. Each object is fetched through this store's sources.

Parameters:

Name Type Description Default
target Path

the folder; created when missing.

required
digests Iterable[Digest]

the objects to land there.

required

Returns:

Type Description
list[Landed]

One landing per digest, in order.

Raises:

Type Description
ManifestError

when target is a store of another layout or algorithm.

MissingObject

when a digest is not obtainable.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def fill(self, target: Path, digests: Iterable[Digest]) -> list[Landed]:
    """Land *digests* into the folder tier at *target*.

    The folder becomes a store of this store's algorithm when it is
    not one already, and afterwards opens as a
    [livery.strongroom.FolderSource][] that serves every filled
    digest. Each object is fetched through this store's sources.

    Args:
        target: the folder; created when missing.
        digests: the objects to land there.

    Returns:
        One landing per digest, in order.

    Raises:
        ManifestError: when *target* is a store of another layout
            or algorithm.
        MissingObject: when a digest is not obtainable.
    """
    if (target / MANIFEST_NAME).exists():
        mirror = Store.open(target)
    else:
        mirror = Store.create(target, algorithm=self.algorithm.name)
    if mirror.algorithm != self.algorithm:
        raise ManifestError(
            f"{target} is a {mirror.algorithm.name} store; this store is"
            f" {self.algorithm.name}"
        )
    landed: list[Landed] = []
    for digest in digests:
        with self.fetch(digest).open("rb") as handle:
            landed.append(mirror.land(handle, expected=digest))
    return landed

tombstone

tombstone(digest: Digest) -> Tombstone | None

The tombstone under the object's path, or None when there is none.

Raises:

Type Description
IntegrityError

when the file under the tombstone's name is not a tombstone.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def tombstone(self, digest: Digest) -> Tombstone | None:
    """The tombstone under the object's path, or None when there is none.

    Raises:
        IntegrityError: when the file under the tombstone's name is
            not a tombstone.
    """
    path = self.object_path(digest)
    try:
        data = path.with_name(path.name + _TOMBSTONE).read_bytes()
    except FileNotFoundError:
        return None
    try:
        return Tombstone.decode(data)
    except FormatError as error:
        raise IntegrityError(f"tombstone of {digest}: {error}") from None

write_tombstone

write_tombstone(stone: Tombstone) -> None

Write stone under its object's path, in this tier.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def write_tombstone(self, stone: Tombstone) -> None:
    """Write *stone* under its object's path, in this tier."""
    path = self.object_path(stone.digest)
    path.parent.mkdir(parents=True, exist_ok=True)
    _write_atomically(path.with_name(path.name + _TOMBSTONE), stone.encode())

evict

evict(digest: Digest) -> None

Remove the object's bytes and its marks; the name is untouched.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def evict(self, digest: Digest) -> None:
    """Remove the object's bytes and its marks; the name is untouched."""
    rungs.remove(self.object_path(digest))
    self._mark(digest).unlink(missing_ok=True)
    self._reference_mark(digest).unlink(missing_ok=True)

clock

clock() -> str

The current instant, from the clock the store was opened with.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def clock(self) -> str:
    """The current instant, from the clock the store was opened with."""
    return self._clock()

publish_begin

publish_begin(target: Digest, *, by: Subject, lease: float = 3600.0) -> Pending

Begin a publish: root target under pending/<id> first.

The pending ref is a root while it exists, so a sweep during the publish keeps every object the target names. Nothing removes a pending ref on a clock; a crashed publish is retired deliberately with livery.strongroom.Store.retire.

Parameters:

Name Type Description Default
target Digest

the tree or version to publish; present here.

required
by Subject

who publishes.

required
lease float

seconds the publish expects to take; recorded in the pending ref's record for whoever retires it.

3600.0

Returns:

Type Description
Pending

The pending publish.

Raises:

Type Description
MissingObject

when target is not present here.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def publish_begin(
    self, target: Digest, *, by: Subject, lease: float = 3600.0
) -> Pending:
    """Begin a publish: root *target* under `pending/<id>` first.

    The pending ref is a root while it exists, so a sweep during the
    publish keeps every object the target names. Nothing removes a
    pending ref on a clock; a crashed publish is retired
    deliberately with [livery.strongroom.Store.retire][].

    Args:
        target: the tree or version to publish; present here.
        by: who publishes.
        lease: seconds the publish expects to take; recorded in the
            pending ref's record for whoever retires it.

    Returns:
        The pending publish.

    Raises:
        MissingObject: when *target* is not present here.
    """
    return publish_begin(self, target, by=by, lease=lease)

publish_commit

publish_commit(pending_id: str, namespace: str, path: str, *, previous: Digest | None, by: Subject, receipt: Digest | None = None, meta: dict[str, Value] | None = None) -> RefRecord

Commit a publish: move the real ref, then drop the pending one.

The move obeys the namespace's mutation class exactly as livery.strongroom.Store.set_ref does.

Raises:

Type Description
NoSuchPending

when pending/<pending_id> does not exist.

RefConflict

when the real ref does not name previous, or its class refuses the move.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def publish_commit(
    self,
    pending_id: str,
    namespace: str,
    path: str,
    *,
    previous: Digest | None,
    by: Subject,
    receipt: Digest | None = None,
    meta: dict[str, Value] | None = None,
) -> RefRecord:
    """Commit a publish: move the real ref, then drop the pending one.

    The move obeys the namespace's mutation class exactly as
    [livery.strongroom.Store.set_ref][] does.

    Raises:
        NoSuchPending: when `pending/<pending_id>` does not exist.
        RefConflict: when the real ref does not name *previous*, or
            its class refuses the move.
    """
    return publish_commit(
        self,
        pending_id,
        namespace,
        path,
        previous=previous,
        by=by,
        receipt=receipt,
        meta=meta,
    )

retire

retire(pending_id: str) -> Digest

Drop a pending ref deliberately, for a publish that will not commit.

Returns:

Type Description
Digest

What the pending ref named.

Raises:

Type Description
NoSuchPending

when it does not exist.

GroupHalfApplied

when it is a group a commit already applied part of; only a commit finishes it.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def retire(self, pending_id: str) -> Digest:
    """Drop a pending ref deliberately, for a publish that will not commit.

    Returns:
        What the pending ref named.

    Raises:
        NoSuchPending: when it does not exist.
        GroupHalfApplied: when it is a group a commit already
            applied part of; only a commit finishes it.
    """
    record = self.record(PENDING, pending_id)
    if record is not None:
        groups.refuse_half_applied(self, pending_id, record)
    return retire(self, pending_id)

pendings

pendings() -> list[str]

Every pending publish's id, sorted; groups included.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def pendings(self) -> list[str]:
    """Every pending publish's id, sorted; groups included."""
    return self.refs(PENDING)

begin

begin(*, by: Subject, lease: float = 3600.0) -> Group

Open a group of ref moves: an empty journal under pending/<id>.

Parameters:

Name Type Description Default
by Subject

who moves the refs.

required
lease float

seconds the group expects to take; recorded for whoever retires it.

3600.0
Source code in packages/strongroom/src/livery/strongroom/_store.py
def begin(self, *, by: Subject, lease: float = 3600.0) -> Group:
    """Open a group of ref moves: an empty journal under `pending/<id>`.

    Args:
        by: who moves the refs.
        lease: seconds the group expects to take; recorded for
            whoever retires it.
    """
    return groups.begin(self, by=by, lease=lease)

add

add(group_id: str, namespace: str, path: str, digest: Digest, *, previous: Digest | None, receipt: Digest | None = None, meta: dict[str, Value] | None = None) -> Group

Append a move to the group; its manifest roots the target from now on.

The arguments are livery.strongroom.Store.set_ref's. The move is checked at commit, not here; only the target's presence is required now.

Raises:

Type Description
NoSuchPending

when the group does not exist.

NotAGroup

when pending/<group_id> is a single publish's.

GroupHalfApplied

when a commit already applied part of it.

MissingObject

when digest is not present here.

RefConflict

when the group already moves that ref.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def add(
    self,
    group_id: str,
    namespace: str,
    path: str,
    digest: Digest,
    *,
    previous: Digest | None,
    receipt: Digest | None = None,
    meta: dict[str, Value] | None = None,
) -> Group:
    """Append a move to the group; its manifest roots the target from now on.

    The arguments are [livery.strongroom.Store.set_ref][]'s. The
    move is checked at commit, not here; only the target's
    presence is required now.

    Raises:
        NoSuchPending: when the group does not exist.
        NotAGroup: when `pending/<group_id>` is a single publish's.
        GroupHalfApplied: when a commit already applied part of it.
        MissingObject: when *digest* is not present here.
        RefConflict: when the group already moves that ref.
    """
    return groups.add(
        self,
        group_id,
        namespace,
        path,
        digest,
        previous=previous,
        receipt=receipt,
        meta=meta,
    )

commit

commit(group_id: str, *, by: Subject) -> tuple[RefRecord, ...]

Apply the group's moves, every one or none, and drop the pending ref.

Holds the maintenance lease and the moved refs' locks for the checks and the moves only. A commit that stopped between two applies is finished by calling this again with the same id.

Returns:

Type Description
tuple[RefRecord, ...]

One record per move, in order.

Raises:

Type Description
NoSuchPending

when the group does not exist.

NotAGroup

when pending/<group_id> is a single publish's.

RefConflict

when a ref does not name its expected previous or its class refuses the move; nothing moved.

LockTimeout

when the lease or a lock outlasts the timeout.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def commit(self, group_id: str, *, by: Subject) -> tuple[RefRecord, ...]:
    """Apply the group's moves, every one or none, and drop the pending ref.

    Holds the maintenance lease and the moved refs' locks for the
    checks and the moves only. A commit that stopped between two
    applies is finished by calling this again with the same id.

    Returns:
        One record per move, in order.

    Raises:
        NoSuchPending: when the group does not exist.
        NotAGroup: when `pending/<group_id>` is a single publish's.
        RefConflict: when a ref does not name its expected
            previous or its class refuses the move; nothing moved.
        LockTimeout: when the lease or a lock outlasts the timeout.
    """
    return groups.commit(self, group_id, by=by)

group

group(group_id: str) -> Group

The group under pending/<group_id>.

Raises:

Type Description
NoSuchPending

when it does not exist.

NotAGroup

when it is a single publish's.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def group(self, group_id: str) -> Group:
    """The group under `pending/<group_id>`.

    Raises:
        NoSuchPending: when it does not exist.
        NotAGroup: when it is a single publish's.
    """
    return groups.read_group(self, group_id)

groups

groups() -> list[Group]

Every group that has begun and not committed, sorted by id.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def groups(self) -> list[Group]:
    """Every group that has begun and not committed, sorted by id."""
    return groups.groups(self)

transaction

transaction(*, by: Subject, lease: float = 3600.0) -> AbstractContextManager[Transaction]

A group under a with: record moves, and the exit commits or retires.

with store.transaction(by=me) as tx:
    tx.move("tools", "bun@1.3", tree, previous=None)
    tx.move("datasets", "corpus/main", version, previous=v1)
records = tx.records

See livery.strongroom.Transaction for what each exit does.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def transaction(
    self, *, by: Subject, lease: float = 3600.0
) -> contextlib.AbstractContextManager[Transaction]:
    """A group under a `with`: record moves, and the exit commits or retires.

    ```python
    with store.transaction(by=me) as tx:
        tx.move("tools", "bun@1.3", tree, previous=None)
        tx.move("datasets", "corpus/main", version, previous=v1)
    records = tx.records
    ```

    See [livery.strongroom.Transaction][] for what each exit does.
    """
    return groups.transaction(self, by=by, lease=lease)

drop_ref

drop_ref(namespace: str, path: str, *, previous: Digest) -> None

Remove a volatile ref and its record by compare-and-swap.

Raises:

Type Description
RefProtected

when the namespace is write-once or monotone.

RefConflict

when the ref does not name previous.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def drop_ref(self, namespace: str, path: str, *, previous: Digest) -> None:
    """Remove a volatile ref and its record by compare-and-swap.

    Raises:
        RefProtected: when the namespace is write-once or monotone.
        RefConflict: when the ref does not name *previous*.
    """
    drop_ref(self, namespace, path, previous=previous)

pin

pin(name: str, digest: Digest, *, by: Subject) -> RefRecord

Root digest under pins/<name>, replacing an earlier pin of that name.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def pin(self, name: str, digest: Digest, *, by: Subject) -> RefRecord:
    """Root *digest* under `pins/<name>`, replacing an earlier pin of that name."""
    return pin(self, name, digest, by=by)

unpin

unpin(name: str) -> Digest

Drop pins/<name> under the maintenance lease, never beside a sweep.

Returns:

Type Description
Digest

What the pin named.

Raises:

Type Description
RefConflict

when the pin does not exist.

LockTimeout

when a sweep holds the lease past the timeout.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def unpin(self, name: str) -> Digest:
    """Drop `pins/<name>` under the maintenance lease, never beside a sweep.

    Returns:
        What the pin named.

    Raises:
        RefConflict: when the pin does not exist.
        LockTimeout: when a sweep holds the lease past the timeout.
    """
    return unpin(self, name)

erase

erase(digest: Digest, *, by: Subject, reason: str, receipt: Digest | None = None) -> Tombstone

Erase the object's bytes and keep the fact as a tombstone.

The name stays valid in every tree that carries it; a path to the object raises naming this tombstone's reason; landing the bytes again is refused while the tombstone stands. An absent object may be erased too, which refuses its future landing.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def erase(
    self, digest: Digest, *, by: Subject, reason: str, receipt: Digest | None = None
) -> Tombstone:
    """Erase the object's bytes and keep the fact as a tombstone.

    The name stays valid in every tree that carries it; a path to
    the object raises naming this tombstone's reason; landing the
    bytes again is refused while the tombstone stands. An absent
    object may be erased too, which refuses its future landing.
    """
    return erase(self, digest, by=by, reason=reason, receipt=receipt)

reachable

reachable(digest: Digest) -> Iterator[Digest]

Every object reachable from digest, itself included.

Structured formats are read by shape, never by namespace: a version yields its tree, parents, receipt and attachments, a tree its entries, anything else nothing. An absent or erased digest is yielded and not walked.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def reachable(self, digest: Digest) -> Iterator[Digest]:
    """Every object reachable from *digest*, itself included.

    Structured formats are read by shape, never by namespace: a
    version yields its tree, parents, receipt and attachments, a
    tree its entries, anything else nothing. An absent or erased
    digest is yielded and not walked.
    """
    return reachable_from(self, digest)

roots

roots() -> list[Digest]

Every root: what every ref on disk names, and every live view's tree.

Namespaces are read from disk, not from the declaration at open, so a consumer's refs root its objects whoever opened the store. A ref file that is not a digest roots nothing. A view whose directory is gone is retired here rather than rooted.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def roots(self) -> list[Digest]:
    """Every root: what every ref on disk names, and every live view's tree.

    Namespaces are read from disk, not from the declaration at
    open, so a consumer's refs root its objects whoever opened the
    store. A ref file that is not a digest roots nothing. A view
    whose directory is gone is retired here rather than rooted.
    """
    found: list[Digest] = live_roots(self)
    base = self.root / "refs"
    for path in sorted(base.rglob("*")):
        if not path.is_file() or path.name.endswith(_RESERVED_SUFFIXES):
            continue
        try:
            found.append(Digest.parse(path.read_text("ascii").strip()))
        except (FormatError, UnicodeDecodeError):
            continue
    return found

sweep

sweep(*, scratch_age: float = 86400.0) -> SweepReport

Remove every object no root reaches, and old scratch.

Runs under the maintenance lease. Pending refs are read again immediately before deleting, so a publish that began during the sweep roots its objects. Tombstones are never removed.

Parameters:

Name Type Description Default
scratch_age float

seconds after which an orphaned .part file is removed; the one age rule.

86400.0

Returns:

Type Description
SweepReport

The report.

Raises:

Type Description
LockTimeout

when another maintenance holds the lease past the timeout.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def sweep(self, *, scratch_age: float = 86400.0) -> SweepReport:
    """Remove every object no root reaches, and old scratch.

    Runs under the maintenance lease. Pending refs are read again
    immediately before deleting, so a publish that began during the
    sweep roots its objects. Tombstones are never removed.

    Args:
        scratch_age: seconds after which an orphaned `.part` file
            is removed; the one age rule.

    Returns:
        The report.

    Raises:
        LockTimeout: when another maintenance holds the lease past
            the timeout.
    """
    return sweep(self, scratch_age=scratch_age, hook=lifecycle.after_mark)

view

view(tree: Digest, at: Path, *, writable: bool = False, allow_hardlink: bool = False) -> ViewRecord

Fill at from the tree by the cheapest safe rung per entry.

The ladder is clone, hardlink, link, copy; a rung that refuses once is not tried again for this view. A hardlink is refused into a writable view unless allow_hardlink says the view's consumers never write in place; a link is refused from a writable view. An executable entry never shares an inode or a target's mode. A symlink entry becomes a real symlink where the platform allows, the within-view target's content as a marked copy where it does not, and a parked refusal when the target escapes the view. Blobs are fetched through the sources.

Parameters:

Name Type Description Default
tree Digest

the tree to present; fetched if absent here.

required
at Path

the directory to fill; missing or empty.

required
writable bool

whether the caller will write in the view.

False
allow_hardlink bool

allow hardlinks into a writable view.

False

Returns:

Type Description
ViewRecord

The record, listing every path and its rung. The view is a

ViewRecord

root while its directory exists.

Raises:

Type Description
FileExistsError

when at exists and is not an empty directory.

MissingObject

when a blob is obtainable from no source, naming the entry's path.

ErasedObject

when a blob was erased, naming the entry's path and the tombstone's reason.

FormatError

when a path in the tree is over the budget.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def view(
    self,
    tree: Digest,
    at: Path,
    *,
    writable: bool = False,
    allow_hardlink: bool = False,
) -> ViewRecord:
    """Fill *at* from the tree by the cheapest safe rung per entry.

    The ladder is clone, hardlink, link, copy; a rung that refuses
    once is not tried again for this view. A hardlink is refused
    into a writable view unless *allow_hardlink* says the view's
    consumers never write in place; a link is refused from a
    writable view. An executable entry never shares an inode or a
    target's mode. A symlink entry becomes a real symlink where the
    platform allows, the within-view target's content as a marked
    copy where it does not, and a parked refusal when the target
    escapes the view. Blobs are fetched through the sources.

    Args:
        tree: the tree to present; fetched if absent here.
        at: the directory to fill; missing or empty.
        writable: whether the caller will write in the view.
        allow_hardlink: allow hardlinks into a writable view.

    Returns:
        The record, listing every path and its rung. The view is a
        root while its directory exists.

    Raises:
        FileExistsError: when *at* exists and is not an empty
            directory.
        MissingObject: when a blob is obtainable from no source,
            naming the entry's path.
        ErasedObject: when a blob was erased, naming the entry's
            path and the tombstone's reason.
        FormatError: when a path in the tree is over the budget.
    """
    return view(self, tree, at, writable=writable, allow_hardlink=allow_hardlink)

views

views() -> list[ViewRecord]

Every recorded view, live or not, by id.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def views(self) -> list[ViewRecord]:
    """Every recorded view, live or not, by id."""
    return views(self)

view_record

view_record(view_id: str) -> ViewRecord | None

The record of one view, or None.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def view_record(self, view_id: str) -> ViewRecord | None:
    """The record of one view, or None."""
    return view_record(self, view_id)

drop_view

drop_view(view_id: str) -> DropReport

Remove what the view's record lists, and nothing else.

A path inside the view that the record does not list is left and named; so is a directory that then stays non-empty, the view's root included. A view whose directory is already gone is retired with nothing removed. The record goes either way.

Raises:

Type Description
FileNotFoundError

when no such view is recorded.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def drop_view(self, view_id: str) -> DropReport:
    """Remove what the view's record lists, and nothing else.

    A path inside the view that the record does not list is left
    and named; so is a directory that then stays non-empty, the
    view's root included. A view whose directory is already gone
    is retired with nothing removed. The record goes either way.

    Raises:
        FileNotFoundError: when no such view is recorded.
    """
    return drop_view(self, view_id)

collect

collect(at: Path, declared: Iterable[str], *, executable: Iterable[str] = ()) -> Tree

Read the declared paths under at back into a tree, landing every object.

A declared directory is collected whole; an undeclared path is not read. Names must be portable and paths within the budget.

A blob's executable bit comes from a ladder: a path named in executable is executable on every platform, so one call lands one tree everywhere; otherwise the file mode where it carries the bit; where it does not, the record of the view that made the path, then the platform's reading of a new file, the extension on Windows; otherwise false.

A symlink whose target stays inside at is content, an absolute target recorded as the relative path from the link's directory. One that leaves at is the link rung's own symlink into this store, read through as the blob it presents, when the view's record says so or the target lands under this store's root; any other is refused.

Parameters:

Name Type Description Default
at Path

the directory to read.

required
declared Iterable[str]

the paths to collect, relative and forward-slashed.

required
executable Iterable[str]

the paths that are executable, spelled the same.

()

Returns:

Type Description
Tree

The root tree, landed here with every subtree.

Raises:

Type Description
FileNotFoundError

when a declared path, or a declared executable, is absent.

FormatError

when a name or a symlink target breaks a rule, a symlink escapes at, or a path is over the budget.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def collect(
    self, at: Path, declared: Iterable[str], *, executable: Iterable[str] = ()
) -> Tree:
    """Read the declared paths under *at* back into a tree, landing every object.

    A declared directory is collected whole; an undeclared path is
    not read. Names must be portable and paths within the budget.

    A blob's executable bit comes from a ladder: a path named in
    *executable* is executable on every platform, so one call lands
    one tree everywhere; otherwise the file mode where it carries
    the bit; where it does not, the record of the view that made
    the path, then the platform's reading of a new file, the
    extension on Windows; otherwise false.

    A symlink whose target stays inside *at* is content, an
    absolute target recorded as the relative path from the link's
    directory. One that leaves *at* is the link rung's own symlink
    into this store, read through as the blob it presents, when
    the view's record says so or the target lands under this
    store's root; any other is refused.

    Args:
        at: the directory to read.
        declared: the paths to collect, relative and forward-slashed.
        executable: the paths that are executable, spelled the same.

    Returns:
        The root tree, landed here with every subtree.

    Raises:
        FileNotFoundError: when a declared path, or a declared
            executable, is absent.
        FormatError: when a name or a symlink target breaks a rule,
            a symlink escapes *at*, or a path is over the budget.
    """
    return collect(self, at, declared, executable)

prefetch

prefetch(digest: Digest) -> list[Digest]

Fetch digest and everything it reaches, for offline use.

Returns:

Type Description
list[Digest]

Every object fetched, the root first.

Raises:

Type Description
MissingObject

naming the first digest no source answered.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def prefetch(self, digest: Digest) -> list[Digest]:
    """Fetch *digest* and everything it reaches, for offline use.

    Returns:
        Every object fetched, the root first.

    Raises:
        MissingObject: naming the first digest no source answered.
    """
    return prefetch(self, digest)

shed

shed(source: Source) -> ShedReport

Evict every local copy source holds, except what a live view needs.

Holding is checked by existence in a folder, a HEAD on an HTTP source, or the digest an origin hint names; never by trust. The next fetch verifies on arrival, so a lying source costs a named miss and never a wrong byte. Objects a live view reaches through a hardlink or a link are kept.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def shed(self, source: Source) -> ShedReport:
    """Evict every local copy *source* holds, except what a live view needs.

    Holding is checked by existence in a folder, a HEAD on an HTTP
    source, or the digest an origin hint names; never by trust.
    The next fetch verifies on arrival, so a lying source costs a
    named miss and never a wrong byte. Objects a live view reaches
    through a hardlink or a link are kept.
    """
    return shed(self, source)

head

head(source: HttpSource, relative: str) -> bool

Whether the HTTP source answers a HEAD for relative.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def head(self, source: HttpSource, relative: str) -> bool:
    """Whether the HTTP source answers a HEAD for *relative*."""
    try:
        with fetch_url(
            source.url(relative),
            connect_timeout=source.connect_timeout,
            transfer_timeout=source.transfer_timeout,
            method="HEAD",
        ):
            return True
    except Unreachable:
        return False

ref_path

ref_path(namespace: str, path: str) -> Path

The file a ref lives at, present or not.

Parameters:

Name Type Description Default
namespace str

a declared namespace.

required
path str

the ref's path within it, forward-slashed; each component is a portable name and none ends in a reserved suffix.

required

Raises:

Type Description
UnknownNamespace

when the namespace was not declared.

FormatError

when a path component breaks a rule.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def ref_path(self, namespace: str, path: str) -> Path:
    """The file a ref lives at, present or not.

    Args:
        namespace: a declared namespace.
        path: the ref's path within it, forward-slashed; each
            component is a portable name and none ends in a
            reserved suffix.

    Raises:
        UnknownNamespace: when the namespace was not declared.
        FormatError: when a path component breaks a rule.
    """
    self._namespace(namespace)
    parts = path.split("/")
    for part in parts:
        check_name(part)
        if part.endswith(_RESERVED_SUFFIXES):
            raise FormatError(
                f"ref path component {part!r} ends in a reserved suffix"
            )
    return self.root.joinpath("refs", namespace, *parts)

ref

ref(namespace: str, path: str) -> Digest | None

What the ref names, or None when it does not exist.

Raises:

Type Description
RefTampered

when the ref and its record disagree, or the record is missing or malformed.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def ref(self, namespace: str, path: str) -> Digest | None:
    """What the ref names, or None when it does not exist.

    Raises:
        RefTampered: when the ref and its record disagree, or the
            record is missing or malformed.
    """
    found = self._read_ref(namespace, path)
    return None if found is None else found[0]

record

record(namespace: str, path: str) -> RefRecord | None

The record beside the ref, or None when there is none.

Raises:

Type Description
RefTampered

when the record's bytes are not a record.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def record(self, namespace: str, path: str) -> RefRecord | None:
    """The record beside the ref, or None when there is none.

    Raises:
        RefTampered: when the record's bytes are not a record.
    """
    target = self.ref_path(namespace, path)
    try:
        data = target.with_name(target.name + _RECORD).read_bytes()
    except FileNotFoundError:
        return None
    try:
        return RefRecord.decode(data)
    except FormatError as error:
        raise RefTampered(f"record of {namespace}/{path}: {error}") from None

refs

refs(namespace: str) -> list[str]

Every ref path in the namespace, sorted.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def refs(self, namespace: str) -> list[str]:
    """Every ref path in the namespace, sorted."""
    self._namespace(namespace)
    base = self.root / "refs" / namespace
    found: list[str] = []
    for path in base.rglob("*"):
        if path.is_file() and not path.name.endswith(_RESERVED_SUFFIXES):
            found.append(path.relative_to(base).as_posix())
    return sorted(found)

set_ref

set_ref(namespace: str, path: str, digest: Digest, *, previous: Digest | None, by: Subject, receipt: Digest | None = None, meta: dict[str, Value] | None = None) -> RefRecord

Move a ref by compare-and-swap, writing its record.

The move happens under the ref's lock. The namespace's mutation class is enforced: write-once refuses a second digest and treats the same digest as done; monotone requires the new target to be a version present here whose parents include the current target; volatile needs only the compare-and-swap.

Parameters:

Name Type Description Default
namespace str

a declared namespace.

required
path str

the ref's path.

required
digest Digest

what the ref names afterwards.

required
previous Digest | None

what the caller believes it names now; None to create.

required
by Subject

who moves it.

required
receipt Digest | None

the receipt of the moving call, if any.

None
meta dict[str, Value] | None

the consumer's own object for the record.

None

Returns:

Type Description
RefRecord

The record written, or the existing record when a

RefRecord

write-once ref already names digest.

Raises:

Type Description
RefConflict

when the ref does not name previous.

WriteOnceRefused

on a write-once ref that names another digest.

NotFastForward

on a monotone ref whose new target does not descend from the current one, or is not a version here.

LockTimeout

when a live lock outlasts the timeout.

RefTampered

when the ref and its record disagree.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def set_ref(
    self,
    namespace: str,
    path: str,
    digest: Digest,
    *,
    previous: Digest | None,
    by: Subject,
    receipt: Digest | None = None,
    meta: dict[str, Value] | None = None,
) -> RefRecord:
    """Move a ref by compare-and-swap, writing its record.

    The move happens under the ref's lock. The namespace's
    mutation class is enforced: write-once refuses a second digest
    and treats the same digest as done; monotone requires the new
    target to be a version present here whose parents include the
    current target; volatile needs only the compare-and-swap.

    Args:
        namespace: a declared namespace.
        path: the ref's path.
        digest: what the ref names afterwards.
        previous: what the caller believes it names now; None to
            create.
        by: who moves it.
        receipt: the receipt of the moving call, if any.
        meta: the consumer's own object for the record.

    Returns:
        The record written, or the existing record when a
        write-once ref already names *digest*.

    Raises:
        RefConflict: when the ref does not name *previous*.
        WriteOnceRefused: on a write-once ref that names another digest.
        NotFastForward: on a monotone ref whose new target does not
            descend from the current one, or is not a version here.
        LockTimeout: when a live lock outlasts the timeout.
        RefTampered: when the ref and its record disagree.
    """
    with self._locked(self.ref_path(namespace, path)):
        return self._move_locked(
            namespace,
            path,
            digest,
            previous=previous,
            by=by,
            receipt=receipt,
            meta=meta,
        )

now

now() -> str

The current instant, RFC 3339 in UTC to the second.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def now() -> str:
    """The current instant, RFC 3339 in UTC to the second."""
    return datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")

silent

silent(message: str) -> None

The default progress sink: report nothing.

Source code in packages/strongroom/src/livery/strongroom/_store.py
def silent(message: str) -> None:
    """The default progress sink: report nothing."""