Using the store¶
A walk through the API in the order a consumer meets it. Every call
is a method of Store; every refusal is a class under StoreError
whose message names what was found and what to do.
Open¶
from pathlib import Path
from livery.strongroom import FolderSource, Namespace, Store
store = Store.create(
Path("~/.cache/mystore").expanduser(),
namespaces=[Namespace("tools", "write-once")],
sources=[FolderSource(Path("/mnt/mirror"))],
)
create writes the root manifest and opens; open opens an existing
store and refuses another layout version or algorithm. Every
namespace the caller will write is declared with its mutation class;
pins and pending are the store's own. Sources are consulted in
order for an object the store lacks: a folder in the same layout with
a copy or reference fill policy, an HttpSource serving the
layout, or an OriginHint naming one URL and the digest it yields.
offline=True never consults an origin.
Objects¶
digest = store.put(b"bytes")
landed = store.land(stream, expected=digest)
path = store.path(digest)
data = store.read(digest)
where = store.fetch(digest)
land streams bytes through the digest into a scratch file and moves
them into place; a mismatch against expected keeps nothing; an
object that already exists is success without a write. path hands
over a read-only path, checking size on every access and hashing an
object this store never verified. fetch answers locally or consults
the sources, verifying every hit on arrival. scrub hashes everything
and removes what does not match its name. prefetch fetches
everything one digest reaches, and shed evicts local copies a named
source holds, except what a live view depends on.
Refs¶
from livery.strongroom import Subject
me = Subject("person", "willem")
record = store.set_ref("tools", "bun@1.3", tree, previous=None, by=me)
current = store.ref("tools", "bun@1.3")
A ref moves by compare-and-swap on previous, under a per-ref lock,
with a record written beside it. The namespace's class is enforced:
write-once refuses a second digest, monotone requires a version
whose parents include the current target, volatile needs only the
compare-and-swap. A ref whose record disagrees is reported as
tampered.
Publishing¶
pending = store.publish_begin(tree, by=me)
# land what the tree names, here or in the tier the publish fills
store.publish_commit(pending.id, "tools", "bun@1.3", previous=None, by=me)
The pending ref roots the target from begin to commit, so a sweep
during the publish keeps every object it names. A refused commit
leaves the pending ref for a retry; retire drops one deliberately.
Several refs move together, or not at all, as a group:
with store.transaction(by=me) as tx:
tx.move("tools", "bun@1.3", tree, previous=None)
tx.move("datasets", "corpus/main", version, previous=v1)
records = tx.records
Entering begins a group under pending/<id>, whose manifest roots
every move's target. A clean exit commits: under the maintenance
lease and the refs' locks, every move is checked first, the
compare-and-swap and the namespace's class, and the first refusal
stops the group with nothing moved; then the moves apply in order.
An exception in the body, or a refused commit, retires the group.
A commit that stops between two applies leaves the group with its
journal; store.commit(id, by=me) on the same id finishes it, and
retire refuses it. begin, add, commit, group and groups
are the pieces underneath, for a group that spans processes.
Views¶
record = store.view(tree, Path("work/bun"))
outputs = store.collect(
Path("work/out"), ["result.bin", "logs", "run.sh"], executable=["run.sh"]
)
store.drop_view(record.id)
view fills a directory by the cheapest safe rung per entry and
records which one it used; the view is a root while its directory
exists. collect reads declared outputs back into a tree, landing
every object; executable names the outputs that are executable
where the file mode cannot say, so one call lands one tree on every
platform. drop_view removes only what the record lists and names
what it leaves.
Maintenance¶
sweep marks from every ref and every live view, reads pending refs
again before deleting, and removes the unreached and old scratch.
erase writes a tombstone and deletes the bytes; the name stays valid
in every tree that carries it, a path to it fails naming the reason,
and landing it again is refused.