livery.git._store¶
livery.git._store ¶
The state store: JSON rows on git refs, on the remote and in the checkout.
A livery.git.StateStore is the checkout and the policy a caller built it with: who may write, under which identity, and which variable a published listing travels in. Every function below takes one; nothing is held between calls, and a snapshot block is the one scoped state, open for its block alone.
State that CI runs keep across runs (timing rows, verified trees,
coverage marks) lives as files on git refs in a namespace outside
refs/heads, so a default clone or fetch never downloads it. Each
ref is one root commit whose tree holds the series' files; a write
reads the current tree, rebuilds it with the kept entries plus the
new ones, and pushes a fresh root commit over the old one, so a
window needs no history and a reader never sees a torn tree.
A livery.git.Series declares one ref's rows: reach
for its rows and row to read and its put to write. The store
parses, stamps the schema and the time, applies the window, and
refuses what the declaration forbids, so no caller does any of
that itself. A livery.git.Keyed family declares one
series per key (a leg and a package; a run and a leg) and lists the
keys the remote holds. The bare read and put are the transport
underneath, and sweep is the janitor. Every failure is a printed
reason, never a boolean: a stamp is best-effort by contract, so the
only way its failure is ever noticed is by being printed.
A series declared local lives under refs/workshop-local/ in
the checkout's own git directory instead of on the remote. No refspec
names that namespace, so a fetch never brings it and a push never
carries it, and a mirror push of refs/workshop/* cannot carry it
either; the checkout's worktrees share it, and a fresh clone starts
empty. Its rows are read and written the same way. Only the transport
differs: the ref moves by an in-process compare-and-swap on the old
value, in place of a push with a lease. Every worktree writes the same
refs, so a write that finds a ref's lock file held waits a bounded
time for it, and a lock that does not clear is the returned reason,
naming the lock file.
The gate reads the remote namespace without the network. fetch_store
mirrors the row series its caller names, and only them, under
refs/workshop-origin/, one ref per series and never a keyed
family, and stamps the fetch and the names on the local FETCHED
series; fetched_snapshot then serves every read of a named series
from that mirror, fails a read or a listing of any other remote ref
naming the ref and the mirror's scope, refuses every write of a
remote series, and, in a checkout the store was never fetched into,
fails every read naming the sync, never as an absence. fm sync and
fm start fetch; fm check reads.
The rules, ported from hse's stamp transport:
- Only CI writes a series declared
ci_only; local runs read. - A write never starts from a state it could not read: an unreachable ref refuses the write, because rebuilding the tree from an empty map would erase every file the writer did not know about. A ref that does not exist yet is not a failure.
- Across runs a shared ref is written by compare-and-swap: the push
carries the sha the writer read (
--force-with-leasewith an explicit value), the server refuses a stale write atomically, and the loser re-reads, merges, and retries. A concurrency loss costs a retry, never a wrong result. - The push's own report is its verdict: git prints which refs moved from the server's report-status, and the compare-and-swap is the guard. A readback through the same git configuration cannot catch what would mislead the push itself.
- The commit carries its own identity and
[skip ci]: the natural writer is a CI container with no global git config, and a push to this namespace must never start a workflow.
Only the wire goes through a git process: fetch, push and
ls-remote. Every tree is read, and every root commit written, in
the checkout's own object store without a process, as bytes, so no
newline translation reaches a row or a file name on Windows. The
working tree and the index are never touched, so a write is safe
from the middle of any job, and a read or a write costs a fixed
handful of processes whatever the series holds: a remote read one
fetch, a remote write one fetch and one push, a local read or write
none.
Classes:
| Name | Description |
|---|---|
StateStore |
A checkout's state store and the policy its builder gave it. |
Skipped |
A file a read passed over: its name and why. |
Row |
One row of a series: the file it came from and its fields. |
Rows |
What a series read found: the rows, what it skipped, and whether it could answer. |
Series |
A shared ref's declaration: its rows' schema, its window, and who may write it. |
Keyed |
A family of series under one prefix, one ref per key. |
Read |
What a read found: the files, the lease, and whether it could answer. |
Fetched |
What |
Functions:
| Name | Description |
|---|---|
slug |
name as a ref-safe and file-safe token: every other character is a dash. |
remote_snapshot |
Read the remote namespace through one listing for the block. |
fetch_store |
Mirror the row series series under |
fetched_snapshot |
Read the remote namespace from the checkout's mirror of origin, for the block. |
read |
Read ref's files, saying whether a miss is absence or failure. |
list_refs |
The refs under prefix by sha, remote or local; |
put |
Write files onto ref, keeping what it already holds; |
drop |
Delete ref, on origin or in the checkout; |
sweep |
The store's janitor: every series bounded, every orphan dropped; the lines. |
Attributes:
| Name | Type | Description |
|---|---|---|
NAMESPACE |
|
|
LOCAL_NAMESPACE |
|
|
TRACE_NAMESPACE |
|
|
FETCHED_NAMESPACE |
|
|
RUN_PREFIX |
|
|
ATTEMPTS |
|
|
Order |
|
|
WHOLE |
tuple[str, ...]
|
|
FETCHED |
|
|
LISTED |
|
|
IMPORT_REF |
|
FETCHED
module-attribute
¶
FETCHED = Series('fetched', window=1, ci_only=False, schema=2, local=True)
StateStore
dataclass
¶
StateStore(root: Path, in_ci: bool, identity: tuple[str, str], snapshot_variable: str, prog: str = 'fm')
A checkout's state store and the policy its builder gave it.
Attributes:
| Name | Type | Description |
|---|---|---|
root |
Path
|
The checkout whose origin holds the remote namespace and whose git directory holds the local one. |
in_ci |
bool
|
Whether this process is a CI run: a put to a series
declared |
identity |
tuple[str, str]
|
The name and address every commit the store makes carries, author and committer. |
snapshot_variable |
str
|
The environment variable a published listing's path travels in, from a parent process to its children. |
prog |
str
|
The runner's command name, for the words a refusal
gives a person ( |
Skipped
dataclass
¶
Row
dataclass
¶
Rows
dataclass
¶
Rows(rows: tuple[Row, ...], skipped: tuple[Skipped, ...] = (), failed: bool = False, reason: str = '', unread: tuple[str, ...] = (), blobs: Mapping[str, str] = dict())
What a series read found: the rows, what it skipped, and whether it could answer.
Attributes:
| Name | Type | Description |
|---|---|---|
rows |
tuple[Row, ...]
|
The rows of the series' schema, newest first. |
skipped |
tuple[Skipped, ...]
|
One entry per file that is not such a row, naming the file and why; printing one gives the line. |
failed |
bool
|
Whether the transport could not answer for a ref that exists, or reach the remote at all. An absent ref reads as no rows and no failure. |
reason |
str
|
Why it could not answer, in the store's one wording. |
unread |
tuple[str, ...]
|
The files a read that named the ones it wants passed over, by name, never read. |
blobs |
Mapping[str, str]
|
The blobs the read was asked for, by name, each the
text its writer wrote (livery.git.Series's |
Series
dataclass
¶
Series(name: str, window: int | None = None, ci_only: bool = True, schema: int = 1, local: bool = False, age: timedelta | None = None, namespace: str = '', blobs: str = '')
A shared ref's declaration: its rows' schema, its window, and who may write it.
A series is one ref holding JSON rows of one schema, one file per
row. rows and row read, put writes; the store stamps the
schema and the time, keeps the window, and refuses what the
declaration forbids, so a caller never parses a row, checks a
schema, or trims a ref itself.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
The ref's name under the namespace. |
window |
int | None
|
The newest rows kept after every write, by the time
the store stamped them; |
ci_only |
bool
|
Whether only a CI run may write it. Local runs read. |
schema |
int
|
The rows' schema. A reader skips a row of another version and names it; a new field is the same version, a changed meaning is the next. |
local |
bool
|
Whether the series lives in the checkout's git directory
under |
age |
timedelta | None
|
How long a row stays before the janitor drops it, by the
store's stamp; |
namespace |
str
|
The prefix the ref lives under, when it is neither
the shared namespace nor the local one:
livery.git.TRACE_NAMESPACE is the one
such prefix, pushed to origin and mirrored by nothing.
Empty means the class |
blobs |
str
|
The directory, ending in a slash, of the files the
series keeps beside its rows as their writer wrote them,
such as a file named by its own content ( |
Methods:
| Name | Description |
|---|---|
__post_init__ |
|
blob |
Whether the file name is one of the series' blobs, not a row. |
rows |
Every row of the series, newest first, with what was skipped and why. |
row |
The row in the file name. |
file |
The text of the file name on the ref, or why it could not be read. |
put |
Write rows, stamped, and blobs by file name; |
__post_init__ ¶
Source code in packages/git/src/livery/git/_store.py
blob ¶
rows ¶
rows(store: StateStore, *, only: Callable[[str], bool] | None = None) -> Rows
Every row of the series, newest first, with what was skipped and why.
An absent ref is no rows and no failure. A ref the transport
could not read is a failure with its reason, so every caller
falls open the same way. A file that does not parse or is of
another schema is skipped and named, and the rest stand. A blob
the read accepts comes back as text in blobs. With
only, the files whose names it accepts are read and parsed,
and every other file is named in unread, never read: a
series of large rows answers a reader of a few of them at the
cost of those few.
Source code in packages/git/src/livery/git/_store.py
row ¶
row(store: StateStore, name: str) -> tuple[Row | None, str]
The row in the file name.
(None, "") when the ref or the file is absent, and
(None, reason) when the ref could not be read or the file
is not a row of the schema. The ref's other files are not read.
Source code in packages/git/src/livery/git/_store.py
file ¶
file(store: StateStore, name: str) -> tuple[str | None, str]
The text of the file name on the ref, or why it could not be read.
For a series whose files are not rows: a leg's trace, kept as
its writer wrote it. (None, "") is absence, of the ref or
of the file. The ref's other files are not read.
Source code in packages/git/src/livery/git/_store.py
put ¶
put(store: StateStore, rows: Mapping[str, Mapping[str, Any]], *, message: str, remove: Iterable[str] = (), blobs: Mapping[str, str] | None = None) -> str
Write rows, stamped, and blobs by file name; "" or the reason.
Every row is written with the schema and the time stamped on
it, and the stamp wins over a schema or when the row
carries. Each blob is written as the text given, so a blob of
the same text is the same object git already holds. The files
the series already holds stand, less the file names in
remove, so a caller replacing a record in place names what
goes and puts what changed.
The window keeps the newest rows by their stamps, so a series
keyed by something other than time (a tree id) keeps its
newest too. A write the declaration forbids is refused with
the rule, and so is a row named as a blob, or a blob named
outside the series' blobs; the transport's refusals come
back as they are.
Source code in packages/git/src/livery/git/_store.py
Keyed
dataclass
¶
Keyed(name: str, keys: tuple[str, ...], window: int | None = None, ci_only: bool = True, schema: int = 1, local: bool = False, namespace: str = '', stale_after: timedelta | None = None, current: Callable[[Path], set[tuple[str, ...]] | None] | None = None, linger: timedelta | None = None, keep: Callable[[Path], int | None] | None = None, holds_rows: bool = True, blobs: str = '')
A family of series under one prefix, one ref per key.
The coverage record keeps one series per base and leg, the
per-run halves one per run and leg. A family declares its key
parts once: series makes the series of one key, with every
part made ref-safe, and listed reads the keys the remote
holds, so no caller spells a ref of the family itself.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
The family's name under the namespace ( |
keys |
tuple[str, ...]
|
The names of the key's parts, in ref order
( |
window |
int | None
|
Every series' window; |
ci_only |
bool
|
Whether only a CI run may write the family. |
schema |
int
|
The rows' schema, shared by every series. |
local |
bool
|
Whether every series of the family is local, as for a livery.git.Series. |
namespace |
str
|
The prefix every series of the family lives under, as for a livery.git.Series. |
stale_after |
timedelta | None
|
How long a key's ref may live, by its commit
time, before the janitor drops it as an orphan; |
current |
Callable[[Path], set[tuple[str, ...]] | None] | None
|
The keys the world still produces, from the checkout
root, or |
linger |
timedelta | None
|
How long a key outside the current ones stays, by its
ref's commit time, before the janitor drops it, so a run
in flight that relies on it can still read it: a branch's
coverage record is read by main's run for the squash that
merged it, minutes after the branch is gone. |
keep |
Callable[[Path], int | None] | None
|
How many values of the key's first part the family
keeps, read from the checkout root; the rest go. The parts
are ranked by value, as numbers where they are numbers, so
a forge's run id ranks by age without reading a commit.
For a family keyed by a run, this is how many runs are
kept. |
holds_rows |
bool
|
Whether the family's series hold JSON rows the
janitor bounds like any other series. A family of blobs (a
leg's trace) says false: its files are not rows, and
|
blobs |
str
|
The directory of every series' blobs, as for a livery.git.Series. |
Methods:
| Name | Description |
|---|---|
__post_init__ |
|
series |
The series of one key, one part per declared key name. |
at |
The series of a key as its ref spells it, every part verbatim. |
listed |
The keys the remote holds under the leading parts head. |
current
class-attribute
instance-attribute
¶
__post_init__ ¶
series ¶
The series of one key, one part per declared key name.
Raises:
| Type | Description |
|---|---|
ValueError
|
When key has another number of parts. |
at ¶
The series of a key as its ref spells it, every part verbatim.
listed returns keys as their refs spell them, and a key
written before the family made its parts ref-safe may spell a
part another way; series would re-spell it and name a ref
that does not exist. A listed key comes back through here.
Raises:
| Type | Description |
|---|---|
ValueError
|
When key has another number of parts. |
Source code in packages/git/src/livery/git/_store.py
listed ¶
listed(store: StateStore, *head: str) -> list[tuple[str, ...]] | None
The keys the remote holds under the leading parts head.
None when the remote cannot be listed. A ref under the
prefix with another number of parts is not the family's and
is left out.
Source code in packages/git/src/livery/git/_store.py
Read
dataclass
¶
Read(files: dict[str, str] | None, sha: str | None, failed: bool, reason: str = '', names: tuple[str, ...] = ())
What a read found: the files, the lease, and whether it could answer.
Attributes:
| Name | Type | Description |
|---|---|---|
files |
dict[str, str] | None
|
The tree's files by name, or |
sha |
str | None
|
The commit the ref points at on the remote, the lease a
write carries; |
failed |
bool
|
Whether |
reason |
str
|
The transport's words when it could not answer. |
names |
tuple[str, ...]
|
The name of every file the tree holds, read or not, sorted; empty when there is nothing to read. |
Fetched ¶
Bases: NamedTuple
What fetch_store did: the mirror's size, why it failed, the refs that moved.
Attributes:
| Name | Type | Description |
|---|---|---|
held |
int
|
The refs the mirror holds after the fetch; zero when it failed. |
why |
str
|
Empty, or why the fetch or the stamp failed, in git's words. |
changed |
int
|
The refs this fetch added, moved or pruned; zero on a quiet origin. |
slug ¶
remote_snapshot ¶
remote_snapshot(store: StateStore, *, fetch: Iterable[str] = (), publish: bool = False) -> Generator[str | None]
Read the remote namespace through one listing for the block.
One ls-remote of the store's namespace and the branches when
the block opens; every read and listing of a remote ref inside
answers from it, and a ref whose commit the checkout already
holds is read with no network at all. The refs whose names start
with a prefix in fetch (spelled below the namespace, metrics
or run/1400/, or in full from refs/ for another namespace,
refs/workshop-trace/about/) and whose commits the checkout
lacks are fetched together, one round trip, when the block opens.
A read of a ref outside them brings the ref's whole key group in
one fetch (_key_group), so a loop over a family's keys costs one
round trip whichever ref it reads first. A ref another writer
moved after the listing is listed again and read at its current
commit, so a concurrent writer never turns a read into a failure.
A write inside keeps its compare-and-swap on the listed sha: a
refused push lists again and retries, and a successful write
records its new sha, so a read after it sees it. A listing the
forge could not answer makes every read inside a failure naming
the reason and every listing unlistable, never an absence. A block
opened inside another for the same checkout shares the outer
listing and brings what its own fetch names against it.
With publish the listing is written to a file whose path the
block yields, for the caller to hand its child processes under
SNAPSHOT_VARIABLE: a block a child opens reads that file
instead of listing, and the child's writes record their shas in
it for the children after it. A listing the parent could not
take is not published and the block yields None, so a child
lists for itself; a file a child cannot read, or one taken for
another checkout, counts as none. Without publish the block
yields None.
Source code in packages/git/src/livery/git/_store.py
fetch_store ¶
fetch_store(store: StateStore, *, series: Iterable[str], branches: bool = False) -> Fetched
Mirror the row series series under FETCHED_NAMESPACE and stamp it.
One fetch, pruned, with one refspec per name below NAMESPACE:
metrics lands at refs/workshop-origin/metrics, a series
origin holds no ref for yet costs nothing, and a ref origin
dropped goes from the mirror. Afterwards the mirror holds the
named series alone: every ref under it that is not one of them,
such as a family's key a fetch of a wider scope left, is deleted
here. Then the origin row of FETCHED records the time, the
count of refs, and the names, sorted. The names are the mirror's
scope: a fetched_snapshot reads a missing ref of a named series
as absent and fails a read of any other remote ref. With
branches the same fetch brings origin's heads into
refs/remotes/origin/, pruned the same way, so a sync that
needs both pays one round trip. A fetch transfers only what the
checkout lacks: on a quiet origin the round trip is the whole
cost and nothing changes. Named no series and no branches, the
fetch is skipped and the mirror is emptied.
Returns:
| Type | Description |
|---|---|
Fetched
|
The refs the mirror holds and how many this fetch changed, or |
Fetched
|
zero of each and the reason when the fetch, the prune or the |
Fetched
|
stamp failed, with git's words; nothing here raises or prints. |
Source code in packages/git/src/livery/git/_store.py
fetched_snapshot ¶
fetched_snapshot(store: StateStore) -> Generator[None]
Read the remote namespace from the checkout's mirror of origin, for the block.
Nothing inside reaches origin. The listing is the mirror
fetch_store took, with origin's branches beside it from the
remote-tracking refs, every commit is local already, and a write
of a remote series is refused with its reason; a local series
reads and writes as ever. The series the stamp names are the
mirror's scope: a read of one of them answers from the mirror, a
missing ref among them is absence, and a read or a listing of any
other remote ref, a keyed family's refs among them, fails naming
the ref and the scope, never as an absence, since the mirror never
held it; a live remote_snapshot reads those. In a checkout
fetch_store never stamped, or stamped under another schema, the
listing is a failure naming the sync, so every read inside fails
the way an unreachable origin fails, never as an absence, and
every caller falls open on it. A block opened inside another
snapshot for the same checkout shares the outer one.
Source code in packages/git/src/livery/git/_store.py
read ¶
Read ref's files, saying whether a miss is absence or failure.
A remote ref is fetched from origin: the fetch lands in
FETCH_HEAD and creates no local ref, so a checkout accrues
only unreachable objects git's own gc prunes. A failed fetch is a
failure only when the ref is there: a ref that does not exist
also fails the fetch, and that is the ordinary first-run case. A
remote that cannot answer the probe either counts as failed: an
unreachable forge must never read as "there was never a stamp".
A local ref, or one of the mirror under FETCHED_NAMESPACE, is
read from the checkout's own git directory: a missing ref is
absence, and anything else git refuses is a failure with git's
words. With only, the files whose names it accepts are read and
the rest are named in names alone, never read.
Source code in packages/git/src/livery/git/_store.py
list_refs ¶
list_refs(store: StateStore, prefix: str) -> dict[str, str] | None
The refs under prefix by sha, remote or local; None when unlistable.
A prefix under the local namespace, the mirror, or the remote-tracking refs is listed from the checkout's git directory; any other answers from the open snapshot when one covers it, and from origin otherwise. A snapshot of the mirror covers the branches and a prefix at or below one of its series; a wider prefix under a remote namespace, a keyed family's, is unlistable from it, never empty, since the mirror holds none of its refs.
Source code in packages/git/src/livery/git/_store.py
put ¶
put(store: StateStore, ref: str, files: dict[str, str], *, message: str, window: int | None = None, ci_only: bool = False, order: Order | None = None, remove: Iterable[str] = (), attempts: int = ATTEMPTS) -> str
Write files onto ref, keeping what it already holds; "" or the reason.
The tree is the ref's current files, less remove, with files laid over them, trimmed to the newest window files when one is given, ranked by order or, without one, by name. The push is a compare-and-swap on the sha the read returned: a ref another writer moved in between refuses the push, and the write re-reads, merges, and retries up to attempts times. ci_only refuses a write from outside CI, naming the rule. Every refusal is the returned reason; nothing here raises or prints.
Source code in packages/git/src/livery/git/_store.py
1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 | |
drop ¶
drop(store: StateStore, ref: str) -> str
Delete ref, on origin or in the checkout; "" when gone, else the reason.
A ref that is already absent counts as gone: the janitor re-runs as its own recovery, and a second sweep must find nothing to do.
Source code in packages/git/src/livery/git/_store.py
sweep ¶
sweep(store: StateStore, declared: tuple[Series | Keyed, ...], *, remote: bool, dry_run: bool = False, now: datetime | None = None) -> list[str]
The store's janitor: every series bounded, every orphan dropped; the lines.
The scope is the run's: the local series always, the remote ones
only under remote, which a caller sets inside CI, since a local
run never writes the remote store. A series is trimmed to its
window, and the rows older than the age it declares go. A
family's keys are listed and each key's series is bounded the
same way, after its orphans go: a key whose ref is older than the
family's stale_after, or one the family's current keys no
longer include. A family that cannot be listed, or whose current
keys cannot be told, drops nothing and says so. Under dry_run
every line says what would go and nothing is written. Every line
names what happened; a failure is a line, never a raise, and a
second sweep finds nothing to do. A remote sweep reads through one
snapshot of the namespace, its caller's when one is open, so a
family's keys, rows and commit times cost a fetch per key group,
never one per key.