toolroom store¶
The pinned tool store over strongroom. A record, one directory per
tool, names the tool, every version tracked, per version and host the
artifact's URL and its sha256 with the deployment resolved through the
record's layers, and per version the surface its command line
accepts; the store lands the artifact by that digest, extracts it,
collects it as a tree, and names the tree under
tools/<name>@<version> in one strongroom store per machine, shared
by every checkout on it.
The store installs, links, emits and fetches:
from pathlib import Path
from livery.strongroom import FolderSource
from livery.toolroom.store import Home, Record, Store, records_in
store = Store(
Home(Path.home() / ".local/share/toolroom"),
sources=[FolderSource(Path("/mirrors/tools/store"))],
)
records = [Record.load(path) for path in records_in(Path("records"))]
ensured = [store.ensure(record, record.versions[-1]) for record in records]
store.link(ensured, Path(".workshop/bin"))
delta = store.delta(ensured, Path(".workshop/bin"))
The record¶
A record is one file of JSON lines, records/<tool>.jsonl, named by
the tool. The first line is the tool axis: the name, the description,
the kind, the version floor a system-check tool must reach, prime,
the oldest version the reading history reaches, the hosts the tool
has, the tool's layout and its override for one host each. A version
line is one version's arrival: the version, its date, per host the
artifact's URL and sha256, the version's override of the layout, for
every host or for one, and read, the platforms and the extractor
that read it and the tool's description when it changed. The
statement lines under a version line are what its reading changed:
one option per line with the option's fields whole, one line for a
verb's own fields when they moved, one line for an option or a verb
withdrawn (gone), and one line per absence, the platforms that read
the version and did not find the option. Versions run in file order,
a release appends lines, and the review diff is the options that
moved. An option unchanged since the previous reading has no line.
A deployment resolves through four layers, most specific winning: the
tool's layout, the tool's override for the host, the version's layout,
the version's override for the host. The layout fields are root, the
directory inside the archive hoisted to the install root; file, the
name a bare download (one file, no archive) is saved under; format,
what the artifact is (zip, tar or file) when its URL's suffix
would sniff wrong; entry_points, the install-relative paths of
the executables the deployment puts on PATH, annotated here and never
discovered by scanning a directory; paths, the install-relative
directories put on PATH; env, with $package standing for the
install root, where a tool's own switches belong (an update check, a
telemetry opt-out), since entering the environment exports them and
every spawn of the tool inherits them; shims, a link name to an
executable the install
carries; and exclude, the archive members left out before import,
fnmatch patterns over the install-relative path.
The surface¶
A delta may carry a surface: what the version's command line
accepts, as it was read, and who read it. The reading is one verb at
a time. verbs names the verbs this version changed, each whole, with
help, wraps, positional, lead and its options, and an option
carries flags, negation, help, type, default and choices;
a verb set to null is withdrawn, and a verb not named is inherited
from the nearest earlier version that has a surface. The tool's own
options hang off the verb named "". help, the tool's description,
is set when it changed and inherited otherwise; the record's first
surface sets it. platforms names who read the version, from Linux,
macOS and Windows, and extractor the generation of the reader;
neither is inherited. absent records, per verb and option, the
platforms that read the version and did not find that option, the
option named "" standing for the verb itself; it names only
platforms among platforms. A version whose delta carries no surface
was never read, and surface_at answers None for it rather than
inheriting a reading. A version with a surface and no artifact is
tracked for its surface alone: it has no host and nothing installs it.
surface_at resolves one version whole, every verb in name order with
its options in name order, and observations resolves every version
that has a surface, in sequence.
The catalogue and the lock¶
A consumer resolves against the catalogue, never against a record. The catalogue lists every tool with its versions in order, per version the hosts it has an artifact for with the digest of each host's deployment, and per version read its surface, from which the store renders that version's stub. It is read on demand, the pointer first, a tool's tree when the tool is asked for and a version's hosts when a lock needs them, from the published index, by URL or from a directory holding one, through the machine's store, which keeps what it fetched so a second read is offline; the authoring site reads its records directly and gets the same catalogue, since a deployment's digest is the digest of its canonical JSON either way.
Every site that names a tool or an extension spells it in one grammar,
name[option,...]?>=floor@scope, and Spec.parse is its one parser:
[...] names options (a tool's profile, an extension's switches), ?
marks it optional, >=floor is the lowest version that satisfies, and
@scope limits it to some hosts. A scope is comma-separated platforms
(windows, every host of it) or host keys (windows-x64), each
excluded with a leading !; any other token refuses at parse.
Scope.hosts(supported) resolves a scope against the hosts a
workspace supports: no inclusion reaches every one of them, and an
exclusion then removes its hosts, so tea@!windows-arm is every
supported host but one. The parser enforces the grammar alone; what a
site takes of it is the site's rule, refused there with the site named.
A tool requirement is a tool's name with a floor, ruff or
ruff>=0.16, and may name the hosts it applies to after @,
dotnet_coverage@windows or tea>=1.1@linux,macos-arm. The lock
takes for each tool the newest version the catalogue lists that
satisfies every floor and resolves on every host the tool is required
on, the union of its requirements' scopes and the whole lock for one
with no scope: a downloaded kind resolves on a host when it has that
host's artifact, a delegated kind everywhere its installer does. A
requirement that cannot be met refuses naming the tool, each floor
with the site that declared it, and for a host no eligible version
has, the first version that has it. The lock is toolroom.lock at the
repository root, one version per tool with the deployment digest per
host it is locked on; an entry locked on fewer hosts than the lock's
names them in on, a scope no locked host matches locks nothing, and
an entry stands while it still satisfies and resolves, moving only
when asked. The graphs of the delegated tools, one resolved graph per
tool, sit beside it in toolroom.graphs/.
The [toolroom] table¶
A directory's tool configuration is one table, [toolroom]: the
catalogue (index), the requirements (requires), the mirrors
(sources), the modes and the host allowances. A workspace the
workshop manages keeps it in workshop.toml; a directory without one
keeps it in toolroom.toml, under the same name, and
livery.toolroom.store.read_table reads it from whichever the
directory has, workshop.toml first:
The table's schema ships with the store,
livery.toolroom.store.SCHEMA_FRAGMENT, as the keys it declares in
a workspace's root contract, a package's contract and an extension's
extension.toml; the workshop composes it into every workspace's
schema, so the editor completes the table and the workshop's judge
refuses a key it does not declare.
What a load refuses¶
Loading a record resolves every host of every version and every
version's surface, and refuses a record that does not resolve whole:
an artifact without a sha256, a layer naming a host or a version the
record does not carry, a layer restating the value it inherits, a
version whose host resolves with no paths, a download that nothing
reaches (no paths, no entry point, no env value under $package), a
path directory with no entry point or an entry point with no paths, a
bare download with no file or whose entry points are not that file
alone, a version with neither an artifact nor a surface, a
surface read on no platform, a surface restating the help or a verb it
inherits, a withdrawn verb no earlier version has, an absence naming a
verb or option the version lacks or a platform that did not read it,
and a delta out of sequence. The schema of both documents is
records/record.schema.json, exported by export_schema.
The store¶
ensure lands a version's artifact through the sources in order and
through the origin URL unless the store is offline; a mismatch at any
tier is refused naming the tier, and an offline miss names
<name>@<version> and the origin. An archive is extracted and its root
hoisted, a bare download saved as its file, the excluded members removed,
the shims made, every declared entry point checked for in the tree,
which refuses naming the tool, the version, the host and the path when
one is absent, the directory collected as a tree with the entry points
as its executables and nothing else, tools/<name>@<version> moved to
it with the deployment that produced it recorded beside the ref, and
the tree viewed at the home's tool directory. A later ensure of the
same version compares the record's deployment with the one beside the
ref: a changed layout is a new tree from the same artifact, the ref
moves to it by compare-and-swap, the old view is dropped and the new
tree viewed, and the old tree is swept as unreached. The artifact
itself never changes under a version, since landing verifies it. One
archive lands one tree digest on every platform by construction: the
annotation decides the executable bit, not the modes the extractor
happened to produce. A second ensure is a probe that answers
offline. A directory the store did not make is never removed.
One process at a time supplies a version. The supply holds the
version's lock, locks/<name>@<version>.lock under the home, from its
probe to its view. A second process that asks for the same version
waits, says so through the progress (wait), and then finds the
version present. The operating system holds the lock for the open file
and lets go when the holder ends, so a holder that died leaves nothing
to clear and a waiter has no timeout.
link fills a bin directory with one link per declared entry point,
a launcher where the platform refuses a link, and removes only links
it made; delta is the one PATH prepend
and the env with $package replaced by the tool directory. fetch
lands every version's artifact for the hosts asked into a store at
another root, a mirror by construction, for an offline install
elsewhere.
supply is the primitive ensure stands on: a tool by name, kind,
version and deployment, which is what a consumer holds after reading
the catalogue. The delegated kinds go through their installer rather
than the objects. A pypi tool is installed by uv at the locked version
into a directory of its own under the home, the record's package
naming what uv installs when it differs from the tool's name, and its
launchers under bin are its entry points. An npm tool is installed
the same way through the runtime its record names, node unless it says
bun, whose executable the caller supplied first: npm run on node, or
bun's own installer. Its launchers under bin are the entry points,
they start with #!/usr/bin/env node, and the runtime's directory
stays on PATH to answer that, bun through the node shim its record
declares. A dotnet tool is installed through the dotnet SDK, the
runtime its record names, with dotnet tool install --tool-path into
a directory of its own under the home, the record's package naming
the NuGet package; the shims the SDK writes under bin are its entry
points, and they find the runtime through DOTNET_ROOT, which the
SDK's own record sets. It has no graph: a .NET tool package carries
its dependencies, and the version pins the whole. A launcher the
runtime placed outside the tool's directory
is refused naming where it points. A system-check tool is the
machine's own, found on PATH and held to the record's min_version
alone, since its locked version is the newest reading the stubs render
for and not a version anyone installs, and the store installs nothing
for it. python refuses naming the kind until it is supplied the
same way. A delegated tool has no tree.
A record's mode says how a materialised tool reaches PATH: link
puts its entry points in the checkout's bin directory, path its own
directories on PATH, none neither, for a tool reached only through a
typed handle; a download with paths takes path, one reached through
its env alone and a system tool take none, unless the record says
otherwise, and every installer's kind takes path.
A version or host the record does not carry refuses by name, naming
what it does carry. The six host keys are macos-arm, macos-x64,
linux-x64, linux-arm, windows-x64 and windows-arm.
A view is a materialisation of a tree the object store keeps, and the
store keeps a record of every view it made: the tree, the directory and
every path. Store.probe answers present from the ref and that record,
without walking the files, so a sync pays no walk. Store.audit(name,
version) walks one view and names the paths the disk lacks;
Store.audit_views() does so for every view under the home;
Store.mend_view(name, version) drops a damaged view's record and views
its tree again, fetching nothing. fm doctor runs the store's doctor,
registered under footman's footman.doctors group, which reports every
damaged view and mends it under --fix.