livery.toolroom.store._fetch¶
livery.toolroom.store._fetch ¶
Reading over HTTP, and unpacking what was read: bytes, JSON, a file, an archive.
One retrying read over livery.strongroom.fetch_url for the store and for every reader built on it. A transient failure (a connection that dropped, a timeout, a 408, a 429, a 5xx) is tried again after a pause; an answer (a 404, a 403) is final at once, since asking again does not change it. Reach for livery.toolroom.store.fetch_bytes for an index, livery.toolroom.store.fetch_json for a forge API with its token, livery.toolroom.store.fetch_file for an asset cached by name, and livery.toolroom.store.unpack to extract an archive whole into a directory.
Classes:
| Name | Description |
|---|---|
FetchError |
A URL not read after the tries; the message names it and the last cause. |
UnpackError |
An archive that will not unpack; the message names the archive and why. |
Functions:
| Name | Description |
|---|---|
api_headers |
The headers a read of url sends: a user agent, and GitHub's token for its API. |
transient |
Whether error is about the connection rather than the object. |
fetch_bytes |
The bytes at url, read whole; a transient failure is tried tries times. |
fetch_json |
The JSON at url, whatever shape the endpoint answers with. |
fetch_file |
Download url into into, named by its last path segment; the file. |
unpack |
Extract archive, a zip or a tar, whole into into. |
Attributes:
| Name | Type | Description |
|---|---|---|
USER_AGENT |
What every read says it is; GitHub and GitLab refuse a request without one. |
|
TRIES |
How many times a transient failure is tried before it is reported. |
|
BACKOFF |
Seconds between tries, multiplied by the try number. |
|
ARCHIVES |
The archive suffixes the store unpacks; the record module's list. |
|
sleep |
Callable[[float], None]
|
Waits between tries. A variable, so a test retries without the pause. |
USER_AGENT
module-attribute
¶
What every read says it is; GitHub and GitLab refuse a request without one.
TRIES
module-attribute
¶
How many times a transient failure is tried before it is reported.
ARCHIVES
module-attribute
¶
ARCHIVES = ARCHIVE_SUFFIXES
The archive suffixes the store unpacks; the record module's list.
sleep
module-attribute
¶
Waits between tries. A variable, so a test retries without the pause.
FetchError ¶
Bases: Unreachable
A URL not read after the tries; the message names it and the last cause.
Source code in packages/strongroom/src/livery/strongroom/_sources.py
UnpackError ¶
Bases: Exception
An archive that will not unpack; the message names the archive and why.
api_headers ¶
The headers a read of url sends: a user agent, and GitHub's token for its API.
The token comes from the environment (GH_TOKEN, then
GITHUB_TOKEN) so nothing here depends on a CLI: GitHub allows 60
unauthenticated API calls an hour per address and 5,000 with a
token, and a shared runner spends the smaller budget on whoever
else is on it. Only a URL on GitHub's API host gets the token;
livery.strongroom.fetch_url keeps it off any other host a
redirect reaches.
Source code in packages/toolroom-store/src/livery/toolroom/store/_fetch.py
transient ¶
transient(error: Unreachable) -> bool
Whether error is about the connection rather than the object.
A failed connection, a timeout and a chain that did not end carry no status and are worth another try; so are a 408, a 429 and any 5xx. Every other status is the source's answer.
Source code in packages/toolroom-store/src/livery/toolroom/store/_fetch.py
fetch_bytes ¶
fetch_bytes(url: str, *, headers: Mapping[str, str] | None = None, tries: int = TRIES, connect_timeout: float = 10.0, transfer_timeout: float = 600.0) -> bytes
The bytes at url, read whole; a transient failure is tried tries times.
headers default to livery.toolroom.store.api_headers for the URL.
Raises:
| Type | Description |
|---|---|
FetchError
|
when the tries are spent, or at once for an answer that says the object is not there. |
Source code in packages/toolroom-store/src/livery/toolroom/store/_fetch.py
fetch_json ¶
The JSON at url, whatever shape the endpoint answers with.
Raises:
| Type | Description |
|---|---|
FetchError
|
when the URL cannot be read, or answers with something that is not JSON. |
Source code in packages/toolroom-store/src/livery/toolroom/store/_fetch.py
fetch_file ¶
Download url into into, named by its last path segment; the file.
A file already there with bytes in it is the download, so a second ask costs nothing. A download that fails part-way leaves nothing behind: a part-written file is never a hit.
Raises:
| Type | Description |
|---|---|
FetchError
|
when the URL cannot be read after the tries. |
Source code in packages/toolroom-store/src/livery/toolroom/store/_fetch.py
unpack ¶
Extract archive, a zip or a tar, whole into into.
name is the archive's file name when the path does not carry it
(an object landed by digest); its suffix decides the format unless
format says zip or tar, for an archive whose name lies. A
zip member's mode is restored where it is set, and a tar is
extracted with the data filter, so a member cannot write outside
into.
Raises:
| Type | Description |
|---|---|
UnpackError
|
for a suffix the store does not unpack, or an archive that will not extract. |