Skip to content

livery.forge._http

livery.forge._http

The HTTP plumbing every REST backend shares.

One JSON client over urllib: authenticated requests, refusal of redirects, error translation into livery.forge.ForgeError with the server's words attached, and pagination that is complete or raises. The network seam is the opener, anything satisfying livery.forge.testing.UrlOpener, so a backend records and replays its traffic by construction argument alone.

Classes:

Name Description
Opener

The network seam: urllib's opener shape, replayable in tests.

JsonClient

An authenticated JSON API client for one server.

Functions:

Name Description
default_opener

The opener a backend uses when none is injected.

spent

Whether a refusal says the API budget is spent, not that access is denied.

reset_time

When the budget returns, as seconds since the epoch; None when unsaid.

read_budget

The budget a response reports, or None when it reports none.

limit_message

The refusal for a spent budget, naming when it returns.

Attributes:

Name Type Description
PAGE_SIZE
PAGE_CAP
REDIRECT_HOPS

PAGE_SIZE module-attribute

PAGE_SIZE = 50

PAGE_CAP module-attribute

PAGE_CAP = 40

REDIRECT_HOPS module-attribute

REDIRECT_HOPS = 5

Opener

Bases: Protocol

The network seam: urllib's opener shape, replayable in tests.

Methods:

Name Description
open

Open request and return a urllib response object.

open

open(request: Request, /, *, timeout: float = 30.0) -> Any

Open request and return a urllib response object.

Source code in packages/forge/src/livery/forge/_http.py
def open(self, request: urllib.request.Request, /, *, timeout: float = 30.0) -> Any:
    """Open *request* and return a urllib response object."""
    ...

JsonClient

JsonClient(api_base: str, *, headers: dict[str, str], opener: Opener | None = None, timeout: float = 30)

An authenticated JSON API client for one server.

Parameters:

Name Type Description Default
api_base str

The API root, no trailing slash.

required
headers dict[str, str]

The authentication headers, sent on every request. An empty mapping reads anonymously.

required
opener Opener | None

The network seam; the redirect-refusing default when omitted.

None

Bind the client to api_base with headers.

timeout bounds each request unless a call overrides it: a heavyweight server (a single-node GitLab under load) needs more headroom than a light one, and the backend knows which it is talking to.

Methods:

Name Description
request

Make one request; the parsed JSON answer, or None per none_on.

text

GET endpoint as plain text (a job log), or None per none_on.

upload

POST data as one raw body to endpoint; the parsed JSON answer.

multipart

POST data as the file part field of a multipart form; the JSON answer.

download

GET endpoint and return its bytes, redirects followed.

paginate

Collect every page from fetch, or raise on an unfinished walk.

pages

Yield the items from fetch page by page, fetching as they are read.

Attributes:

Name Type Description
api_base
budget RateBudget | None
Source code in packages/forge/src/livery/forge/_http.py
def __init__(
    self,
    api_base: str,
    *,
    headers: dict[str, str],
    opener: Opener | None = None,
    timeout: float = 30,
) -> None:
    """Bind the client to *api_base* with *headers*.

    *timeout* bounds each request unless a call overrides it: a
    heavyweight server (a single-node GitLab under load) needs
    more headroom than a light one, and the backend knows which
    it is talking to.
    """
    self.api_base = api_base.rstrip("/")
    self._headers = headers
    self._opener = opener if opener is not None else default_opener()
    self._timeout = timeout
    #: The API budget the server reported on its last response; None
    #: before any, or from a server that reports none.
    self.budget: RateBudget | None = None

api_base instance-attribute

api_base = api_base.rstrip('/')

budget instance-attribute

budget: RateBudget | None = None

request

request(endpoint: str, *, method: str = 'GET', data: dict[str, Any] | None = None, none_on: tuple[int, ...] = (), timeout: float | None = None) -> Any

Make one request; the parsed JSON answer, or None per none_on.

Returns the parsed JSON body ({} for an empty one). Statuses in none_on return None instead of raising; every other HTTP error raises livery.forge.ForgeError with the status as a number and the body verbatim in detail. An unreachable server raises the same type with no status. A redirect is refused rather than followed and raises with the 3xx status.

data is sent as a JSON body whenever it is not None, so an empty mapping means "an empty object", not "no body".

Source code in packages/forge/src/livery/forge/_http.py
def request(
    self,
    endpoint: str,
    *,
    method: str = "GET",
    data: dict[str, Any] | None = None,
    none_on: tuple[int, ...] = (),
    timeout: float | None = None,
) -> Any:
    """Make one request; the parsed JSON answer, or None per *none_on*.

    Returns the parsed JSON body ({} for an empty one). Statuses in
    *none_on* return None instead of raising; every other HTTP
    error raises livery.forge.ForgeError with the status as a
    number and the body verbatim in ``detail``. An unreachable
    server raises the same type with no status. A redirect is
    refused rather than followed and raises with the 3xx status.

    *data* is sent as a JSON body whenever it is not None, so an
    empty mapping means "an empty object", not "no body".
    """
    raw = self._raw(
        endpoint,
        method=method,
        data=data,
        none_on=none_on,
        timeout=self._timeout if timeout is None else timeout,
    )
    if raw is None:
        return None
    # strict=False: user-authored fields (PR bodies) have been
    # observed carrying literal control characters, which the
    # default strict decoder rejects mid-poll.
    return json.loads(raw, strict=False) if raw else {}

text

text(endpoint: str, *, none_on: tuple[int, ...] = (), timeout: float | None = None) -> str | None

GET endpoint as plain text (a job log), or None per none_on.

Source code in packages/forge/src/livery/forge/_http.py
def text(
    self,
    endpoint: str,
    *,
    none_on: tuple[int, ...] = (),
    timeout: float | None = None,
) -> str | None:
    """GET *endpoint* as plain text (a job log), or None per *none_on*."""
    raw = self._raw(
        endpoint,
        method="GET",
        data=None,
        none_on=none_on,
        timeout=self._timeout if timeout is None else timeout,
    )
    return None if raw is None else raw.decode("utf-8", errors="replace")

upload

upload(endpoint: str, data: bytes, *, content_type: str = 'application/octet-stream', timeout: float | None = None) -> Any

POST data as one raw body to endpoint; the parsed JSON answer.

endpoint is relative to the API root, or absolute when it starts with http: GitHub takes uploads on a host of its own. Every error raises as request does.

Source code in packages/forge/src/livery/forge/_http.py
def upload(
    self,
    endpoint: str,
    data: bytes,
    *,
    content_type: str = "application/octet-stream",
    timeout: float | None = None,
) -> Any:
    """POST *data* as one raw body to *endpoint*; the parsed JSON answer.

    *endpoint* is relative to the API root, or absolute when it
    starts with ``http``: GitHub takes uploads on a host of its
    own. Every error raises as `request` does.
    """
    raw = self._raw(
        endpoint,
        method="POST",
        data=None,
        none_on=(),
        timeout=self._timeout if timeout is None else timeout,
        raw_body=data,
        content_type=content_type,
    )
    return json.loads(raw, strict=False) if raw else {}

multipart

multipart(endpoint: str, field: str, filename: str, data: bytes, *, timeout: float | None = None) -> Any

POST data as the file part field of a multipart form; the JSON answer.

Source code in packages/forge/src/livery/forge/_http.py
def multipart(
    self,
    endpoint: str,
    field: str,
    filename: str,
    data: bytes,
    *,
    timeout: float | None = None,
) -> Any:
    """POST *data* as the file part *field* of a multipart form; the JSON answer."""
    boundary = f"----livery-forge-{uuid.uuid4().hex}"
    body = b"".join(
        (
            f"--{boundary}\r\n".encode(),
            f'Content-Disposition: form-data; name="{field}";'
            f' filename="{filename}"\r\n'.encode(),
            b"Content-Type: application/octet-stream\r\n\r\n",
            data,
            f"\r\n--{boundary}--\r\n".encode(),
        )
    )
    raw = self._raw(
        endpoint,
        method="POST",
        data=None,
        none_on=(),
        timeout=self._timeout if timeout is None else timeout,
        raw_body=body,
        content_type=f"multipart/form-data; boundary={boundary}",
    )
    return json.loads(raw, strict=False) if raw else {}

download

download(endpoint: str, *, accept: str = '') -> bytes

GET endpoint and return its bytes, redirects followed.

The one read that must follow a redirect: a forge hands the bytes of a release asset to a signed URL on a store of its own. The credential rides the first hop alone. A hop to another host drops it, because the signed URL carries its own authorisation and a second one is rejected by the store.

accept sets the Accept header, which is how GitHub is asked for the bytes of an asset rather than its JSON.

Raises:

Type Description
ForgeError

for any HTTP error, an unreachable server, or a redirect chain longer than livery.forge._http.REDIRECT_HOPS.

Source code in packages/forge/src/livery/forge/_http.py
def download(self, endpoint: str, *, accept: str = "") -> bytes:
    """GET *endpoint* and return its bytes, redirects followed.

    The one read that must follow a redirect: a forge hands the
    bytes of a release asset to a signed URL on a store of its
    own. The credential rides the first hop alone. A hop to
    another host drops it, because the signed URL carries its own
    authorisation and a second one is rejected by the store.

    *accept* sets the Accept header, which is how GitHub is asked
    for the bytes of an asset rather than its JSON.

    Raises:
        ForgeError: for any HTTP error, an unreachable server, or
            a redirect chain longer than
            `livery.forge._http.REDIRECT_HOPS`.
    """
    url = endpoint if endpoint.startswith("http") else f"{self.api_base}{endpoint}"
    headers = dict(self._headers)
    if accept:
        headers["Accept"] = accept
    for _hop in range(REDIRECT_HOPS):
        request = urllib.request.Request(url, method="GET", headers=headers)
        try:
            response = self._opener.open(request, timeout=self._timeout)
        except urllib.error.HTTPError as exc:
            location = exc.headers.get("Location", "") if exc.headers else ""
            if 300 <= exc.code < 400 and location:
                target = urllib.parse.urljoin(url, location)
                if (
                    urllib.parse.urlsplit(target).netloc
                    != urllib.parse.urlsplit(url).netloc
                ):
                    headers = {"Accept": accept} if accept else {}
                url = target
                continue
            detail = exc.read().decode(errors="replace")
            raise ForgeError(
                f"HTTP {exc.code} on GET {endpoint}: {detail}",
                status=exc.code,
                method="GET",
                endpoint=endpoint,
                detail=detail,
            ) from exc
        except (urllib.error.URLError, OSError, http.client.HTTPException) as exc:
            raise ForgeError(
                f"server unreachable on GET {endpoint}: {exc}",
                method="GET",
                endpoint=endpoint,
            ) from exc
        read: bytes = response.read()
        return read
    raise ForgeError(
        f"GET {endpoint} redirected more than {REDIRECT_HOPS} times",
        method="GET",
        endpoint=endpoint,
    )

paginate

paginate(fetch: Callable[[int], list[Any]], *, subject: str, size: int = PAGE_SIZE) -> list[Any]

Collect every page from fetch, or raise on an unfinished walk.

fetch takes a 1-based page number and returns that page's items; a batch shorter than size, the page size fetch asks for, ends the walk. Hitting livery.forge._http.PAGE_CAP with a full last page raises livery.forge.ForgeError naming subject: the listing is not complete, and a prefix is never the answer.

Source code in packages/forge/src/livery/forge/_http.py
def paginate(
    self,
    fetch: Callable[[int], list[Any]],
    *,
    subject: str,
    size: int = PAGE_SIZE,
) -> list[Any]:
    """Collect every page from *fetch*, or raise on an unfinished walk.

    *fetch* takes a 1-based page number and returns that page's
    items; a batch shorter than *size*, the page size *fetch* asks
    for, ends the walk. Hitting livery.forge._http.PAGE_CAP with a
    full last page raises livery.forge.ForgeError naming *subject*:
    the listing is not complete, and a prefix is never the answer.
    """
    return list(self.pages(fetch, subject=subject, size=size))

pages

pages(fetch: Callable[[int], list[Any]], *, subject: str, size: int = PAGE_SIZE) -> Iterator[Any]

Yield the items from fetch page by page, fetching as they are read.

The walk livery.forge._http.JsonClient.paginate collects, for a caller that stops at its first match: no page after the one holding it is fetched. size is the page size fetch asks for, and a shorter page is the last. The walk raises as paginate does, and only when a caller reads past livery.forge._http.PAGE_CAP full pages.

Source code in packages/forge/src/livery/forge/_http.py
def pages(
    self,
    fetch: Callable[[int], list[Any]],
    *,
    subject: str,
    size: int = PAGE_SIZE,
) -> Iterator[Any]:
    """Yield the items from *fetch* page by page, fetching as they are read.

    The walk [livery.forge._http.JsonClient.paginate][] collects,
    for a caller that stops at its first match: no page after the
    one holding it is fetched. *size* is the page size *fetch* asks
    for, and a shorter page is the last. The walk raises as
    ``paginate`` does, and only when a caller reads past
    [livery.forge._http.PAGE_CAP][] full pages.
    """
    page = 1
    while page <= PAGE_CAP:
        batch = fetch(page)
        yield from batch
        if len(batch) < size:
            return
        page += 1
    raise ForgeError(
        f"scanned {PAGE_CAP * PAGE_SIZE} entries of {subject} and the"
        " server has more: reporting a truncated listing would be a"
        " guess. Narrow the query."
    )

default_opener

default_opener() -> Opener

The opener a backend uses when none is injected.

Keeps every default handler (proxies included) except the redirect one, which livery.forge._http._RefuseRedirect replaces.

Source code in packages/forge/src/livery/forge/_http.py
def default_opener() -> Opener:
    """The opener a backend uses when none is injected.

    Keeps every default handler (proxies included) except the redirect
    one, which livery.forge._http._RefuseRedirect replaces.
    """
    return urllib.request.build_opener(_RefuseRedirect())

spent

spent(status: int, headers: Message | None) -> bool

Whether a refusal says the API budget is spent, not that access is denied.

A 429 always is. A 403 is when the primary budget reads zero (GitHub's X-RateLimit-Remaining: 0) or the server asks the caller to come back later (Retry-After, GitHub's secondary limit); any other 403 is a permission the token lacks.

Source code in packages/forge/src/livery/forge/_http.py
def spent(status: int, headers: Message | None) -> bool:
    """Whether a refusal says the API budget is spent, not that access is denied.

    A 429 always is. A 403 is when the primary budget reads zero
    (GitHub's ``X-RateLimit-Remaining: 0``) or the server asks the
    caller to come back later (``Retry-After``, GitHub's secondary
    limit); any other 403 is a permission the token lacks.
    """
    if status == 429:
        return True
    if status != 403:
        return False
    remaining = _header(headers, "X-RateLimit-Remaining", "RateLimit-Remaining")
    return remaining == "0" or bool(_header(headers, "Retry-After"))

reset_time

reset_time(headers: Message | None, *, now: float) -> float | None

When the budget returns, as seconds since the epoch; None when unsaid.

Retry-After wins, in seconds or as an HTTP date; else the reset the budget headers name (GitHub's X-RateLimit-Reset, GitLab's RateLimit-Reset), which are epoch seconds.

Source code in packages/forge/src/livery/forge/_http.py
def reset_time(headers: Message | None, *, now: float) -> float | None:
    """When the budget returns, as seconds since the epoch; None when unsaid.

    ``Retry-After`` wins, in seconds or as an HTTP date; else the reset
    the budget headers name (GitHub's ``X-RateLimit-Reset``, GitLab's
    ``RateLimit-Reset``), which are epoch seconds.
    """
    retry = _header(headers, "Retry-After")
    if retry:
        if retry.isdigit():
            return now + int(retry)
        try:
            return email.utils.parsedate_to_datetime(retry).timestamp()
        except (TypeError, ValueError):
            pass
    reset = _header(headers, "X-RateLimit-Reset", "RateLimit-Reset")
    if reset.isdigit():
        return float(reset)
    return None

read_budget

read_budget(headers: Message | None) -> RateBudget | None

The budget a response reports, or None when it reports none.

Source code in packages/forge/src/livery/forge/_http.py
def read_budget(headers: Message | None) -> RateBudget | None:
    """The budget a response reports, or None when it reports none."""
    remaining = _header(headers, "X-RateLimit-Remaining", "RateLimit-Remaining")
    limit = _header(headers, "X-RateLimit-Limit", "RateLimit-Limit")
    if not (remaining.isdigit() and limit.isdigit()):
        return None
    return RateBudget(
        remaining=int(remaining),
        limit=int(limit),
        reset_at=reset_time(headers, now=time.time()),
    )

limit_message

limit_message(status: int, method: str, endpoint: str, reset_at: float | None, now: float) -> str

The refusal for a spent budget, naming when it returns.

Source code in packages/forge/src/livery/forge/_http.py
def limit_message(
    status: int, method: str, endpoint: str, reset_at: float | None, now: float
) -> str:
    """The refusal for a spent budget, naming when it returns."""
    if reset_at is None:
        when = "the forge did not say when it resets"
    else:
        clock = time.strftime("%H:%M", time.localtime(reset_at))
        minutes = max(0, math.ceil((reset_at - now) / 60))
        when = f"it resets at {clock}, in {minutes} min"
    return (
        f"HTTP {status} on {method} {endpoint}: the forge's API rate limit is"
        f" spent; {when}"
    )