Skip to content

livery.forge.testing._conformance

livery.forge.testing._conformance

The one conformance suite every backend must pass, the fake included.

The suite is data plus a harness seam: livery.forge.testing.SCENARIOS is the data, a tuple of named livery.forge.testing.Scenario records, and each backend supplies a livery.forge.testing.ForgeDriver so the scenarios can drive the world around the protocol (pushes, tags, CI settling) in that backend's own way. A test harness runs every applicable scenario against every driver it has; the scenarios themselves never change per backend, which is what makes passing them mean something.

The driver seam is shaped by what a real forge can actually be made to do. CI's verdict is decided by the commit that was pushed, so the scenario states the intended outcome at push time and the driver arranges it (the fake stores it; a real driver pushes a commit whose seeded workflow produces it). Blocking lives only in the driver: settle and await_run return immediately on the fake and poll on a real forge, and no scenario ever sleeps.

Scenarios assert the contract and only the contract: where forges may legitimately differ, the scenario either probes livery.forge.Forge.supports or does not look.

Classes:

Name Description
ForgeDriver

What a backend's test rig supplies to run the shared scenarios.

Scenario

One conformance case: a name, its capability gates, and its body.

Attributes:

Name Type Description
Outcome TypeAlias

What CI will do with a pushed commit.

SCENARIOS tuple[Scenario, ...]

Every conformance scenario, in the order the groups are documented.

Outcome module-attribute

Outcome: TypeAlias = Literal['success', 'failure', 'hang', 'skipped']

What CI will do with a pushed commit.

success and failure are verdicts the run reaches once settled. hang is a held run: it stays live until the scenario either cancels it or settles it, and settling releases it to success. The held run is what makes the pending state, cancellation, and arming deterministic on every backend, however fast its runners are.

SCENARIOS module-attribute

SCENARIOS: tuple[Scenario, ...] = (Scenario('identity', _identity), Scenario('repo-lifecycle', _repo_lifecycle), Scenario('branches', _branches), Scenario('tags', _tags), Scenario('addresses', _addresses), Scenario('governance-reads', _governance_reads), Scenario('configure-is-idempotent', _configure_is_idempotent), Scenario('configure-required-contexts', _configure_required_contexts, requires=('required_contexts',)), Scenario('configure-required-contexts-declined', _configure_required_contexts_declined, forbids=('required_contexts',)), Scenario('configure-secrets', _configure_secrets, requires=('ci_secrets',)), Scenario('configure-secrets-declined', _configure_secrets_declined, forbids=('ci_secrets',)), Scenario('pr-open-find-get', _pr_open_find_get), Scenario('pr-list', _pr_list), Scenario('pr-title-close-reopen', _pr_title_close_reopen), Scenario('pr-merge-now', _pr_merge_now), Scenario('pr-comment', _pr_comment), Scenario('pr-labels', _pr_labels), Scenario('arm-disarm', _arm_disarm, requires=('auto_merge',)), Scenario('arm-long-title', _arm_long_title, requires=('auto_merge',)), Scenario('armed-pr-merges-on-green', _armed_pr_merges_on_green, requires=('auto_merge',)), Scenario('status-progression', _status_progression), Scenario('runs-jobs-log', _runs_jobs_log), Scenario('rerun', _rerun), Scenario('cancel-run', _cancel_run), Scenario('cancel-run-forced', _cancel_run_forced, requires=('force_cancel',)), Scenario('cancel-run-force-declined', _cancel_run_force_declined, forbids=('force_cancel',)), Scenario('dispatch', _dispatch), Scenario('schedules', _schedules, requires=('pipeline_schedules',)), Scenario('schedules-declined', _schedules_declined, forbids=('pipeline_schedules',)), Scenario('release-by-tag', _release_by_tag), Scenario('release-assets', _release_assets), Scenario('issue-text-both-ways', _issue_text_both_ways), Scenario('issue-update', _issue_update), Scenario('issue-list-and-search', _issue_list_and_search), Scenario('issue-assign', _issue_assign), Scenario('issue-comment', _issue_comment), Scenario('issue-close', _issue_close), Scenario('issue-reopen', _issue_reopen), Scenario('governance-listings', _governance_listings), Scenario('configure-approvals', _configure_approvals, requires=('min_approvals',)), Scenario('approvals-declined', _approvals_declined, forbids=('min_approvals',)))

Every conformance scenario, in the order the groups are documented.

ForgeDriver

Bases: Protocol

What a backend's test rig supplies to run the shared scenarios.

The protocol under test only observes the forge; the driver moves the world the protocol observes. The fake moves dictionaries; a real driver creates commits, seeds workflow files, and polls. The scenarios cannot tell the difference, which is the point.

Methods:

Name Description
unused_repo_name

An owner and name no repository has yet.

fresh_repo

A new repository with an initialised default branch.

push

Push branch (creating it when new) and return its head sha.

create_tag

Create tag at the default branch's head.

settle

Return once every run for sha is terminal.

await_run

The one run matching the filters, once it exists.

comment_bodies

The comments on one pull request or issue, oldest first.

await_mergeable

Return once the forge has computed the pull request's mergeability.

await_merged

Return once the merge of number and its aftermath have landed.

await_branch

Return once branch's existence matches present.

await_issue

Return once listings serve issue number in its current state.

required_context

How this forge spells the seeded workflow's check context.

Attributes:

Name Type Description
forge Forge

The forge under test.

forge property

forge: Forge

The forge under test.

unused_repo_name

unused_repo_name() -> tuple[str, str]

An owner and name no repository has yet.

Source code in packages/forge/src/livery/forge/testing/_conformance.py
def unused_repo_name(self) -> tuple[str, str]:
    """An owner and name no repository has yet."""
    ...

fresh_repo

fresh_repo() -> Repository

A new repository with an initialised default branch.

The repository carries a dispatchable CI workflow the scenarios address as conf.yml, however the backend spells that underneath.

Source code in packages/forge/src/livery/forge/testing/_conformance.py
def fresh_repo(self) -> Repository:
    """A new repository with an initialised default branch.

    The repository carries a dispatchable CI workflow the
    scenarios address as ``conf.yml``, however the backend spells
    that underneath.
    """
    ...

push

push(repo_owner: str, repo_name: str, branch: str, *, outcome: Outcome = 'success') -> str

Push branch (creating it when new) and return its head sha.

CI starts on the pushed commit and will reach outcome once settled; a hang outcome stays live until cancelled. A real driver arranges this with the workflow it seeds (the branch or commit carries the outcome); the fake stores it on the run.

Source code in packages/forge/src/livery/forge/testing/_conformance.py
def push(
    self,
    repo_owner: str,
    repo_name: str,
    branch: str,
    *,
    outcome: Outcome = "success",
) -> str:
    """Push *branch* (creating it when new) and return its head sha.

    CI starts on the pushed commit and will reach *outcome* once
    settled; a ``hang`` outcome stays live until cancelled. A real
    driver arranges this with the workflow it seeds (the branch or
    commit carries the outcome); the fake stores it on the run.
    """
    ...

create_tag

create_tag(repo_owner: str, repo_name: str, tag: str) -> None

Create tag at the default branch's head.

Source code in packages/forge/src/livery/forge/testing/_conformance.py
def create_tag(self, repo_owner: str, repo_name: str, tag: str) -> None:
    """Create *tag* at the default branch's head."""
    ...

settle

settle(repo_owner: str, repo_name: str, sha: str) -> None

Return once every run for sha is terminal.

A held (hang) run is released and concludes success; the other outcomes conclude as pushed. The fake applies the outcome; a real driver releases its hold and polls.

Source code in packages/forge/src/livery/forge/testing/_conformance.py
def settle(self, repo_owner: str, repo_name: str, sha: str) -> None:
    """Return once every run for *sha* is terminal.

    A held (``hang``) run is released and concludes success; the
    other outcomes conclude as pushed. The fake applies the
    outcome; a real driver releases its hold and polls.
    """
    ...

await_run

await_run(repo_owner: str, repo_name: str, *, head_sha: str = '', event: str = '') -> Run

The one run matching the filters, once it exists.

Runs appear asynchronously on a real forge; this blocks until the matching run is listed. Exactly one run must match, and the run comes back as the listing named it.

Source code in packages/forge/src/livery/forge/testing/_conformance.py
def await_run(
    self, repo_owner: str, repo_name: str, *, head_sha: str = "", event: str = ""
) -> Run:
    """The one run matching the filters, once it exists.

    Runs appear asynchronously on a real forge; this blocks until
    the matching run is listed. Exactly one run must match, and
    the run comes back as the listing named it.
    """
    ...

comment_bodies

comment_bodies(repo_owner: str, repo_name: str, number: int, *, kind: Literal['pr', 'issue']) -> tuple[str, ...]

The comments on one pull request or issue, oldest first.

Source code in packages/forge/src/livery/forge/testing/_conformance.py
def comment_bodies(
    self,
    repo_owner: str,
    repo_name: str,
    number: int,
    *,
    kind: Literal["pr", "issue"],
) -> tuple[str, ...]:
    """The comments on one pull request or issue, oldest first."""
    ...

await_mergeable

await_mergeable(repo_owner: str, repo_name: str, number: int) -> None

Return once the forge has computed the pull request's mergeability.

A forge recomputes mergeability asynchronously after a pull request opens, and merging or arming inside that window is refused with a 405. Scenarios wait here before either verb; workflows retry on the 405 instead, which is the same contract from the other side.

Source code in packages/forge/src/livery/forge/testing/_conformance.py
def await_mergeable(self, repo_owner: str, repo_name: str, number: int) -> None:
    """Return once the forge has computed the pull request's mergeability.

    A forge recomputes mergeability asynchronously after a pull
    request opens, and merging or arming inside that window is
    refused with a 405. Scenarios wait here before either verb;
    workflows retry on the 405 instead, which is the same
    contract from the other side.
    """
    ...

await_merged

await_merged(repo_owner: str, repo_name: str, number: int) -> None

Return once the merge of number and its aftermath have landed.

A scheduled merge is performed by the forge after the checks go green, and even an immediate merge finishes its aftermath (the configured source branch deletion) asynchronously; the caller only ever observes both. This is the driver-side bound on that observation.

Source code in packages/forge/src/livery/forge/testing/_conformance.py
def await_merged(self, repo_owner: str, repo_name: str, number: int) -> None:
    """Return once the merge of *number* and its aftermath have landed.

    A scheduled merge is performed by the forge after the checks
    go green, and even an immediate merge finishes its aftermath
    (the configured source branch deletion) asynchronously; the
    caller only ever observes both. This is the driver-side bound
    on that observation.
    """
    ...

await_branch

await_branch(repo_owner: str, repo_name: str, branch: str, *, present: bool) -> None

Return once branch's existence matches present.

Branch reads are eventually consistent on github.com (a deleted branch answers for seconds afterwards); the fake is immediate, a real driver polls.

Source code in packages/forge/src/livery/forge/testing/_conformance.py
def await_branch(
    self, repo_owner: str, repo_name: str, branch: str, *, present: bool
) -> None:
    """Return once *branch*'s existence matches *present*.

    Branch reads are eventually consistent on github.com (a
    deleted branch answers for seconds afterwards); the fake is
    immediate, a real driver polls.
    """
    ...

await_issue

await_issue(repo_owner: str, repo_name: str, number: int, *, assignee: str = '') -> None

Return once listings serve issue number in its current state.

Issue listings are eventually consistent on some forges (about nine seconds behind on github.com), so a scenario that creates or assigns and then reads through a listing waits here first. When assignee is given, the served row must carry it.

Source code in packages/forge/src/livery/forge/testing/_conformance.py
def await_issue(
    self, repo_owner: str, repo_name: str, number: int, *, assignee: str = ""
) -> None:
    """Return once listings serve issue *number* in its current state.

    Issue listings are eventually consistent on some forges (about
    nine seconds behind on github.com), so a scenario that creates
    or assigns and then reads through a listing waits here first.
    When *assignee* is given, the served row must carry it.
    """
    ...

required_context

required_context() -> str

How this forge spells the seeded workflow's check context.

The string branch protection must name for the gate job: forges spell it differently (a bare job name, or workflow / job (event)), and the spelling is rig knowledge.

Source code in packages/forge/src/livery/forge/testing/_conformance.py
def required_context(self) -> str:
    """How this forge spells the seeded workflow's check context.

    The string branch protection must name for the gate job:
    forges spell it differently (a bare job name, or
    workflow / job (event)), and the spelling is rig knowledge.
    """
    ...

Scenario dataclass

Scenario(name: str, run: Callable[[ForgeDriver], None], requires: tuple[Capability, ...] = (), forbids: tuple[Capability, ...] = ())

One conformance case: a name, its capability gates, and its body.

Attributes:

Name Type Description
name str

The scenario's stable identifier, used as the test id.

run Callable[[ForgeDriver], None]

The body. Raises AssertionError when the backend breaks the contract.

requires tuple[Capability, ...]

Capabilities the forge must support for the scenario to apply.

forbids tuple[Capability, ...]

Capabilities the forge must NOT support: the scenario asserts the honest decline.

Methods:

Name Description
applies_to

Whether forge's capabilities put this scenario in scope.

name instance-attribute

name: str

run instance-attribute

run: Callable[[ForgeDriver], None]

requires class-attribute instance-attribute

requires: tuple[Capability, ...] = ()

forbids class-attribute instance-attribute

forbids: tuple[Capability, ...] = ()

applies_to

applies_to(forge: Forge) -> bool

Whether forge's capabilities put this scenario in scope.

Source code in packages/forge/src/livery/forge/testing/_conformance.py
def applies_to(self, forge: Forge) -> bool:
    """Whether *forge*'s capabilities put this scenario in scope."""
    return all(forge.supports(c) for c in self.requires) and not any(
        forge.supports(c) for c in self.forbids
    )