livery.forge.testing._conformance¶
livery.forge.testing._conformance ¶
The one conformance suite every backend must pass, the fake included.
The suite is data plus a harness seam: livery.forge.testing.SCENARIOS is the data, a tuple of named livery.forge.testing.Scenario records, and each backend supplies a livery.forge.testing.ForgeDriver so the scenarios can drive the world around the protocol (pushes, tags, CI settling) in that backend's own way. A test harness runs every applicable scenario against every driver it has; the scenarios themselves never change per backend, which is what makes passing them mean something.
The driver seam is shaped by what a real forge can actually be made to
do. CI's verdict is decided by the commit that was pushed, so the
scenario states the intended outcome at push time and the driver
arranges it (the fake stores it; a real driver pushes a commit whose
seeded workflow produces it). Blocking lives only in the driver:
settle and await_run return immediately on the fake and poll
on a real forge, and no scenario ever sleeps.
Scenarios assert the contract and only the contract: where forges may legitimately differ, the scenario either probes livery.forge.Forge.supports or does not look.
Classes:
| Name | Description |
|---|---|
ForgeDriver |
What a backend's test rig supplies to run the shared scenarios. |
Scenario |
One conformance case: a name, its capability gates, and its body. |
Attributes:
| Name | Type | Description |
|---|---|---|
Outcome |
TypeAlias
|
What CI will do with a pushed commit. |
SCENARIOS |
tuple[Scenario, ...]
|
Every conformance scenario, in the order the groups are documented. |
Outcome
module-attribute
¶
What CI will do with a pushed commit.
success and failure are verdicts the run reaches once settled.
hang is a held run: it stays live until the scenario either
cancels it or settles it, and settling releases it to success. The
held run is what makes the pending state, cancellation, and arming
deterministic on every backend, however fast its runners are.
SCENARIOS
module-attribute
¶
SCENARIOS: tuple[Scenario, ...] = (Scenario('identity', _identity), Scenario('repo-lifecycle', _repo_lifecycle), Scenario('branches', _branches), Scenario('tags', _tags), Scenario('addresses', _addresses), Scenario('governance-reads', _governance_reads), Scenario('configure-is-idempotent', _configure_is_idempotent), Scenario('configure-required-contexts', _configure_required_contexts, requires=('required_contexts',)), Scenario('configure-required-contexts-declined', _configure_required_contexts_declined, forbids=('required_contexts',)), Scenario('configure-secrets', _configure_secrets, requires=('ci_secrets',)), Scenario('configure-secrets-declined', _configure_secrets_declined, forbids=('ci_secrets',)), Scenario('pr-open-find-get', _pr_open_find_get), Scenario('pr-list', _pr_list), Scenario('pr-title-close-reopen', _pr_title_close_reopen), Scenario('pr-merge-now', _pr_merge_now), Scenario('pr-comment', _pr_comment), Scenario('pr-labels', _pr_labels), Scenario('arm-disarm', _arm_disarm, requires=('auto_merge',)), Scenario('arm-long-title', _arm_long_title, requires=('auto_merge',)), Scenario('armed-pr-merges-on-green', _armed_pr_merges_on_green, requires=('auto_merge',)), Scenario('status-progression', _status_progression), Scenario('runs-jobs-log', _runs_jobs_log), Scenario('rerun', _rerun), Scenario('cancel-run', _cancel_run), Scenario('cancel-run-forced', _cancel_run_forced, requires=('force_cancel',)), Scenario('cancel-run-force-declined', _cancel_run_force_declined, forbids=('force_cancel',)), Scenario('dispatch', _dispatch), Scenario('schedules', _schedules, requires=('pipeline_schedules',)), Scenario('schedules-declined', _schedules_declined, forbids=('pipeline_schedules',)), Scenario('release-by-tag', _release_by_tag), Scenario('release-assets', _release_assets), Scenario('issue-text-both-ways', _issue_text_both_ways), Scenario('issue-update', _issue_update), Scenario('issue-list-and-search', _issue_list_and_search), Scenario('issue-assign', _issue_assign), Scenario('issue-comment', _issue_comment), Scenario('issue-close', _issue_close), Scenario('issue-reopen', _issue_reopen), Scenario('governance-listings', _governance_listings), Scenario('configure-approvals', _configure_approvals, requires=('min_approvals',)), Scenario('approvals-declined', _approvals_declined, forbids=('min_approvals',)))
Every conformance scenario, in the order the groups are documented.
ForgeDriver ¶
Bases: Protocol
What a backend's test rig supplies to run the shared scenarios.
The protocol under test only observes the forge; the driver moves the world the protocol observes. The fake moves dictionaries; a real driver creates commits, seeds workflow files, and polls. The scenarios cannot tell the difference, which is the point.
Methods:
| Name | Description |
|---|---|
unused_repo_name |
An owner and name no repository has yet. |
fresh_repo |
A new repository with an initialised default branch. |
push |
Push branch (creating it when new) and return its head sha. |
create_tag |
Create tag at the default branch's head. |
settle |
Return once every run for sha is terminal. |
await_run |
The one run matching the filters, once it exists. |
comment_bodies |
The comments on one pull request or issue, oldest first. |
await_mergeable |
Return once the forge has computed the pull request's mergeability. |
await_merged |
Return once the merge of number and its aftermath have landed. |
await_branch |
Return once branch's existence matches present. |
await_issue |
Return once listings serve issue number in its current state. |
required_context |
How this forge spells the seeded workflow's check context. |
Attributes:
| Name | Type | Description |
|---|---|---|
forge |
Forge
|
The forge under test. |
unused_repo_name ¶
fresh_repo ¶
fresh_repo() -> Repository
A new repository with an initialised default branch.
The repository carries a dispatchable CI workflow the
scenarios address as conf.yml, however the backend spells
that underneath.
Source code in packages/forge/src/livery/forge/testing/_conformance.py
push ¶
Push branch (creating it when new) and return its head sha.
CI starts on the pushed commit and will reach outcome once
settled; a hang outcome stays live until cancelled. A real
driver arranges this with the workflow it seeds (the branch or
commit carries the outcome); the fake stores it on the run.
Source code in packages/forge/src/livery/forge/testing/_conformance.py
create_tag ¶
settle ¶
Return once every run for sha is terminal.
A held (hang) run is released and concludes success; the
other outcomes conclude as pushed. The fake applies the
outcome; a real driver releases its hold and polls.
Source code in packages/forge/src/livery/forge/testing/_conformance.py
await_run ¶
The one run matching the filters, once it exists.
Runs appear asynchronously on a real forge; this blocks until the matching run is listed. Exactly one run must match, and the run comes back as the listing named it.
Source code in packages/forge/src/livery/forge/testing/_conformance.py
comment_bodies ¶
comment_bodies(repo_owner: str, repo_name: str, number: int, *, kind: Literal['pr', 'issue']) -> tuple[str, ...]
The comments on one pull request or issue, oldest first.
Source code in packages/forge/src/livery/forge/testing/_conformance.py
await_mergeable ¶
Return once the forge has computed the pull request's mergeability.
A forge recomputes mergeability asynchronously after a pull request opens, and merging or arming inside that window is refused with a 405. Scenarios wait here before either verb; workflows retry on the 405 instead, which is the same contract from the other side.
Source code in packages/forge/src/livery/forge/testing/_conformance.py
await_merged ¶
Return once the merge of number and its aftermath have landed.
A scheduled merge is performed by the forge after the checks go green, and even an immediate merge finishes its aftermath (the configured source branch deletion) asynchronously; the caller only ever observes both. This is the driver-side bound on that observation.
Source code in packages/forge/src/livery/forge/testing/_conformance.py
await_branch ¶
Return once branch's existence matches present.
Branch reads are eventually consistent on github.com (a deleted branch answers for seconds afterwards); the fake is immediate, a real driver polls.
Source code in packages/forge/src/livery/forge/testing/_conformance.py
await_issue ¶
Return once listings serve issue number in its current state.
Issue listings are eventually consistent on some forges (about nine seconds behind on github.com), so a scenario that creates or assigns and then reads through a listing waits here first. When assignee is given, the served row must carry it.
Source code in packages/forge/src/livery/forge/testing/_conformance.py
required_context ¶
required_context() -> str
How this forge spells the seeded workflow's check context.
The string branch protection must name for the gate job: forges spell it differently (a bare job name, or workflow / job (event)), and the spelling is rig knowledge.
Source code in packages/forge/src/livery/forge/testing/_conformance.py
Scenario
dataclass
¶
Scenario(name: str, run: Callable[[ForgeDriver], None], requires: tuple[Capability, ...] = (), forbids: tuple[Capability, ...] = ())
One conformance case: a name, its capability gates, and its body.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
The scenario's stable identifier, used as the test id. |
run |
Callable[[ForgeDriver], None]
|
The body. Raises AssertionError when the backend breaks the contract. |
requires |
tuple[Capability, ...]
|
Capabilities the forge must support for the scenario to apply. |
forbids |
tuple[Capability, ...]
|
Capabilities the forge must NOT support: the scenario asserts the honest decline. |
Methods:
| Name | Description |
|---|---|
applies_to |
Whether forge's capabilities put this scenario in scope. |
applies_to ¶
Whether forge's capabilities put this scenario in scope.