Skip to content

livery-extensions-pytest

pytest as steps of the workshop. A workspace turns it on by listing the extension:

[workspace]
extensions = ["pytest"]

fm test then runs the suites, fm examples the documentation examples, and fm check runs both with the rest of the gate:

  • test.pytest runs the suites of the python packages a run reaches in one pytest call, across cores, with the workspace's own tests directory beside them while the root's set holds python: that directory holds the root's set. A root whose set holds no python runs none of its own tests, and the step prints a line saying why. While the coverage extension is listed, every run is measured: on a machine the run prints each package's number beside its floor, and in CI the legs' measurements combine into the union the floors judge. Without it, the suites run with no meter.
  • examples.pytest runs each python package's documentation examples, the files under docs/examples/, one test per file.

Both steps run through the python extension. Each asks it for the function that runs what the step judges: its answer to python.test.runner runs the suites, and its answer to python.examples.runner runs a package's examples. The python extension answers while the root's set or a package's set holds it. In a workspace where neither holds python, no python suite runs, the workspace's own tests directory included, and the test step says so; the examples step names each package it skips.

While the coverage extension is listed, this extension measures for it, through its [for.coverage] table, and coverage judges what it measures. A workspace that lists coverage with nothing that measures has coverage dormant, and fm extensions names pytest among what would wake it.

This extension also reduces coverage.py's measurements to parts in the coverage extension's line format, per file coverage.py's own analysis of it. On each check leg the coverage extension asks it once for every suite the leg ran: the suite's part holds the files of its closure, as the suite's own tests and the code outside every test reached them, with the tests that reached each line, each named by its node id and the code outside every test as the empty context, and that part is what the leg stores. On a machine the data the last run left at the root becomes one part, which the coverage pages and the preview read. Each file's numbers are checked against coverage.py's before a part is handed over, and a file whose numbers differ stops the reduction, named.

fm test.pytest -- -k name hands pytest the words after --, and fm examples.pytest takes them the same way.

A package whose suite is not safe to run across cores turns that off in its own workshop.toml, and its suite then runs in a pytest call of its own under one worker:

[phases.test.pytest]
parallel = false

That call puts -n 0 after the words from the verb, so fm test.pytest -- -n 4 cannot spread that suite across cores.

On a check leg that runs as several shards, test.pytest runs on every shard, each over its share of the test files. The step packs every test file the workspace holds, every test_*.py and *_test.py under each package's tests directory and the workspace's own tests directory, into the shard's count of slices by the platform leg's durations rows: longest first, each file into the slice with the least summed seconds, ties by name, a file no row names at the rows' median. The first slice's sum starts at the rows' head, the test seconds the steps only the first shard runs hold back, so its slice is lighter by them. Of the files of the suites the run reaches, the ones the run selects for a suite or all of a suite's, the shard runs those in its own slice. The packing is over the whole workspace and never over one run's suites, so a file has one shard however many runs of the step reach it on a leg, the gate's and the coverage top-up's. The shard hands python's runner its slice as the selection the runner already takes, a suite with no file in the slice left out of the call; a suite that is not worker-safe is sliced the same way and still runs under -n 0. The shard records its slice, the suites with a file in the set, the digest of the whole set and the rows it read in .workshop/fm-shard.json, for the gate job's proof that the shards of a platform partitioned one set from one row. A suite with no python test file, a native suite the cmake extension measures, is not partitioned: it runs whole on the first shard, with every step that does not read the shard, and the coverage leg expects its parts there alone. With no row before the checkout's committer time, or a store that cannot be read, the line says why, the first shard runs every file and the others run none. fm check --shard=2of2 on a machine runs the share by name alone, since no leg names a durations series there. The coverage leg's top-up of the suites no record can supply runs through the same step, so it is sliced the same way and added to the shard's record.

The configuration

fm sync writes pytest.toml at the workspace root, and .coveragerc beside it while coverage is listed, each under the name its tool looks for, so a bare pytest or coverage call reads the same settings as the gate:

  • pytest.toml: where the tests are, the import mode that names a test module by its path, the options every run takes, the cache, kept under .workshop/.cache/pytest/, and the tmp_path retention: a passing test's directory goes at its teardown, a failing test's stays for inspection, so a session ends when its last test ends instead of deleting an older session's whole tree.
  • .coveragerc: what coverage measures (the project's namespace, with branches), each file named from the workspace root, the names the union reads, and how the data of Windows and POSIX runs combine.

The options every run takes are this extension's own, followed by the union of the listed extensions' answers to the python.test.addopts query, in composition order. This extension declares the query, at the workspace level.

While coverage is listed, the dev group also takes coverage, whose startup hook meters every python the tests start once the runner arms it, and pytest-cov for a machine's measured run, and the root .gitignore takes the lines for the data the meter leaves.

Each file ends in a region that is the repository's own: settings written there stay when the render runs again, and an edit anywhere else is drift.