livery-extensions-pytest¶
pytest as steps of the workshop. A workspace turns it on by listing the extension:
fm test then runs the suites, fm examples the documentation examples,
and fm check runs both with the rest of the gate:
test.pytestruns the suites of the python packages a run reaches in one pytest call, across cores, with the workspace's owntestsdirectory beside them while the root's set holds python: that directory holds the root's set. A root whose set holds no python runs none of its own tests, and the step prints a line saying why. While the coverage extension is listed, every run is measured: on a machine the run prints each package's number beside its floor, and in CI the legs' measurements combine into the union the floors judge. Without it, the suites run with no meter.examples.pytestruns each python package's documentation examples, the files underdocs/examples/, one test per file.
Both steps run through the python extension. Each asks it for the
function that runs what the step judges: its answer to
python.test.runner runs the suites, and its answer to
python.examples.runner runs a package's examples. The python extension
answers while the root's set or a package's set holds it. In a workspace
where neither holds python, no python suite runs, the workspace's own
tests directory included, and the test step says so; the examples step
names each package it skips.
While the coverage extension is listed, this extension measures for it,
through its [for.coverage] table, and coverage judges what it measures.
A workspace that lists coverage with nothing that measures has coverage
dormant, and fm extensions names pytest among what would wake it.
This extension also reduces coverage.py's measurements to parts in the coverage extension's line format, per file coverage.py's own analysis of it. On each check leg the coverage extension asks it once for every suite the leg ran: the suite's part holds the files of its closure, as the suite's own tests and the code outside every test reached them, with the tests that reached each line, each named by its node id and the code outside every test as the empty context, and that part is what the leg stores. On a machine the data the last run left at the root becomes one part, which the coverage pages and the preview read. Each file's numbers are checked against coverage.py's before a part is handed over, and a file whose numbers differ stops the reduction, named.
fm test.pytest -- -k name hands pytest the words after --, and
fm examples.pytest takes them the same way.
A package whose suite is not safe to run across cores turns that off in its
own workshop.toml, and its suite then runs in a pytest call of its own
under one worker:
That call puts -n 0 after the words from the verb, so
fm test.pytest -- -n 4 cannot spread that suite across cores.
On a check leg that runs as several shards, test.pytest runs on every
shard, each over its share of the test files. The step packs every
test file the workspace holds, every test_*.py and *_test.py under
each package's tests directory and the workspace's own tests
directory, into the shard's count of slices by the platform leg's
durations rows: longest first, each file into the slice with the
least summed seconds, ties by name, a file no row names at the rows'
median. The first slice's sum starts at the rows' head, the test
seconds the steps only the first shard runs hold back, so its slice
is lighter by them. Of the files of the suites the run reaches, the ones the run
selects for a suite or all of a suite's, the shard runs those in its
own slice. The packing is over the whole workspace and never over one
run's suites, so a file has one shard however many runs of the step
reach it on a leg, the gate's and the coverage top-up's. The shard
hands python's runner its slice as the selection the runner already
takes, a suite with
no file in the slice left out of the call; a suite that is not
worker-safe is sliced the same way and still runs under -n 0. The
shard records its slice, the suites with a file in the set, the digest
of the whole set and the rows it read in .workshop/fm-shard.json, for
the gate job's proof that the shards of a platform partitioned one set
from one row. A suite with no python test file, a native suite the
cmake extension measures, is not partitioned: it runs whole on the
first shard, with every step that does not read the shard, and the
coverage leg expects its parts there alone. With no row
before the checkout's committer time, or a store that cannot be read,
the line says why, the first shard runs every file and the others run
none. fm check --shard=2of2 on a machine runs the share by name alone,
since no leg names a durations series there. The coverage leg's top-up
of the suites no record can supply runs through the same step, so it is
sliced the same way and added to the shard's record.
The configuration¶
fm sync writes pytest.toml at the workspace root, and .coveragerc
beside it while coverage is listed, each under the name its tool looks
for, so a bare pytest or coverage call reads the same settings as the
gate:
pytest.toml: where the tests are, the import mode that names a test module by its path, the options every run takes, the cache, kept under.workshop/.cache/pytest/, and thetmp_pathretention: a passing test's directory goes at its teardown, a failing test's stays for inspection, so a session ends when its last test ends instead of deleting an older session's whole tree..coveragerc: what coverage measures (the project's namespace, with branches), each file named from the workspace root, the names the union reads, and how the data of Windows and POSIX runs combine.
The options every run takes are this extension's own, followed by the
union of the listed extensions' answers to the python.test.addopts
query, in composition order. This extension declares the query, at the
workspace level.
While coverage is listed, the dev group also takes coverage, whose
startup hook meters every python the tests start once the runner arms
it, and pytest-cov for a machine's measured run, and the root
.gitignore takes the lines for the data the meter leaves.
Each file ends in a region that is the repository's own: settings written there stay when the render runs again, and an edit anywhere else is drift.