Read a tool's own --help and turn it into a ToolSpec.
The structural extractors in _toolspec are better when they apply — click
and argparse hand over their parameters as data. But most of the tools a
task actually calls are Rust, Go or Node binaries with no Python parser to
introspect: ruff and uv (clap), docker (cobra), cspell and markdownlint
(commander), git (its own). For those, the tool's --help is the only
description it offers, and it is far more regular than it looks.
Every one of those help formats prints an option as a line that starts with
a dash, followed by help text that is either on the same line past a run of
spaces, or on the lines below indented deeper:
clap --fix
Apply fixes to resolve lint violations. Use `--no-fix`
to disable
optparse -d DIR, --directory=DIR
Write the output files to DIR.
cobra -f, --file stringArray Compose configuration files
So one parser reads them all: find the lines that start an option, split the
flag spellings from the prose, and glue on the continuation lines. What
differs between the families is only how they spell a default and a
negation, and those are small dialects on top ([default: 3],
(default true), "Use --no-fix to disable").
Nothing here runs at task time. The extractor runs when a maintainer
regenerates the stubs, and fm toolroom.audit compares what it finds
against what is checked in.
Functions:
| Name |
Description |
parse_help |
One verb's options, from its --help output or (with man) manual.
|
subcommands |
{name: summary} from the Commands: section of a tool's help.
|
run_help |
<tool> ... --help, as text. Empty when the tool isn't installed.
|
man_version |
The version a fetched manual belongs to, from its own header.
|
from_help |
A ToolSpec for name by asking the installed binary.
|
Attributes:
DECIDED = {'DOCKER_BUILDKIT': '1'}
A choice the tool would otherwise take from the machine reading it.
docker build is buildx wherever buildx is installed, which is why the
driver installs it — but with DOCKER_BUILDKIT unset the CLI only routes
build that way for a Linux daemon, and a reader has no daemon at all.
So Linux and macOS read buildx while Windows fell back to the builder
docker shipped with before 2019, and one verb arrived as two surfaces:
--compress and --cpu-shares from one leg, --platform and --push
from the others, each recorded as missing on the platforms that saw the
other. Twelve options of a per-platform difference that no user has.
Docker's own notes retire the classic builder for Linux images and name
the exception — native Windows containers — which is not the docker
this describes: a Windows user on the usual backend talks to a Linux
daemon and gets buildx like everyone else. Pinning the opt-in reads the
builder that is actually installed, the same way COLUMNS and
help.format are pinned so a reading describes the tool and not the
machine that took it.
parse_help(text: str, *, name: str = '', man: bool = False, shorts: str = 'only') -> Verb
One verb's options, from its --help output or (with man) manual.
The option grammar is the same either way — the man page states a flag
and its help exactly as --help does. Only the positional shape reads
from a different place: a usage: line normally, the SYNOPSIS forms
for a manual.
Source code in packages/toolroom-bench/src/livery/toolroom/bench/_toolhelp.py
| def parse_help(
text: str, *, name: str = "", man: bool = False, shorts: str = "only"
) -> Verb:
"""One verb's options, from its `--help` output or (with `man`) manual.
The option grammar is the same either way — the man page states a flag
and its help exactly as `--help` does. Only the positional shape reads
from a different place: a `usage:` line normally, the `SYNOPSIS` forms
for a manual.
"""
from livery.toolroom.store import Verb
sections = _sections(text)
# The flags the usage line has already given a value to. Where it has
# spoken, the block's bare-lowercase-word rule is redundant at best and
# wrong at worst — see `_spellings`.
stated = set() if man else _grammar_values(_usage_line(text))
options: list[Option] = []
seen: dict[str, int] = {}
for title, lines in sections.items():
if _NOT_OPTIONS.search(title):
continue # `Commands:`, `Examples:` — dashes there aren't flags
if not title:
# The preamble is the one section a title cannot excuse, and the
# usage line lives there. Wrapped, its continuation is an
# indented line of bracketed flags — the shape of an option row.
# `build` wraps at any width, and every flag on the continuation
# was swept into one option carrying six flags and no help.
lines = _drop_usage(lines)
for head, help_text in _blocks(lines, man=man):
first = _FIRST_LONG.search(head)
option = _option(
head,
help_text,
strict=man,
shorts=shorts,
bare_meta=not (first and first["flag"] in stated),
)
if option is None or option.name in _NOISE:
continue
if (kept := seen.get(option.name)) is not None:
# A manual may state one option as several complete forms on
# consecutive head lines (ssh's `-L` gives four), and only
# the last carries the description. The first form stays the
# option; a later twin only donates the help the kept one
# lacks.
if not options[kept].help and option.help:
options[kept] = replace(options[kept], help=option.help)
continue
seen[option.name] = len(options)
options.append(option)
if not options:
# Go's `flag` prints its options under `Usage of <prog>:` — a section
# `_NOT_OPTIONS` skips. Nothing parsed anywhere else, so scan every
# section, including that one. Guarded on emptiness, so a tool that
# parses normally never reaches here and can't regress.
for _title, lines in sections.items():
for head, help_text in _blocks(lines, man=man):
option = _option(head, help_text, strict=man, shorts=shorts)
if option is not None and option.name not in _NOISE:
seen[option.name] = len(options)
options.append(option)
options = list({o.name: o for o in options}.values())
if shorts == "all":
options = _with_short_aliases(options)
if not man:
options = _arity_from_grammar(options, _usage_line(text))
positional, lead = _synopsis_shape(text) if man else _usage_shape(text)
return Verb(
name=name,
help=_summary(text),
options=tuple(sorted(_pair_negations(options), key=lambda o: o.name)),
positional=positional,
lead=lead,
wraps=_synopsis_wraps(text) if man else _wraps(text),
)
|
{name: summary} from the Commands: section of a tool's help.
Source code in packages/toolroom-bench/src/livery/toolroom/bench/_toolhelp.py
| def subcommands(text: str) -> dict[str, str]:
"""`{name: summary}` from the `Commands:` section of a tool's help."""
found: dict[str, str] = {}
for title, lines in _sections(text).items():
if not re.search(r"command|subcommand", title):
continue
for line in lines:
match = re.match(
r"^\s+(?P<name>[a-z][a-z0-9-]*)(?:,\s*[a-z0-9-]+)*"
r"(?:\s{2,}(?P<help>.*))?$",
line.rstrip(),
)
if match:
found.setdefault(match["name"], (match["help"] or "").strip())
return found
|
<tool> ... --help, as text. Empty when the tool isn't installed.
argv[0] is the executable to run — a bare name resolved on PATH, or the
absolute path a caller already resolved (from_help(..., binary=…)).
Help goes to stdout for every tool footman curates, but a few print
usage to stderr on older versions, so both are read.
man reads the manual instead — git help <verb> — for a tool whose
terse -h omits most of its flags (git's -h shows about half). It
runs only at stub-generation time, never at task time, so its heavier
footprint (a rendered man page) costs a user nothing.
Source code in packages/toolroom-bench/src/livery/toolroom/bench/_toolhelp.py
| def run_help(
argv: list[str], *, flag: str = "--help", man: bool = False, timeout: float = 30.0
) -> str:
"""`<tool> ... --help`, as text. Empty when the tool isn't installed.
`argv[0]` is the executable to run — a bare name resolved on `PATH`, or the
absolute path a caller already resolved (`from_help(..., binary=…)`).
Help goes to stdout for every tool footman curates, but a few print
usage to stderr on older versions, so both are read.
`man` reads the manual instead — `git help <verb>` — for a tool whose
terse `-h` omits most of its flags (git's `-h` shows about half). It
runs only at stub-generation time, never at task time, so its heavier
footprint (a rendered man page) costs a user nothing.
"""
if man:
# A manual can be read without the tool: `man` renders the pages a
# release shipped, and for a fetched tree there is no binary at all.
return _run_man(argv, timeout)
if shutil.which(argv[0]) is None:
return ""
try:
# UTF-8 rather than the locale codec, a hidden console, and a bound
# that kills the tree: all of it is what `run()` does for a captured
# call now, so the read says only what is particular to it. `recorded=False`
# keeps a probe out of the run's story.
done = _run(
[*argv, flag],
recorded=False,
timeout=timeout,
nofail=True,
# A wide, dumb, colourless terminal: every family honours one
# of these, and a narrow wrap costs nothing but re-joined prose
# while a wide one keeps `[default: …]` on the line it belongs to.
env={
**os.environ,
**DECIDED,
"COLUMNS": "200",
"TERM": "dumb",
"NO_COLOR": "1",
},
)
except (OSError, subprocess.SubprocessError):
return ""
out, err = done.stdout or "", done.stderr or ""
return _decoded(out if len(out) > len(err) else err, argv, flag, timeout)
|
The version a fetched manual belongs to, from its own header.
The installer stamps the tree with the release it fetched, per tool
(VERSION-ssh) because the provision tier merges every manual into one
tree and git's release is not ssh's — and the stamp is the only
statement there is for mdoc pages, which say no version anywhere.
git's older trees predate the stamp and carry it in their .TH line
instead: .TH "GIT" "1" "2025-06-15" "Git 2\&.50\&.1" "Git Manual".
Either way the tree says which release it documents, so a reading
names its own version the way a binary does, and the guard against
describing the wrong release works unchanged.
Source code in packages/toolroom-bench/src/livery/toolroom/bench/_toolhelp.py
| def man_version(tree: Path, name: str = "git") -> str:
"""The version a fetched manual belongs to, from its own header.
The installer stamps the tree with the release it fetched, per tool
(`VERSION-ssh`) because the provision tier merges every manual into one
tree and git's release is not ssh's — and the stamp is the only
statement there is for mdoc pages, which say no version anywhere.
git's older trees predate the stamp and carry it in their `.TH` line
instead: `.TH "GIT" "1" "2025-06-15" "Git 2\\&.50\\&.1" "Git Manual"`.
Either way the tree says which release it documents, so a reading
names its own version the way a binary does, and the guard against
describing the wrong release works unchanged.
"""
stamp = tree / f"VERSION-{name}"
try:
stamped = stamp.read_text(encoding="utf-8").strip()
except OSError:
stamped = ""
if stamped:
return stamped
page = tree / "man1" / "git.1"
try:
head = page.read_text(encoding="utf-8", errors="replace")[:4000]
except OSError:
return ""
found = re.search(r'^\.TH\s+"[^"]*"\s+"[^"]*"\s+"[^"]*"\s+"([^"]*)"', head, re.M)
if not found: # pragma: no cover - every page carries a .TH line
return ""
title = found.group(1).replace("\\&", "")
number = _VERSION_IN_TITLE.search(title)
return number.group(0) if number else ""
|
from_help(name: str, *, binary: str | None = None, verbs: tuple[str, ...] = (), version: str = '', in_process: bool = False, flag: str = '--help', man: bool = False, shorts: str = 'only') -> ToolSpec
A ToolSpec for name by asking the installed binary.
binary is the executable to run (the caller may have resolved it, e.g. to
a Homebrew keg); it defaults to name, resolved on PATH. The tool's own
verb names still ride as name/verbs in each argv, only the executable
differs.
Each verb costs one <tool> <verb> --help (or <tool> help <verb>
with man); the root call supplies the tool's summary and its global
options (verb ""). With man, per-verb manuals are read but the root
stays on --help, which is where a tool prints its verb list.
Source code in packages/toolroom-bench/src/livery/toolroom/bench/_toolhelp.py
| def from_help(
name: str,
*,
binary: str | None = None,
verbs: tuple[str, ...] = (),
version: str = "",
in_process: bool = False,
flag: str = "--help",
man: bool = False,
shorts: str = "only",
) -> ToolSpec:
"""A `ToolSpec` for *name* by asking the installed binary.
*binary* is the executable to run (the caller may have resolved it, e.g. to
a Homebrew keg); it defaults to *name*, resolved on `PATH`. The tool's own
verb names still ride as `name`/verbs in each argv, only the executable
differs.
Each verb costs one `<tool> <verb> --help` (or `<tool> help <verb>`
with `man`); the root call supplies the tool's summary and its global
options (verb `""`). With `man`, per-verb manuals are read but the root
stays on `--help`, which is where a tool prints its verb list.
"""
from livery.toolroom.store import ToolSpec
cmd = binary or name
# Against a fetched manual there is no binary to ask for a usage line,
# and asking the machine's own would describe a different release.
root = run_help([cmd], flag=flag, man=man and bool(_fetched_manpath()))
if not root:
return ToolSpec(name=name, version=version)
root_verb = parse_help(root, name="", shorts=shorts)
if verbs:
# A multi-command tool's bare usage line (`docker [OPTIONS] COMMAND`)
# describes the subcommand slot, not arguments to `docker` itself —
# so `tools.docker(...)` must not be constrained by it. Only a
# single-command tool's root verb carries a real positional shape.
root_verb = replace(root_verb, positional="any", lead="", wraps=False)
if man:
# The terse root help (`git -h`) lists subcommands, not globals; the
# tool's own manual (`git help git`) lists the options that must
# precede the verb — what `.opts()` binds. Read them from there.
manual = run_help([cmd, name], man=True)
if manual:
from_manual = parse_help(manual, man=True, shorts=shorts)
root_verb = replace(root_verb, options=from_manual.options)
if not verbs:
# A verb-less manual tool (ssh) *is* its root: the manual's
# SYNOPSIS is the only statement of its shape, and whether
# it wraps a trailing command — the terse root read above
# had no usage line to say either.
root_verb = replace(
root_verb,
positional=from_manual.positional,
lead=from_manual.lead,
wraps=from_manual.wraps,
)
parsed = [root_verb]
for verb in verbs:
text = run_help([cmd, *verb.split(".")], flag=flag, man=man)
if text and not _is_the_root_again(text, root):
# `git rev-parse` is spelled `tools.git.rev_parse(...)`: the
# bridge turns the underscore back into a dash when it calls.
parsed.append(
parse_help(text, name=verb.replace("-", "_"), man=man, shorts=shorts)
)
# The tool's own line, and the root verb's when it is the same line —
# a verb-less tool describes itself once and it lands in both.
banner = _summary(root)
summary = _unstamped(banner, version)
if summary != banner:
parsed = [
replace(verb, help=summary) if verb.help == banner else verb
for verb in parsed
]
return ToolSpec(
name=name,
help=summary,
version=version,
verbs=tuple(parsed),
in_process=in_process,
)
|