The agent-facing site files: llms.txt and llms-full.txt.
Ported from footman's generator. llms.txt
(https://llmstxt.org) is the index: every page of the merged nav in
order, each with a one-line description pulled from its first prose
paragraph and an absolute link. llms-full.txt is the authored
text in one file, snippet includes resolved inline. The
machine-appended sections (package changelogs, the release view, the
API trees) are excluded from the full file by construction: a
context file that concatenates history teaches superseded behaviour
as current, so only what an author put in a nav belongs in a
model's context. The index still links every page, the changelogs
included.
Both files are written into the built site's root after the site
build; they are derived, never hand-edited, and never committed.
Functions:
| Name |
Description |
page_url |
The published URL for page: a directory index lives at the directory.
|
first_sentence |
The first sentence of the page's first prose paragraph.
|
resolve_snippets |
Inline --8<-- "path" includes, the way the site build does.
|
llms_files |
The two files' bodies: (index, full).
|
write_llms_files |
Write both files into the built site's root; the names written.
|
Attributes:
MACHINE_PREFIXES
module-attribute
MACHINE_PREFIXES = ('releases/',)
page_url
The published URL for page: a directory index lives at the directory.
Source code in packages/extensions/docs/src/livery/extensions/docs/_llms.py
| def page_url(site: str, page: str) -> str:
"""The published URL for *page*: a directory index lives at the directory."""
if page == "index.md" or page.endswith("/index.md"):
stem = page.removesuffix("index.md").rstrip("/")
return f"{site}{stem + '/' if stem else ''}"
return f"{site}{page.removesuffix('.md')}/"
|
first_sentence
The first sentence of the page's first prose paragraph.
Prose, strictly: headings, list items, table rows, admonition
bodies, and fenced code are structure, not description; served as
a page's one-line summary they read as junk in every agent index
built on the file.
Source code in packages/extensions/docs/src/livery/extensions/docs/_llms.py
| def first_sentence(text: str, site: str, page: str) -> str:
"""The first sentence of the page's first prose paragraph.
Prose, strictly: headings, list items, table rows, admonition
bodies, and fenced code are structure, not description; served as
a page's one-line summary they read as junk in every agent index
built on the file.
"""
if text.startswith("---\n"):
_, _, text = text.partition("\n---\n")
skip = ("#", "---", "[![", "!!!", ">", "<", "--8<--", "|", "- ", "* ")
fenced = False
admonition = False
hanging = False
paragraph: list[str] = []
for raw in text.splitlines():
line = raw.strip()
if admonition:
if not line or raw.startswith((" ", "\t")):
continue
admonition = False
if line.startswith("!!!"):
admonition = True
continue
if line.startswith("```"):
fenced = not fenced
continue
digits = line.split(".", 1)[0]
numbered = digits.isdigit() and line.startswith(f"{digits}.")
if fenced or line.startswith(skip) or numbered:
if paragraph:
break
# A wrapped list item or table row continues on the next
# line without its marker; serving that tail as prose put
# mid-sentence fragments in the index.
hanging = True
continue
if not line:
if paragraph:
break
hanging = False
continue
if hanging:
continue
paragraph.append(line)
prose = _absolute_links(" ".join(paragraph), site, page)
end = prose.find(". ")
return prose[: end + 1] if end != -1 else prose
|
resolve_snippets
Inline --8<-- "path" includes, the way the site build does.
Paths are repo-relative, matching the snippets configuration; a
missing file keeps the directive, so the gap is visible rather
than silent.
Source code in packages/extensions/docs/src/livery/extensions/docs/_llms.py
| def resolve_snippets(text: str, root: Path) -> str:
"""Inline ``--8<-- "path"`` includes, the way the site build does.
Paths are repo-relative, matching the snippets configuration; a
missing file keeps the directive, so the gap is visible rather
than silent.
"""
def inline(match: re.Match[str]) -> str:
path = root / match.group(1)
if not path.is_file():
return match.group(0)
return resolve_snippets(path.read_text(encoding="utf-8").rstrip(), root)
return re.sub(r"""^--8<-- ["']([^"']+)["']$""", inline, text, flags=re.MULTILINE)
|
llms_files
The two files' bodies: (index, full).
Derived from the same emitted config the site builds from, so
the index can never disagree with the nav. Without a declared
site URL the links are root-relative, which still resolves on
whatever host serves the site.
Source code in packages/extensions/docs/src/livery/extensions/docs/_llms.py
| def llms_files(root: Path) -> tuple[str, str]:
"""The two files' bodies: (index, full).
Derived from the same emitted config the site builds from, so
the index can never disagree with the nav. Without a declared
site URL the links are root-relative, which still resolves on
whatever host serves the site.
"""
table = docs_table(root)
title = str(table.get("title", "")) or project_name(root)
description = str(table.get("description", ""))
site = str(table.get("site-url", "")) or "/"
if not site.endswith("/"):
site += "/"
nav = tomllib.loads(zensical_config(root))["project"]["nav"]
pages = _nav_pages(nav)
members = tuple(package.member for package in discover_packages(root))
index = [f"# {title}", ""]
if description:
index += [f"> {description}", ""]
index += ["## Docs", ""]
full = [f"# {title} - full documentation", ""]
for page_title, page in pages:
source = root / "docs" / page
if not source.is_file():
continue
resolved = resolve_snippets(source.read_text(encoding="utf-8").rstrip(), root)
url = page_url(site, page)
summary = first_sentence(resolved, site, page)
index.append(
f"- [{page_title}]({url}): {summary}"
if summary
else f"- [{page_title}]({url})"
)
if _machine_page(page, members):
continue
full += ["", "---", "", f"<!-- {page_title} - {url} -->", "", resolved]
return ("\n".join(index) + "\n", "\n".join(full) + "\n")
|
write_llms_files
Write both files into the built site's root; the names written.
The site must already be built: these land beside its pages the
way any root asset does, and a clean rebuild regenerates them.
Source code in packages/extensions/docs/src/livery/extensions/docs/_llms.py
| def write_llms_files(root: Path) -> list[str]:
"""Write both files into the built site's root; the names written.
The site must already be built: these land beside its pages the
way any root asset does, and a clean rebuild regenerates them.
"""
site_dir = root / "site"
site_dir.mkdir(exist_ok=True)
index, full = llms_files(root)
(site_dir / "llms.txt").write_text(index, encoding="utf-8")
(site_dir / "llms-full.txt").write_text(full, encoding="utf-8")
return ["llms.txt", "llms-full.txt"]
|