Skip to main content

LLM Access to Documentation

The user docs site (docs.rat.gd) is legible to AI agents: every doc page's raw Markdown source is published at the same URL as its HTML page (same permalink, .md suffix), and a generated /llms.txt index lists all pages with descriptions. Shipped and verified live 2026-07-11.

https://docs.rat.gd/docs/activity-templates      ← HTML page (humans)
https://docs.rat.gd/docs/activity-templates.md   ← raw Markdown (agents)
https://docs.rat.gd/llms.txt                     ← index of all .md URLs

Scope​

  • User docs only (docs/user, built by website/docusaurus.config.ts). The dev docs site (docusaurus.dev.config.ts) is deliberately excluded: it is gated behind Cloudflare Access, and publishing an AI-consumption index for gated content would defeat the gate.
  • Default locale (en) only. No per-locale .md trees or llms.txt variants — consuming agents can translate. (9 of 10 locales have no translated docs/user content anyway; only docs/user/legal is translated today.)
  • No llms-full.txt (single concatenated file). Easy to add later from the same doc list if a real consumer asks for it.

Architecture​

Custom plugin: website/src/plugins/rawMarkdownPlugin.ts​

A ~1-file in-house Docusaurus plugin. Third-party packages were evaluated and rejected — docusaurus-plugin-llms (the most mature candidate) was tested against this site's real config (docsDir: "../docs/user", routeBasePath: "docs", 10 locales) and failed four ways: .md files written at source paths instead of route paths (guaranteed 404s), ../ path-traversal fragments baked into llms.txt URLs, locale passes silently overwriting each other, and stray output outside outDir. Root cause: it reverse-engineers URLs from the filesystem instead of reading Docusaurus's own route data.

The custom plugin avoids all of that by construction — it never derives a URL from a file path:

  1. allContentLoaded — collects { source, permalink, title, description } from every docusaurus-plugin-content-docs instance's loaded data. permalink is Docusaurus's own already-correct route path (includes routeBasePath); source is @site-aliased and resolved against siteDir. Note this is the allContentLoaded lifecycle, not contentLoaded — allContent (other plugins' data) is only available there. Collected docs live in the plugin factory's closure, which is per-build-pass since Docusaurus re-instantiates plugins for each locale.

  2. postBuild — guarded with if (props.i18n.currentLocale !== props.i18n.defaultLocale) return; so only the en pass writes output. For each doc it copies the raw source file to path.join(outDir, permalink + ".md") (trailing-slash permalinks fall back to /index), then writes llms.txt from the same in-memory doc list — link URLs and file destinations both derive from the one mdPermalink() helper, so they cannot diverge.

llms.txt format​

# ROPA Docs

> Record of Processing Activities tool — user documentation.

## Docs

- [Activity-Scoped Templates](https://docs.rat.gd/docs/activity-templates.md): How to configure and use...

## Legal

- [Privacy Policy](https://docs.rat.gd/docs/legal/privacy.md): How GDPR Labs collects...

Docs whose permalink contains /legal/ go in the Legal section, the rest in Docs. Each line's description is the doc's description metadata (frontmatter description, falling back to the first content line), with the title as final fallback. Absolute URLs are built from siteConfig.url.

Registration​

Registered in the plugins: [] array of website/docusaurus.config.ts only. Docusaurus calls each entry as a factory with (context, options), matching the plugin's (context: LoadContext) signature. Do not add it to docusaurus.dev.config.ts (see Scope).

Content-Type: _headers (written by the plugin)​

Deploys go to Cloudflare Pages, which reads a _headers file from the build output. The plugin writes it in the same postBuild step as the .md files and llms.txt:

/*.md
  Content-Type: text/markdown; charset=utf-8
/llms.txt
  Content-Type: text/plain; charset=utf-8

Cloudflare's * matches across /, so nested paths like /docs/legal/privacy.md are covered.

The file is deliberately not in website/static/ — static assets are shared by both sites, and shipping the /*.md rule to the dev site made Cloudflare serve the dev site's 404 page as text/markdown for .md URLs (raw HTML in the browser instead of the styled Page Not Found).

Verifying​

The plugin has a no-mock Vitest suite at website/tests/rawMarkdownPlugin.test.ts (run with npm run docs:test; chained into test:run) — see Docs Website Tests.

After cd website && npm run build:

  • build/docs/<slug>.md exists next to build/docs/<slug>/index.html.
  • build/llms.txt exists; link count matches the docs/user doc count; no ../ or double slashes in any URL.
  • No .md output under any locale directory (build/es/, build/fr/, …).
  • No stray output outside build/ (e.g. no website/docs/ created).
  • npm run build:dev produces no .md/llms.txt/_headers in build-dev/.

Against the live site:

curl -s https://docs.rat.gd/llms.txt
curl -s https://docs.rat.gd/docs/activity-templates.md
curl -sI https://docs.rat.gd/docs/activity-templates.md   # Content-Type: text/markdown

Maintenance​

  • New user docs need no wiring — the plugin reads whatever the content-docs plugin loaded, so any page added under docs/user appears in llms.txt and gets a .md URL automatically. Give each doc a frontmatter description; it becomes the llms.txt link description.
  • Rollback is fully additive-reverse: remove the plugin registration from docusaurus.config.ts and delete rawMarkdownPlugin.ts (the _headers file goes with it). No existing routes or files are touched.

Possible follow-ups (not planned)​

  • llms-full.txt — single concatenated file, generated from the same doc list, if an actual consumer asks for it.
  • A "View as Markdown" link in the doc page toolbar (swizzle DocItem/Layout) so humans can find the raw view GitHub-style.
  • Locale-specific .md/llms.txt if a real use case for non-English raw content appears.