Colophon

Implementation

The contract everything negotiates through

One schema sits between the publisher, the backend, the frontend and the MCP tools. If it is right, the four can be built independently; if it is wrong, every one of them acquires a workaround.

01

The manifest

The design rule is that a manifest is a complete index of a revision. Titles, descriptions, the navigation tree, and every heading anchor live in it, so the backend can render navigation, build a table of contents, validate cross-page links and plan retrieval chunks without fetching a single Markdown blob. Only page bodies require a blob read.

manifest.json — abridged, from a real publish
{
  "schemaVersion": 1,
  "bundleId": "github.com/yorch/colophon",
  "revisionId": "c25e73f7eee0c052eeccc2db0e1de2dfc06c391c6c514c381f3cfeab9d029d7a",
  "createdAt": "2026-08-21T17:11:32.452Z",
  "source": {
    "type": "git",
    "url": "https://github.com/yorch/colophon",
    "ref": "main",
    "commit": "4c47e41",
    "path": "docs"
  },
  "publisher": { "name": "colophon-cli" },
  "title": "Colophon",
  "description": "Markdown documentation for the Backstage software catalog, …",
  "pages": [
    {
      "path": "architecture.md",
      "slug": "architecture",
      "title": "Architecture",
      "description": "How a docs directory becomes a bundle, and how Backstage serves it to both people and agents.",
      "type": "explanation",
      "status": "current",
      "tags": ["architecture"],
      "headings": [
        { "depth": 1, "text": "Architecture",         "anchor": "architecture" },
        { "depth": 2, "text": "Pipeline",             "anchor": "pipeline" },
        { "depth": 2, "text": "Two storage systems",  "anchor": "two-storage-systems" }
      ],
      "contentHash": "1f08ced741e5626a59950c29ffe9cdb5a1139c33df90be5114d12417bc29b36b",
      "size": 2187
    }
  ],
  "nav": [
    { "title": "Colophon",     "slug": "" },
    { "title": "Architecture", "slug": "architecture" },
    {
      "title": "Guides",
      "children": [
        { "title": "Publishing documentation", "slug": "guides/publishing" },
        { "title": "Writing documentation",    "slug": "guides/writing-docs" }
      ]
    }
  ],
  "assets": []
}

Abridged for the page: one of five pages entries is shown, with three of its six headings, and short arrays are compacted onto single lines. Every value shown is as generated.

What is deliberately absent

There is no channel field. A manifest describes content; a channel describes routing, and it is recorded in the database at publish time instead. That is what keeps revisions genuinely immutable and lets a rollback be a pointer move rather than a rebuild.

createdAt is canonical UTC with a trailing Z, and offsets are rejected rather than normalised. Canonical UTC sorts lexicographically and carries no offset ambiguity, so publishers convert before they publish.

02

Revision identity

The revision id is the sha-256 of the canonicalised manifest body — but not of the whole manifest. Two fields are excluded, and the reason is a bug found by publishing this repository's own docs twice.

Field In the id? Why
pages, nav, assets Yes This is the documentation.
bundleId, title Yes Identity of the thing being published.
source.commit Yes So identical docs built from different commits remain distinct revisions.
createdAt No Describes the run, not the documentation.
publisher No Same — and it carries runUrl, which differs on every retry.

With the timestamp inside the hash, a retried pipeline produced a brand new revision every time and history grew without bound — precisely the duplicate history that content-addressing exists to prevent. Excluding it makes publishing idempotent: the repeat publish on the home page emits the same id and uploads nothing.

Keys

Bundle ids allow / so they can be used verbatim as a storage key prefix. bundles/github.com/brnby/payments-api/… is far easier to debug in a bucket browser than an opaque hash, and the charset is restricted to what is safe in an S3 key, a URL path and a filesystem path at once.

object-storage key layout
blobs/<ab>/<sha256>                                   content-addressed, shared
bundles/<bundleId>/revisions/<revisionId>/manifest.json

Lowercase is enforced, not merely recommended. TechDocs still carries a legacyPathCasing compatibility flag because it allowed case-sensitive keys and then met case-insensitive storage; there is no reason to re-learn that lesson.

03

Chunking

Retrieval chunks are what an agent actually receives from colophon:search. A page is cut into sections at heading boundaries, and each section carries the heading trail it sits under.

Cutting a page into breadcrumb-carrying chunks On the left, a Markdown page whose title heading is followed by three level-two headings with paragraphs beneath them. On the right, three retrieval chunks, one per heading. Each chunk carries a breadcrumb that starts with the page title and ends with its own heading, plus its ordinal position in the page and the heading anchor used for deep linking. ONE PAGE THREE RETRIEVAL CHUNKS # Publishing documentation ## Link the entity to a bundle ## Publish from CI ## Release branches … Publishing documentation ▸ Link the entity to a bundle ordinal 0 · anchor link-the-entity-to-a-bundle Publishing documentation ▸ Publish from CI ordinal 1 · anchor publish-from-ci Publishing documentation ▸ Release branches ordinal 2 · anchor release-branches
The page is a real one from this repository's own docs corpus; the chunk cards show the fields the chunker produces for it.

H2 and H3, not H2 alone

Reference pages put one endpoint or one configuration option per H3. Splitting only on H2 would glue thirty unrelated options into a single chunk and destroy retrieval precision. The size ceiling then catches prose sections that run long, and the floor merges heading stubs that carry no content of their own.

Option Default Behaviour
splitDepths [2, 3] Heading depths that start a new chunk.
maxChars 1500 Soft ceiling. Blocks are packed greedily; a single block larger than the ceiling is emitted alone rather than cut, because slicing through a fenced code block or a table row produces a chunk worse than an oversized one.
minChars 200 Short sections merge into the following sibling, keeping the following section's anchor and breadcrumb — that is where the content is, and a deep link should land where the reader is looking.
overlapChars 0 Optional continuity tail. Taken from the previous chunk's own text, never from its already-overlapped form, so repetition does not compound down a page.

The breadcrumb is the cheap defence

Every chunk carries the trail from the page title down to its own heading — ["Payments API", "Operations", "Rotating credentials"] — and that trail is prepended to the chunk text at retrieval time. A chunk read in isolation still says what it is about. This matters more to retrieval quality than any ranking algorithm: a chunk that does not stand alone is useless no matter how well it was retrieved.

In the backend, at index time

Chunking runs in the backend when a revision is indexed, not in the CLI when it is published. The strategy will change as we learn what agents retrieve well, and re-chunking must never require every repository in the organisation to re-run its CI. Changing chunking.maxChars in app-config re-chunks on the next indexing run and nothing else moves.

Two details that only show up once you run it on real Markdown. Frontmatter is stripped before parsing rather than parsed as content — without that, title: Foo followed by --- reads as a setext heading and invents a heading that does not exist. And every heading consumes a slug whether or not it starts a chunk, so duplicate headings get the same -1 suffixes GitHub and the publisher CLI produce.

04

Rejecting at the boundary

validate and publish run the same build, so the two cannot drift into disagreeing about what is publishable. Validation rejects at the boundary rather than letting three downstream consumers each invent their own handling.

  • Relative links that resolve to nothing.
  • Heading anchors that match no heading.
  • Referenced assets that are not in the tree.
  • Duplicate slugs — the trap where guides/index.md and guides.md both address guides, and one author's edits silently go nowhere.

A missing description warns by default and fails under --strict, because a weak description is what makes an agent fetch a whole page it did not need.

Found by dogfooding. The docs-root index.md has the empty slug and therefore no segment in the directory tree. It was being dropped from the derived navigation and then reported as an orphan by the same run. It now leads the nav.

05

Five packages

The roles follow Backstage conventions, which is not cosmetic: the role drives the build output shape and which dependency field @backstage/no-undeclared-imports expects.

Package Role Owns
colophon-common common-library Manifest schema, identifiers, storage key layout, chunk contract.
colophon-cli cli Scan, validate, hash, upload, register. Also shipped as an image.
plugin-colophon-backend backend-plugin Storage abstraction, Knex schema, chunker, channels, retention, router, search collator, MCP actions.
plugin-colophon-react web-library Markdown renderer, component override registry, sanitize schema, theme-aware Mermaid.
plugin-colophon frontend-plugin Entity docs tab, cross-repository docs home, channel picker.

The component override registry in colophon-react is the extension point that keeps the theming promise. Because the stored artifact is Markdown, a portal can replace the renderer for any node type with its own component instead of patching CSS at a DOM it does not control.

The dependency-hygiene rules are worth the second linter. They catch imports that only resolve because Yarn hoisted a package to the workspace root — which installs cleanly here and then fails for anyone consuming the published package. Two such imports have already been caught this way.