Skip to content

Make DocsForge work for the models it was built for - #22

Merged
R204570 merged 1 commit into
mainfrom
docs-architecture
Aug 20, 2026
Merged

Make DocsForge work for the models it was built for#22
R204570 merged 1 commit into
mainfrom
docs-architecture

Conversation

@R204570

@R204570 R204570 commented Aug 20, 2026

Copy link
Copy Markdown
Owner

From a real session against the web chat with Ollama running qwen3.5:9b — the
audience DocsForge exists for. The user asked five times for Astro's
documentation. Four times the model answered by listing the knowledge base back
at them; on the fifth it harvested Astro's blog.

Four defects. The accuracy fixture passed throughout and could not have caught
any of them.

F10 — why it works in Claude Code and not with Ollama

The system prompt said:

Before learning anything, check list_knowledge_base … (learn_technology
checks this for you.)

An imperative cancelled by a parenthetical. Claude weighs the two and skips
the call. A 9B model obeys the imperative
, receives 2 KB of listing, and then
answers about the most recent large blob of text in its context instead of the
question — four turns running, exactly as the transcript shows.

The parenthetical was true: tool_learn_technology already calls
stored_name() and returns without fetching. So the instruction was redundant
and actively harmful, and harmful only to the models the product exists to
serve
. Nothing was wrong with the tools — the guidance around them was written
for a reader who could resolve a contradiction.

Now leads with the rule instead of the caveat: call learn_technology
immediately, never call list_knowledge_base to decide what to do, never stop
at a tool result. MAX_ROUNDS 4 → 6, because three rounds is the happy path, a
small model reliably wastes one, and at four it then ran out and was forced to
answer without having read anything
.

F11 — the docs root thrown away, again

astro resolved to astro.build with five identity signals. Right project,
wrong page. The docs root was found and discarded for the second time, for a
different reason than F3:

https://docs.astro.build/   ->  200,  0 characters of visible text

Client-rendered. No meta-refresh to follow, no text to measure, so the content
floor rejected it — correct by its own rule, wrong in fact.

A project's own documentation host is now exempt from the floor. Conditional on
the host also owning the name, because docs.rs is a "docs." host too and
that is where htmx went wrong.

F12 — the defect that actually produced the blog

docs_scope("https://astro.build")   ->  "/"
40 pages harvested                  ->  34 blog, 0 documentation

Any technology resolving to a homepage harvested marketing. A whole-host
sitemap is now narrowed to its documentation — and only when scope is the entire
host, since a caller who named a section meant that section.

F13 — found while verifying F12

docs.astro.build/sitemap-index.xml  ->  5,880 URLs, every translation
first 25 pages                      ->  all Arabic

Locale-sorted, /ar/ first, so a capped harvest never reaches English. Keeping
every language is no better — twenty times the corpus, and search returning the
same page in languages the caller cannot read. Matched against a curated list of
locale codes rather than "any two letters", because /go/, /js/ and /ai/
are sections.

The fixture was too lenient, and that is why this survived

It accepted astro.build as a correct answer for astro — settling for the
right project instead of the right page. It now requires docs.astro.build.
That one line is the difference between a regression the suite catches and a bug
a user has to report.

Measured after

astro  ->  https://docs.astro.build/      identified by own-domain, docs-host
           40 pages, 423,325 characters, all /en/, 0 blog
           reported honestly: INCOMPLETE, 345+ pages still queued

What this changed in the proposal

A new §4.6, because the architecture had nothing to say about whether a weak
model can drive it:

Guidance written for a strong model is not neutral for a weak one. A
caveat a capable reader silently applies is an instruction a small model
follows literally. The tool surface has to be legible to the least capable
caller in the target audience, not merely unambiguous to the most capable.

Every defect here was found by using the product as its actual audience would —
a small model, a plain question — and none by testing a component. A resolution
fixture measures resolution; it says nothing about whether a 9B model can get an
answer out of the thing.

Verified

  • 356 passed, 22 skipped across both backends, with new regression tests for
    each of the four.
  • Live fixture: 22 passed in 3m59s with the stricter astro expectation.
  • End to end through the real tool, reproducing the user's exact request.
  • All four mermaid diagrams rendered in Chromium via mmdc.

A real session against the web chat with Ollama running qwen3.5:9b. The user
asked five times for Astro's documentation. Four times the model answered by
listing the knowledge base back at them, and on the fifth it harvested Astro's
blog. Four defects, none of which the accuracy fixture could see.

F10, and the answer to "why does it work in Claude Code but not Ollama".
SYSTEM_PROMPT said "Before learning anything, check list_knowledge_base ...
(learn_technology checks this for you)" -- an imperative cancelled by a
parenthetical. A strong model weighs the two and skips the call; a 9B model
obeys the imperative, gets 2 KB of listing back, and then answers about the
most recent blob of text in its context instead of the question. The
parenthetical was true, so the instruction was redundant as well as harmful,
and harmful only to the models this product exists to serve. Nothing was wrong
with the tools. The guidance around them was written for a reader who could
resolve a contradiction.

The prompt now leads with the rule instead of the caveat: call
learn_technology immediately, never call list_knowledge_base to decide what to
do, never stop at a tool result. MAX_ROUNDS goes 4 -> 6, because three rounds
is the happy path, a small model reliably wastes one, and at four it then ran
out and was forced to answer without having read anything.

F11. astro resolved to astro.build with five identity signals -- right
project, wrong page. The docs root was found and discarded for the second time
and a different reason: docs.astro.build returns 200 with zero characters of
visible text because it renders client-side, so the content floor added for F3
rejected it. Correct by its own rule, wrong in fact. A project's own
documentation host is now exempt from the floor, conditional on the host also
owning the name -- docs.rs is a "docs." host too, and it is where htmx went
wrong.

F12, the defect that actually produced the blog. docs_scope("astro.build") is
"/", so the sitemap crawl took the whole host: 34 of 40 pages were blog posts
and none were documentation. Any technology resolving to a homepage harvested
marketing. A whole-host sitemap is now narrowed to its documentation, and only
when scope is the entire host -- a caller who named a section meant it.

F13, found while verifying F12. docs.astro.build lists 5,880 URLs across every
translation, locale-sorted, so a capped harvest returned Arabic and stopped
before English. Keeping every language is no better: twenty times the corpus
and search returning the same page in languages the caller cannot read.
Matched against a curated list of locale codes rather than any two letters,
because /go/, /js/ and /ai/ are sections.

The fixture is tightened to require docs.astro.build for astro. Accepting
astro.build is what let F11 and F12 survive -- it settled for the right
project instead of the right page, and that one line is the difference between
a regression the suite catches and a bug a user has to report.

Measured after: astro resolves to docs.astro.build, harvests 40 pages and
423,325 characters, all English, zero blog, and reports itself INCOMPLETE with
345+ pages still queued.

356 passed, 22 skipped across both backends. The live fixture passes 22 checks
in 3m59s with the stricter expectation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APThnqHvfP2oBjBCBxBhd6
@R204570
R204570 merged commit 70a14be into main Aug 20, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant