Make DocsForge work for the models it was built for - #22
Merged
Conversation
A real session against the web chat with Ollama running qwen3.5:9b. The user
asked five times for Astro's documentation. Four times the model answered by
listing the knowledge base back at them, and on the fifth it harvested Astro's
blog. Four defects, none of which the accuracy fixture could see.
F10, and the answer to "why does it work in Claude Code but not Ollama".
SYSTEM_PROMPT said "Before learning anything, check list_knowledge_base ...
(learn_technology checks this for you)" -- an imperative cancelled by a
parenthetical. A strong model weighs the two and skips the call; a 9B model
obeys the imperative, gets 2 KB of listing back, and then answers about the
most recent blob of text in its context instead of the question. The
parenthetical was true, so the instruction was redundant as well as harmful,
and harmful only to the models this product exists to serve. Nothing was wrong
with the tools. The guidance around them was written for a reader who could
resolve a contradiction.
The prompt now leads with the rule instead of the caveat: call
learn_technology immediately, never call list_knowledge_base to decide what to
do, never stop at a tool result. MAX_ROUNDS goes 4 -> 6, because three rounds
is the happy path, a small model reliably wastes one, and at four it then ran
out and was forced to answer without having read anything.
F11. astro resolved to astro.build with five identity signals -- right
project, wrong page. The docs root was found and discarded for the second time
and a different reason: docs.astro.build returns 200 with zero characters of
visible text because it renders client-side, so the content floor added for F3
rejected it. Correct by its own rule, wrong in fact. A project's own
documentation host is now exempt from the floor, conditional on the host also
owning the name -- docs.rs is a "docs." host too, and it is where htmx went
wrong.
F12, the defect that actually produced the blog. docs_scope("astro.build") is
"/", so the sitemap crawl took the whole host: 34 of 40 pages were blog posts
and none were documentation. Any technology resolving to a homepage harvested
marketing. A whole-host sitemap is now narrowed to its documentation, and only
when scope is the entire host -- a caller who named a section meant it.
F13, found while verifying F12. docs.astro.build lists 5,880 URLs across every
translation, locale-sorted, so a capped harvest returned Arabic and stopped
before English. Keeping every language is no better: twenty times the corpus
and search returning the same page in languages the caller cannot read.
Matched against a curated list of locale codes rather than any two letters,
because /go/, /js/ and /ai/ are sections.
The fixture is tightened to require docs.astro.build for astro. Accepting
astro.build is what let F11 and F12 survive -- it settled for the right
project instead of the right page, and that one line is the difference between
a regression the suite catches and a bug a user has to report.
Measured after: astro resolves to docs.astro.build, harvests 40 pages and
423,325 characters, all English, zero blog, and reports itself INCOMPLETE with
345+ pages still queued.
356 passed, 22 skipped across both backends. The live fixture passes 22 checks
in 3m59s with the stricter expectation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APThnqHvfP2oBjBCBxBhd6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From a real session against the web chat with Ollama running
qwen3.5:9b— theaudience DocsForge exists for. The user asked five times for Astro's
documentation. Four times the model answered by listing the knowledge base back
at them; on the fifth it harvested Astro's blog.
Four defects. The accuracy fixture passed throughout and could not have caught
any of them.
F10 — why it works in Claude Code and not with Ollama
The system prompt said:
An imperative cancelled by a parenthetical. Claude weighs the two and skips
the call. A 9B model obeys the imperative, receives 2 KB of listing, and then
answers about the most recent large blob of text in its context instead of the
question — four turns running, exactly as the transcript shows.
The parenthetical was true:
tool_learn_technologyalready callsstored_name()and returns without fetching. So the instruction was redundantand actively harmful, and harmful only to the models the product exists to
serve. Nothing was wrong with the tools — the guidance around them was written
for a reader who could resolve a contradiction.
Now leads with the rule instead of the caveat: call
learn_technologyimmediately, never call
list_knowledge_baseto decide what to do, never stopat a tool result.
MAX_ROUNDS4 → 6, because three rounds is the happy path, asmall model reliably wastes one, and at four it then ran out and was forced to
answer without having read anything.
F11 — the docs root thrown away, again
astroresolved toastro.buildwith five identity signals. Right project,wrong page. The docs root was found and discarded for the second time, for a
different reason than F3:
Client-rendered. No meta-refresh to follow, no text to measure, so the content
floor rejected it — correct by its own rule, wrong in fact.
A project's own documentation host is now exempt from the floor. Conditional on
the host also owning the name, because
docs.rsis a "docs." host too andthat is where
htmxwent wrong.F12 — the defect that actually produced the blog
Any technology resolving to a homepage harvested marketing. A whole-host
sitemap is now narrowed to its documentation — and only when scope is the entire
host, since a caller who named a section meant that section.
F13 — found while verifying F12
Locale-sorted,
/ar/first, so a capped harvest never reaches English. Keepingevery language is no better — twenty times the corpus, and search returning the
same page in languages the caller cannot read. Matched against a curated list of
locale codes rather than "any two letters", because
/go/,/js/and/ai/are sections.
The fixture was too lenient, and that is why this survived
It accepted
astro.buildas a correct answer forastro— settling for theright project instead of the right page. It now requires
docs.astro.build.That one line is the difference between a regression the suite catches and a bug
a user has to report.
Measured after
What this changed in the proposal
A new §4.6, because the architecture had nothing to say about whether a weak
model can drive it:
Every defect here was found by using the product as its actual audience would —
a small model, a plain question — and none by testing a component. A resolution
fixture measures resolution; it says nothing about whether a 9B model can get an
answer out of the thing.
Verified
each of the four.
mmdc.