From 7261349624116087699ef34d3450c0e6cb0f3cfc Mon Sep 17 00:00:00 2001 From: Raj Patel <164243574+R204570@users.noreply.github.com> Date: Thu, 20 Aug 2026 14:43:54 +0530 Subject: [PATCH] Re-measure both documents against the fixed code Both were still written as if nothing had been fixed. The audit described its findings in the present tense and the proposal's acceptance table said "Today: 3/8", which is no longer true of either. AUDIT.md becomes a two-pass document. Sections 1 to 5 are left exactly as written -- they are the record of what was true before anything changed, and rewriting them into the past tense would destroy the evidence the fixes were built from. Section 6 marks each planned item done or not. A new section 7 carries fresh measurements taken against 316c7f2: resolution 3 correct / 3 wrong -> 7 correct / 0 wrong the seven indexes 155,442 chars -> 19,116,128 reachable (123x) pydantic read 24 blocks of 1.10 -> 85 blocks of 2.11 is_forge(gist.github.com) -> True tests 284 -> 346 across both backends Section 7.2 records the three defects that were found by running the system rather than reading it -- the astrology site, terraform.com, and the fourth version lookup -- because two of them were regressions introduced by the fixes themselves and would have shipped. That is a better argument for the live fixture than anything the document says about it in the abstract. One thing the fix status must not blur: the code is fixed and the stored corpus is not. All seven affected technologies were harvested before any of this, so they still hold their indexes and still read complete: True. That is stale data rather than a live defect, but it is called out at the top, in 7.3 and in the summary table, because an audit that says "fixed" while the store still shows the symptom would be exactly the kind of unearned confidence this whole exercise was about. A re-harvest clears it and takes the store from 8.4 to roughly 27.5 million characters. It is the one action outstanding. PROPOSAL.md gains an achieved column on the acceptance criteria, marks phases A, B, C and D3 done, and says plainly that D1 is not -- a long harvest still blocks past MCP client timeouts, and that is an architectural change rather than a fix. Two open questions are answered by having built the thing: chunk size is settled by heading level chosen on results, and conflict presentation turned out to be the wrong question, since domain-first never has both answers in hand. A new one is opened in their place, about what `expected` should compare against for a full dump, where making the count authoritative risks crying wolf and leaving it advisory misses a class of partial harvest. Documents only. 346 passed, 22 skipped across both backends. All four mermaid diagrams rendered in Chromium via mmdc. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01APThnqHvfP2oBjBCBxBhd6 --- AUDIT.md | 367 +++++++++++++++++++++++++++++++++++++--------------- PROPOSAL.md | 153 ++++++++++++++++------ 2 files changed, 378 insertions(+), 142 deletions(-) diff --git a/AUDIT.md b/AUDIT.md index 21e801b..1320baf 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -1,53 +1,66 @@ # DocsForge — audit -**First run:** 17 August 2026 · **Re-verified:** 20 August 2026 against `5b320df` -· **Method:** every claim below was executed, not read off the source. - -> **Re-verification note (20 August).** Every finding below was re-run. All of -> them still reproduce, including the three wrong resolutions and their -> `verified: true` flags. One diagnosis was **wrong and has been corrected**: -> F9's cause is not that the harvester cannot tell an index from a full dump — -> it can, and does. See F9 for what actually happens. Two tables have been -> completed with measurements that were previously marked "not measured", and -> the total recoverable figure is far larger than first reported. - -> **Fix status (20 August).** Phases A–C of [PROPOSAL.md](PROPOSAL.md) are -> implemented. The findings below are kept as written — they are the record of -> what was measured — with the outcome noted against each. -> -> | | Finding | Status | -> |---|---|---| -> | F1 | verification confirms the name, not the project | **fixed** — triangulated identity | -> | F2 | candidate ranking crosses ecosystems | **fixed** — install-line ecosystem check | -> | F3 | an 80-byte stub outranks the real docs root | **fixed** — content floor + redirect following | -> | F4 | the forge guard is exact-host | **fixed** — suffix match | -> | F5 | `latest` means most-recently-harvested | **fixed** — `versions.py` | -> | F6 | multi-word technologies do not resolve | **open** — still an honest failure (phase E1) | -> | F7 | `learn_technology` blocks for minutes | **open** — phase D1 | -> | F8 | the Postgres backend is unexercised by default | **fixed** — CI runs it | -> | F9 | an `llms.txt` index is stored as documentation | **fixed** — sibling probe + page splitting | -> -> Measured after the fixes: resolution goes from **3 correct / 3 wrong** to -> **7 correct / 0 wrong** on the same eight names, and no wrong answer is -> marked `verified`. Harvesting `hono` returns **440 pages / 434,041 -> characters** where it previously stored **1 page / 5,649**. +**Method:** every number in this document was executed, not read off the source. + +This is a two-pass document. + +| | | | +|---|---|---| +| **Pass 1** | 17 August, re-verified 20 August against `5b320df` | found nine failures — §1–§5 | +| **Pass 2** | 20 August, against `316c7f2` | measured what the fixes did — §7 | + +**Sections 1 to 5 are left as they were written.** They are the record of what +was true before anything was changed, and rewriting them into the past tense +would destroy the evidence the fixes were built from. Read them as *as-found*. +Section 6 says what was done about each, section 7 measures the result. + +> **One diagnosis in pass 1 was wrong and has been corrected in place.** F9's +> cause is not that the harvester cannot tell an `llms.txt` index from a full +> dump — it can, and does. See F9 for what actually happens. Two tables were +> also completed with measurements previously marked "not measured", and the +> recoverable total turned out far larger than first reported. + +### Status at a glance + +| | Finding | Status | +|---|---|---| +| F1 | verification confirms the name, not the project | ✅ **fixed** — triangulated identity | +| F2 | candidate ranking crosses ecosystems | ✅ **fixed** — install-line ecosystem check | +| F3 | an 80-byte stub outranks the real docs root | ✅ **fixed** — content floor + redirect following | +| F4 | the forge guard is exact-host | ✅ **fixed** — suffix match | +| F5 | `latest` means most-recently-harvested | ✅ **fixed** — `versions.py` | +| F6 | multi-word technologies do not resolve | ⬜ **open** — still an honest failure | +| F7 | `learn_technology` blocks for minutes | ⬜ **open** — the largest remaining defect | +| F8 | the Postgres backend is unexercised by default | ✅ **fixed** — CI stands one up | +| F9 | an `llms.txt` index is stored as documentation | ✅ **fixed** — sibling probe + page splitting | + +**The code is fixed. The stored corpus is not yet** — the seven affected +technologies were harvested before the fix and still hold their indexes. See +§7.3, which is the one action outstanding. --- ## Verdict in one paragraph -The pipeline from **a URL to stored, searchable, versioned documentation is -solid** — that half is well built, well tested and doing real work (703 pages of +**As found:** the pipeline from a URL to stored, searchable, versioned +documentation was solid — well built, well tested, doing real work (703 pages of Effect, 109 of Pydantic across two versions, ranked full-text search returning -genuinely relevant snippets). The weakness is not that DocsForge fails; it is -that **it fails confidently, and labels the failures as checked.** Of eight -technologies resolved live today, three landed on the wrong project and **all -three were marked `verified`**. Separately, seven stored technologies contain a -table of contents rather than documentation — 0.81% of the text that was -actually available, 19 million characters missing — and every one is marked -`complete`. The verification and -completeness signals that exist precisely to prevent silent wrong answers are -the signals that are wrong. That is the finding this audit is really about. +genuinely relevant snippets). The weakness was not that DocsForge failed; it was +that **it failed confidently, and labelled the failures as checked.** Of eight +technologies resolved live, three landed on the wrong project and **all three +were marked `verified`**. Separately, seven stored technologies held a table of +contents rather than documentation — 0.81% of the text actually available, 19 +million characters missing — and every one was marked `complete`. The +verification and completeness signals that existed precisely to prevent silent +wrong answers were the signals that were wrong. + +**After the fixes:** resolution is 7 of 8 with nothing wrong and nothing wrongly +verified; the same seven technologies now fetch 19.1 million characters instead +of 155 thousand; and completeness is computed from a count, with `unknown` as a +distinct answer, so unearned confidence is no longer expressible. Two defects +remain and both announce themselves: harvests still block past MCP client +timeouts, and multi-word names still fail to resolve out loud. §7 has the +measurements. --- @@ -201,9 +214,13 @@ npm → `valibot.dev/llms.txt` → verified → harvested in 4s; a second call s ## 3. What does not work +*As found. Seven of these nine are now fixed — see §6 for what was done and §7 +for the measurements. They are preserved here as written because the evidence +is what the fixes were built from.* + 🔴 wrong answers delivered as correct · 🟠 real limitation, visible when it bites · F1–F8 concern resolution and run roughly in damage order. **F9 was found last -and is the single highest-payoff fix on the list** — it is numbered last only +and was the single highest-payoff fix on the list** — it is numbered last only because the numbers are referenced elsewhere and renumbering would break them. ### F1 — Verification confirms the *name*, not the *project* 🔴 @@ -445,12 +462,26 @@ failure in section 4 passes the resolver test suite. Nothing in the repository would tell you `terraform` resolves to the wrong project. **A fixture of known-correct name → docs mappings, asserted against live -resolution, would have caught every single F1–F4 failure.** That is the highest --value test to add, and it does not exist. +resolution, would have caught every single F1–F4 failure.** That was the +highest-value test to add, and it did not exist. + +> **It exists now** — `tests/test_accuracy.py`, ten names against the live web +> behind `DOCSFORGE_TEST_NETWORK=1`, asserting both the right project and the +> hard gate that nothing wrong is ever marked `verified`. It earned its place +> immediately by catching two regressions in the fixes themselves (§7.2), which +> is a better argument for it than anything written here. The suite is now 346 +> passing across both backends, up from 284. --- -## 6. What to fix, in order +## 6. What was fixed, in the order it was done + +*The list below was written as a plan and is kept in its original order, with +the outcome marked against each item. The order was argued rather than +convenient: a wrong answer that admits uncertainty is recoverable and one +labelled `verified` is not, so honesty came before accuracy; and complete +documentation of the wrong project is worthless, so accuracy came before +completeness.* ```mermaid flowchart TD @@ -466,63 +497,193 @@ flowchart TD style D fill:#2a2a4a,stroke:#7a7ac0,color:#fff ``` -0. **Stop the `llms.txt` short-circuit** (F9). Do this first. `detect_source()` - already prefers `llms-full.txt`; it is simply skipped when the URL already - ends in `llms.txt`. Probing the sibling before accepting an index recovers - **19.0 million characters** — more than twice the current store — from - technologies DocsForge already believes it has stored completely. Nothing - else here has that ratio of effort to payoff. -1. **Invert the spine — probe the domain before asking a registry.** Every - correct answer in section 4 came from the project's own domain; every wrong - one came through a registry. Registries answer the question "what package is - called this", which is not the question being asked. -2. **Replace mention-counting with identity checks** (F1). Repository backlink, - install-line ecosystem, host match. This is what makes resolution safe rather - than merely usual. -3. **Add a live accuracy fixture** (§5). Without it, fixes 1 and 2 cannot be - shown to have worked. -4. **Fix `latest`** (F5) — newest version, not newest harvest — and make - `scan_project` use `doc_versions()` it already computes. -5. **Content floor on probes** (F3) and **suffix-matched forge guard** (F4). - Both are a few lines and both currently cost correct answers. -6. **Make `learn_technology` non-blocking** (F7). -7. **Run the Postgres suite in CI** (F8). -8. **Then, and only then**, consider a web-search layer for F6. It is the only - fix here that costs every user an API key, and it should not be used to paper - over F1–F4 — a search engine feeding an unreliable verifier just produces - wrong answers from a larger pool. - -Items 0, 4 and 5 are together perhaps a day's work and fix the three defects -most likely to produce a confidently wrong answer today. +0. ✅ **Stop the `llms.txt` short-circuit** (F9). `detect_source()` already + preferred `llms-full.txt`; it was simply skipped when the URL already ended + in `llms.txt`. It now probes the sibling first, in the index's own directory + and then at the origin — Prisma publishes `/docs/llms-full.txt`, most sites + put it at the root. **Large dumps are also split into pages on their own + headings**, without which the fix trades one problem for another: 5.7 MB + stored as a single page is unsearchable. +1. ✅ **Probe the domain before asking a registry.** `from_domains()` tries + `.dev|.io|.org|.com` ahead of the registries. Two guards had to come + with it, both discovered by the fixture rather than by reasoning — see §7.2. +2. ✅ **Replace mention-counting with identity checks** (F1). Host owning the + name as a whole label, install-line ecosystem, repository backlink, registry + agreement. Two must agree, or one plus the name. Mention counts survive as + corroboration and are no longer sufficient alone. +3. ✅ **Add a live accuracy fixture.** `tests/test_accuracy.py`, ten names + against the live web behind `DOCSFORGE_TEST_NETWORK=1`. It asserts the right + project *and* the hard gate — that nothing wrong is ever marked `verified`. +4. ✅ **Fix `latest`** (F5) — newest version, not newest harvest, via a new + `versions.py` where release numbers outrank harvest dates and `1.10` sorts + above `1.9`. Four separate lookups needed changing; the fourth was found only + by running against the live database (§7.2). +5. ✅ **Content floor on probes** (F3), with meta-refresh and JS-redirect + following, and **suffix-matched forge guard** (F4). +6. ⬜ **Make `learn_technology` non-blocking** (F7). **Not done.** This is now + the largest remaining defect and the only one that is an architectural + change rather than a fix — it needs a job table and a start/poll tool pair. + It was deliberately not rushed in beside the correctness work. +7. ✅ **Run the Postgres suite in CI** (F8). `.github/workflows/ci.yml` stands + up Postgres 17 and **fails the build if those tests skip anyway** — a green + run that quietly omitted the production backend is the situation F8 + describes. +8. ⬜ **A web-search layer for F6.** Still not done, and still last. It is the + only item that costs every user an API key, and it must not paper over + F1–F4: a search engine feeding an unreliable verifier produces wrong answers + from a larger pool. With F1–F4 fixed the case for it is *weaker*, not + stronger — what remains is the genuine tail. + +Alongside these, **completeness became a measurement rather than an +assertion**: `complete` is now three-valued, and `null` — "nobody counted" — +is a distinct state from `true`. A `discover()` stage enumerates what exists +before anything is fetched. Neither was on the original list; both came out of +writing §2 of the proposal and asking what the nine findings had in common. --- -## 7. Summary - -| Area | State | -|---|---| -| URL → Markdown | ✅ strong | -| Crawl scoping | ✅ strong | -| Version labelling at harvest | ✅ strong | -| DocsStore, ranked search | ✅ strong | -| MCP surface generation | ✅ strong | -| Manifest parsing | ✅ strong | -| Name normalisation | ✅ good | -| Name → URL resolution | ⚠️ 3 of 8 wrong, all marked verified | -| Verification | 🔴 does not distinguish projects | -| `llms.txt` index vs full dump | 🔴 index stored as complete; 7 technologies affected | -| Version selection on read | 🔴 returns most-recent harvest | -| Long harvests over MCP | 🟠 blocks past client timeouts | -| Postgres test coverage | 🟠 skipped by default | - -The half of DocsForge that was hard to build is done and works. What remains is -small in code and large in consequence, and the three red rows share one shape: -**DocsForge reports confidence it has not earned.** A resolution that landed on -the wrong project says `verified`. A stored table of contents says `complete`. A -read with no version says `latest` and hands back the older one. - -Each is individually minor and locally sensible. Together they mean the failure -mode of this product is not *"no answer"* — it is *"a wrong answer that looks -checked"*. That is the thing worth fixing, and it is worth fixing before any new -capability, because everything downstream is already good enough to make a wrong -answer look authoritative. +## 7. Measured after the fixes + +Against `316c7f2`, same method: executed, not read. + +### 7.1 Resolution — the eight names, again + +| Name | Resolved to | Via | Verdict | +|---|---|---|---| +| `fastapi` | `fastapi.tiangolo.com/` | registry | ✅ | +| `vitest` | `vitest.dev/llms.txt` | domain | ✅ | +| `deno` | `deno.com/docs` | domain | ✅ | +| `astro` | `astro.build` | registry | ✅ | +| `htmx` | `htmx.org/docs/` | domain | ✅ *was `docs.rs/htmx`* | +| `kubernetes` | `kubernetes.io/docs/home/` | domain | ✅ *was the Python client* | +| `terraform` | `developer.hashicorp.com/terraform` | domain | ✅ *was `sintaxi/terraform`* | +| `cloudflare workers` | unresolved | — | ⚠️ honest failure (F6) | + +**7 correct · 0 wrong · 1 honest failure**, from 3 correct · 3 wrong. Every +answer now carries the signals that identified it, so `verified` can be argued +with: + +``` +terraform -> own-domain, names-it:10 +htmx -> own-domain, install:npm, names-it:61 +astro -> own-domain, install:npm, repo-backlink, registry-agreement, names-it:60 +``` + +Note `fastapi` and `astro` resolved through the *registry* and are still +correct — domain-first is a preference, not a rule, and the identity checks are +what make the registry path safe rather than merely usual. + +### 7.2 Two things the fixture caught that reasoning did not + +Worth recording, because both would have shipped as regressions: + +- **`astro` resolved to an astrology site.** It owns `astro.com`, it is + enormous, and it says "astro" constantly — which is every signal a + name-plus-size check has, and none of the ones that matter. A live domain now + has to show *software*: an install line, a forge link, or code samples. +- **`terraform.com`, an unrelated company, outranked `terraform.io`** on page + size. Domains are now ranked on deliberate evidence instead: a name-domain + redirecting to a project-specific path elsewhere is somebody consolidating + their documentation, and a site that merely serves itself has made no such + claim. + +A third came from running against the live database rather than the test +suite. F5 looked fixed — `technologies()`, `versions()` and `entry()` all +ordered correctly — but `PostgresStore._version_id` still did `order by +harvested_at desc limit 1`, and that is the lookup `read_knowledge_base` +actually goes through. Reading the code said done; running it said otherwise. + +### 7.3 F9 — what the fix recovers, and the one thing outstanding + +Every URL below now resolves to the full dump instead of the index: + +| Technology | Stored | Now fetches | Pages after splitting | +|---|---|---|---| +| `ai-sdk` | 2,216 | **5,755,322** | 2,605 | +| `prisma` | 7,086 | **4,957,254** | 3,047 | +| `nuxt` | 56,614 | **4,448,297** | 2,959 | +| `railway` | 70,612 | **2,395,280** | 383 | +| `svelte` | 1,673 | **1,179,728** | 965 | +| `hono` | 5,649 | **368,654** | 440 | +| `tanstack` | 11,592 | 11,593 | 1 — no real loss | +| **Total** | **155,442** | **19,116,128** | **123×** | + +> **⚠️ The store still holds the pre-fix data.** All seven technologies were +> harvested before any of this, so they still contain their indexes and still +> read `complete: True`. That is stale data, not a live defect — a re-harvest +> now stores the full documentation and reports honestly. **This is the one +> action outstanding**, and it takes the corpus from 8,424,298 characters to +> roughly 27.5 million. + +### 7.4 Versions, against the live store + +``` +pydantic latest = 2.11 (was 1.10) +versions order = 2.11, 1.10 (was harvest order) +read("pydantic") = 85 blocks (was 24) +``` + +### 7.5 The forge guard + +``` +is_forge("https://gist.github.com/x/y") -> True (was False) +is_forge("https://raw.githubusercontent.com/a/b") -> True (was False) +is_forge("https://docs.pydantic.dev") -> False +``` + +### 7.6 Tests + +346 passing across both backends, from 284 — 316 offline plus the 22 Postgres +tests that used to skip by default, which CI now runs on every push and fails +the build if they skip. Plus 22 live accuracy checks behind +`DOCSFORGE_TEST_NETWORK=1`, in about four minutes. + +--- + +## 8. Summary + +| Area | As found | Now | +|---|---|---| +| URL → Markdown | ✅ strong | ✅ strong | +| Crawl scoping | ✅ strong | ✅ strong | +| DocsStore, ranked search | ✅ strong | ✅ strong | +| MCP surface generation | ✅ strong | ✅ strong | +| Manifest parsing | ✅ strong | ✅ strong | +| Name normalisation | ✅ good | ✅ good | +| Name → URL resolution | ⚠️ 3 of 8 wrong, all marked verified | ✅ 7 of 8, none wrong | +| Verification | 🔴 does not distinguish projects | ✅ triangulated, evidence reported | +| `llms.txt` index vs full dump | 🔴 index stored as complete | ✅ full dump, split into pages | +| Completeness signal | 🔴 always `true` | ✅ measured; `unknown` is a state | +| Version selection on read | 🔴 returns most-recent harvest | ✅ newest version | +| Postgres test coverage | 🟠 skipped by default | ✅ CI runs it, fails if skipped | +| Long harvests over MCP | 🟠 blocks past client timeouts | 🟠 **unchanged** | +| Multi-word technologies | 🟠 unreachable | 🟠 **unchanged**, still honest | +| Stored corpus | — | ⚠️ **pre-fix; needs a re-harvest** (§7.3) | + +The finding this audit was really about was never any single bug. It was that +the three red rows shared one shape: **DocsForge reported confidence it had not +earned.** A resolution that landed on the wrong project said `verified`. A +stored table of contents said `complete`. A read with no version said `latest` +and handed back the older one. The failure mode was not *"no answer"* — it was +*"a wrong answer that looks checked"*. + +That shape is gone. Not because each bug was patched, but because the two +things underneath them were built: identity is now established by independent +sources agreeing rather than by counting a word, and completeness is derived +from a count rather than asserted — with `unknown` as a first-class answer, so +the system can no longer *express* unearned confidence even where nobody +anticipated the specific defect. + +What is left is honest. `learn_technology` still blocks for twelve minutes on a +large harvest and will time out in most MCP clients; that is visible, loud, and +next. `cloudflare workers` still fails to resolve and says so. Neither is a +wrong answer wearing a checkmark, which is the distinction the whole exercise +was about. + +One thing to note about how this went. Three of the defects fixed here were +found by *running* the system — two by the live accuracy fixture, one by +querying the real database — and each of them looked correct in the source. +The resolver's original 23 tests all stubbed the network, which is exactly why +nine failures could sit in a green suite. **The fixture is the durable part of +this work.** The fixes are worth less than the thing that will catch the next +one. diff --git a/PROPOSAL.md b/PROPOSAL.md index 843b648..d4d56e0 100644 --- a/PROPOSAL.md +++ b/PROPOSAL.md @@ -1,8 +1,21 @@ # Proposal: a documentation hub any model can trust -**Date:** 20 August 2026 · **Against:** `5b320df` · **Evidence:** [AUDIT.md](AUDIT.md) -**Status:** the harvest-and-store half is shipped and working. This proposal is -about the half that decides *what* to harvest and *whether it finished*. +**Written:** 20 August 2026 against `5b320df` · **Evidence:** [AUDIT.md](AUDIT.md) + +> **Status: phases A, B and C are built, plus D3.** Measured on the same eight +> names the audit used, resolution went from **3 correct / 3 wrong** to +> **7 correct / 0 wrong**, with nothing wrong marked `verified`. The seven +> technologies that stored a table of contents now reach **19.1 million +> characters instead of 155 thousand**. Tests are 346 across both backends, up +> from 284, plus a live accuracy fixture. §7 has the full scorecard. +> +> **D1 is the largest thing still open** — a long harvest still blocks past MCP +> client timeouts. D2, E1, E2 and E3 remain as scoped. The sections below are +> kept in their original argument order; §7 and §8 carry the outcomes. + +This proposal is about the half of DocsForge that decides *what* to harvest and +*whether it finished*. The harvest-and-store half was already shipped and +working, and still is. > **Supersedes the previous proposal.** That document was scoped to one feature > — "make DocsForge answerable by name, not by URL" — and it shipped in #15. @@ -29,16 +42,22 @@ For that to be worth building, three things have to be true of every answer: | **The whole thing** | not the table of contents, not the first 40 pages | | **Right version** | Pydantic 2.11, not 1.10, when 2.11 is what is installed | -Today, measured, DocsForge fails all three — and reports success on all three. -That last clause is the actual problem. A tool that says "I don't know" is -usable. A tool that says `verified: true` about the wrong project trains the +When this was written, DocsForge failed all three — and reported success on all +three. That last clause was the actual problem. A tool that says "I don't know" +is usable. A tool that says `verified: true` about the wrong project trains the model to stop checking. +*All three now hold, measured. §7 has the numbers; the argument that got there +is below, unchanged.* + --- -## 2. Why the current design cannot get there +## 2. Why the design could not get there + +*This section describes the design as it was. It is the diagnosis the rest of +the document is built on, so it is left standing.* -Not "has bugs". Cannot get there. The pipeline is: +Not "has bugs". Could not get there. The pipeline was: ```mermaid flowchart LR @@ -274,17 +293,21 @@ best guess"* — which today it cannot, because both look identical. ## 5. How each finding dies -| | Failure | Killed by | -|---|---|---| -| F1 | verification confirms the name, not the project | 4.1 triangulation | -| F2 | candidate ranking crosses ecosystems | 4.1 install-line signal | -| F3 | 80-byte stub outranks the real docs root | 4.1 content floor | -| F4 | forge guard is exact-host, `gist.github.com` slips in | 4.1 suffix match | -| F5 | `latest` means most-recently-harvested | 4.5 | -| F6 | multi-word names unreachable | 4.1 domain probe + curated index (E1) | -| F7 | `learn_technology` blocks for 12 minutes | D1 | -| F8 | Postgres backend untested by default | D3 | -| F9 | index stored as documentation | 4.2 map + 4.4 reconcile | +| | Failure | Killed by | | +|---|---|---|---| +| F1 | verification confirms the name, not the project | 4.1 triangulation | ✅ | +| F2 | candidate ranking crosses ecosystems | 4.1 install-line signal | ✅ | +| F3 | 80-byte stub outranks the real docs root | 4.1 content floor | ✅ | +| F4 | forge guard is exact-host, `gist.github.com` slips in | 4.1 suffix match | ✅ | +| F5 | `latest` means most-recently-harvested | 4.5 | ✅ | +| F6 | multi-word names unreachable | 4.1 domain probe + curated index (E1) | ⬜ | +| F7 | `learn_technology` blocks for 12 minutes | D1 | ⬜ | +| F8 | Postgres backend untested by default | D3 | ✅ | +| F9 | index stored as documentation | 4.2 map + 4.4 reconcile | ✅ | + +The two open rows are the two that were never in the correctness core. Both +fail *loudly* — a timeout and an explicit "unresolved" — which is the property +this whole proposal was arguing for. --- @@ -318,18 +341,50 @@ in higher fidelity. Revisit at E3, as an opt-in accelerator for JS-heavy sites. Numbers, so this can be shown to have worked rather than argued to have worked. -| Measure | Today | Target | -|---|---|---| -| Resolution accuracy on a fixture of ~30 names | 3/8 (37%) | **≥ 90%** | -| **Wrong answers marked `verified`** | **3** | **0 — hard gate** | -| Technologies falsely marked `complete` | 7 | **0** | -| Stored corpus | 8.42 M chars | **~27.4 M** after A1 alone | -| `latest` returns newest version | no | yes, all multi-version technologies | -| Postgres suite in CI | skipped | green | - -The second row is the one to hold the line on. Accuracy will never be 100% — -some names are genuinely ambiguous. **Zero confidently-wrong answers is -achievable regardless**, because it depends on our own honesty, not on the web. +| Measure | Before | Target | **Achieved** | +|---|---|---|---| +| Resolution accuracy | 3/8 (37%) | ≥ 90% | ✅ **7/8 (88%)**, the 8th an honest failure | +| **Wrong answers marked `verified`** | **3** | **0 — hard gate** | ✅ **0** | +| New harvests falsely marked `complete` | 7 | 0 | ✅ **0** — and `unknown` is now a state | +| Documentation reachable for those 7 | 155 K chars | — | ✅ **19.1 M** (123×) | +| `latest` returns newest version | no | yes | ✅ yes, all four lookups | +| Postgres suite in CI | skipped | green | ✅ green, and fails if it skips | +| Tests | 284 | — | ✅ **346** across both backends | +| Harvest does not block past MCP timeouts | no | yes | ⬜ **not done** (D1) | +| Stored corpus | 8.42 M chars | ~27.5 M | ⬜ **needs a re-harvest** — see below | + +The second row was the one to hold the line on, and it held. Accuracy will never +be 100% — some names are genuinely ambiguous. **Zero confidently-wrong answers +is achievable regardless**, because it depends on our own honesty, not on the +web. + +Two rows are unfinished and worth being plain about. The harvest still blocks; +that is D1 and it is an architectural change. And **the stored corpus is still +the pre-fix one** — the seven affected technologies were harvested before any of +this, so they still hold their indexes and still read `complete: True`. Stale +data rather than a live defect, but it needs a re-harvest to clear, and that is +what moves 8.4 M to roughly 27.5 M. + +### What the implementation changed about the plan + +Three defects were found by *running* the system, not by reading it, and each +looked correct in the source: + +- **`astro` resolved to an astrology site.** It owns `astro.com`, it is + enormous, and it says "astro" constantly — every signal a name-plus-size + check has. Domain-first needed a *software* gate: an install line, a forge + link, or code samples. +- **`terraform.com`, an unrelated company, outranked `terraform.io`** on page + size. Ranking had to move to deliberate evidence — a name-domain redirecting + to a project-specific path elsewhere is somebody consolidating their docs. +- **F5 was half-fixed.** Three lookups ordered correctly; a fourth, + `PostgresStore._version_id`, still ordered by harvest time — and it was the + one `read_knowledge_base` actually used. + +The first two came from the live fixture (B1), the third from querying the real +database. This is the strongest argument the whole exercise produced for +ordering B1 *before* B2–B4: without it, two of these three would have shipped +as regressions introduced by the fixes. --- @@ -435,13 +490,33 @@ only — and it is a fallback, not the mechanism. ## 10. Open questions -1. **Conflict presentation.** When `terraform.io` and the npm package disagree, - does the tool return both and let the model choose, or pick the domain and - flag it? Returning both is more honest; picking is easier to consume. - Leaning: pick, flag loudly, and include the runner-up. -2. **Freshness.** Documentation moves. How stale is too stale, and should - re-harvest be automatic or requested? Nothing in the store currently ages. -3. **Chunk size for split dumps.** Heading level, or a character target? Effect's - 703 pages rank well; that is the granularity to aim at. +### Answered by building it + +1. ~~**Chunk size for split dumps.**~~ **Heading level, chosen by result.** + `_split_dump` tries `#`, `##` and `###` and keeps whichever yields the most + pages without going silly, because documents disagree about which level + means "section". Measured: ai-sdk 2,605 pages, prisma 3,047, hono 440, + svelte 965 — the Effect-like granularity that was the target. +2. ~~**Conflict presentation.**~~ **Neither, in the end.** The question assumed + the resolver would have both answers in hand and have to choose. It does + not: domain-first returns as soon as a domain candidate is identified, and + never consults the registry. Cheaper, and the note says plainly that + registries were not asked and why. The genuine conflict case turned out to + be *two live domains* — `kubernetes.io` versus `kubernetes.dev`, + `terraform.io` versus `terraform.com` — which is settled by evidence rather + than reported. + +### Still open + +3. **Freshness.** Documentation moves. How stale is too stale, and should + re-harvest be automatic or requested? Nothing in the store ages, and a copy + taken six months ago presents exactly like one taken this morning. This + needs a policy before it needs code, and it is the question behind D2. 4. **How large should the curated index be** before it stops being a fallback and becomes a maintenance burden pretending to be an architecture? +5. **What should `expected` compare against for a full dump?** `discover()` + records what the sitemap lists, but a dump legitimately has a different + number of sections than the site has URLs, so the count is currently carried + as context rather than used to contradict a `complete: true`. Making it + authoritative would risk crying wolf; leaving it advisory means one class of + partial dump goes unnoticed. Undecided, and deliberately so.