Re-measure the audit and proposal against the fixed code - #21
Merged
Conversation
Both were still written as if nothing had been fixed. The audit described its findings in the present tense and the proposal's acceptance table said "Today: 3/8", which is no longer true of either. AUDIT.md becomes a two-pass document. Sections 1 to 5 are left exactly as written -- they are the record of what was true before anything changed, and rewriting them into the past tense would destroy the evidence the fixes were built from. Section 6 marks each planned item done or not. A new section 7 carries fresh measurements taken against 316c7f2: resolution 3 correct / 3 wrong -> 7 correct / 0 wrong the seven indexes 155,442 chars -> 19,116,128 reachable (123x) pydantic read 24 blocks of 1.10 -> 85 blocks of 2.11 is_forge(gist.github.com) -> True tests 284 -> 346 across both backends Section 7.2 records the three defects that were found by running the system rather than reading it -- the astrology site, terraform.com, and the fourth version lookup -- because two of them were regressions introduced by the fixes themselves and would have shipped. That is a better argument for the live fixture than anything the document says about it in the abstract. One thing the fix status must not blur: the code is fixed and the stored corpus is not. All seven affected technologies were harvested before any of this, so they still hold their indexes and still read complete: True. That is stale data rather than a live defect, but it is called out at the top, in 7.3 and in the summary table, because an audit that says "fixed" while the store still shows the symptom would be exactly the kind of unearned confidence this whole exercise was about. A re-harvest clears it and takes the store from 8.4 to roughly 27.5 million characters. It is the one action outstanding. PROPOSAL.md gains an achieved column on the acceptance criteria, marks phases A, B, C and D3 done, and says plainly that D1 is not -- a long harvest still blocks past MCP client timeouts, and that is an architectural change rather than a fix. Two open questions are answered by having built the thing: chunk size is settled by heading level chosen on results, and conflict presentation turned out to be the wrong question, since domain-first never has both answers in hand. A new one is opened in their place, about what `expected` should compare against for a full dump, where making the count authoritative risks crying wolf and leaving it advisory misses a class of partial harvest. Documents only. 346 passed, 22 skipped across both backends. All four mermaid diagrams rendered in Chromium via mmdc. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APThnqHvfP2oBjBCBxBhd6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both documents were still written as if nothing had been fixed. The audit
described its findings in the present tense — "Live, right now: pydantic →
1.10" — and the proposal's acceptance table said "Today: 3/8". Neither is true
any more. Re-measured against
316c7f2and updated.AUDIT.md— now a two-pass documentSections 1 to 5 are left exactly as written. They are the record of what was
true before anything changed, and rewriting them into the past tense would
destroy the evidence the fixes were built from. They are marked as-found.
Section 6 marks each planned item done or not. A new section 7 carries fresh
measurements:
read_knowledge_base("pydantic")is_forge("gist.github.com/…")§7.1 also records something worth noticing:
fastapiandastroresolvethrough the registry and are still correct. Domain-first is a preference,
not a rule — the identity checks are what make the registry path safe.
The thing the fix status must not blur
The code is fixed. The stored corpus is not. All seven affected
technologies were harvested before any of this, so they still hold their
indexes and still read
complete: True.That is stale data rather than a live defect, but it is called out at the top,
in §7.3, and in the summary table — because an audit that says "fixed" while
the store still shows the symptom would be exactly the kind of unearned
confidence this whole exercise was about. A re-harvest clears it and takes
the store from 8.4 M to roughly 27.5 M characters. It is the one action
outstanding.
§7.2 — three defects found by running, not reading
Recorded because two of them were regressions introduced by the fixes and
would otherwise have shipped:
astroresolved to an astrology site. It ownsastro.com, it isenormous, and it says "astro" constantly — every signal a name-plus-size
check has, and none of the ones that matter.
terraform.com, an unrelated company, outrankedterraform.ioon pagesize.
PostgresStore._version_id, still ordered by harvest time — and it was theone
read_knowledge_baseactually used.The first two came from the live fixture, the third from querying the real
database. Each looked correct in the source. That is a better argument for the
fixture than anything the document says about it in the abstract, and it is now
the closing point of the summary: the fixes are worth less than the thing that
will catch the next one.
PROPOSAL.mdnot.
harvest still blocks past MCP client timeouts, and that is an architectural
change rather than a fix.
settled — heading level chosen by result, giving ai-sdk 2,605 pages and hono
440. Conflict presentation turned out to be the wrong question: domain-first
returns before consulting the registry, so it never has both answers in hand.
The real conflict case was two live domains, settled by evidence.
expectedshould compareagainst for a full dump. Making the sitemap count authoritative risks crying
wolf; leaving it advisory misses a class of partial harvest. Undecided, and
deliberately so.
Verified
only.
mmdc, not parsed in Node,which reports false failures on flowcharts.