Skip to content

Re-measure the audit and proposal against the fixed code - #21

Merged
R204570 merged 1 commit into
mainfrom
docs-architecture
Aug 20, 2026
Merged

Re-measure the audit and proposal against the fixed code#21
R204570 merged 1 commit into
mainfrom
docs-architecture

Conversation

@R204570

@R204570 R204570 commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Both documents were still written as if nothing had been fixed. The audit
described its findings in the present tense — "Live, right now: pydantic →
1.10" — and the proposal's acceptance table said "Today: 3/8". Neither is true
any more. Re-measured against 316c7f2 and updated.

AUDIT.md — now a two-pass document

Sections 1 to 5 are left exactly as written. They are the record of what was
true before anything changed, and rewriting them into the past tense would
destroy the evidence the fixes were built from. They are marked as-found.
Section 6 marks each planned item done or not. A new section 7 carries fresh
measurements:

Before Now
Resolution 3 correct / 3 wrong 7 correct / 0 wrong
The seven index technologies 155,442 chars 19,116,128 reachable (123×)
read_knowledge_base("pydantic") 24 blocks of 1.10 85 blocks of 2.11
is_forge("gist.github.com/…") False True
Tests 284 346 across both backends

§7.1 also records something worth noticing: fastapi and astro resolve
through the registry and are still correct. Domain-first is a preference,
not a rule — the identity checks are what make the registry path safe.

The thing the fix status must not blur

The code is fixed. The stored corpus is not. All seven affected
technologies were harvested before any of this, so they still hold their
indexes and still read complete: True.

That is stale data rather than a live defect, but it is called out at the top,
in §7.3, and in the summary table — because an audit that says "fixed" while
the store still shows the symptom would be exactly the kind of unearned
confidence this whole exercise was about. A re-harvest clears it and takes
the store from 8.4 M to roughly 27.5 M characters. It is the one action
outstanding.

§7.2 — three defects found by running, not reading

Recorded because two of them were regressions introduced by the fixes and
would otherwise have shipped:

  • astro resolved to an astrology site. It owns astro.com, it is
    enormous, and it says "astro" constantly — every signal a name-plus-size
    check has, and none of the ones that matter.
  • terraform.com, an unrelated company, outranked terraform.io on page
    size.
  • F5 was half-fixed. Three version lookups ordered correctly; a fourth,
    PostgresStore._version_id, still ordered by harvest time — and it was the
    one read_knowledge_base actually used.

The first two came from the live fixture, the third from querying the real
database. Each looked correct in the source. That is a better argument for the
fixture than anything the document says about it in the abstract, and it is now
the closing point of the summary: the fixes are worth less than the thing that
will catch the next one.

PROPOSAL.md

  • Acceptance criteria gain an achieved column — seven rows met, two openly
    not.
  • Phases A, B, C and D3 marked done; D1 marked plainly not done. A long
    harvest still blocks past MCP client timeouts, and that is an architectural
    change rather than a fix.
  • Two open questions answered by having built the thing. Chunk size is
    settled — heading level chosen by result, giving ai-sdk 2,605 pages and hono
    440. Conflict presentation turned out to be the wrong question: domain-first
    returns before consulting the registry, so it never has both answers in hand.
    The real conflict case was two live domains, settled by evidence.
  • One new question opened in their place: what expected should compare
    against for a full dump. Making the sitemap count authoritative risks crying
    wolf; leaving it advisory misses a class of partial harvest. Undecided, and
    deliberately so.

Verified

  • 346 passed, 22 skipped across both backends. No code changed — documents
    only.
  • All four mermaid diagrams rendered in Chromium via mmdc, not parsed in Node,
    which reports false failures on flowcharts.
  • Every number was measured today against the live store and live network.

Both were still written as if nothing had been fixed. The audit described its
findings in the present tense and the proposal's acceptance table said "Today:
3/8", which is no longer true of either.

AUDIT.md becomes a two-pass document. Sections 1 to 5 are left exactly as
written -- they are the record of what was true before anything changed, and
rewriting them into the past tense would destroy the evidence the fixes were
built from. Section 6 marks each planned item done or not. A new section 7
carries fresh measurements taken against 316c7f2:

  resolution        3 correct / 3 wrong  ->  7 correct / 0 wrong
  the seven indexes 155,442 chars        ->  19,116,128 reachable (123x)
  pydantic read     24 blocks of 1.10    ->  85 blocks of 2.11
  is_forge(gist.github.com)              ->  True
  tests             284                  ->  346 across both backends

Section 7.2 records the three defects that were found by running the system
rather than reading it -- the astrology site, terraform.com, and the fourth
version lookup -- because two of them were regressions introduced by the fixes
themselves and would have shipped. That is a better argument for the live
fixture than anything the document says about it in the abstract.

One thing the fix status must not blur: the code is fixed and the stored
corpus is not. All seven affected technologies were harvested before any of
this, so they still hold their indexes and still read complete: True. That is
stale data rather than a live defect, but it is called out at the top, in 7.3
and in the summary table, because an audit that says "fixed" while the store
still shows the symptom would be exactly the kind of unearned confidence this
whole exercise was about. A re-harvest clears it and takes the store from 8.4
to roughly 27.5 million characters. It is the one action outstanding.

PROPOSAL.md gains an achieved column on the acceptance criteria, marks phases
A, B, C and D3 done, and says plainly that D1 is not -- a long harvest still
blocks past MCP client timeouts, and that is an architectural change rather
than a fix. Two open questions are answered by having built the thing: chunk
size is settled by heading level chosen on results, and conflict presentation
turned out to be the wrong question, since domain-first never has both answers
in hand. A new one is opened in their place, about what `expected` should
compare against for a full dump, where making the count authoritative risks
crying wolf and leaving it advisory misses a class of partial harvest.

Documents only. 346 passed, 22 skipped across both backends. All four mermaid
diagrams rendered in Chromium via mmdc.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APThnqHvfP2oBjBCBxBhd6
@R204570
R204570 merged commit bb4c1cf into main Aug 20, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant