W4/honest readme - #234
Merged
Merged
Conversation
The version badge in README said v1.5.0 while the package shipped 1.5.0 and was about to ship 1.6.0; the tests badge said 2075 against 2252 collected. Both are hand-written statements of a fact the code already knows, and neither had a gate, so they drifted across a release without failing anything. tests/unit/test_cli_version.py does not cover this. It reads __version__ and asserts the CLI prints it, which stays true whatever the badge claims. The badge is a second, independent statement aimed at a reader who will never run the command -- and it was the wrong one. The new architecture test pins the badge to __version__ and nothing else: the test count is left ungated on purpose, because it changes with almost every commit and a gate on it would fail on green work. The launch-post draft carried the same class of stale number: 24,169 lines of production and 24,088 of tests against 25,498 and 28,569 measured now, and 2001 tests at v1.5.0. It is tracked in the public repository, so it is corrected here rather than left for a publication date. CHANGELOG: the Unreleased section becomes 1.6.0, dated. Its contents were already written by W5 and W2 -- the licence change and the contamination fix are what this release is.
README published the probe score. By the project's own docstring the probe engine measures the targets and the harness, not the product -- so the number on the front page answered a question nobody asked, while the pipeline's own 4/4 sat unmentioned in examples/. The two engines are now separated by what they answer rather than by how they are driven, and the pipeline's run metrics are on the page: four endpoints, 21 requests, four in-band proofs and one out-of-band. The stronger claim is the one that was missing entirely: the pipeline reaches 4/4 without contacting a model. That is measured, not assumed -- the agent engine constructs no LLM client on either path, and the card carries a proven zero with its cause rather than a blank. README said 28 requests against 21 measured, which is the figure the decontamination of W2 moved; both scorecards were regenerated at 1.6.0 against live targets and the totals did not move. Two documents now quote the same measurement, which doubles the surface a stale number can hide on -- CJ in the tail list, where a published card lagged the code by two days and nothing failed. The new architecture test compares README's table to the committed card cell by cell, ignoring the availability column README drops and the emphasis it adds. It does not re-run the bench: the card is the artifact of a real run by construction, and the gap that actually opened was between that artifact and the prose quoting it. docs/benchmarks/local-suite.md carried 2026-08-13 at 1.5.0 in a table two paragraphs above its own rule that these numbers come from a measured run and never by hand. Corrected to the run that produced the cards.
README described the security design of the code before W1. It said every phase output is injection-scanned before it propagates, that the orchestrator neutralizes a malicious banner before the data reaches the model, and that the trust boundary is an injection-scan at every phase edge. The phase scan runs after the agent has already called the provider -- the code says so in its own docstring, and calls itself an audit signal rather than a barrier. The banner sentence was worse than wrong: recon contacts no model at all, so it described a defence on a path with nothing to defend. What is true is now on the page. TrustGuard sits inside LLMClient ahead of the provider branch on all four entry points; it scores the raw message and sanitises the copy it sends, in that order, because the reverse was measured to blind the detector -- sanitisation strips three of the categories the detector scores on, so a template marker lowered an injection's score. The three policies are documented with the reason annotate is the default and quarantine is not: quarantine mutates content, and the detector it would mutate on flagged 42 of 43 real scan reports. The phase scan stays described, as the audit signal it is. The pattern count was wrong in the other direction. README said 33, which is correct; docs/security/adversarial-robustness.md and the module docstring of core/safety.py said thirty-eight. Two security documents named a number a reviewer can check in one line, and nothing failed. The new architecture test pins all three spellings -- one digit, two English words -- to len(INJECTION_PATTERNS), and rejects a stale word left beside a corrected one. Only the count is pinned: a test over the wording would fail on every paragraph edit and train the reviewer to regenerate without reading.
The documentation table listed five files. README's own prose linked to seven more -- adversarial-robustness, local-suite, cve-bench, contamination-2026-08, licensing among them -- so a reader who scrolled to the table saw a smaller project than the one they had just read about. The contamination write-up in particular was cited in the benchmarks section and absent from the index, which is the arrangement that looks like hiding even when it is only an oversight. Twelve entries now, including the two walkthroughs, the command recipes and the known-issues list. Six documents stay out on purpose: an EVMBench adapter note, a competitive landscape, a phantom-stack integration note, a scoring-criterion essay, a live-recon workflow and an older benchmarks index. They belong to threads a first-time reader is not on, and an index that lists everything indexes nothing. The shipped table stopped at v1.4 while the repository was tagging 1.5 and this branch releases 1.6. Both rows added. The new architecture test resolves every repository-relative link in README against the filesystem: thirty links, twenty-three distinct paths, none of them checked by anything until now. A renamed document leaves a 404 on the page a reviewer reads first, which reads as an abandoned project rather than as a typo. External URLs are deliberately not fetched -- a test that reaches the network fails for reasons that have nothing to do with the change, and gets ignored. Anchors are stripped: whether a heading still exists is a question about someone else's document. The W4.7 item is closed by checking rather than by editing. `mcp-scan` is a real command, agents/mcp_scan carries poisoning, attestation and trust analysis, and W1 put a guard in front of the model, so the line about red-teaming AI rather than only building with it is now backed by shipped code and stays.
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
How it was measured
Checklist
ruff format --check cyberai/ tests/andruff check cyberai/ tests/passpytest -W ignore::DeprecationWarning -m "not slow and not smoke"passes