Restructure + compress instruction set for a 200K-context budget - #551
Merged
Conversation
Reviewed the whole auto-loaded instruction set (CLAUDE.md + .claude/rules/*.md, the files that cost context every session regardless of what's touched) for duplicated or restated content, keeping every invariant, pitfall, rationale, and Tests: pointer intact. Found the ruleset already written under the same "cut narration, keep invariants" discipline this pass applies, so gains are real but modest: - Collapsed same-file restatements into cross-references (agent-hooks.md, agent-tunnel.md, turma.md, turma-archive.md, turma-sessions.md, agent.md, agent-archive.md). - Collapsed cross-file restatements where a paths:-scoped file fully repeated content its always-co-loaded sibling already carries (dsh-input.md, dsh-delegation.md, qwen-delegation.md, qwen-migration.md, qwen.md, dsh.md). - Trimmed CLAUDE.md's XERK-348 cross-cutting bullet, which restated turma.md's orgBound assignment mechanics almost verbatim — that detail now lives only in turma.md, with CLAUDE.md keeping the spanning rule. Total: 579,198 -> 576,942 chars (~0.4%). Every file stays under the 40,000-char cap; dsh.md and qwen.md (closest to it) gained a little headroom.
…dget The prior commit's prose-trim pass (0.4%) was nowhere near enough once the target model dropped from 1M to 200K context. Touching agent/hub-agent.py alone auto-loaded 16 rules files (336K chars, ~84K tokens = 42% of a 200K budget) before any code was read. This pass attacks the actual cause: 1. Narrowed over-broad `paths:` matches. agent.md and dsh.md both listed `agent/**`, so editing any agent-side file (even a pure-qwen or pure-native one) pulled in content that had nothing to do with it. Narrowed both to the specific files they actually govern. 2. Relocated settled ADR narrative out of the auto-loaded set. dsh.md and qwen.md were architecture-decision records for now-fully-shipped work; their D1-D5 rationale, G0/G1 spike history, and open questions moved to new docs/dsh-adr.md and docs/qwen-adr.md (read on demand, never auto-loaded), while each file keeps only the operative invariants and a pointer to the ADR for "why". dsh.md 39,678 -> 18,287 chars; qwen.md 39,631 -> 28,751 (later further compressed, see below). 3. Deep compression pass across every remaining file, explicitly overriding the repo's own "never delete rationale" convention (user-authorized given the budget constraint): historical bug narration, rejected-alternative prose, and worked-example arithmetic collapsed to the rule plus a short reason. Every "never/must/always" rule, identifier, XERK-nnn tag, and Tests: pointer was preserved -- verified by diffing every backtick identifier in the old ruleset against the new one and restoring the ~39 that had been dropped as a side effect (env vars, constants, test names). Total instruction-set size: 579,198 -> 377,743 chars (-35%), all files under the 40,000-char cap (closest: CLAUDE.md 32,302, qwen.md 28,828). Worst-case co-load on the most common single-file touches: - agent/hub-agent.py: 336K -> 222K chars (~84K -> ~55K tokens, 16 files) - turma/server.js: 252K -> 161K chars (~63K -> ~40K tokens, 9 files) - sessions.html: 189K -> 124K chars (~47K -> ~31K tokens, 5 files) - board.js: 176K -> 122K chars (~44K -> ~30K tokens, 6 files) hub-agent.py remains the worst case because it is a genuine monolith that all 15 other files' invariants legitimately depend on -- closing that further would mean splitting the code file itself, out of scope here. Verified: every cross-reference between files still resolves, every rules file's YAML frontmatter parses, no file is over the size cap, and no identifier present in the pre-compression ruleset is now absent from it.
4 tasks
…on-md # Conflicts: # .claude/rules/qwen.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Started as a prose-trim pass on the auto-loaded instruction set (
CLAUDE.md+.claude/rules/*.md)for a 1M-context model; that only bought 0.4%. Mid-PR the target changed to a 200K-context model,
which needed real structural changes, not just tighter wording:
paths:matches.agent.mdanddsh.mdboth listedagent/**, soediting any agent-side file - even a pure-qwen or pure-native one - loaded content that had
nothing to do with it. Narrowed both to the specific files they actually govern.
dsh.md/qwen.mdwerearchitecture-decision records for now-fully-shipped work; their rationale, spike history and open
questions moved to new
docs/dsh-adr.md/docs/qwen-adr.md(read on demand, never auto-loaded),leaving each rules file with only the operative invariants + a pointer to the ADR for "why".
rationale" convention (user-authorized given the budget constraint): historical bug narration,
rejected-alternative prose and worked-example arithmetic collapsed to the rule + a short reason.
Every
never/must/alwaysrule, identifier,XERK-nnntag andTests:pointer survives -verified by diffing every backtick identifier old vs. new and restoring the ~39 dropped as a
side effect (env vars, constants, test names).
Total: 579,198 -> 377,743 chars (-35%), every file under the 40,000-char cap.
Worst-case co-load (files that auto-load when touching the given path):
agent/hub-agent.pyturma/server.jssessions.htmlboard.jshub-agent.pystays the worst case because it's a genuine monolith every other file's invariantslegitimately depend on - closing that further means splitting the code file itself, out of scope.
Test plan
wc -mon every file - none at/over the 40,000-char cap (repo's own CI gate re-checks this).Every cross-reference between rules files (and into
docs/dsh-adr.md/docs/qwen-adr.md)resolves to a real file.
Every rules file's YAML frontmatter (
paths:block) parses and is well-formed.Diffed every backtick-wrapped identifier (env vars, constants, function names, test names)
across the whole old vs. new ruleset; the ~39 that had been dropped as a side effect of
compression were individually restored into the file where their fact now lives.
Read every diff by hand across both commits.
node --test turma/tests/*.test.js agent/tests/*.test.js .github/scripts/tests/*.test.js—1695/1695 pass.
python3 -m unittest discover -s agent/tests— 2194 tests, 1 pre-existing failure unrelatedto this change (below).
git statusconfirms the change is markdown-only — no source file is touched.Pre-existing test failure, reported not fixed
test_qwen_guard.TestQwenGuardShimEndToEnd.test_replace_in_aws_is_deniedfails on this dev hostonly, and fails identically on
origin/main— this PR changes no Python.Cause: the qwen guard shim matches its credential globs as an
expanduser'd prefix against arealpath'd target. On this machine~/.awsis a symlink to/mnt/c/Users/mhabeeb/.aws(WSL), sothe realpathed target escapes the
~/.aws/**prefix and the write is not denied. The sibling~/.sshcase passes because that directory is real.This is exactly the residual gap
.claude/rules/qwen.mdalready records as accepted ("a symlinkedHOME could dodge a credential glob") — it is just not hypothetical on a WSL host whose
~/.awspoints at the Windows filesystem. CI is green because GitHub runners have no such symlink. Flagged
rather than fixed: it is a real, if low-severity, guard gap and out of scope for a docs change.