Skip to content

Restructure + compress instruction set for a 200K-context budget - #551

Merged
xerhab merged 3 commits into
mainfrom
consolidate-instruction-md
Aug 29, 2026
Merged

Restructure + compress instruction set for a 200K-context budget#551
xerhab merged 3 commits into
mainfrom
consolidate-instruction-md

Conversation

@xerhab

@xerhab xerhab commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Started as a prose-trim pass on the auto-loaded instruction set (CLAUDE.md + .claude/rules/*.md)
for a 1M-context model; that only bought 0.4%. Mid-PR the target changed to a 200K-context model,
which needed real structural changes, not just tighter wording:

  1. Narrowed over-broad paths: matches. agent.md and dsh.md both listed agent/**, so
    editing any agent-side file - even a pure-qwen or pure-native one - loaded content that had
    nothing to do with it. Narrowed both to the specific files they actually govern.
  2. Relocated settled ADR narrative out of the auto-loaded set. dsh.md/qwen.md were
    architecture-decision records for now-fully-shipped work; their rationale, spike history and open
    questions moved to new docs/dsh-adr.md / docs/qwen-adr.md (read on demand, never auto-loaded),
    leaving each rules file with only the operative invariants + a pointer to the ADR for "why".
  3. A deep compression pass across every file, explicitly overriding the repo's own "never delete
    rationale" convention (user-authorized given the budget constraint): historical bug narration,
    rejected-alternative prose and worked-example arithmetic collapsed to the rule + a short reason.
    Every never/must/always rule, identifier, XERK-nnn tag and Tests: pointer survives -
    verified by diffing every backtick identifier old vs. new and restoring the ~39 dropped as a
    side effect (env vars, constants, test names).

Total: 579,198 -> 377,743 chars (-35%), every file under the 40,000-char cap.

Worst-case co-load (files that auto-load when touching the given path):

Touch Before After
agent/hub-agent.py 336K chars / ~84K tok / 16 files 222K chars / ~55K tok / 16 files
turma/server.js 252K / ~63K tok / 9 files 161K / ~40K tok / 9 files
sessions.html 189K / ~47K tok / 5 files 124K / ~31K tok / 5 files
board.js 176K / ~44K tok / 6 files 122K / ~30K tok / 6 files

hub-agent.py stays the worst case because it's a genuine monolith every other file's invariants
legitimately depend on - closing that further means splitting the code file itself, out of scope.

Test plan

  • wc -m on every file - none at/over the 40,000-char cap (repo's own CI gate re-checks this).

  • Every cross-reference between rules files (and into docs/dsh-adr.md/docs/qwen-adr.md)
    resolves to a real file.

  • Every rules file's YAML frontmatter (paths: block) parses and is well-formed.

  • Diffed every backtick-wrapped identifier (env vars, constants, function names, test names)
    across the whole old vs. new ruleset; the ~39 that had been dropped as a side effect of
    compression were individually restored into the file where their fact now lives.

  • Read every diff by hand across both commits.

  • node --test turma/tests/*.test.js agent/tests/*.test.js .github/scripts/tests/*.test.js
    1695/1695 pass.

  • python3 -m unittest discover -s agent/tests — 2194 tests, 1 pre-existing failure unrelated
    to this change
    (below).

  • git status confirms the change is markdown-only — no source file is touched.

Pre-existing test failure, reported not fixed

test_qwen_guard.TestQwenGuardShimEndToEnd.test_replace_in_aws_is_denied fails on this dev host
only
, and fails identically on origin/main — this PR changes no Python.

Cause: the qwen guard shim matches its credential globs as an expanduser'd prefix against a
realpath'd target. On this machine ~/.aws is a symlink to /mnt/c/Users/mhabeeb/.aws (WSL), so
the realpathed target escapes the ~/.aws/** prefix and the write is not denied. The sibling
~/.ssh case passes because that directory is real.

This is exactly the residual gap .claude/rules/qwen.md already records as accepted ("a symlinked
HOME could dodge a credential glob") — it is just not hypothetical on a WSL host whose ~/.aws
points at the Windows filesystem. CI is green because GitHub runners have no such symlink. Flagged
rather than fixed: it is a real, if low-severity, guard gap and out of scope for a docs change.

xerhab added 2 commits August 29, 2026 10:08
Reviewed the whole auto-loaded instruction set (CLAUDE.md + .claude/rules/*.md,
the files that cost context every session regardless of what's touched) for
duplicated or restated content, keeping every invariant, pitfall, rationale,
and Tests: pointer intact.

Found the ruleset already written under the same "cut narration, keep
invariants" discipline this pass applies, so gains are real but modest:
- Collapsed same-file restatements into cross-references (agent-hooks.md,
  agent-tunnel.md, turma.md, turma-archive.md, turma-sessions.md, agent.md,
  agent-archive.md).
- Collapsed cross-file restatements where a paths:-scoped file fully repeated
  content its always-co-loaded sibling already carries (dsh-input.md,
  dsh-delegation.md, qwen-delegation.md, qwen-migration.md, qwen.md, dsh.md).
- Trimmed CLAUDE.md's XERK-348 cross-cutting bullet, which restated turma.md's
  orgBound assignment mechanics almost verbatim — that detail now lives only
  in turma.md, with CLAUDE.md keeping the spanning rule.

Total: 579,198 -> 576,942 chars (~0.4%). Every file stays under the 40,000-char
cap; dsh.md and qwen.md (closest to it) gained a little headroom.
…dget

The prior commit's prose-trim pass (0.4%) was nowhere near enough once the
target model dropped from 1M to 200K context. Touching agent/hub-agent.py
alone auto-loaded 16 rules files (336K chars, ~84K tokens = 42% of a 200K
budget) before any code was read. This pass attacks the actual cause:

1. Narrowed over-broad `paths:` matches. agent.md and dsh.md both listed
   `agent/**`, so editing any agent-side file (even a pure-qwen or
   pure-native one) pulled in content that had nothing to do with it.
   Narrowed both to the specific files they actually govern.

2. Relocated settled ADR narrative out of the auto-loaded set. dsh.md and
   qwen.md were architecture-decision records for now-fully-shipped work;
   their D1-D5 rationale, G0/G1 spike history, and open questions moved to
   new docs/dsh-adr.md and docs/qwen-adr.md (read on demand, never
   auto-loaded), while each file keeps only the operative invariants and a
   pointer to the ADR for "why". dsh.md 39,678 -> 18,287 chars; qwen.md
   39,631 -> 28,751 (later further compressed, see below).

3. Deep compression pass across every remaining file, explicitly overriding
   the repo's own "never delete rationale" convention (user-authorized given
   the budget constraint): historical bug narration, rejected-alternative
   prose, and worked-example arithmetic collapsed to the rule plus a short
   reason. Every "never/must/always" rule, identifier, XERK-nnn tag, and
   Tests: pointer was preserved -- verified by diffing every backtick
   identifier in the old ruleset against the new one and restoring the ~39
   that had been dropped as a side effect (env vars, constants, test names).

Total instruction-set size: 579,198 -> 377,743 chars (-35%), all files under
the 40,000-char cap (closest: CLAUDE.md 32,302, qwen.md 28,828).

Worst-case co-load on the most common single-file touches:
- agent/hub-agent.py: 336K -> 222K chars (~84K -> ~55K tokens, 16 files)
- turma/server.js:    252K -> 161K chars (~63K -> ~40K tokens, 9 files)
- sessions.html:      189K -> 124K chars (~47K -> ~31K tokens, 5 files)
- board.js:           176K -> 122K chars (~44K -> ~30K tokens, 6 files)

hub-agent.py remains the worst case because it is a genuine monolith that
all 15 other files' invariants legitimately depend on -- closing that
further would mean splitting the code file itself, out of scope here.

Verified: every cross-reference between files still resolves, every rules
file's YAML frontmatter parses, no file is over the size cap, and no
identifier present in the pre-compression ruleset is now absent from it.
@xerhab xerhab changed the title Tighten CLAUDE.md + .claude/rules/*.md to cut redundancy Restructure + compress instruction set for a 200K-context budget Aug 29, 2026
@xerhab
xerhab merged commit 8803456 into main Aug 29, 2026
4 checks passed
@xerhab
xerhab deleted the consolidate-instruction-md branch August 29, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant