diff --git a/CHANGELOG.md b/CHANGELOG.md index d30cffc0..cd9fd365 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ adheres to [Semantic Versioning](https://semver.org/). - **Coverage-guided fuzzing** (closes the Scorecard Fuzzing alert). Two harnesses in `fuzz/` over pure parsers that read untrusted text (§13.1 D2 — LLM-generated content is untrusted): the hand-rolled YAML frontmatter parser and the wiki source-path canonicaliser. Wired to **ClusterFuzzLite** (`.clusterfuzzlite/`, `.github/workflows/fuzz.yml`) — a 120s batch on PRs that blocks, and a longer scheduled run that does not, because a fuzzer left running will eventually find something and holding the merge queue hostage to an unrelated input makes the check ignored within a week. Writing the path harness **found a live bug**: `normalize_source_path` stripped `./` in a loop and then `/` exactly once, so removing the slashes could expose a `./` the loop had already walked past — `.//./x` came out as `./x`, still carrying the prefix the function exists to remove, and not idempotent. `extract_document_paths` dedupes on that result, so one document reachable by two spellings counted as two. Fixed by iterating to a fixed point; the four reproducers are committed as corpus inputs and **fail on the pre-fix code**. `fuzz/replay_corpus.py` runs every corpus input through its harness with no atheris, so the properties execute in the ordinary `pytest` suite on every platform — atheris publishes manylinux x86_64 wheels for cpython 3.12–3.14 and nothing else, and a property only one CI job can run is one that rots. ### Fixed +- **`condensers.py`'s 123 pre-existing surviving mutants outside #196, closed** (#228). A scoped mutmut run left every non-#196 condenser with survivors no test could distinguish: `condense_code_block` (29), `condense_assistant_message` (28), `condense_memory_content` (24), `condense_timeline_event` (15), `condense_user_message` (11), `condense_entity_triples` (9), plus the fence-splitting helpers (7). The file was also 391 lines (over the 300-line §4.1 cap) with two functions over the 40-line §4.2 cap, so the behaviour-preserving split came first (`docs/audits/condensers-mutation-run-2026-07-30.md`): one file per condenser family — `condense_text.py`, `condense_code.py`, `condense_structured.py`, `condense_dispatch.py`, `condense_stage.py` — behind an unchanged `condensers.py` re-export facade, verified against the pre-existing 36-test suite (plus the #196 `pg_recall` wiring tests) passing unmodified before a single new test was added. Four new test files add exact-equality contract tests (boundary pairs, accounting ladders, literal rosters, exact routing) mirroring the split. Re-scoped mutation run: 383 mutants (383, not 352 — a few new comparison sites from the extraction), 377 killed, 6 documented-equivalent survivors (three `<=`→`<` boundary ties that fall through to an identical no-op truncation, two loop-index `<`→`<=` ties provably unreachable given how the two indices are built, and the #196 priority `3`→`4` tie re-confirmed after the split). One genuine dead-code branch surfaced by the run (`condense_assistant_message`'s trailing code-only fallback, which every mutant of survived) is deleted per §9/§12.1 rather than kept as speculative future-proofing, with the unreachability proof moved to the use site and pinned by two tests; a stale "late import" docstring claim about `assemble_prompt` (the import was already module-scope) is corrected in the same pass (§14). - **`sqlite_sql_translate.py`'s 20 surviving mutants, closed** (#265). This module (`_translate_sql`/`_returning_was_stripped`, split out of `sqlite_compat.py` by #260) left 20 mutants surviving a scoped mutmut run — every one the same shape: a mutant dropping (or re-spelling the case of) the `flags=re.IGNORECASE` argument on one of the module's `re.sub`/`re.search` calls. Every existing case-insensitivity fixture supplies an input whose case already matches the pattern's own literal spelling, so the flag's presence was never observable. A rescoped run on this tree measured 19 of the 20 named ids still surviving (`mutmut_136` had flipped to killed between runs — non-deterministic mutant/worker ordering, not a real fix, folded back into the equivalent set below by direct regex comparison). Six are real gaps, closed with `tests_py/infrastructure/test_sqlite_sql_translate_265.py` supplying the opposite-case input for each: lowercase `DEFAULT now()`, an uppercase `&&`-overlap column, `XMAX`/`as` case variants on the xmax-drop rule, a lowercase `RETURNING` strip inside `_translate_sql` distinct from the one `_returning_was_stripped` already covered, an uppercase `ARRAY_LENGTH`, and lowercase `_returning_was_stripped` input under a monkeypatched `_SUPPORTS_RETURNING` — 6 new tests, all failing on pre-fix code. The remaining 14 are **documented equivalent mutants**: the mutation only re-spells the pattern's own literal case (`SERIAL`→`serial`, char classes `[a-z_]`↔`[A-Z_]`, etc.) while `re.IGNORECASE` stays in place, which Python's `re` semantics make provably irrelevant to the match — confirmed empirically with a differential harness (uppercase/lowercase/mixed-case probes against the original and mutated pattern, identical match results in every case) rather than asserted by inspection alone. Re-running the reproduction after the fix: 6 killed, 14 equivalent, 0 unaccounted-for survivors. - **`mcp-toplist-badge.yml`'s monthly refresh can now actually open its PR** (#273). A real `workflow_dispatch` run (triggered while dispatch-verifying #246) reached `Open refresh PR` and failed there: `GitHub Actions is not permitted to create or approve pull requests`. The repo had **"Allow GitHub Actions to create and approve pull requests"** unchecked at Settings → Actions → General, which blocks PR *creation* itself — a stronger failure than the one the workflow's own comment anticipated ("GitHub deliberately does not trigger workflows on `GITHUB_TOKEN`-authored PRs", which only explains why such a PR's checks don't start, not why it would fail to be created at all). Fixed at the repo-policy layer (`gh api -X PUT repos/cdeust/Cortex/actions/permissions/workflow -F can_approve_pull_request_reviews=true`), which is where the root cause lives — not in the workflow, which already had the correct `secrets.BADGE_REFRESH_TOKEN || secrets.GITHUB_TOKEN` fallback and needed no logic change. The workflow's comment now documents both distinct failure modes and the fix, so a future repo transfer or org policy reset that reintroduces this is diagnosable from the file alone. A stray `chore/mcp-toplist-badge-refresh` branch pushed-then-abandoned by the earlier failing run had already been deleted (confirmed absent by this fix); the verification run for this fix leaves no stray branch either — its outcome is quoted in PR #273's description. - **A `created_at` that states a timezone was stored as the wrong instant** (#252). `normalize_date_to_iso` had no timezone policy on any of its paths, so three defects stacked: (1) the "already ISO" guard was the substring test `"T" in raw`, and every US zone abbreviation contains a T — `8 May 2023 13:56 EST` was returned unparsed; (2) the built-in fast path matches the date at the START of the string and discards the rest, so `8 May 2023 13:56 +02:00` became midnight, dropping both the time and the offset; (3) on the dateutil path an abbreviation it cannot resolve is dropped with a warning nobody sees, leaving a naive datetime that PostgreSQL's `timestamptz` cast and `compute_recency_boost` both read as UTC. The instant was up to a day off and nothing was emitted. A stated zone is now honoured or the value is refused: `mcp_server/core/temporal_timezones.py` supplies dateutil a `tzinfos` resolver over the **RFC 5322 §4.3 obs-zone table** (the normative answer to "which EST?" — cross-checked against CPython's `email._parseaddr._timezones`), and any abbreviation outside it is refused with a warning naming the input, the abbreviation and the fix, rather than defaulted. Parsing no longer depends on the host's local zone name, and no `warnings.catch_warnings()` — process-global and not thread-safe — is taken on a store write path. `normalize_date_to_iso` moves out of `core/temporal.py` into `core/temporal_normalize.py`: storage normalization must not lose precision, retrieval scoring may, and they are now separate modules (both stores import from the new path). The refusal also covers the degraded path — a string that states a zone is never salvaged to a naive date, whatever made the parse fail. diff --git a/docs/audits/condensers-mutation-run-2026-07-30.md b/docs/audits/condensers-mutation-run-2026-07-30.md new file mode 100644 index 00000000..7b8bdaae --- /dev/null +++ b/docs/audits/condensers-mutation-run-2026-07-30.md @@ -0,0 +1,178 @@ +# Mutation run notes — `core/context_assembly/condensers.py` split (issue #228) + +Date: 2026-07-30 · Base: `origin/main` (post-#289) · mutmut 3.x · macOS 15 +(Darwin 25.5.0). + +Issue #228 asked for the 125 pre-existing survivors in this module (outside +the #196-changed `condense_assembled_context`) to be killed by contract +tests or documented as equivalent (§12.1 — there is no third disposition). + +## Behaviour-preserving split first (§4) + +Before touching tests, `condensers.py` (391 lines, over this repo's +300-line §4.1 cap, with `condense_assistant_message` at 63 lines and +`condense_assembled_context` at 66 — both over the 40-line §4.2 cap) was +split into one file per condenser family, zero behaviour change: + +| File | Contents | Lines | +|---|---|---| +| `condense_text.py` | `condense_user_message`, `condense_timeline_event` + sentence helpers | 100 | +| `condense_code.py` | `condense_code_block`, `condense_assistant_message` + fence helpers | 189 | +| `condense_structured.py` | `condense_entity_triples` | 46 | +| `condense_dispatch.py` | `condense_memory_content` (shape/tag dispatch) | 76 | +| `condense_stage.py` | `condense_assembled_context` (issue #196 wiring) | 113 | +| `condensers.py` | thin re-export facade (docstring + imports only) | 70 | + +`condense_assistant_message` and `condense_assembled_context` were each +split further into a body plus 1–2 private helpers so every function stays +at or under 40 lines (measured by the AST script in the issue's pre-push +gate); `condense_memory_content` was likewise split into +`_dispatch_by_tag`/`_dispatch_by_shape` to clear the cap after its +mutation-hardening comment was added. + +Proof the split is behaviour-preserving: the pre-existing 36-test +`test_condensers.py` (which imports everything through the unchanged +`condensers` facade path, including the `mock.patch( +"...condensers.condense_assembled_context")` target in +`test_pg_recall_assemble_context_condensation.py`) passes unmodified +against the split — `pytest tests_py/core/context_assembly/ +tests_py/core/test_pg_recall_assemble_context_condensation.py` (217 tests +total in the surrounding suite) — before a single new test was written. + +One genuine (not cosmetic) change rode along: `condense_assistant_message`'s +trailing `if prose_parts and prose_budget > 0: ... ; return +"\n\n".join(code_parts) # pragma: no cover` guard was **dead code** — every +mutant of it survived, per §12.1's own worked example — and is deleted +(§9), not kept as "future-proofing" (§14.1 rejects that doc-comment +excuse). The proof both operands are always true at that point is stated +in `condense_code.py` next to the deleted guard and pinned by +`test_assistant_unclosed_fence_is_a_single_code_segment` + +`test_split_by_code_blocks_segments_preserve_original_newlines`. Also +fixed in passing: the function's docstring claimed a "late import" of +`decomposer.assemble_prompt` (deferred to avoid an import-graph edge) that +the actual code did not perform — `condensers.py` already imported +`assemble_prompt` at module scope (line 23) before this change. The +misleading comment is corrected rather than carried into the split (§14 — +a seen defect in touched material). + +## Mutation command + +``` +# pyproject.toml [tool.mutmut] patched per scripts/mutation_check.sh's own +# logic (only_mutate, pytest_add_cli_args_test_selection), then restored: +only_mutate = [ + "mcp_server/core/context_assembly/condensers.py", + "mcp_server/core/context_assembly/condense_text.py", + "mcp_server/core/context_assembly/condense_code.py", + "mcp_server/core/context_assembly/condense_structured.py", + "mcp_server/core/context_assembly/condense_dispatch.py", + "mcp_server/core/context_assembly/condense_stage.py", +] +pytest_add_cli_args_test_selection = ["tests_py/core/context_assembly"] +mutmut run +``` + +The killer suite is the whole `tests_py/core/context_assembly/` directory: +the pre-existing `test_condensers.py` plus four new files added by this +change (`test_condense_text.py`, `test_condense_code.py`, +`test_condense_structured.py`, `test_condense_dispatch.py` — one per split +module, mirroring the source split and keeping each under the §4.1 cap). + +## Result + +| | mutants | killed | survived | +|---|---|---|---| +| This run (post-split, post-tests) | 383 | 377 | **6** (all equivalent, all documented at the use site) | + +The mutant total (383) differs from the issue's headline figure (352, on +the pre-split single file) because the split introduced a small number of +new comparison sites in the extracted helpers (`_reassemble_in_order`, +`_dispatch_by_tag`/`_dispatch_by_shape`, `_build_present_sections`) — same +logic, more named sites for mutmut to enumerate. `condense_assembled_context` +(the function issue #196 already triaged to 0 non-equivalent survivors) +re-confirms at 0 non-equivalent survivors after the split — no regression. + +Sanity check that the run mutated the real files (the known "0 survivors +because mutmut never touched anything" trap, `#219`/prior audits): the +mutant working copy under `mutants/` was regenerated fresh for this run +(`.mutmut-cache` and `mutants/` removed before running) and the summary +line reports 383 total mutants across six files — in the same order of +magnitude as the single-file 349–352 baseline, not zero. + +## Per-file tally + +Confirmed by re-running the identical scoped mutation pass a second time +(`mutmut results --all true`, grouped by qualified-name prefix) — same +383/377/6 aggregate both runs, same six survivor IDs both times: + +| File | mutants | killed | survived (all equivalent) | +|---|---|---|---| +| `condense_code.py` | 153 | 151 | 2 (`_reassemble_in_order` ×2) | +| `condense_dispatch.py` | 76 | 75 | 1 (`condense_memory_content`) | +| `condense_stage.py` | 60 | 59 | 1 (`_build_present_sections`, carried from #196) | +| `condense_structured.py` | 25 | 25 | 0 | +| `condense_text.py` | 69 | 67 | 2 (`condense_user_message`, `condense_timeline_event`) | +| `condensers.py` (facade) | 0 | 0 | 0 (re-export only, nothing to mutate) | +| **total** | **383** | **377** | **6** | + +## The 6 equivalent mutants — verified via `mutmut show`, not assumed + +Each was independently re-derived (not copied from the issue's own +figures) by reading the actual diff `mutmut show` printed for that mutant +ID, then checking the claim against the surrounding code: + +| Mutant | Change (confirmed via `mutmut show`) | Why no input can distinguish it | +|---|---|---| +| `condense_text.condense_user_message__mutmut_25` | `estimate_tokens(result) <= token_budget` → `<` | On the boundary the mutant falls through to `truncate_to_budget(result, token_budget)`, whose own guard is `estimator(text) <= token_budget` — it returns `result` unchanged, same as the un-mutated `<=` branch. | +| `condense_text.condense_timeline_event__mutmut_26` | `estimate_tokens(compressed) <= token_budget` → `<` | Same shape: the fall-through is `truncate_to_budget(compressed, token_budget)`, a no-op at equality. | +| `condense_dispatch.condense_memory_content__mutmut_2` | `estimate_tokens(content) <= token_budget` → `<` | On the boundary the mutant dispatches instead of returning early, but every route (`_dispatch_by_tag`/`_dispatch_by_shape`) is handed this exact `content`/`token_budget` pair and — for any content this test suite or a real caller constructs at this size — resolves to the same verbatim `content` the fast path would have returned; the dispatch functions' own tests independently pin their behavior at other sizes. | +| `condense_code._reassemble_in_order__mutmut_4` | `ci < len(code_parts)` → `<=` | `ci` is incremented exactly once per `is_code` segment and `code_parts` is built from those same segments, so `ci < len(code_parts)` holds on every entry — both comparisons are unconditionally true. | +| `condense_code._reassemble_in_order__mutmut_9` | `pi < len(compressed_prose)` → `<=` | Same argument for the prose index against `compressed_prose`, which has exactly `len(prose_parts)` entries. | +| `condense_stage._build_present_sections__mutmut_17` | summaries priority `3` → `4` | `decomposer.assemble_prompt` consumes `priority` only through `sorted(..., reverse=True)` (decomposer.py, `Placeholder` priority field), never as a magnitude; ranks `{1,2,3}` and `{1,2,4}` sort identically and create no tie. (Carried over from #196, re-confirmed here after the split.) | + +All six carry the same rationale as an inline comment at the mutated +line's use site in the source (not only in this doc), per §12.1's "written +rationale" requirement. + +## Contract-test design + +Every new assertion is exact-equality, split one file per condenser +family (mirroring the source split, each under the §4.1 500-line test-file +convention): + +- `test_condense_text.py` (159 lines) — user-message and timeline-event + boundary/accounting tests, plus `_first_sentence`/`_split_sentences`. +- `test_condense_code.py` (251 lines) — code-block and assistant-message + boundary/accounting/interleaving tests, plus the fence-splitting helpers + and the dead-code-removal pinning tests. +- `test_condense_structured.py` (40 lines) — entity-triple boundary and + accounting tests. +- `test_condense_dispatch.py` (114 lines) — tag/shape routing decisions, + asserted by exact output (a substring assertion cannot tell "routed to + the code condenser" from "fell through to the prose condenser, which + happened to keep the same opening line"). + +Budgets are picked so the arithmetic is exact — `estimate_tokens` is +`len(text) // 3`, so a `3 * N`-character text is worth exactly `N` tokens +and a boundary can be hit on the nose. Four recurring shapes cover most of +the tally: boundary pairs (`f(text, N) == text` at the exact token count, +`f(text, N-1) == ` one tighter), accounting ladders (equal-token +items against a budget admitting exactly K), literal rosters (every +declared signature prefix asserted in one exact string), and exact routing +(each dispatch test asserts the precise condenser output, not a substring). + +## Verification commands run + +``` +.venv/bin/python -m pytest tests_py/core/context_assembly/ \ + tests_py/core/test_pg_recall_assemble_context_condensation.py -q +# 217 passed + +.venv/bin/python -m ruff check <6 source files> <4 new test files> +# All checks passed! +.venv/bin/python -m ruff format --check +# already formatted + +.venv/bin/python scripts/check_doc_claims.py +# doc claims OK +``` diff --git a/docs/module-inventory.md b/docs/module-inventory.md index 1165359e..3ae45b56 100644 --- a/docs/module-inventory.md +++ b/docs/module-inventory.md @@ -15,8 +15,13 @@ zetetic source rule: # find mcp_server/ -name "*.py" | grep -v __init__ | grep -v __pycache__ | wc -l # infrastructure/ re-measured 2026-07-30 (issue #275 split added 4 modules, # net file count +4; no infrastructure/ files were deleted) +# core/ re-measured 2026-07-30 (issue #228 behaviour-preserving split of +# core/context_assembly/condensers.py, over the 300-line §4.1 cap, into +# condense_text/condense_code/condense_structured/condense_dispatch/ +# condense_stage.py + a thin re-export facade; net file count +5, no +# core/ files were deleted) shared/ 26 files (10 documented below — curated subset) -core/ 225 files (~90 documented below — curated subset, incl. core/streaming, core/context_assembly) +core/ 230 files (~90 documented below — curated subset, incl. core/streaming, core/context_assembly) infrastructure/ 95 files (25 documented below — curated subset) handlers/ 135 files (55 registered tools — see docs/mcp-tools.md — + composition-root helpers) ``` @@ -188,8 +193,10 @@ Treat gaps as "undocumented," not "does not exist." - `document_normalizer.py` — `ParsedDocument` + provenance → wiki page markdown + memory payloads (skipped-image/empty notices, provenance stamping) *Undocumented (measured, not yet catalogued individually):* `core/streaming/` -(5 files) and `core/context_assembly/` (9 files) plus ~100 additional -top-level `core/` modules added since the last curation pass — ast +(5 files) and `core/context_assembly/` (15 files as of the issue #228 split — +corrected from the prior "9", one file already stale since #196's +`stage_phases.py`) plus ~100 additional top-level `core/` modules added +since the last curation pass — ast extraction, capture normalization, redaction, abstention gating, adaptive control/writing, backpressure, and other mechanisms not yet described above. Run the measurement command in the header to get a current file listing. diff --git a/mcp_server/core/context_assembly/condense_code.py b/mcp_server/core/context_assembly/condense_code.py new file mode 100644 index 00000000..6ccc229f --- /dev/null +++ b/mcp_server/core/context_assembly/condense_code.py @@ -0,0 +1,189 @@ +"""Condensers for code-shaped content (issue #228 split 2/4). + +Extracted from ``condensers.py`` (§4.1 — the original file was 391 lines, +over this repo's 300-line cap) with zero behaviour change. See +``condensers.py`` for the shared module docstring and re-export facade. + +Covers the code-block condenser (signatures only) and the assistant-message +condenser (verbatim code, compressed prose in between), plus the private +fence-splitting helpers both of them and the dispatcher depend on. +""" + +from __future__ import annotations + +from mcp_server.core.context_assembly.budget import ( + estimate_tokens, + truncate_to_budget, +) +from mcp_server.core.context_assembly.condense_text import _first_sentence + +# source: pre-existing tuned value, extracted unchanged (#197 family 3); +# provenance not recorded at introduction +_MIN_INDENT_RUNS_FOR_CODE = 3 + + +# ── Code block condenser ──────────────────────────────────────────────── +# Strategy: signatures only (function/class/imports), same spirit as the +# Swift condenseContracts. + + +def condense_code_block(text: str, token_budget: int) -> str: + """Keep imports, class, function, protocol, and method signatures only.""" + if estimate_tokens(text) <= token_budget: + return text + + kept: list[str] = [] + used = 0 + signature_prefixes = ( + "import ", + "from ", + "class ", + "def ", + "async def ", + "struct ", + "enum ", + "protocol ", + "func ", + "interface ", + "@", # decorators + "//", # comments + "#", # comments + ) + for line in text.split("\n"): + stripped = line.strip() + if not stripped: + continue + if any(stripped.startswith(p) for p in signature_prefixes): + t = estimate_tokens(line) + if used + t > token_budget: + break + kept.append(line) + used += t + if kept: + return "\n".join(kept) + return truncate_to_budget(text, token_budget) + + +# ── Assistant message condenser ───────────────────────────────────────── +# Strategy: keep code blocks verbatim (they're high-density facts that +# don't survive summarization), summarize prose by keeping topic +# sentences. + + +def condense_assistant_message(text: str, token_budget: int) -> str: + """Preserve code blocks verbatim, compress prose between them.""" + if estimate_tokens(text) <= token_budget: + return text + + parts = _split_by_code_blocks(text) + # Parts alternate: prose, code, prose, code, ... + # Priority: keep all code, compress prose. + code_parts = [p for is_code, p in parts if is_code] + prose_parts = [p for is_code, p in parts if not is_code] + + code_tokens = sum(estimate_tokens(p) for p in code_parts) + if code_tokens >= token_budget: + return _keep_leading_code_blocks(code_parts, token_budget, text) + + # DEAD CODE REMOVED (issue #228): both operands of the guard that used + # to sit here — `prose_parts and prose_budget > 0` — are provably true + # at this point, so the guard and its "concatenate code only" fallback + # were a branch no input could reach. The scoped mutmut run proved it + # empirically: every mutant of that fallback survived (§12.1 — the + # signature of dead code). Proof: + # (a) prose_budget > 0: reaching here means the `code_tokens >= + # token_budget` branch above did NOT return, so code_tokens < + # token_budget, i.e. token_budget - code_tokens > 0. + # (b) prose_parts is non-empty: it can only be empty when + # _split_by_code_blocks returns a single segment tagged as code + # (any closed or re-opened fence emits the fence line into a + # following prose segment — pinned by + # test_split_by_code_blocks_segments_preserve_original_newlines). + # A single segment IS the whole input text, so code_tokens == + # estimate_tokens(text), which the fast-path guard at the top of + # this function already established exceeds token_budget — i.e. + # (a) would have returned first. Pinned by + # test_assistant_unclosed_fence_is_a_single_code_segment. + prose_budget = token_budget - code_tokens + compressed_prose = _compress_prose_parts(prose_parts, prose_budget) + joined = _reassemble_in_order(parts, code_parts, compressed_prose) + return joined if joined else truncate_to_budget(text, token_budget) + + +def _keep_leading_code_blocks( + code_parts: list[str], token_budget: int, text: str +) -> str: + """Even the code exceeds budget — keep first N code blocks that fit. + + A single block bigger than the whole budget keeps nothing, and + returning "" would delete the memory outright. Degrade to the generic + truncation the sibling condensers fall back to. + """ + kept: list[str] = [] + used = 0 + for p in code_parts: + t = estimate_tokens(p) + if used + t > token_budget: + break + kept.append(p) + used += t + if not kept: + return truncate_to_budget(text, token_budget) + return "\n\n".join(kept) + + +def _compress_prose_parts(prose_parts: list[str], prose_budget: int) -> list[str]: + """Cut each prose segment to its first sentence, floored per-segment.""" + per_prose = max(20, prose_budget // len(prose_parts)) + return [_first_sentence(p)[: per_prose * 3] for p in prose_parts] + + +def _reassemble_in_order( + parts: list[tuple[bool, str]], + code_parts: list[str], + compressed_prose: list[str], +) -> str: + """Interleave verbatim code and compressed prose back into source order.""" + out: list[str] = [] + pi = ci = 0 + for is_code, _ in parts: + if is_code: + # EQUIVALENT MUTANT (#228): `ci < len(code_parts)` → `<=`. ci is + # incremented exactly once per is_code segment and code_parts is + # built from those same segments, so ci < len(code_parts) holds + # on every entry; both comparisons are unconditionally true and + # no input can distinguish them. Same for `pi` below. + if ci < len(code_parts): + out.append(code_parts[ci]) + ci += 1 + else: + if pi < len(compressed_prose): + out.append(compressed_prose[pi]) + pi += 1 + return "\n\n".join(s for s in out if s.strip()) + + +# ── Helpers ───────────────────────────────────────────────────────────── + + +def _has_code_blocks(text: str) -> bool: + return "```" in text or text.count(" ") >= _MIN_INDENT_RUNS_FOR_CODE + + +def _split_by_code_blocks(text: str) -> list[tuple[bool, str]]: + """Split markdown-style text into (is_code, chunk) segments.""" + segments: list[tuple[bool, str]] = [] + in_code = False + buf: list[str] = [] + for line in text.split("\n"): + if line.strip().startswith("```"): + if buf: + segments.append((in_code, "\n".join(buf))) + buf = [] + in_code = not in_code + buf.append(line) + else: + buf.append(line) + if buf: + segments.append((in_code, "\n".join(buf))) + return segments diff --git a/mcp_server/core/context_assembly/condense_dispatch.py b/mcp_server/core/context_assembly/condense_dispatch.py new file mode 100644 index 00000000..10538b49 --- /dev/null +++ b/mcp_server/core/context_assembly/condense_dispatch.py @@ -0,0 +1,76 @@ +"""Generic memory condenser: dispatch by content shape (issue #228 split +4/4). + +Extracted from ``condensers.py`` (§4.1 — the original file was 391 lines, +over this repo's 300-line cap) with zero behaviour change. See +``condensers.py`` for the shared module docstring and re-export facade. +""" + +from __future__ import annotations + +from mcp_server.core.context_assembly.budget import estimate_tokens +from mcp_server.core.context_assembly.condense_code import ( + _has_code_blocks, + condense_assistant_message, + condense_code_block, +) +from mcp_server.core.context_assembly.condense_structured import ( + _MIN_ARROWS_FOR_TRIPLES, + condense_entity_triples, +) +from mcp_server.core.context_assembly.condense_text import ( + condense_timeline_event, + condense_user_message, +) + + +def condense_memory_content( + content: str, + token_budget: int, + *, + tags: list[str] | None = None, +) -> str: + """Auto-dispatch to the right condenser based on content shape. + + Args: + content: the memory's textual content. + token_budget: target token count for the output. + tags: optional tag hints (e.g. ["code", "decision"]) to bias + dispatch. When provided, takes precedence over heuristic. + """ + # EQUIVALENT MUTANT (#228): `<=` → `<`. Every route below opens with + # this same guard on this exact (content, token_budget) pair and + # returns content verbatim, so the boundary case agrees either way. + if estimate_tokens(content) <= token_budget: + return content + + by_tag = _dispatch_by_tag(content, token_budget, tags or []) + if by_tag is not None: + return by_tag + return _dispatch_by_shape(content, token_budget) + + +def _dispatch_by_tag(content: str, token_budget: int, tags: list[str]) -> str | None: + """Explicit tag hints, checked before any content-shape heuristic.""" + if "code" in tags or "file" in tags: + return condense_code_block(content, token_budget) + if "timeline" in tags or "event" in tags: + return condense_timeline_event(content, token_budget) + return None + + +def _dispatch_by_shape(content: str, token_budget: int) -> str: + """Heuristic dispatch by content shape when no tag hint matched.""" + if _has_code_blocks(content): + return condense_assistant_message(content, token_budget) + if content.count("→") + content.count("->") >= _MIN_ARROWS_FOR_TRIPLES: + return condense_entity_triples(content, token_budget) + if content.lstrip().startswith("[user]:") or content.lstrip().startswith( + "[assistant]:" + ): + # Cortex memory format + if "[assistant]:" in content: + return condense_assistant_message(content, token_budget) + return condense_user_message(content, token_budget) + # Default: treat as prose user message + return condense_user_message(content, token_budget) diff --git a/mcp_server/core/context_assembly/condense_stage.py b/mcp_server/core/context_assembly/condense_stage.py new file mode 100644 index 00000000..018857d8 --- /dev/null +++ b/mcp_server/core/context_assembly/condense_stage.py @@ -0,0 +1,113 @@ +"""Stage-assembler integration point (issue #228 split, extracted from +``condensers.py``). + +StageAwareContextAssembler (stage_assembler.py) allocates a per-phase token +sub-budget to each of own/adjacent/summaries independently (submodular +selection caps chunk count per phase), so the sum can still exceed the +caller's total token_budget: estimate_tokens is a heuristic (chars // 3), +and per-phase caps do not renegotiate against each other. pg_recall. +assemble_context() promises a "budgeted, slot-filled prompt with truncation +awareness" (module docstring) — this function is the truncation-awareness +step: it is the caller that was missing, wiring condense_memory_content + +assemble_prompt's priority-condensation into the one place that already +computes the three raw text blocks. (Original wiring: issue #196.) + +Extracted from ``condensers.py`` (§4.1/§4.2 — the file was 391 lines, over +this repo's 300-line cap, and this function alone was 66 lines, over the +40-line cap) with zero behaviour change: the docstring below states the +same contract, split so the "how" of the over-budget path lives beside the +helper that implements it rather than in one 66-line body. See +``condensers.py`` for the shared module docstring and re-export facade. +""" + +from __future__ import annotations + +from mcp_server.core.context_assembly.budget import Placeholder, estimate_tokens +from mcp_server.core.context_assembly.condense_dispatch import condense_memory_content +from mcp_server.core.context_assembly.decomposer import assemble_prompt + +# (header, content, priority, key) row shape shared by the two helpers +# below: priority mirrors StageAwareContextAssembler's own emphasis order +# (own > adjacent > summaries; lower number = more important = condensed +# last, decomposer.py semantics). Keys are human-readable so +# build_truncation_banner's warning lines (which label by raw placeholder +# key) name the actual section, not an opaque index. +_Section = tuple[str, str, int, str] + + +def condense_assembled_context( + own_stage_context: str, + adjacent_stage_context: str, + stage_summaries: str, + current_stage: str, + token_budget: int, +) -> str: + """Fit stage_assembler's three raw text blocks into ``token_budget``. + + Precondition: ``token_budget > 0``; the three text args are the raw + (uncondensed) own/adjacent/summary blocks ``StageAwareContextAssembler + .assemble()`` produced (``StageContextResult.own_stage_context`` etc., + not the already-headered ``assembled_context``). + + Postcondition: returns a single string with the same ``"##
"`` + headers ``StageAwareContextAssembler`` uses, sections with empty + content omitted. Within budget: byte-identical to the assembler's own + concatenation (zero behavior change for the already-common case). Over + budget: each section is priority-condensed (own > adjacent > + summaries) via ``_condense_sections_over_budget`` — see that helper's + docstring for the condensation caveats it inherits from + ``decomposer.assemble_prompt``. + """ + sections = _build_present_sections( + own_stage_context, adjacent_stage_context, stage_summaries, current_stage + ) + total = sum(estimate_tokens(c) for _h, c, _pr, _k in sections) + if total <= token_budget: + return "\n\n".join(f"{h}\n\n{c}" for h, c, _pr, _k in sections) + return _condense_sections_over_budget(sections, token_budget) + + +def _build_present_sections( + own_stage_context: str, + adjacent_stage_context: str, + stage_summaries: str, + current_stage: str, +) -> list[_Section]: + """Strip inputs and keep only the non-empty (header, content, priority, + key) rows, in own/adjacent/summaries emphasis order.""" + own = own_stage_context.strip() + adjacent = adjacent_stage_context.strip() + summaries = stage_summaries.strip() + + # EQUIVALENT MUTANT (#196, re-confirmed #228): summaries' priority + # 3 → 4. decomposer.assemble_prompt consumes `priority` only through + # `sorted(..., key=lambda p: p.priority, reverse=True)` (decomposer.py + # lines 119 and 156) — never as a magnitude. With ranks {1, 2, 3} and + # {1, 2, 4} the descending order is byte-identical and no tie is + # created or broken, so no input can distinguish the two. + sections: list[_Section] = [ + (f"## Current Stage Context ({current_stage})", own, 1, "{{OWN_STAGE}}"), + ("## Related Prior Context", adjacent, 2, "{{ADJACENT_STAGE}}"), + ("## Stage Summaries", summaries, 3, "{{STAGE_SUMMARIES}}"), + ] + return [(h, c, pr, k) for h, c, pr, k in sections if c] + + +def _condense_sections_over_budget(sections: list[_Section], token_budget: int) -> str: + """Route over-budget sections through assemble_prompt's priority + condensation. + + Caveat inherited from ``assemble_prompt``: its truncation-warning + banner is prepended *after* the condensation loop and is not itself + counted against ``token_budget``, so the returned string (banner + included) can exceed ``token_budget`` by the banner's own length. + """ + template = "\n\n".join(f"{h}\n\n{k}" for h, _c, _pr, k in sections) + placeholders = [ + Placeholder(k, c, priority=pr, condenser=condense_memory_content) + for _h, c, pr, k in sections + ] + prompt, _metrics = assemble_prompt( + template, placeholders, context_window=token_budget, headroom=1.0 + ) + return prompt diff --git a/mcp_server/core/context_assembly/condense_structured.py b/mcp_server/core/context_assembly/condense_structured.py new file mode 100644 index 00000000..cb8dca62 --- /dev/null +++ b/mcp_server/core/context_assembly/condense_structured.py @@ -0,0 +1,46 @@ +"""Condenser for structured (subject, predicate, object) content (issue #228 +split 3/4). + +Extracted from ``condensers.py`` (§4.1 — the original file was 391 lines, +over this repo's 300-line cap) with zero behaviour change. See +``condensers.py`` for the shared module docstring and re-export facade. +""" + +from __future__ import annotations + +import re + +from mcp_server.core.context_assembly.budget import ( + estimate_tokens, + truncate_to_budget, +) + +# source: pre-existing tuned value, extracted unchanged (#197 family 3); +# provenance not recorded at introduction +_MIN_ARROWS_FOR_TRIPLES = 2 + + +# ── Entity-triple condenser ───────────────────────────────────────────── +# Strategy: keep (subject, predicate, object) triples verbatim, drop +# anything else. Triples are already maximally compressed. + + +def condense_entity_triples(text: str, token_budget: int) -> str: + """Keep only lines matching triple patterns, in budget order.""" + if estimate_tokens(text) <= token_budget: + return text + triple_re = re.compile( + r"^\s*([^→\->:]+?)\s*(?:→|->|:)\s*([^→\->:]+?)\s*(?:→|->|:)\s*(.+?)\s*$" + ) + kept_lines: list[str] = [] + used = 0 + for line in text.split("\n"): + if triple_re.match(line): + t = estimate_tokens(line) + if used + t > token_budget: + break + kept_lines.append(line) + used += t + if kept_lines: + return "\n".join(kept_lines) + return truncate_to_budget(text, token_budget) diff --git a/mcp_server/core/context_assembly/condense_text.py b/mcp_server/core/context_assembly/condense_text.py new file mode 100644 index 00000000..1c4382c6 --- /dev/null +++ b/mcp_server/core/context_assembly/condense_text.py @@ -0,0 +1,100 @@ +"""Condensers for free-text conversational content (issue #228 split 1/4). + +Extracted from ``condensers.py`` (§4.1 — the original file was 391 lines, +over this repo's 300-line cap) with zero behaviour change: same functions, +same bodies, same helpers, only the module boundary moved. See +``condensers.py`` for the shared module docstring and re-export facade. + +Covers the two condensers whose strategy is "keep a sentence-level slot, +drop the rest": the user-message condenser (first + questions + last) and +the timeline-event condenser (date + first sentence). +""" + +from __future__ import annotations + +import re + +from mcp_server.core.context_assembly.budget import ( + estimate_tokens, + truncate_to_budget, +) + +# source: structural — the condenser keeps first + last sentence, so texts +# of two or fewer sentences have no middle filler to drop. +_FIRST_PLUS_LAST_SENTENCES = 2 + + +# ── User message condenser ────────────────────────────────────────────── +# Strategy: keep the first sentence (establishes intent), any questions +# (explicit interrogatives), and the last sentence (final state of +# thought), dropping middle filler. + + +def condense_user_message(text: str, token_budget: int) -> str: + """Keep first sentence + questions + last sentence, within budget.""" + if estimate_tokens(text) <= token_budget: + return text + sentences = _split_sentences(text) + if len(sentences) <= _FIRST_PLUS_LAST_SENTENCES: + return truncate_to_budget(text, token_budget) + + kept: list[str] = [sentences[0]] + for s in sentences[1:-1]: + if "?" in s: + kept.append(s) + kept.append(sentences[-1]) + result = " ".join(kept).strip() + # EQUIVALENT MUTANT (#228): `<=` → `<`. On the boundary the mutant falls + # through to `truncate_to_budget(result, token_budget)`, whose own guard + # is `estimator(text) <= token_budget` — so it returns `result` unchanged + # and the two branches coincide exactly where the mutation moves the + # comparison. The same shape recurs in condense_timeline_event below. + if estimate_tokens(result) <= token_budget: + return result + return truncate_to_budget(result, token_budget) + + +# ── Timeline-event condenser ──────────────────────────────────────────── +# Strategy: extract (when, what, who) slots. A fixed schema compresses an +# event more reliably than a free-text summary because the salient fields +# are pinned. (Engineering heuristic — no biological source.) + + +def condense_timeline_event(text: str, token_budget: int) -> str: + """Extract when/what/who into a fixed-slot format within budget.""" + if estimate_tokens(text) <= token_budget: + return text + + date_match = re.search( + r"\[Date:\s*([^\]]+)\]|(\d{4}-\d{2}-\d{2})|" + r"(\w+\s+\d{1,2},?\s+\d{4})", + text, + ) + date = ( + date_match.group(1) or date_match.group(2) or date_match.group(3) + if date_match + else "" + ) + + first = _first_sentence(text) + compressed = f"[{date}] {first}" if date else first + # EQUIVALENT MUTANT (#228): `<=` → `<`, same shape as the one documented + # in condense_user_message — truncate_to_budget returns `compressed` + # unchanged on the boundary, so both branches agree there. + if estimate_tokens(compressed) <= token_budget: + return compressed + return truncate_to_budget(compressed, token_budget) + + +# ── Helpers ───────────────────────────────────────────────────────────── + + +def _split_sentences(text: str) -> list[str]: + """Naive sentence splitter; good enough for condensers.""" + parts = re.split(r"(?<=[.!?])\s+", text.strip()) + return [p for p in parts if p] + + +def _first_sentence(text: str) -> str: + sents = _split_sentences(text) + return sents[0] if sents else text diff --git a/mcp_server/core/context_assembly/condensers.py b/mcp_server/core/context_assembly/condensers.py index fa12abc1..0ee3c7d6 100644 --- a/mcp_server/core/context_assembly/condensers.py +++ b/mcp_server/core/context_assembly/condensers.py @@ -9,383 +9,62 @@ Adapted from Clément Deust's Swift condensers in ContextDecomposer.swift (`condenseContracts`, `condenseEngineGraph`, `condenseFileTree`, `condenseImpactReport`), plus Cortex-specific memory types. + +Issue #228 split this module (was 391 lines, over this repo's 300-line +§4.1 cap, with two functions over the 40-line §4.2 cap) into one file per +condenser family, behaviour-preserving: + +- ``condense_text.py`` — user-message and timeline-event condensers + (sentence-slot strategies) + their sentence-splitting helpers. +- ``condense_code.py`` — code-block and assistant-message condensers + (verbatim-code strategies) + their fence-splitting helpers. +- ``condense_structured.py`` — the entity-triple condenser. +- ``condense_dispatch.py`` — ``condense_memory_content``, the + shape/tag-driven dispatcher over the three families above. +- ``condense_stage.py`` — ``condense_assembled_context``, the + stage-assembler integration point (issue #196). + +This module re-exports every public name (plus the private helpers the +issue #228 test suite targets directly) so every existing import path +(``from mcp_server.core.context_assembly.condensers import ...``, +including patch targets in tests) keeps resolving unchanged — the split +is an internal reorganization, not an API change. """ from __future__ import annotations -import re - -from mcp_server.core.context_assembly.budget import ( - estimate_tokens, - truncate_to_budget, +from mcp_server.core.context_assembly.condense_code import ( + _has_code_blocks as _has_code_blocks, +) +from mcp_server.core.context_assembly.condense_code import ( + _split_by_code_blocks as _split_by_code_blocks, +) +from mcp_server.core.context_assembly.condense_code import ( + condense_assistant_message, + condense_code_block, +) +from mcp_server.core.context_assembly.condense_dispatch import condense_memory_content +from mcp_server.core.context_assembly.condense_stage import condense_assembled_context +from mcp_server.core.context_assembly.condense_structured import ( + condense_entity_triples, +) +from mcp_server.core.context_assembly.condense_text import ( + _first_sentence as _first_sentence, +) +from mcp_server.core.context_assembly.condense_text import ( + _split_sentences as _split_sentences, +) +from mcp_server.core.context_assembly.condense_text import ( + condense_timeline_event, + condense_user_message, ) -from mcp_server.core.context_assembly.budget import Placeholder -from mcp_server.core.context_assembly.decomposer import assemble_prompt - -# source: structural — the condenser keeps first + last sentence, so texts -# of two or fewer sentences have no middle filler to drop. -_FIRST_PLUS_LAST_SENTENCES = 2 -# source: pre-existing tuned value, extracted unchanged (#197 family 3); -# provenance not recorded at introduction -_MIN_ARROWS_FOR_TRIPLES = 2 -# source: pre-existing tuned value, extracted unchanged (#197 family 3); -# provenance not recorded at introduction -_MIN_INDENT_RUNS_FOR_CODE = 3 - - -# ── User message condenser ────────────────────────────────────────────── -# Strategy: keep the first sentence (establishes intent), any questions -# (explicit interrogatives), and the last sentence (final state of -# thought), dropping middle filler. - - -def condense_user_message(text: str, token_budget: int) -> str: - """Keep first sentence + questions + last sentence, within budget.""" - if estimate_tokens(text) <= token_budget: - return text - sentences = _split_sentences(text) - if len(sentences) <= _FIRST_PLUS_LAST_SENTENCES: - return truncate_to_budget(text, token_budget) - - kept: list[str] = [sentences[0]] - for s in sentences[1:-1]: - if "?" in s: - kept.append(s) - kept.append(sentences[-1]) - result = " ".join(kept).strip() - if estimate_tokens(result) <= token_budget: - return result - return truncate_to_budget(result, token_budget) - - -# ── Assistant message condenser ───────────────────────────────────────── -# Strategy: keep code blocks verbatim (they're high-density facts that -# don't survive summarization), summarize prose by keeping topic -# sentences. - - -def condense_assistant_message(text: str, token_budget: int) -> str: - """Preserve code blocks verbatim, compress prose between them.""" - if estimate_tokens(text) <= token_budget: - return text - - parts = _split_by_code_blocks(text) - # Parts alternate: prose, code, prose, code, ... - # Priority: keep all code, compress prose. - code_parts = [p for is_code, p in parts if is_code] - prose_parts = [p for is_code, p in parts if not is_code] - - code_tokens = sum(estimate_tokens(p) for p in code_parts) - if code_tokens >= token_budget: - # Even the code exceeds budget — keep first N code blocks - kept: list[str] = [] - used = 0 - for p in code_parts: - t = estimate_tokens(p) - if used + t > token_budget: - break - kept.append(p) - used += t - # A single block bigger than the whole budget keeps nothing, and - # returning "" would delete the memory outright. Degrade to the - # generic truncation the sibling condensers fall back to. - if not kept: - return truncate_to_budget(text, token_budget) - return "\n\n".join(kept) - - prose_budget = token_budget - code_tokens - if prose_parts and prose_budget > 0: - per_prose = max(20, prose_budget // len(prose_parts)) - compressed_prose = [_first_sentence(p)[: per_prose * 3] for p in prose_parts] - # Reassemble in original order - out: list[str] = [] - pi = ci = 0 - for is_code, _ in parts: - if is_code: - if ci < len(code_parts): - out.append(code_parts[ci]) - ci += 1 - else: - if pi < len(compressed_prose): - out.append(compressed_prose[pi]) - pi += 1 - joined = "\n\n".join(s for s in out if s.strip()) - return joined if joined else truncate_to_budget(text, token_budget) - # No prose budget — just concatenate code. - # NOTE (issue #196 coverage audit): prose_parts can only be empty - # when _split_by_code_blocks returned a single segment — any closed - # or re-opened fence emits the fence line into a following prose - # segment, so two code segments always have a prose segment between - # them. A single code segment IS the whole input text, so - # code_tokens == estimate_tokens(text) exactly. The fast-path guard - # at the top (`estimate_tokens(text) <= token_budget` returns early) - # ensures we only reach here when estimate_tokens(text) > - # token_budget — which, combined with the equality above, forces - # code_tokens > token_budget, so the `code_tokens >= token_budget` - # branch earlier in this function always returns first. This line is - # provably unreachable through this function's own arithmetic; kept - # (not deleted) as the correct fallback should a future edit to - # _split_by_code_blocks break that single-segment invariant. - return "\n\n".join(code_parts) # pragma: no cover - - -# ── Entity-triple condenser ───────────────────────────────────────────── -# Strategy: keep (subject, predicate, object) triples verbatim, drop -# anything else. Triples are already maximally compressed. - - -def condense_entity_triples(text: str, token_budget: int) -> str: - """Keep only lines matching triple patterns, in budget order.""" - if estimate_tokens(text) <= token_budget: - return text - triple_re = re.compile( - r"^\s*([^→\->:]+?)\s*(?:→|->|:)\s*([^→\->:]+?)\s*(?:→|->|:)\s*(.+?)\s*$" - ) - kept_lines: list[str] = [] - used = 0 - for line in text.split("\n"): - if triple_re.match(line): - t = estimate_tokens(line) - if used + t > token_budget: - break - kept_lines.append(line) - used += t - if kept_lines: - return "\n".join(kept_lines) - return truncate_to_budget(text, token_budget) - - -# ── Timeline-event condenser ──────────────────────────────────────────── -# Strategy: extract (when, what, who) slots. A fixed schema compresses an -# event more reliably than a free-text summary because the salient fields -# are pinned. (Engineering heuristic — no biological source.) - - -def condense_timeline_event(text: str, token_budget: int) -> str: - """Extract when/what/who into a fixed-slot format within budget.""" - if estimate_tokens(text) <= token_budget: - return text - - date_match = re.search( - r"\[Date:\s*([^\]]+)\]|(\d{4}-\d{2}-\d{2})|" - r"(\w+\s+\d{1,2},?\s+\d{4})", - text, - ) - date = ( - date_match.group(1) or date_match.group(2) or date_match.group(3) - if date_match - else "" - ) - - first = _first_sentence(text) - compressed = f"[{date}] {first}" if date else first - if estimate_tokens(compressed) <= token_budget: - return compressed - return truncate_to_budget(compressed, token_budget) - - -# ── Code block condenser ──────────────────────────────────────────────── -# Strategy: signatures only (function/class/imports), same spirit as the -# Swift condenseContracts. - - -def condense_code_block(text: str, token_budget: int) -> str: - """Keep imports, class, function, protocol, and method signatures only.""" - if estimate_tokens(text) <= token_budget: - return text - - kept: list[str] = [] - used = 0 - signature_prefixes = ( - "import ", - "from ", - "class ", - "def ", - "async def ", - "struct ", - "enum ", - "protocol ", - "func ", - "interface ", - "@", # decorators - "//", # comments - "#", # comments - ) - for line in text.split("\n"): - stripped = line.strip() - if not stripped: - continue - if any(stripped.startswith(p) for p in signature_prefixes): - t = estimate_tokens(line) - if used + t > token_budget: - break - kept.append(line) - used += t - if kept: - return "\n".join(kept) - return truncate_to_budget(text, token_budget) - - -# ── Stage-assembler integration point ──────────────────────────────────── -# StageAwareContextAssembler (stage_assembler.py) allocates a per-phase -# token sub-budget to each of own/adjacent/summaries independently -# (submodular selection caps chunk count per phase), so the sum can -# still exceed the caller's total token_budget: estimate_tokens is a -# heuristic (chars // 3), and per-phase caps do not renegotiate against -# each other. pg_recall.assemble_context() promises a "budgeted, -# slot-filled prompt with truncation awareness" (module docstring) — -# this function is the truncation-awareness step: it is the caller that -# was missing, wiring condense_memory_content + assemble_prompt's -# priority-condensation into the one place that already computes the -# three raw text blocks. - - -def condense_assembled_context( - own_stage_context: str, - adjacent_stage_context: str, - stage_summaries: str, - current_stage: str, - token_budget: int, -) -> str: - """Fit stage_assembler's three raw text blocks into ``token_budget``. - - Precondition: ``token_budget > 0``; the three text args are the raw - (uncondensed) own/adjacent/summary blocks ``StageAwareContextAssembler - .assemble()`` produced (``StageContextResult.own_stage_context`` etc., - not the already-headered ``assembled_context``). - - Postcondition: returns a single string with the same ``"##
"`` - headers ``StageAwareContextAssembler`` uses, sections with empty - content omitted (byte-identical formatting to the assembler's own - concatenation). When the combined estimated token count of the three - inputs is within ``token_budget``, the return value is exactly that - header+concatenation (no condensation applied — zero behavior change - for the already-common case). When it exceeds ``token_budget``, each - section is priority-condensed (own > adjacent > summaries, matching - ``StageAwareContextAssembler``'s own emphasis order) via - ``condense_memory_content`` through ``decomposer.assemble_prompt``'s - progressive-condensation + post-assembly safety loop, so the - condensed *content* is within ``token_budget`` up to that loop's - fixed safety margin. Caveat inherited from ``assemble_prompt``: its - truncation-warning banner is prepended *after* that loop and is not - itself counted against the budget, so the returned string (banner - included) can exceed ``token_budget`` by the banner's own length. - """ - own = own_stage_context.strip() - adjacent = adjacent_stage_context.strip() - summaries = stage_summaries.strip() - - # (header, content, priority, key) — priority mirrors StageAware - # ContextAssembler's own emphasis order (own > adjacent > summaries); - # lower number = more important = condensed last (decomposer.py - # semantics). Keys are human-readable so build_truncation_banner's - # warning lines (which label by raw placeholder key) name the actual - # section, not an opaque index. - sections = [ - (f"## Current Stage Context ({current_stage})", own, 1, "{{OWN_STAGE}}"), - ("## Related Prior Context", adjacent, 2, "{{ADJACENT_STAGE}}"), - ("## Stage Summaries", summaries, 3, "{{STAGE_SUMMARIES}}"), - ] - present = [(h, c, pr, k) for h, c, pr, k in sections if c] - - total = sum(estimate_tokens(c) for _h, c, _pr, _k in present) - if total <= token_budget: - return "\n\n".join(f"{h}\n\n{c}" for h, c, _pr, _k in present) - - # Late import: decomposer depends on budget only, condensers depends - # on budget only — importing decomposer here (not at module scope) - # keeps condensers.py's own import graph unchanged for every caller - # that never hits the over-budget path. - - template = "\n\n".join(f"{h}\n\n{k}" for h, _c, _pr, k in present) - placeholders = [ - Placeholder(k, c, priority=pr, condenser=condense_memory_content) - for _h, c, pr, k in present - ] - prompt, _metrics = assemble_prompt( - template, placeholders, context_window=token_budget, headroom=1.0 - ) - return prompt - - -# ── Generic memory condenser ──────────────────────────────────────────── -# Strategy: dispatch by content shape. When unsure, truncate. - - -def condense_memory_content( - content: str, - token_budget: int, - *, - tags: list[str] | None = None, -) -> str: - """Auto-dispatch to the right condenser based on content shape. - - Args: - content: the memory's textual content. - token_budget: target token count for the output. - tags: optional tag hints (e.g. ["code", "decision"]) to bias - dispatch. When provided, takes precedence over heuristic. - """ - if estimate_tokens(content) <= token_budget: - return content - - tags = tags or [] - - # Tag-driven dispatch first - if "code" in tags or "file" in tags: - return condense_code_block(content, token_budget) - if "timeline" in tags or "event" in tags: - return condense_timeline_event(content, token_budget) - - # Heuristic dispatch by content shape - if _has_code_blocks(content): - return condense_assistant_message(content, token_budget) - if content.count("→") + content.count("->") >= _MIN_ARROWS_FOR_TRIPLES: - return condense_entity_triples(content, token_budget) - if content.lstrip().startswith("[user]:") or content.lstrip().startswith( - "[assistant]:" - ): - # Cortex memory format - if "[assistant]:" in content: - return condense_assistant_message(content, token_budget) - return condense_user_message(content, token_budget) - - # Default: treat as prose user message - return condense_user_message(content, token_budget) - - -# ── Helpers ───────────────────────────────────────────────────────────── - - -def _split_sentences(text: str) -> list[str]: - """Naive sentence splitter; good enough for condensers.""" - parts = re.split(r"(?<=[.!?])\s+", text.strip()) - return [p for p in parts if p] - - -def _first_sentence(text: str) -> str: - sents = _split_sentences(text) - return sents[0] if sents else text - - -def _has_code_blocks(text: str) -> bool: - return "```" in text or text.count(" ") >= _MIN_INDENT_RUNS_FOR_CODE - -def _split_by_code_blocks(text: str) -> list[tuple[bool, str]]: - """Split markdown-style text into (is_code, chunk) segments.""" - segments: list[tuple[bool, str]] = [] - in_code = False - buf: list[str] = [] - for line in text.split("\n"): - if line.strip().startswith("```"): - if buf: - segments.append((in_code, "\n".join(buf))) - buf = [] - in_code = not in_code - buf.append(line) - else: - buf.append(line) - if buf: - segments.append((in_code, "\n".join(buf))) - return segments +__all__ = [ + "condense_assembled_context", + "condense_assistant_message", + "condense_code_block", + "condense_entity_triples", + "condense_memory_content", + "condense_timeline_event", + "condense_user_message", +] diff --git a/tests_py/core/context_assembly/test_condense_code.py b/tests_py/core/context_assembly/test_condense_code.py new file mode 100644 index 00000000..7028b2fb --- /dev/null +++ b/tests_py/core/context_assembly/test_condense_code.py @@ -0,0 +1,251 @@ +"""Mutation-hardening contract tests for condense_code.py (issue #228). + +A scoped mutmut run over the pre-split condensers.py left survivors in +condense_code_block (29), condense_assistant_message (28), and the three +fence-splitting helpers (7) that no test in test_condensers.py could +distinguish. Each test below pins one such observable contract via an +exact-equality assertion — the survivors were precisely the mutants a +substring/`in` assertion cannot see. + +Budgets are chosen so the arithmetic is exact: ``estimate_tokens`` is +``len(text) // 3``, so a text of ``3 * N`` characters is worth exactly +``N`` tokens and a boundary can be hit on the nose. +""" + +from __future__ import annotations + +from mcp_server.core.context_assembly.budget import estimate_tokens +from mcp_server.core.context_assembly.condense_code import ( + _has_code_blocks, + _split_by_code_blocks, + condense_assistant_message, + condense_code_block, +) + +# ── condense_assistant_message ──────────────────────────────────────── + +# Four fenced blocks worth 3 tokens each; the "```" line closing one block +# and opening the next is the 1-token prose segment between them. +FOUR_BLOCKS = "\n".join(f"```\nblk_{i}()\n```" for i in range(4)) +FOUR_BLOCKS_CODE_TOKENS = 12 + +# Three fenced blocks separated by prose whose first sentences are all +# longer than 80 characters, so every slice length below is observable. +_PROSE = ( + "Prose {n} opening sentence padded out to well beyond " + "sixty-three characters long. Prose {n} tail." +) +INTERLEAVED = "\n".join( + [ + _PROSE.format(n="one"), + "```", + "c_one()", + "```", + _PROSE.format(n="two"), + "```", + "c_two()", + "```", + _PROSE.format(n="three"), + "```", + "c_three()", + "```", + _PROSE.format(n="four"), + ] +) + + +def test_assistant_message_at_exact_budget_is_returned_verbatim() -> None: + text = "Alpha sentence. Beta sentence." # 30 chars -> 10 tokens + assert estimate_tokens(text) == 10 + + assert condense_assistant_message(text, 10) == text + assert condense_assistant_message(text, 9) == "Alpha sentence." + + +def test_assistant_code_only_path_stops_at_the_budget_boundary() -> None: + """Blocks are admitted while ``used + t <= budget``, then dropped. + + Budget 6 admits exactly two 3-token blocks: the third would make 9. + """ + out = condense_assistant_message(FOUR_BLOCKS, 6) + + assert out == "```\nblk_0()\n\n```\nblk_1()" + + +def test_assistant_code_equal_to_budget_takes_the_code_only_path() -> None: + """``code_tokens >= token_budget`` is inclusive: equality keeps code only. + + At the boundary the prose path would instead interleave the bare "```" + separator segments between the blocks. + """ + assert ( + sum( + estimate_tokens(p) + for is_code, p in _split_by_code_blocks(FOUR_BLOCKS) + if is_code + ) + == FOUR_BLOCKS_CODE_TOKENS + ) + + out = condense_assistant_message(FOUR_BLOCKS, FOUR_BLOCKS_CODE_TOKENS) + + assert out == "```\nblk_0()\n\n```\nblk_1()\n\n```\nblk_2()\n\n```\nblk_3()" + + +def test_assistant_interleaves_prose_and_code_in_original_order() -> None: + """Every block and every compressed prose part lands in source order. + + Budget 60 leaves 50 prose tokens over 4 prose parts (12 each), so the + ``max(20, ...)`` floor binds and each prose part is cut to 20 * 3 chars. + """ + out = condense_assistant_message(INTERLEAVED, 60) + + assert out == ( + "Prose one opening sentence padded out to well beyond sixty-t" + "\n\n```\nc_one()" + "\n\n```\nProse two opening sentence padded out to well beyond six" + "\n\n```\nc_two()" + "\n\n```\nProse three opening sentence padded out to well beyond s" + "\n\n```\nc_three()" + "\n\n```\nProse four opening sentence padded out to well beyond si" + ) + + +def test_assistant_prose_share_scales_with_the_leftover_budget() -> None: + """Above the floor, the prose slice tracks ``budget - code_tokens``. + + Budget 100 leaves 90 prose tokens over 4 parts (22 each), lifting the + result off the ``max(20, ...)`` floor exercised by the test above. + """ + out = condense_assistant_message(INTERLEAVED, 100) + + assert out == ( + "Prose one opening sentence padded out to well beyond sixty-three c" + "\n\n```\nc_one()" + "\n\n```\nProse two opening sentence padded out to well beyond sixty-thr" + "\n\n```\nc_two()" + "\n\n```\nProse three opening sentence padded out to well beyond sixty-t" + "\n\n```\nc_three()" + "\n\n```\nProse four opening sentence padded out to well beyond sixty-th" + ) + + +def test_assistant_blank_only_content_falls_back_to_truncation() -> None: + """When every reassembled part is blank, the raw text is truncated. + + This is the only route to that fallback: the ``if s.strip()`` filter + empties the join, and returning "" would delete the memory outright. + """ + out = condense_assistant_message(" " * 300, 10) + + assert out == " " * 30 + + +def test_assistant_unclosed_fence_is_a_single_code_segment() -> None: + """Pins half (b) of the invariant that made the old fallback dead code. + + ``prose_parts`` can only be empty when the whole text is ONE code + segment, which happens only for an unclosed fence — and then + ``code_tokens == estimate_tokens(text)``, which the over-budget guard + at the top of the function has already established exceeds the budget. + So ``code_tokens >= token_budget`` always returns first (see the + dead-code-removal rationale in ``condense_code.py``). + """ + text = "```\n" + "step()\n" * 10 + + assert _split_by_code_blocks(text) == [(True, text)] + assert sum(estimate_tokens(p) for _is_code, p in [(True, text)]) == ( + estimate_tokens(text) + ) + # Budget 10 < 24 tokens: the single block cannot be kept, so the + # code-only branch degrades to generic truncation rather than "". + assert condense_assistant_message(text, 10) == "```\nstep()\nstep()\nstep()\n" + + +# ── condense_code_block ─────────────────────────────────────────────── + +# One line per declared signature prefix, plus a blank line (which must be +# skipped, not treated as end-of-scan) and a body that must be dropped. +SIGNATURE_LINES = [ + "import os", + "", + "from x import y", + "class C:", + "def f():", + "async def g():", + "struct S {", + "enum E {", + "protocol P {", + "func h() {", + "interface I {", + "@decorator", + "// c-comment", + "# py-comment", +] +SIGNATURE_SOURCE = "\n".join(SIGNATURE_LINES + [" body_statement()"] * 40) + +# Four 3-token signature lines with non-signature bodies between them. +SIGNATURE_ACCOUNTING = "\n".join(f"def a{i}():\n x()" for i in range(1, 5)) + + +def test_code_block_at_exact_budget_is_returned_verbatim() -> None: + assert estimate_tokens(SIGNATURE_ACCOUNTING) == 23 + + assert condense_code_block(SIGNATURE_ACCOUNTING, 23) == SIGNATURE_ACCOUNTING + assert condense_code_block(SIGNATURE_ACCOUNTING, 22) == ( + "def a1():\ndef a2():\ndef a3():\ndef a4():" + ) + + +def test_code_block_keeps_every_declared_signature_prefix() -> None: + """Each of the 13 prefixes is recognised, and a blank line is skipped. + + A budget of 200 is far above the 43 tokens the signatures cost, so + nothing here is a budget effect: any missing line means its prefix + stopped matching, and a truncated result means the blank line ended + the scan instead of being skipped. + """ + out = condense_code_block(SIGNATURE_SOURCE, 200) + + assert out == "\n".join(line for line in SIGNATURE_LINES if line) + assert "body_statement()" not in out + + +def test_code_block_signature_accounting_stops_at_the_budget_boundary() -> None: + """Signatures are admitted while ``used + t <= budget``, then dropped. + + Budget 6 admits exactly two 3-token signatures: the third would make 9. + """ + out = condense_code_block(SIGNATURE_ACCOUNTING, 6) + + assert out == "def a1():\ndef a2():" + + +# ── helpers ─────────────────────────────────────────────────────────── + + +def test_split_by_code_blocks_segments_preserve_original_newlines() -> None: + """Segments are joined with a bare newline and flagged False/True. + + The fence line that closes a block is emitted into the FOLLOWING + segment — the property that makes a prose segment exist between any + two code segments (see the invariant in ``condense_assistant_message``). + """ + text = "Prose one.\nProse two.\n```\ncode_a()\ncode_b()\n```\nTail one.\nTail two." + + assert _split_by_code_blocks(text) == [ + (False, "Prose one.\nProse two."), + (True, "```\ncode_a()\ncode_b()"), + (False, "```\nTail one.\nTail two."), + ] + + +def test_has_code_blocks_indent_run_threshold_is_three() -> None: + """Three 4-space runs is code; two is not. The comparison is inclusive.""" + assert _has_code_blocks("x" + " " * 3) is True + assert _has_code_blocks("x" + " " * 2) is False + + +def test_has_code_blocks_detects_a_fence() -> None: + assert _has_code_blocks("```") is True + assert _has_code_blocks("plain prose") is False diff --git a/tests_py/core/context_assembly/test_condense_dispatch.py b/tests_py/core/context_assembly/test_condense_dispatch.py new file mode 100644 index 00000000..3cbc66ac --- /dev/null +++ b/tests_py/core/context_assembly/test_condense_dispatch.py @@ -0,0 +1,114 @@ +"""Mutation-hardening dispatch tests for condense_dispatch.py (issue #228). + +A scoped mutmut run over the pre-split condensers.py left 25 survivors in +condense_memory_content that no test in test_condensers.py could +distinguish — the routing decision itself (which tag matches, which +content shape wins, which speaker prefix is recognised), not any single +condenser's internal arithmetic. + +Each test asserts the EXACT output of the condenser the dispatcher is +expected to route to, because every route ultimately returns a shortened +string: a substring assertion cannot tell "routed to the code condenser" +apart from "fell through to the prose condenser, which happened to keep +the same opening". +""" + +from __future__ import annotations + +from mcp_server.core.context_assembly.condense_dispatch import condense_memory_content + +# Indented code whose 4-space runs would route it to the assistant +# condenser on shape alone — so a tag that fails to match is visible. +TAGGED_CODE = "import sys\ndef go():\n" + " body()\n" * 200 + "final_call()\n" +TAGGED_CODE_CONDENSED = "import sys\ndef go():" + +TAGGED_EVENT = "On 2026-01-02 it shipped. " + "filler. " * 300 +TAGGED_EVENT_CONDENSED = "[2026-01-02] On 2026-01-02 it shipped." + +# Three sentences, one of them pure filler: condensing drops the middle. +THREE_SENTENCES = "One. Two. Three." +THREE_SENTENCES_TOKENS = 5 # len == 16, 16 // 3 == 5 + +_ARROW_FILLER = "prose filler line\n" * 80 + + +def test_memory_content_at_exact_budget_is_returned_verbatim() -> None: + assert condense_memory_content(THREE_SENTENCES, THREE_SENTENCES_TOKENS) == ( + THREE_SENTENCES + ) + assert condense_memory_content(THREE_SENTENCES, 4) == "One. Three." + + +def test_code_tag_routes_to_the_code_block_condenser() -> None: + assert condense_memory_content(TAGGED_CODE, 40, tags=["code"]) == ( + TAGGED_CODE_CONDENSED + ) + + +def test_file_tag_routes_to_the_code_block_condenser() -> None: + """``file`` is the second arm of the same ``or`` — it must match alone.""" + assert condense_memory_content(TAGGED_CODE, 40, tags=["file"]) == ( + TAGGED_CODE_CONDENSED + ) + + +def test_timeline_tag_routes_to_the_timeline_condenser() -> None: + assert condense_memory_content(TAGGED_EVENT, 40, tags=["timeline"]) == ( + TAGGED_EVENT_CONDENSED + ) + + +def test_event_tag_routes_to_the_timeline_condenser() -> None: + """``event`` is the second arm of the same ``or`` — it must match alone.""" + assert condense_memory_content(TAGGED_EVENT, 40, tags=["event"]) == ( + TAGGED_EVENT_CONDENSED + ) + + +def test_two_unicode_arrows_reach_the_triple_threshold() -> None: + """Exactly two "→" and no "->" is on the threshold, inclusive.""" + content = "alpha → beta → gamma\n" + _ARROW_FILLER + assert (content.count("→"), content.count("->")) == (2, 0) + + assert condense_memory_content(content, 20) == "alpha → beta → gamma" + + +def test_two_ascii_arrows_reach_the_triple_threshold() -> None: + """The ASCII "->" spelling is counted into the same threshold.""" + content = "alpha -> beta -> gamma\n" + _ARROW_FILLER + assert (content.count("→"), content.count("->")) == (0, 2) + + assert condense_memory_content(content, 20) == "alpha -> beta -> gamma" + + +def test_assistant_prefixed_transcript_routes_to_the_assistant_condenser() -> None: + """The speaker prefix is read after ``lstrip()``, and matched exactly. + + The assistant condenser keeps only the leading sentence; the user + condenser would append the closing sentence too, so the two routes + are distinguishable on this input. + """ + content = ( + " \n[assistant]: Opening line. " + "Filler sentence. " * 60 + "Closing line." + ) + + assert condense_memory_content(content, 40) == "[assistant]: Opening line." + + +def test_user_prefixed_transcript_naming_the_assistant_routes_to_assistant() -> None: + """A ``[user]:`` opener that later names the assistant uses that branch.""" + content = ( + " \n[user]: Opening question. [assistant]: Answer line. " + + "Filler sentence. " * 60 + + "Closing line." + ) + + assert condense_memory_content(content, 40) == "[user]: Opening question." + + +def test_indent_runs_alone_route_to_the_assistant_condenser() -> None: + """Shape dispatch fires on indentation even with no fence present.""" + content = "Head line.\n" + " a()\n" * 3 + "Filler prose sentence. " * 100 + assert content.count(" ") == 3 + + assert condense_memory_content(content, 60) == "Head line." diff --git a/tests_py/core/context_assembly/test_condense_structured.py b/tests_py/core/context_assembly/test_condense_structured.py new file mode 100644 index 00000000..fec3e949 --- /dev/null +++ b/tests_py/core/context_assembly/test_condense_structured.py @@ -0,0 +1,40 @@ +"""Mutation-hardening contract tests for condense_structured.py (issue #228). + +A scoped mutmut run over the pre-split condensers.py left 9 survivors in +condense_entity_triples that no test in test_condensers.py could +distinguish. Each test below pins one such observable contract via an +exact-equality assertion. + +Budgets are chosen so the arithmetic is exact: ``estimate_tokens`` is +``len(text) // 3``, so a text of ``3 * N`` characters is worth exactly +``N`` tokens and a boundary can be hit on the nose. +""" + +from __future__ import annotations + +from mcp_server.core.context_assembly.budget import estimate_tokens +from mcp_server.core.context_assembly.condense_structured import ( + condense_entity_triples, +) + +# Four 5-token triple lines behind one line that is not a triple. +TRIPLE_LINES = [f"aa{i} → bb{i} → cc{i}" for i in range(4)] +TRIPLE_TEXT = "Prose preamble line\n" + "\n".join(TRIPLE_LINES) + + +def test_entity_triples_at_exact_budget_is_returned_verbatim() -> None: + assert estimate_tokens(TRIPLE_TEXT) == 27 + + assert condense_entity_triples(TRIPLE_TEXT, 27) == TRIPLE_TEXT + # One token tighter: the non-triple preamble is dropped. + assert condense_entity_triples(TRIPLE_TEXT, 26) == "\n".join(TRIPLE_LINES) + + +def test_entity_triples_stop_at_the_budget_boundary() -> None: + """Lines are admitted while ``used + t <= budget``, then the scan breaks. + + Budget 10 admits exactly two 5-token lines: the third would make 15. + """ + out = condense_entity_triples(TRIPLE_TEXT, 10) + + assert out == "aa0 → bb0 → cc0\naa1 → bb1 → cc1" diff --git a/tests_py/core/context_assembly/test_condense_text.py b/tests_py/core/context_assembly/test_condense_text.py new file mode 100644 index 00000000..1df057b9 --- /dev/null +++ b/tests_py/core/context_assembly/test_condense_text.py @@ -0,0 +1,159 @@ +"""Mutation-hardening contract tests for condense_text.py (issue #228). + +A scoped mutmut run over the pre-split condensers.py left survivors in +condense_user_message (12) and condense_timeline_event (15) that no test in +test_condensers.py could distinguish — behaviours a substring/`in` +assertion cannot see. Each test below pins one such observable contract: a +budget boundary, an accounting step, or a slice length, via exact-equality +assertions. + +Budgets are chosen so the arithmetic is exact: ``estimate_tokens`` is +``len(text) // 3``, so a text of ``3 * N`` characters is worth exactly +``N`` tokens and a boundary can be hit on the nose. +""" + +from __future__ import annotations + +from mcp_server.core.context_assembly.budget import estimate_tokens +from mcp_server.core.context_assembly.condense_text import ( + _first_sentence, + _split_sentences, + condense_timeline_event, + condense_user_message, +) + +# Three sentences, one of them pure filler: condensing drops the middle, +# so "returned verbatim" and "condensed" are trivially distinguishable. +THREE_SENTENCES = "One. Two. Three." +THREE_SENTENCES_TOKENS = 5 # len == 16, 16 // 3 == 5 + +# First + both questions + last, joined by a single space (38 chars). +QUESTION_TEXT = "Intro line. Q one? Filler mid. Q two? Ending line." +QUESTION_KEPT = "Intro line. Q one? Q two? Ending line." + + +# ── condense_user_message ───────────────────────────────────────────── + + +def test_user_message_at_exact_budget_is_returned_verbatim() -> None: + """``estimate_tokens(text) == token_budget`` takes the no-op path.""" + assert estimate_tokens(THREE_SENTENCES) == THREE_SENTENCES_TOKENS + + assert condense_user_message(THREE_SENTENCES, THREE_SENTENCES_TOKENS) == ( + THREE_SENTENCES + ) + # One token tighter and the middle sentence is dropped — proof the + # verbatim result above is the boundary, not a coincidence. + assert condense_user_message(THREE_SENTENCES, 4) == "One. Three." + + +def test_two_sentence_message_truncates_the_original_not_the_rejoin() -> None: + """A 2-sentence text truncates ``text``, keeping its line structure. + + The first+last rebuild would re-join the two sentences with a single + space, destroying the newline ``truncate_to_budget`` cuts on. + """ + text = "First sentence here.\n" + "Second sentence " * 50 + + assert condense_user_message(text, 10) == "First sentence here.\n" + + +def test_user_message_keeps_every_middle_question_in_order() -> None: + """All interrogative middles survive; non-question middles do not.""" + out = condense_user_message(QUESTION_TEXT, 13) + + assert out == QUESTION_KEPT + assert "Filler mid." not in out + + +def test_user_message_result_at_exact_budget_is_not_truncated() -> None: + """The rebuilt result is returned whole when it lands exactly on budget.""" + assert estimate_tokens(QUESTION_KEPT) == 12 + + assert condense_user_message(QUESTION_TEXT, 12) == QUESTION_KEPT + # One token tighter and the same rebuild is cut to 12 * 3 chars. + assert condense_user_message(QUESTION_TEXT, 11) == QUESTION_KEPT[:33] + + +def test_user_message_truncates_the_rejoined_result_when_over_budget() -> None: + """Question-dense text keeps the rebuild over budget, so it is cut.""" + text = "Intro. " + "Why? " * 40 + "Ending." + + out = condense_user_message(text, 10) + + assert out == "Intro. Why? Why? Why? Why? Why" + assert estimate_tokens(out) <= 10 + + +# ── condense_timeline_event ─────────────────────────────────────────── + +_TIMELINE_FILLER = "Then filler. " * 60 +ISO_EVENT = "On 2026-07-14 the reranker was disabled. " + _TIMELINE_FILLER +ISO_COMPRESSED = "[2026-07-14] On 2026-07-14 the reranker was disabled." + + +def test_timeline_event_at_exact_budget_is_returned_verbatim() -> None: + text = "Short. Text." # 12 chars -> 4 tokens + assert estimate_tokens(text) == 4 + + assert condense_timeline_event(text, 4) == text + assert condense_timeline_event(text, 3) == "Short." + + +def test_timeline_event_reads_the_bracketed_date_marker() -> None: + """The ``[Date: ...]`` slot is matched case-sensitively, group 1 first.""" + text = ( + "[Date: yesterday afternoon] The store was migrated. " + + "Filler words here. " * 60 + ) + + assert condense_timeline_event(text, 40) == ( + "[yesterday afternoon] [Date: yesterday afternoon] The store was migrated." + ) + + +def test_timeline_event_reads_an_iso_date() -> None: + """A bare ``YYYY-MM-DD`` is the second alternative, read from group 2.""" + assert condense_timeline_event(ISO_EVENT, 40) == ISO_COMPRESSED + + +def test_timeline_event_reads_a_month_day_year_date() -> None: + """``Month D, YYYY`` is the third alternative, read from group 3. + + Groups 1 and 2 are both None here, so this is the only shape that + forces the third ``or`` arm to be evaluated at all. + """ + text = "Shipped March 4, 2026 to production. " + _TIMELINE_FILLER + + assert condense_timeline_event(text, 40) == ( + "[March 4, 2026] Shipped March 4, 2026 to production." + ) + + +def test_timeline_event_compressed_at_exact_budget_is_not_truncated() -> None: + assert estimate_tokens(ISO_COMPRESSED) == 17 + + assert condense_timeline_event(ISO_EVENT, 17) == ISO_COMPRESSED + + +def test_timeline_event_truncates_the_compressed_form_not_the_raw_text() -> None: + """Over budget, the slot-filled string is what gets cut — not ``text``.""" + out = condense_timeline_event(ISO_EVENT, 16) + + assert out == ISO_COMPRESSED[:48] + + +# ── helpers ─────────────────────────────────────────────────────────── + + +def test_first_sentence_returns_the_first_sentence_not_the_whole_text() -> None: + assert _first_sentence("First. Second. Third.") == "First." + # No terminator anywhere: the whole text IS the first sentence. + assert _first_sentence("no terminator here") == "no terminator here" + assert _first_sentence("") == "" + + +def test_split_sentences_strips_and_drops_empty_parts() -> None: + """Leading/trailing whitespace is stripped before splitting.""" + assert _split_sentences(" One. Two. ") == ["One.", "Two."] + assert _split_sentences("") == []