From 2ac63ccd4d9fd76cb31d72886e98eb2c35a448a3 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 05:02:14 +0530 Subject: [PATCH 01/83] docs(tally): four findings left open when #306 was merged too early MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit These arrived on #306 after it was merged. Two are corrections to text I wrote, and one is the same failure in two places: a qualification stated in prose while the thing beside it does the unqualified thing anyway. **The case clause was symmetric while the note admitted it was not.** `ascii_lower(candidate) == ascii_lower(tally_name)` also accepts an UPPERCASE candidate against a lowercase master, a direction never sent — so the symmetric form asserts a second experiment, exactly as a canonical form does for the separator. The note below it said so and the predicate kept the clause. Now `candidate == ascii_lower(tally_name)`, which is what the capture sent: a lowercase candidate against a master carrying uppercase. All four clauses are directional and each is one measured result. **`PROMPT_PLAYBOOK.md` still mandated NFC normalization for name keys**, and it is the active Phase 2 implementation authority — so the reference could carry the MEASURED row saying normalising is wrong while the document people build from told them to normalise. Withdrawn with a dated deviation citing the measurement: an NFD spelling of a UI-created NFC ledger was rejected with `EXCEPTIONS=1` while the NFC spelling created it, and an NFD create read back with identical codepoints. Case folding and the non-Latin fixtures stay; normalisation goes. **`IMPLEMENTATION_GUIDE.md` §3.4 kept the fingerprint alive as duplicate prevention.** My sentence — "it is still true everywhere §3.3a does not reach" — preserved the withdrawn mandate in every case §3.3a excludes, while §3.4a of the same patch says the tuple cannot distinguish a retry from a legitimate identical payment and so prevents no duplicate anywhere. Two paragraphs of one patch contradicting each other is how a withdrawn mandate comes back. What is true outside §3.3a's reach is that there is no proven mechanism there at all, which is narrower and less comfortable. **`IMPROVEMENT_PLAN_2026H2.md` cited both sections wrongly.** The signal table is `IMPLEMENTATION_GUIDE.md` §3.4a, not §6.2 (Unicode round-trip), and the reference's `REMOTEID` rule is §9.3 — `§3.3a` is a section of the guide, not of the reference, so the citation pointed at a section that does not exist. A supersession note that lands readers away from the decision it claims to identify makes the old mandate harder to disambiguate, not easier. Reference resealed: 1 changed file, pins unchanged at 211. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 16 ++++++++++++---- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 7 +++++-- docs/tally/PROMPT_PLAYBOOK.md | 17 ++++++++++++++--- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 16 +++++++++++----- 4 files changed, 42 insertions(+), 14 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index 2b7ecd74..b8fc9797 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -675,10 +675,18 @@ duplicate. So on that path a crash-retry of the *same file* is safe on its own, that used to stand here — that the fingerprint plus an embedded key is "the only thing" preventing a duplicate — is no longer true where §3.3a applies. -It is still true everywhere §3.3a does not reach, and that is most places: a **different** payload -under the same key is untested (it may overwrite, partially update or duplicate), as is any -non-Journal voucher type, any other SKU, and a retry across a Tally restart or a company boundary. -Name which case you are in before relying on either mechanism. +**And it is not true anywhere else either.** An earlier revision of this paragraph — mine — said it +"is still true everywhere §3.3a does not reach", which quietly kept the fingerprint alive as a +duplicate-prevention mechanism in every case §3.3a excludes. §3.4a establishes the opposite: a +`(date, amount, ledger-set, voucher-type)` tuple **cannot** distinguish a retry from a legitimate +identical payment, so it prevents no duplicate anywhere. Two paragraphs of one patch contradicting +each other is how a withdrawn mandate comes back. + +What is actually true outside §3.3a's reach is narrower and less comfortable: **there is no proven +duplicate-prevention mechanism there at all.** A **different** payload under the same key is +untested (it may overwrite, partially update or duplicate), as is any non-Journal voucher type, any +other SKU, and a retry across a Tally restart or a company boundary. Name which case you are in, +and where it is not §3.3a's, stop and involve a human rather than reaching for the tuple. ### 3.4a Undefined UDF fields are silently discarded — **the plan's primary idempotency key does not work as written** diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index cfa20791..39c9ba50 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -734,8 +734,11 @@ not say so. That is the failure this plan is least able to detect, because nothi no counter is wrong. The correction, with the full table of what each signal does support, is in -[`IMPLEMENTATION_GUIDE.md`](./IMPLEMENTATION_GUIDE.md) §6.2, and -`TALLY_PROTOCOL_REFERENCE.md` §3.3a states the same rule from the other side: +[`IMPLEMENTATION_GUIDE.md`](./IMPLEMENTATION_GUIDE.md) **§3.4a** — §6.2 is the Unicode +round-trip result and carries none of this — and +[`TALLY_PROTOCOL_REFERENCE.md`](./TALLY_PROTOCOL_REFERENCE.md) **§9.3** states the same rule from +the other side. (`§3.3a` is a section of the implementation guide, not of the reference; citing it +under the wrong document sent readers to a section that does not exist.) | Signal | What it supports | | --- | --- | diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 2211d14b..c8515905 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -298,9 +298,20 @@ Implement: fix-it list. 5. Encoding/normalization hardening: UTF-8/UTF-16LE/BOM fixtures; non-English (Devanagari, Gujarati, Tamil) company/ledger/narration - fixtures in the simulator corpus; NFC normalization + case-insensitive - collation for name keys (Tally name uniqueness is effectively - case-insensitive). + fixtures in the simulator corpus; case-insensitive collation for name keys + (Tally name uniqueness is effectively case-insensitive). + + **DEVIATION 2026-09-12 — NFC normalization for name keys is withdrawn.** + This item previously required it. `TALLY_PROTOCOL_REFERENCE.md` §9.4b records + the measurement: a voucher naming a UI-created **NFC** ledger in its + canonically equivalent **NFD** spelling was rejected — `EXCEPTIONS=1`, + `LINEERROR` saying the ledger does not exist — while the NFC spelling created + it, and an NFD create read back with identical NFD codepoints. **Tally matches + and stores exact codepoints.** Normalising before comparing therefore resolves + a name onto a master Tally itself keeps apart, which is a silent misbinding. + Keep the non-Latin fixtures and the case folding; do **not** normalise. It is + the only transformation in that table with evidence pointing the wrong way, + which is why it needs a deviation here rather than a note. 6. Migration: versioned mirror schema evolution for the new fields (voucher lines, bill allocations, inventory lines, tax lines) with rollback notes. diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index 56fed873..2e683e78 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1120,7 +1120,10 @@ accepts(candidate, tally_name): # `tally_name` is the spelling Tally holds. Each line is one measured # result. Do not compose them; do not add a line without a capture. return candidate == tally_name # exact — VERIFIED - or ascii_lower(candidate) == ascii_lower(tally_name) # ASCII case — VERIFIED, see below + or candidate == ascii_lower(tally_name) # candidate is the master lowercased — VERIFIED + # NOT included: ascii_lower(candidate) == ascii_lower(tally_name). + # That also accepts an UPPERCASE candidate against a lowercase master, + # a direction never sent. See the third note below. or drop_one_trailing_space(candidate) == tally_name # ONE trailing space — VERIFIED or candidate == tally_name.replace("-", " ") # space for Tally's hyphen — VERIFIED ``` @@ -1133,10 +1136,13 @@ Three things this spelling is careful about, each of which was wrong in an earli - **The separator substitution is applied to `tally_name` only.** `tally_name="A-B"` accepts `candidate="A B"`; `tally_name="A B"` does **not** accept `candidate="A-B"`. That asymmetry is the entire point of the clause and is what a canonical form cannot express. -- **The case clause folds both sides, and that is broader than the capture.** The measurement sent - a lowercase name against a master carrying uppercase; the reverse was not sent. It is written - symmetrically because "Tally folds ASCII case" is the claim the capture supports, but a consumer - relying on the *uppercase-candidate* direction is relying on an inference. Qualify it before +- **The case clause is directional, because the capture was.** The measurement sent a **lowercase** + candidate against a master carrying uppercase. `ascii_lower(candidate) == ascii_lower(tally_name)` + also accepts an **uppercase** candidate against a lowercase master, which was never sent — so the + symmetric form asserts a second experiment, exactly as a canonical form does for the separator. + An earlier draft admitted that in this note and left the symmetric clause in the predicate + anyway; a qualification in the prose does not qualify the code beside it. Written as + `candidate == ascii_lower(tally_name)`, the predicate now says only what was sent. Qualify it before building on it. If a further direction is later measured, one clause is added and the table row changes. Until From 22990f8bf868e8b5e2dc35985b4a56b277299521 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:16:56 +0530 Subject: [PATCH 02/83] docs(tally): directional case fold, NFC withdrawal reaches every site MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five review findings, docs-only. Defect A (2 findings, P1): PROMPT_PLAYBOOK.md Phase 2 item 5 prescribed a symmetric case-insensitive collation for name keys. §9.4b's accepts(candidate, tally_name) predicate only measured a lowercase candidate against an uppercase master; the reverse was never sent. A symmetric fold accepts that unverified direction and can bind a voucher to the wrong master. Restated as the directional predicate. Defect B (2 findings, P1): the NFC/NFD withdrawal did not reach three other sites that all instructed matching or reading masters back by 'normalized name' without excluding NFC, or flagged NFC normalization only when applied asymmetrically: - PROMPT_PLAYBOOK.md Phase 2 item 5 (paired with the case fix above) - PROMPT_PLAYBOOK.md Phase 4 step 4 (readback verification) - PROMPT_PLAYBOOK.md Phase 2 adversarial review item 4 - IMPROVEMENT_PLAN_2026H2.md §8.6 Each now carries a dated deviation note pointing at TALLY_PROTOCOL_REFERENCE.md §9.4b, in the style already used for the existing 2026-09-11 fingerprint deviation. Over-correction (1 finding, P2): IMPLEMENTATION_GUIDE.md §3.4a claimed no proven duplicate-prevention mechanism exists outside §3.3a's REMOTEID path. TALLY_PROTOCOL_REFERENCE.md §9.8 is VERIFIED that Manual numbering with PREVENTDUPLICATES=Yes cleanly rejects a duplicate (CREATED=0, ALTERED=0, EXCEPTIONS=1) where Automatic numbering silently creates one — a second proven mechanism, already noted in §3.3 of the same file. Narrowed the claim to what's actually unproven (an automatic dedupe decision from the fingerprint tuple) while keeping the correct part: the fingerprint must never auto-suppress. No code changes. TALLY_PROTOCOL_REFERENCE.md (pinned) is untouched; compatibility surface pin count unchanged at 211. Section-number gate scripts pass. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 16 ++++++--- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 4 ++- docs/tally/PROMPT_PLAYBOOK.md | 51 +++++++++++++++++---------- 3 files changed, 47 insertions(+), 24 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index b8fc9797..f6fe5c8c 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -781,11 +781,17 @@ fingerprint is **co-primary** rather than secondary. Withdrawn: promoting it doe to do the job. The tuple is identical for a legitimate recurring or same-day repeat payment, so as an automatic dedupe it suppresses real vouchers no matter which tier it is placed in — see §3.4a. -What follows instead is narrower and less comfortable: **there is no proven automatic -write-confirmation mechanism for Phase 4.** `REMOTEID` upsert covers a byte-identical repeat on the -Journal path (§3.3a) and nothing beyond it; a destroyed narration marker leaves a write -unattributable, and the honest response to that is to stop and ask a human, not to substitute a -signal that cannot tell the two cases apart. +What follows instead is narrower and less comfortable: **there is no proven mechanism that lets an +automatic dedupe *decision* be made from the fingerprint tuple.** That is not the same claim as "no +proven duplicate-prevention mechanism outside §3.3a" — it overstates the gap. Two mechanisms are +proven, each with a narrow scope: `REMOTEID` upsert on a byte-identical repeat on the Journal path +(§3.3a), and Manual numbering with `PREVENTDUPLICATES=Yes`, which cleanly rejects a duplicate +voucher number instead of silently creating one — `CREATED=0, ALTERED=0, EXCEPTIONS=1` (§3.3; +[`TALLY_PROTOCOL_REFERENCE.md` §9.8](TALLY_PROTOCOL_REFERENCE.md#98-voucher-numbering-method-changes-everything--use-manual)). +Neither reaches a destroyed narration marker or a differently-numbered duplicate under automatic +numbering; for those cases the honest response is still to stop and ask a human, not to substitute +the fingerprint as an automatic suppressor — it cannot tell a retry from a legitimate second +payment no matter which carrier is missing. ### 3.5 Identity after write diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index 39c9ba50..cd32c9cc 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -413,7 +413,9 @@ Re-sending the identical ledger `ACTION="Create"` returned `CREATED=0, ALTERED=1 ### 8.6 `LASTMID` is 0 on successful master creates; `LASTVCHID` works -Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by normalized name. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. +Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name, matched via `TALLY_PROTOCOL_REFERENCE.md` §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. + +DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints plus only the directional case fold. ### 8.7 AlterID high-water marks move — Drift Sentinel's mechanism is sound diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index c8515905..7d88e065 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -298,20 +298,20 @@ Implement: fix-it list. 5. Encoding/normalization hardening: UTF-8/UTF-16LE/BOM fixtures; non-English (Devanagari, Gujarati, Tamil) company/ledger/narration - fixtures in the simulator corpus; case-insensitive collation for name keys - (Tally name uniqueness is effectively case-insensitive). - - **DEVIATION 2026-09-12 — NFC normalization for name keys is withdrawn.** - This item previously required it. `TALLY_PROTOCOL_REFERENCE.md` §9.4b records - the measurement: a voucher naming a UI-created **NFC** ledger in its - canonically equivalent **NFD** spelling was rejected — `EXCEPTIONS=1`, - `LINEERROR` saying the ledger does not exist — while the NFC spelling created - it, and an NFD create read back with identical NFD codepoints. **Tally matches - and stores exact codepoints.** Normalising before comparing therefore resolves - a name onto a master Tally itself keeps apart, which is a silent misbinding. - Keep the non-Latin fixtures and the case folding; do **not** normalise. It is - the only transformation in that table with evidence pointing the wrong way, - which is why it needs a deviation here rather than a note. + fixtures in the simulator corpus; name-key matching via §9.4b's + `accepts(candidate, tally_name)` predicate — directional ASCII case + folding only (`candidate == ascii_lower(tally_name)`; the reverse, an + uppercase candidate against a lowercase master, was never measured and + must not be accepted). Never a symmetric case-insensitive collation: + that accepts the unverified direction and can bind a voucher to the + wrong master. + DEVIATION 2026-09-12 (TALLY_PROTOCOL_REFERENCE.md §9.4b): NFC + normalization of name keys is WITHDRAWN. §9.4b is MEASURED, not + inferred: an NFD spelling of a UI-created NFC ledger was rejected + (`EXCEPTIONS=1`, ledger does not exist) while the NFC spelling created + it — Tally matches on exact codepoints. Normalizing before comparing + resolves a name onto a master Tally itself keeps apart. Name keys + compare on exact codepoints; do not NFC/NFD-normalize either side. 6. Migration: versioned mirror schema evolution for the new fields (voucher lines, bill allocations, inventory lines, tax lines) with rollback notes. @@ -356,9 +356,16 @@ Hunt specifically for: 3. Amount fidelity: any new tax/inventory line parsed through anything but ExactDecimal; sign conventions (IsDeemedPositive) mishandled on new line types; Dr/Cr balance invariant not re-checked with lines present. -4. Identity/normalization traps: NFC normalization applied on read but not - on the keys used for diffing (same ledger counted twice); case-collation - asymmetry between mirror and reconciliation. +4. Identity/normalization traps: ANY NFC/NFD normalization of name keys, + anywhere in the read or diff path — applied consistently on both reads + and diff keys is still a confirmed finding, not only when applied + asymmetrically (§9.4b: Tally matches exact codepoints; normalizing + resolves a name onto a master Tally itself keeps apart, whether or not + both sides agree). A symmetric case-insensitive collation is likewise + a finding: §9.4b's `accepts(candidate, tally_name)` folds ASCII case + in one direction only (candidate lowered against an uppercase master); + a fold that also accepts an uppercase candidate against a lowercase + master accepts the unverified direction. 5. Bounded-resource regressions: new list explosions (AllInventoryEntries on huge vouchers) versus the 32 MiB response cap — is there a paging or windowing story? Does a capped response get honestly labeled Partial? @@ -574,7 +581,9 @@ Implement — write core (masters): 3. Single-writer actor owns the import surface; reads gated during dispatch→readback windows; queue depth visible. 4. Readback verification: after counters accept, re-export the object - (masters by normalized name; vouchers by LASTVCHID) and + (masters matched by name via §9.4b's `accepts(candidate, tally_name)` + predicate only — directional ASCII case folding; vouchers by + LASTVCHID) and ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign @@ -582,6 +591,12 @@ Implement — write core (masters): else OUTCOME_UNKNOWN. Persist the BridgeID ↔ GUID/MasterID binding. Field-diff readback vs intent; divergence → CONFIRMED_WITH_DIVERGENCE, surfaced in the Gap Map, never silent. + DEVIATION 2026-09-12 (TALLY_PROTOCOL_REFERENCE.md §9.4b): "matched by + name" never means NFC/NFD-normalized. An NFD create read back against + a pre-existing NFC master would resolve as a match and promote the + wrong object to CONFIRMED — §9.4b measured Tally keeping the two + apart. Compare master names on exact codepoints plus only the + directional ASCII-case fold; never normalize either side first. 5. OutcomeUnknown recovery: on restart, DISPATCHING rows → probe by key + fingerprint. A probe MATCH is not itself a confirmation: run the SAME full field-level readback diff as the normal dispatch path (step 4) and From e8d2bad521392b7a37b0b88b175c83d7821c7ff6 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:31:39 +0530 Subject: [PATCH 03/83] docs(tally): scope the name fold to its baseline, keep the one proven dedupe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings left open on #314, each checked against the tree before being accepted. **The name fold is prescribed outside the SKU it was measured on.** Phase 4's readback step told implementers to match masters with §9.4b's `accepts()` predicate. §9.4b's case-folding and hyphen-for-space rows carry no licensed-SKU qualification, so they inherit §0's Edit Log 7.0 Educational baseline; the only row measured on a licensed 7.1 instance is NFC/NFD, and it points the other way — exact codepoints. Phase 4 runs against licensed TallyPrime. Applying an Educational-baseline fold there can bind a write to an account Tally keeps distinct. Phase 4 now matches on exact codepoints and widens only where a licensed capture or the connected instance's compatibility result qualifies the predicate. **The fingerprint correction stopped at the plan.** IMPROVEMENT_PLAN was repointed to §3.4a but Phase 4 item 6 still sent implementers to IMPLEMENTATION_GUIDE §6.2 — the Unicode round-trip result, which says nothing about fingerprints — and therefore away from the signal table and its automatic-suppression constraints. **Withdrawing the tuple withdrew a mechanism it had no claim over.** §3.4's "there is no proven duplicate-prevention mechanism there at all" is absolute, and false: §9.8 is VERIFIED that Manual numbering with `PREVENTDUPLICATES=Yes` preserves the supplied number and rejects a duplicate cleanly (`CREATED=0, ALTERED=0, EXCEPTIONS=1`). The claim is now scoped to what lies outside both §3.3a's REMOTEID path and §9.8's, with a sentence saying explicitly that §9.8 supplies a different mechanism rather than rehabilitating the fingerprint — the tuple stays withdrawn everywhere. Surface resealed (rehash → seal → repoint) after the rebase onto #289; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 16 +++++++++++++--- docs/tally/PROMPT_PLAYBOOK.md | 14 +++++++++++++- 2 files changed, 26 insertions(+), 4 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index f6fe5c8c..2c4a041a 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -682,11 +682,21 @@ duplicate-prevention mechanism in every case §3.3a excludes. §3.4a establishes identical payment, so it prevents no duplicate anywhere. Two paragraphs of one patch contradicting each other is how a withdrawn mandate comes back. -What is actually true outside §3.3a's reach is narrower and less comfortable: **there is no proven -duplicate-prevention mechanism there at all.** A **different** payload under the same key is +What is actually true outside §3.3a's reach is narrower and less comfortable — with **one** +qualified exception. `TALLY_PROTOCOL_REFERENCE.md` §9.8 is VERIFIED: a voucher type set to +**Manual** numbering with `PREVENTDUPLICATES=Yes` preserves the supplied number verbatim and +rejects a duplicate **cleanly** (`CREATED=0, ALTERED=0, EXCEPTIONS=1`). A flow that carries +voucher-number identity under that configuration therefore *does* have a proven mechanism, and this +paragraph must not discard it. + +Outside **both** §3.3a's `REMOTEID` path and §9.8's Manual + `PREVENTDUPLICATES=Yes` path, **there +is no proven duplicate-prevention mechanism at all.** A **different** payload under the same key is untested (it may overwrite, partially update or duplicate), as is any non-Journal voucher type, any other SKU, and a retry across a Tally restart or a company boundary. Name which case you are in, -and where it is not §3.3a's, stop and involve a human rather than reaching for the tuple. +and where it is neither of those two, stop and involve a human rather than reaching for the tuple. + +The tuple is withdrawn in every case. §9.8 does not rehabilitate the fingerprint — it supplies a +*different* mechanism, one that rejects at the gateway instead of guessing at the caller. ### 3.4a Undefined UDF fields are silently discarded — **the plan's primary idempotency key does not work as written** diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 7d88e065..eee46411 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -584,6 +584,16 @@ Implement — write core (masters): (masters matched by name via §9.4b's `accepts(candidate, tally_name)` predicate only — directional ASCII case folding; vouchers by LASTVCHID) and + SCOPE GATE (§9.4b, §0): §9.4b's case-folding and hyphen-for-space rows + were measured on the **Edit Log 7.0 Educational** baseline and carry no + licensed-SKU qualification; only the NFC/NFD row was measured on a + licensed 7.1 instance, and it points the other way (exact codepoints). + Phase 4 runs against licensed TallyPrime, so on a licensed SKU match + master names on **exact codepoints** and let a case or separator + difference fail loudly. Widen to `accepts()` only where a licensed + capture has qualified the predicate, or where the compatibility result + for the connected instance says it holds. A fold applied on an + unqualified SKU can bind a write to an account Tally keeps distinct. ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign @@ -625,7 +635,9 @@ Implement — voucher writes (after masters CONFIRMED-path is soak-tested): tuple cannot tell a retry from a legitimate second payment — a standing instruction, or two invoices settled to one supplier on one day, produce the identical tuple — so automatic suppression is a - silent under-write. See IMPLEMENTATION_GUIDE.md §6.2. + silent under-write. See IMPLEMENTATION_GUIDE.md §3.4a — the signal + table and its automatic-suppression constraints. (§6.2 is the Unicode + round-trip result and says nothing about fingerprints.) 7. Cancel qualified as the compensation primitive (ACTION=Cancel by REMOTEID/GUID). Alter-by-GUID qualified per version; where flaky, the fallback is a Cancel+Create saga bound in one outbox transaction with From 0f3710d189abd9e4a48c7288149ea44d1cc93ea4 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:49:23 +0530 Subject: [PATCH 04/83] docs(tally): one name-matching rule per step, and let it be the gated one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scope gate added in `40dee282` was inserted *between* two directives that still prescribed the fold unconditionally, so Phase 4 step 4 stated its name-matching rule three times: unqualified before the gate, gated, and unqualified again in the closing deviation. An implementer reading to the end of the step finds the fold reinstated. Codex flagged the third; the first was the same defect and is fixed with it. Step 4 now names the gate as the only matching rule in the step, and the NFC deviation says explicitly that it *removes* normalization without widening what remains — on an unqualified licensed SKU, exact codepoints and nothing else. Phase 2 item 5 carried the same unqualified prescription and was not flagged. Fixed anyway: Phase 2 reads a licensed instance too, and a read is the worse place to be wrong, because the mirror rows built there are what later binding decisions resolve against — a fold that merges two masters on read merges them everywhere downstream. The lesson is the doc-level form of "a fix that adds a branch must assert on the new branch": a qualifier inserted between two unqualified statements qualifies neither. Add the gate, then audit every sibling directive in the same step for whether it inherits it. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index eee46411..e5803e3a 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -305,6 +305,14 @@ Implement: must not be accepted). Never a symmetric case-insensitive collation: that accepts the unverified direction and can bind a voucher to the wrong master. + And the same SCOPE GATE that governs Phase 4 step 4 governs here: + §9.4b's case and separator rows sit on §0's **Edit Log 7.0 + Educational** baseline. Where this phase reads a licensed instance, + match on exact codepoints; widen to `accepts()` only where a licensed + capture or the connected instance's compatibility result qualifies it. + A read is not a safe place to be wrong about this — the mirror rows + built here are what later binding decisions resolve against, so a fold + that merges two masters here merges them everywhere downstream. DEVIATION 2026-09-12 (TALLY_PROTOCOL_REFERENCE.md §9.4b): NFC normalization of name keys is WITHDRAWN. §9.4b is MEASURED, not inferred: an NFD spelling of a UI-created NFC ledger was rejected @@ -581,10 +589,11 @@ Implement — write core (masters): 3. Single-writer actor owns the import surface; reads gated during dispatch→readback windows; queue depth visible. 4. Readback verification: after counters accept, re-export the object - (masters matched by name via §9.4b's `accepts(candidate, tally_name)` - predicate only — directional ASCII case folding; vouchers by + (masters matched by name under the SCOPE GATE below — never by a + broader rule stated anywhere else in this step; vouchers by LASTVCHID) and - SCOPE GATE (§9.4b, §0): §9.4b's case-folding and hyphen-for-space rows + SCOPE GATE (§9.4b, §0) — THE ONLY NAME-MATCHING RULE IN THIS STEP: + §9.4b's case-folding and hyphen-for-space rows were measured on the **Edit Log 7.0 Educational** baseline and carry no licensed-SKU qualification; only the NFC/NFD row was measured on a licensed 7.1 instance, and it points the other way (exact codepoints). @@ -605,8 +614,11 @@ Implement — write core (masters): name" never means NFC/NFD-normalized. An NFD create read back against a pre-existing NFC master would resolve as a match and promote the wrong object to CONFIRMED — §9.4b measured Tally keeping the two - apart. Compare master names on exact codepoints plus only the - directional ASCII-case fold; never normalize either side first. + apart. Never normalize either side first. This deviation removes + normalization; it does not widen what remains — whether any case or + separator fold is permitted on top of exact codepoints is decided by + the SCOPE GATE above and by nothing in this sentence. On an + unqualified licensed SKU that leaves exact codepoints and nothing else. 5. OutcomeUnknown recovery: on restart, DISPATCHING rows → probe by key + fingerprint. A probe MATCH is not itself a confirmation: run the SAME full field-level readback diff as the normal dispatch path (step 4) and From 3ba9a43b19be65fa47e7ca7f448aaac02ff4ff6c Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:28:44 +0530 Subject: [PATCH 05/83] docs(tally): narrow the 9.8 exception to the case 9.8 actually measured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit withdrew an over-broad claim and replaced it with another one. It said Manual numbering with `PREVENTDUPLICATES=Yes` is "a proven duplicate-prevention mechanism". §9.8 measured one thing: how a **failed `Alter`** behaves under that setting. Its own rule says *"Do not apply the failed-`Alter` observation to a different request identity mechanism"*, and a crash retry — the case §3.4 was discussing — sends a `Create`. It overreaches on SKU as well. §9.8 carries no licensed qualification for the numbering path; the scope clarification beneath it covers a licensed *Journal* `ACTION="Create"` repeat carrying `REMOTEID` and says in terms that it does not establish voucher-number identity, the configured numbering method, or other request shapes. Both sites now state the result as what it is — for a failed `Alter` on §9.8's baseline, Manual numbering converts a silent duplicate into a clean rejection — with request shape, SKU and voucher type named as the three limits. The tuple stays withdrawn everywhere; §9.8 reports how one failure mode behaves under one setting, which is a different kind of thing from a mechanism. A paragraph is added saying why this needed two corrections: **a claim is not made safe by being a correction.** Withdrawing one over-broad statement is the moment you are most likely to write another, because the writing feels like narrowing while the sentence is widening something else. `IMPROVEMENT_PLAN_2026H2.md` §8.6 carried the same defect twice over: it prescribed the directional fold with no scope gate, and its NFC deviation then reinstated the fold unconditionally — the identical shape fixed in the playbook last commit, in a file the playbook's own fix did not reach. §8.6 now defers to the playbook's SCOPE GATE and its deviation says explicitly that removing normalization does not decide whether any fold is permitted on top. Audited rather than assumed: all three `accepts()` prescription sites across the tally docs now carry a gate or an exact-codepoint instruction within eight lines. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 58 ++++++++++++++++++--------- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 4 +- 2 files changed, 42 insertions(+), 20 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index 2c4a041a..718f6a9d 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -683,20 +683,39 @@ identical payment, so it prevents no duplicate anywhere. Two paragraphs of one p each other is how a withdrawn mandate comes back. What is actually true outside §3.3a's reach is narrower and less comfortable — with **one** -qualified exception. `TALLY_PROTOCOL_REFERENCE.md` §9.8 is VERIFIED: a voucher type set to -**Manual** numbering with `PREVENTDUPLICATES=Yes` preserves the supplied number verbatim and -rejects a duplicate **cleanly** (`CREATED=0, ALTERED=0, EXCEPTIONS=1`). A flow that carries -voucher-number identity under that configuration therefore *does* have a proven mechanism, and this -paragraph must not discard it. - -Outside **both** §3.3a's `REMOTEID` path and §9.8's Manual + `PREVENTDUPLICATES=Yes` path, **there -is no proven duplicate-prevention mechanism at all.** A **different** payload under the same key is -untested (it may overwrite, partially update or duplicate), as is any non-Journal voucher type, any -other SKU, and a retry across a Tally restart or a company boundary. Name which case you are in, -and where it is neither of those two, stop and involve a human rather than reaching for the tuple. - -The tuple is withdrawn in every case. §9.8 does not rehabilitate the fingerprint — it supplies a -*different* mechanism, one that rejects at the gateway instead of guessing at the caller. +narrowly qualified exception, and the qualification is tighter than the first correction made it +look. `TALLY_PROTOCOL_REFERENCE.md` §9.8 measured **one thing**: how a **failed `Alter`** behaves +under Manual numbering with `PREVENTDUPLICATES=Yes`. It was cleanly rejected — `CREATED=0, +ALTERED=0, EXCEPTIONS=1` — where automatic numbering silently duplicated. That is the whole result. + +Three limits come with it, and §9.8 states two of them itself: + +- **Request shape.** The observation is about a failed `Alter`. §9.8's own rule says *"Do not apply + the failed-`Alter` observation to a different request identity mechanism."* A crash retry sends a + `Create`, which is a different request shape and is **UNVERIFIED**. +- **SKU.** §9.8 carries no licensed qualification for the numbering path. Its later scope + clarification covers a licensed *Journal* `ACTION="Create"` repeat carrying `REMOTEID` and says + in terms that it does **not** establish voucher-number identity, the configured numbering method, + or other request shapes. +- **Voucher type.** Journal only, as everywhere else in this section. + +So the honest statement is: **for a failed `Alter` on the measured baseline, Manual numbering +converts a silent duplicate into a clean rejection.** It is not a general duplicate-prevention +mechanism, and a `Create` retry is not covered by it. + +Outside §3.3a's `REMOTEID` path and outside that one measured case, **there is no proven +duplicate-prevention mechanism at all.** A **different** payload under the same key is untested (it +may overwrite, partially update or duplicate), as is any non-Journal voucher type, any other SKU, +and a retry across a Tally restart or a company boundary. Name which case you are in, and where it +is neither, stop and involve a human rather than reaching for the tuple. + +The tuple is withdrawn in every case. §9.8 does not rehabilitate the fingerprint — it reports how +one failure mode behaves under one setting, which is a different kind of thing entirely. + +**Why this needed two corrections.** The first revision withdrew an over-broad claim ("no proven +mechanism anywhere") and replaced it with another one ("Manual + `PREVENTDUPLICATES` is a proven +mechanism"), widening §9.8 past both its request shape and its SKU in the act of narrowing +something else. A claim is not made safe by being a correction. ### 3.4a Undefined UDF fields are silently discarded — **the plan's primary idempotency key does not work as written** @@ -793,11 +812,14 @@ an automatic dedupe it suppresses real vouchers no matter which tier it is place What follows instead is narrower and less comfortable: **there is no proven mechanism that lets an automatic dedupe *decision* be made from the fingerprint tuple.** That is not the same claim as "no -proven duplicate-prevention mechanism outside §3.3a" — it overstates the gap. Two mechanisms are -proven, each with a narrow scope: `REMOTEID` upsert on a byte-identical repeat on the Journal path -(§3.3a), and Manual numbering with `PREVENTDUPLICATES=Yes`, which cleanly rejects a duplicate -voucher number instead of silently creating one — `CREATED=0, ALTERED=0, EXCEPTIONS=1` (§3.3; +proven duplicate-prevention mechanism outside §3.3a" — it overstates the gap, but only just. One +mechanism is proven: `REMOTEID` upsert on a byte-identical repeat on the Journal path (§3.3a). One +narrower observation sits beside it: under Manual numbering with `PREVENTDUPLICATES=Yes`, a +**failed `Alter`** is cleanly rejected rather than silently duplicated — `CREATED=0, ALTERED=0, +EXCEPTIONS=1` (§3.3; [`TALLY_PROTOCOL_REFERENCE.md` §9.8](TALLY_PROTOCOL_REFERENCE.md#98-voucher-numbering-method-changes-everything--use-manual)). +That is a failed-`Alter` result on §9.8's own baseline, not a general rejection mechanism: §9.8 +forbids carrying it to a different request identity mechanism, and a crash retry sends a `Create`. Neither reaches a destroyed narration marker or a differently-numbered duplicate under automatic numbering; for those cases the honest response is still to stop and ask a human, not to substitute the fingerprint as an automatic suppressor — it cannot tell a retry from a legitimate second diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index cd32c9cc..e6c43435 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -413,9 +413,9 @@ Re-sending the identical ledger `ACTION="Create"` returned `CREATED=0, ALTERED=1 ### 8.6 `LASTMID` is 0 on successful master creates; `LASTVCHID` works -Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name, matched via `TALLY_PROTOCOL_REFERENCE.md` §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. +Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen to §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only — where a licensed capture has qualified it, or where the connected instance's compatibility result says it holds. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. -DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints plus only the directional case fold. +DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints. This deviation removes normalization; it does not decide whether any case or separator fold is permitted on top — that is the SCOPE GATE's question, answered above, and on an unqualified licensed SKU the answer is exact codepoints and nothing else. ### 8.7 AlterID high-water marks move — Drift Sentinel's mechanism is sound From 96fce0f1bebe319b47eb758b3a07816ff0178151 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:36:43 +0530 Subject: [PATCH 06/83] docs(tally): withdraw a licence claim I inferred rather than measured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **The NFC/NFD capture is not licensed evidence.** The scope gate added two commits ago said "only the NFC/NFD row was measured on a licensed 7.1 instance". It was not. `TALLY_PROTOCOL_REFERENCE.md` records that capture as "2026-08-19, TallyPrime 7.1, port 9001" and says nothing about licence mode; `tests/fixtures/encoding/PROVENANCE.md` records that day's instance as **EDU**. I supplied "licensed" from a stored note about what port 9001 usually runs, not from the document, and then used it as the premise of an argument about which evidence is qualified. Correcting it makes the gate **stricter**, not weaker: with that row withdrawn, **no** row of §9.4b is qualified on a licensed SKU, so there is no licensed evidence to widen towards at all. The gate's instruction is unchanged — exact codepoints — and now rests on something the tree actually says. **Two identity authorities still said "normalised name".** The withdrawal was carried to four sites and I reported it complete; `IMPLEMENTATION_GUIDE.md` §3.5 and `TALLY_PROTOCOL_REFERENCE.md`'s `LASTMID` entry were not among them, and both are exactly where an implementer looks for the readback rule — so an implementer following the identity guidance rather than the deviation could still resolve an NFD create onto a distinct pre-existing NFC master. Both now say normalised never means NFC/NFD-normalised and defer to the scope gate. Audited rather than counted this time: no `by normalised name` or `by normalized name` directive remains anywhere in the tally docs. The lesson is one already written down and not applied here: a recalled fact is not a measured one. A note saying what an instance usually is cannot qualify a specific capture — only that capture's provenance can. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 10 +++++++--- docs/tally/PROMPT_PLAYBOOK.md | 8 ++++++-- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 2 +- 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index 718f6a9d..eae66b0d 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -827,9 +827,13 @@ payment no matter which carrier is missing. ### 3.5 Identity after write -`LASTMID` is **0** on successful master creates — unusable. Read masters back by normalised -name. `LASTVCHID` is populated for vouchers and usable, subject to a foreign-writer -cross-check. It also accepts non-numeric text without error when parsed back, so validate it. +`LASTMID` is **0** on successful master creates — unusable. Read masters back by name — and +**normalised never means NFC/NFD-normalised**: §9.4b measured Tally matching on exact +codepoints, so normalising before comparing resolves an NFD create onto a distinct +pre-existing NFC master and promotes the wrong object. Which name rule applies is the +SCOPE GATE's question (`PROMPT_PLAYBOOK.md` Phase 4 step 4); on an unqualified licensed SKU +it is exact codepoints and nothing else. `LASTVCHID` is populated for vouchers and usable, +subject to a foreign-writer cross-check. It also accepts non-numeric text without error when parsed back, so validate it. ### 3.6 Master re-create is a silent Alter diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index e5803e3a..cd4de901 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -595,8 +595,12 @@ Implement — write core (masters): SCOPE GATE (§9.4b, §0) — THE ONLY NAME-MATCHING RULE IN THIS STEP: §9.4b's case-folding and hyphen-for-space rows were measured on the **Edit Log 7.0 Educational** baseline and carry no - licensed-SKU qualification; only the NFC/NFD row was measured on a - licensed 7.1 instance, and it points the other way (exact codepoints). + licensed-SKU qualification. NOR DOES THE NFC/NFD ROW: an earlier + revision of this gate called that capture licensed, and it is not — + `tests/fixtures/encoding/PROVENANCE.md` records the 2026-08-19 + instance behind it as **EDU**. Correcting that makes this gate + stricter, not weaker: **no** row of §9.4b is qualified on a licensed + SKU, so there is no licensed evidence to widen towards. Phase 4 runs against licensed TallyPrime, so on a licensed SKU match master names on **exact codepoints** and let a case or separator difference fail loudly. Widen to `accepts()` only where a licensed diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index 2e683e78..11c2102f 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1398,7 +1398,7 @@ habits, not against real operator input. ### 9.5 Identity after write **VERIFIED.** `LASTMID` is **0** on successful master creates — unusable for master identity; -read masters back by normalised name. `LASTVCHID` is populated for vouchers and usable, +read masters back by name — and **normalised never means NFC/NFD-normalised**: §9.4b measured Tally matching on exact codepoints, so normalising before comparing resolves a name onto a master Tally itself keeps apart. Match under the scope gate in `PROMPT_PLAYBOOK.md` Phase 4 step 4, which on an unqualified licensed SKU is exact codepoints and nothing else. `LASTVCHID` is populated for vouchers and usable, subject to a foreign-writer cross-check. `LASTVCHID` also accepts non-numeric text without error when parsed back, so validate it. From 32f56139ec572f0c53d7474ab25b74f2af734ba3 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:39:24 +0530 Subject: [PATCH 07/83] docs(tally): exact-only is the right rule for binding and the wrong one for creating MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scope gate said "match on exact codepoints" and stopped there, as though strictness were automatically safe. It is not, and the failure is in the opposite direction from the one the gate was written to prevent. If the connected SKU **does** share the Educational case behaviour, an existing `FOO` reads as ABSENT for a requested `foo` under exact matching. A step that creates whatever it finds missing then writes a SECOND master differing only by case — a duplicate in a client's book, produced by the conservative rule. The loose rule misbinds; the strict rule duplicates. Neither is safe, and the gate offered no third option. It has three outcomes now: **bind** on an exact match; **create** only when no master differs from the requested name by case or separator alone; otherwise **refuse and raise it for a human**. A near-collision on an unqualified SKU is exactly where neither automatic answer is defensible. The near-collision set is computed with §9.4b's fold used as a **detector**, never as a binder — and that distinction is what makes it legitimate on an unqualified SKU. Folding to conclude "something similar exists, stop" needs no licensed qualification, because the conclusion is a refusal rather than a write. Unverified evidence can justify caution; it cannot justify action. Separately, `TALLY_PROTOCOL_REFERENCE.md`'s crash-retry paragraph still called Manual + `PREVENTDUPLICATES=Yes` a qualified duplicate-rejection mechanism for a repeated number, which is the generalisation corrected in the guide two commits ago — the correction had not reached the behavioural authority, which is the document an implementer trusts most. It now names the failed-`Alter` scope, the UNVERIFIED `Create` retry and the UNVERIFIED licensed SKU, and says plainly not to read it as promising a crash retry is safe. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 15 +++++++++++++++ docs/tally/TALLY_PROTOCOL_REFERENCE.md | 9 +++++++-- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index cd4de901..d6e38646 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -607,6 +607,21 @@ Implement — write core (masters): capture has qualified the predicate, or where the compatibility result for the connected instance says it holds. A fold applied on an unqualified SKU can bind a write to an account Tally keeps distinct. + AND EXACT-ONLY MATCHING HAS ITS OWN FAILURE, WHICH IS NOT "SAFE": + if the connected SKU *does* share the Educational behaviour, an + existing `FOO` reads as ABSENT for a requested `foo`, and a step that + creates what it finds missing then makes a SECOND master differing + only by case — a duplicate in the client's book rather than a + misbinding. Exact-only is the right rule for BINDING and the wrong + rule for CREATING, so the gate has three outcomes, not two: + **bind** on an exact match; **create** only when no master differs + from the requested name by case or separator alone; otherwise + **REFUSE and raise it for a human** — a near-collision on an + unqualified SKU is precisely the case where neither automatic answer + is defensible. Compute the near-collision set with the §9.4b fold + used only as a *detector*, never as a binder: folding to decide + "something similar exists, stop" needs no licensed qualification, + because the conclusion is a refusal rather than a write. ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index 11c2102f..cefcebf4 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -867,8 +867,13 @@ Read that precisely, because the obvious paraphrase — "Tally does not dedupe o number" — is false in two directions. Under **automatic** numbering the supplied number is *discarded* (§9.8), so the two sends never shared a stored voucher number and nothing could have deduped on it. Under **Manual + `PREVENTDUPLICATES=Yes`**, §9.8 records that a repeated number is -**cleanly rejected** — a qualified duplicate-rejection mechanism that a reader of this sentence -would otherwise never look for. +**cleanly rejected** — a qualified rejection that a reader of this sentence would otherwise never +look for. Qualified narrowly, though: §9.8 measured a **failed `Alter`**, and its own rule forbids +carrying that observation to a different request identity mechanism. A crash retry sends a +`Create`, which is **UNVERIFIED** here, as is the behaviour on any licensed SKU — §9.8's scope +clarification covers a licensed Journal `REMOTEID` repeat and says in terms that it establishes +neither voucher-number identity nor the configured numbering method. Do not read this sentence as +promising a crash-retry is safe under Manual numbering. So: on the numbering method measured here, a crash-retry duplicates client data unless the integrator prevents it. From 3f7e43290096a003c6a3a5e79a03ba4fbf1418b2 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:49:30 +0530 Subject: [PATCH 08/83] docs(tally): a detector must be wider than a binder, not the same rule reversed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The near-collision guard added last commit reused §9.4b's `accepts()` as its detector, and `accepts()` is **directional**. For a requested `FOO` against an existing `foo`, `accepts(FOO, foo)` is false — so the guard misses precisely the collision it exists to catch, and Phase 4 goes on to create the duplicate master. The hazard survived inside its own fix. The detector now folds symmetrically and deliberately over-wide: case-insensitive both ways, hyphen and space interchangeable both ways, leading and trailing whitespace ignored, internal whitespace runs collapsed. Several of those rows are UNVERIFIED as *matching* behaviour, and that is the point rather than an oversight. **An unverified equivalence cannot justify a write, but it is ample reason to stop and ask.** The asymmetry in consequences sets the direction: a detector that misses a collision creates a duplicate in a client's book, while a detector that over-fires costs one question to a human. Fail toward the question. That generalises past this guard — wherever the same fold serves two purposes, the safe width differs by purpose, and reusing one predicate for both silently adopts the wrong width for one of them. Also: the licence correction cited `tests/fixtures/encoding/PROVENANCE.md`, which exists at no such path. The evidence is at `src-tauri/crates/bridge-tally-protocol/tests/fixtures/encoding/PROVENANCE.md`, verified present. I took the short path from the review comment that raised the finding instead of resolving it — in a commit whose whole subject was citing provenance accurately. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index d6e38646..7feb4348 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -597,8 +597,8 @@ Implement — write core (masters): were measured on the **Edit Log 7.0 Educational** baseline and carry no licensed-SKU qualification. NOR DOES THE NFC/NFD ROW: an earlier revision of this gate called that capture licensed, and it is not — - `tests/fixtures/encoding/PROVENANCE.md` records the 2026-08-19 - instance behind it as **EDU**. Correcting that makes this gate + `src-tauri/crates/bridge-tally-protocol/tests/fixtures/encoding/` + `PROVENANCE.md` records the 2026-08-19 instance behind it as **EDU**. Correcting that makes this gate stricter, not weaker: **no** row of §9.4b is qualified on a licensed SKU, so there is no licensed evidence to widen towards. Phase 4 runs against licensed TallyPrime, so on a licensed SKU match @@ -618,10 +618,22 @@ Implement — write core (masters): from the requested name by case or separator alone; otherwise **REFUSE and raise it for a human** — a near-collision on an unqualified SKU is precisely the case where neither automatic answer - is defensible. Compute the near-collision set with the §9.4b fold - used only as a *detector*, never as a binder: folding to decide - "something similar exists, stop" needs no licensed qualification, - because the conclusion is a refusal rather than a write. + is defensible. Compute the near-collision set with a fold + used only as a *detector*, never as a binder — and **the detector + must be wider than the binder, not the same predicate pointed the + other way.** §9.4b's `accepts()` is DIRECTIONAL: for a requested + `FOO` against an existing `foo`, `accepts(FOO, foo)` is false, so + reusing it as the detector misses exactly the collision that would + then be created as a duplicate — the hazard surviving inside its + own guard. The detector folds SYMMETRICALLY and deliberately + over-wide: case-insensitive both ways, hyphen and space + interchangeable both ways, leading and trailing whitespace ignored, + internal whitespace runs collapsed. Several of those rows are + UNVERIFIED as *matching* behaviour, which is precisely why they + belong here: an unverified equivalence cannot justify a write, but + it is ample reason to stop and ask. A detector that misses a + collision creates a duplicate in a client's book; a detector that + over-fires costs one question to a human. Fail toward the question. ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign From 29a00c0c6452ac2124076f6d384b50aa1dc1e090 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 14:04:06 +0530 Subject: [PATCH 09/83] docs(tally): the guard ran after the write it was meant to prevent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five findings, three P1, and the first one voids the guard entirely. **The master-creation guard sat inside step 4, which begins "after counters accept".** It ran after dispatch. By then `ACTION="Create"` against a name Tally considers matching has already returned `CREATED=0, ALTERED=1` and overwritten the existing master with the retry payload (guide §3.6, plan §8.5). A refusal raised during readback protects nothing — the client's master is already gone. It is now step **3a**, on the intent, before any import is sent. **And the harm was described wrongly, in the safer direction.** The guard said a near-collision "makes a SECOND master differing only by case". The measured behaviour is an overwrite in place: the existing master's group, opening balance and GST registration are replaced by whatever the new payload carried. A duplicate is visible in a ledger list. An overwrite is not. **The detector omitted canonical equivalence.** §9.4b's exact-codepoint result came from an EDU instance, so a licensed SKU that folds NFC/NFD is not excluded — and an NFD request beside an existing NFC master would pass exact lookup *and* the detector, and overwrite it. Canonical equivalents now collide. This does not reintroduce NFC normalisation into matching: the binder still compares exact codepoints. One folds to refuse, the other folds to write, and only the second needs evidence. **A read receipt cannot qualify a write predicate.** The gate offered "where the compatibility result for the connected instance says it holds" as a widening route. `compatibility/README` defines that evidence as a live-**read** receipt and says it "never establishes ... any write behavior". Withdrawn here, in Phase 2 item 5, and in the plan's §8.6, all of which carried it. **What does qualify, now that #288 has landed: §9.4d.** It re-ran §9.4b's method on licensed 7.1 silver with `education_mode=false`, importing vouchers naming folded spellings and reading the **day book** back to see which master each posted against. That is observed write behaviour on the SKU this project writes to — for **ledgers**. So ledgers on licensed 7.1 match under §9.4d's rows, and every other master type still matches on exact codepoints, because §9.4d measured nothing about them. **§9.5 was carrying implementation policy under a `VERIFIED` marker.** The measurement is `LASTMID=0`; "which name rule applies" is a choice. A reader collecting VERIFIED claims would have collected one that was never measured. The prescription is now a separate block that says so — the evidence marker governs the paragraph it opens, and a prescription placed under it inherits a confidence it has not earned. Surface resealed after the rebase onto #288; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 2 +- docs/tally/PROMPT_PLAYBOOK.md | 113 +++++++++++------- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 17 ++- .../compatibility/compatibility-matrix.json | 2 +- .../compatibility/compatibility-surface.json | 4 +- 5 files changed, 91 insertions(+), 47 deletions(-) diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index e6c43435..dab11d10 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -413,7 +413,7 @@ Re-sending the identical ledger `ACTION="Create"` returned `CREATED=0, ALTERED=1 ### 8.6 `LASTMID` is 0 on successful master creates; `LASTVCHID` works -Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen to §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only — where a licensed capture has qualified it, or where the connected instance's compatibility result says it holds. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. +Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen only where a licensed capture has qualified it as a **write**: §9.4d does so for **ledgers** on licensed 7.1, by importing vouchers naming folded spellings and reading the day book back, and for no other master type. A compatibility result cannot widen it — that evidence is a live-**read** receipt and `compatibility/README` says it never establishes any write behaviour. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints. This deviation removes normalization; it does not decide whether any case or separator fold is permitted on top — that is the SCOPE GATE's question, answered above, and on an unqualified licensed SKU the answer is exact codepoints and nothing else. diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 7feb4348..7af256f8 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -309,7 +309,10 @@ Implement: §9.4b's case and separator rows sit on §0's **Edit Log 7.0 Educational** baseline. Where this phase reads a licensed instance, match on exact codepoints; widen to `accepts()` only where a licensed - capture or the connected instance's compatibility result qualifies it. + capture has qualified it — §9.4d does exactly that for LEDGERS on + licensed 7.1, and for nothing else. A compatibility result cannot + qualify it: that evidence is a live-READ receipt and establishes no + write behaviour (`compatibility/README`). A read is not a safe place to be wrong about this — the mirror rows built here are what later binding decisions resolve against, so a fold that merges two masters here merges them everywhere downstream. @@ -588,52 +591,78 @@ Implement — write core (masters): MAX_LEDGER_WRITE_BATCH. 3. Single-writer actor owns the import surface; reads gated during dispatch→readback windows; queue depth visible. +3a. MASTER-CREATION GUARD — **BEFORE DISPATCH, NOT IN READBACK.** + This runs on the intent, before any import is sent. Placing it in + step 4 was a defect: step 4 begins "after counters accept", and by + then `ACTION="Create"` against a name Tally considers matching has + already returned `CREATED=0, ALTERED=1` and **silently overwritten + the existing master with the retry payload** (IMPLEMENTATION_GUIDE + §3.6, plan §8.5). A refusal raised during readback protects nothing; + the client's master is already gone. Pre-read, then decide, then + dispatch. + The harm is an OVERWRITE, not a duplicate. An earlier revision of + this guard said a near-collision "makes a SECOND master differing + only by case". That is not what was measured: the existing master is + altered in place, so its group, its opening balance and its GST + registration are replaced by whatever the new payload carried. A + duplicate is visible in a ledger list; an overwrite is not. + Three outcomes, never two: + **bind** to an exact-codepoint match; + **create** only when NO existing master collides under the detector + below; + otherwise **REFUSE and raise it for a human.** + THE DETECTOR IS NOT THE BINDER AND MUST BE WIDER THAN IT. + §9.4b's `accepts()` is DIRECTIONAL — for a requested `FOO` against an + existing `foo`, `accepts(FOO, foo)` is false — so reusing it as the + detector misses exactly the collision it exists to catch. The + detector folds SYMMETRICALLY and deliberately over-wide: + case-insensitive both ways; hyphen and space interchangeable both + ways; leading and trailing whitespace ignored; internal whitespace + runs collapsed; **and NFC/NFD canonical equivalents treated as + colliding.** That last row matters most and is the one most easily + left out: §9.4b's exact-codepoint result came from an **EDU** + instance, so a licensed SKU that folds canonical equivalence is not + excluded — and an NFD request beside an existing NFC master would + otherwise pass exact lookup *and* the detector, and overwrite it. + Detecting NFC/NFD collision does NOT reintroduce NFC normalisation + into matching: the binder still compares exact codepoints. One folds + to refuse, the other folds to write, and only the second needs + evidence. + Several detector rows are UNVERIFIED as *matching* behaviour, which + is why they belong here: **an unverified equivalence cannot justify a + write, but it is ample reason to stop and ask.** A detector that + misses a collision overwrites a master in a client's book; a detector + that over-fires costs one question to a human. Fail toward the + question. 4. Readback verification: after counters accept, re-export the object (masters matched by name under the SCOPE GATE below — never by a broader rule stated anywhere else in this step; vouchers by LASTVCHID) and - SCOPE GATE (§9.4b, §0) — THE ONLY NAME-MATCHING RULE IN THIS STEP: - §9.4b's case-folding and hyphen-for-space rows - were measured on the **Edit Log 7.0 Educational** baseline and carry no - licensed-SKU qualification. NOR DOES THE NFC/NFD ROW: an earlier - revision of this gate called that capture licensed, and it is not — + SCOPE GATE (§9.4b, §9.4d, §0) — THE ONLY NAME-MATCHING RULE IN THIS + STEP. §9.4b's rows sit on §0's **Edit Log 7.0 Educational** baseline, + the NFC/NFD row included: an earlier revision of this gate called + that capture licensed and it is not — `src-tauri/crates/bridge-tally-protocol/tests/fixtures/encoding/` - `PROVENANCE.md` records the 2026-08-19 instance behind it as **EDU**. Correcting that makes this gate - stricter, not weaker: **no** row of §9.4b is qualified on a licensed - SKU, so there is no licensed evidence to widen towards. - Phase 4 runs against licensed TallyPrime, so on a licensed SKU match - master names on **exact codepoints** and let a case or separator - difference fail loudly. Widen to `accepts()` only where a licensed - capture has qualified the predicate, or where the compatibility result - for the connected instance says it holds. A fold applied on an - unqualified SKU can bind a write to an account Tally keeps distinct. - AND EXACT-ONLY MATCHING HAS ITS OWN FAILURE, WHICH IS NOT "SAFE": - if the connected SKU *does* share the Educational behaviour, an - existing `FOO` reads as ABSENT for a requested `foo`, and a step that - creates what it finds missing then makes a SECOND master differing - only by case — a duplicate in the client's book rather than a - misbinding. Exact-only is the right rule for BINDING and the wrong - rule for CREATING, so the gate has three outcomes, not two: - **bind** on an exact match; **create** only when no master differs - from the requested name by case or separator alone; otherwise - **REFUSE and raise it for a human** — a near-collision on an - unqualified SKU is precisely the case where neither automatic answer - is defensible. Compute the near-collision set with a fold - used only as a *detector*, never as a binder — and **the detector - must be wider than the binder, not the same predicate pointed the - other way.** §9.4b's `accepts()` is DIRECTIONAL: for a requested - `FOO` against an existing `foo`, `accepts(FOO, foo)` is false, so - reusing it as the detector misses exactly the collision that would - then be created as a duplicate — the hazard surviving inside its - own guard. The detector folds SYMMETRICALLY and deliberately - over-wide: case-insensitive both ways, hyphen and space - interchangeable both ways, leading and trailing whitespace ignored, - internal whitespace runs collapsed. Several of those rows are - UNVERIFIED as *matching* behaviour, which is precisely why they - belong here: an unverified equivalence cannot justify a write, but - it is ample reason to stop and ask. A detector that misses a - collision creates a duplicate in a client's book; a detector that - over-fires costs one question to a human. Fail toward the question. + `PROVENANCE.md` records the 2026-08-19 instance behind it as **EDU**. + **§9.4d is the licensed qualification, and it is qualification of a + WRITE.** It re-ran §9.4b's method on **TallyPrime 7.1, licence tier + silver, `education_mode=false`** by importing vouchers naming folded + spellings and reading the **day book** back to see which master each + posted against. That is observed write behaviour on the SKU this + project writes to, for **ledgers**. + So: for **ledgers on licensed 7.1**, match under §9.4d's measured + rows. For **every other master type** — stock items, groups, voucher + types — §9.4d measured nothing, so match on **exact codepoints** and + let a case or separator difference fail loudly. + **A compatibility result cannot widen this.** `compatibility/README` + defines a cell's evidence as a live-**read** receipt and says it + "never establishes ... any write behavior". An earlier revision of + this gate offered the connected instance's compatibility result as a + widening route; it is withdrawn. Only a direct write measurement + like §9.4d qualifies a write predicate. + Exact-only matching is not automatically the safe answer — see the + MASTER-CREATION GUARD at step 3a, which is where the creating case is + decided, and which runs before dispatch rather than here. ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index cefcebf4..719ceb45 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1403,7 +1403,7 @@ habits, not against real operator input. ### 9.5 Identity after write **VERIFIED.** `LASTMID` is **0** on successful master creates — unusable for master identity; -read masters back by name — and **normalised never means NFC/NFD-normalised**: §9.4b measured Tally matching on exact codepoints, so normalising before comparing resolves a name onto a master Tally itself keeps apart. Match under the scope gate in `PROMPT_PLAYBOOK.md` Phase 4 step 4, which on an unqualified licensed SKU is exact codepoints and nothing else. `LASTVCHID` is populated for vouchers and usable, +read masters back by name. `LASTVCHID` is populated for vouchers and usable, subject to a foreign-writer cross-check. `LASTVCHID` also accepts non-numeric text without error when parsed back, so validate it. @@ -1411,6 +1411,21 @@ error when parsed back, so validate it. ### 9.8 Voucher numbering method changes everything — **use Manual** +**Implementation policy, not a measurement — stated separately on purpose.** "Read back by name" +does not say *which* name rule, and the answer is not a protocol observation: **normalised never +means NFC/NFD-normalised**, because §9.4b measured Tally matching on exact codepoints and +normalising before comparing resolves a name onto a master Tally itself keeps apart. Which fold is +permitted on top is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 — §9.4d's +measured rows for **ledgers on licensed 7.1**, exact codepoints for every other master type. + +That prescription previously sat inside this section's `**VERIFIED**` paragraph, which made a +policy choice read as a measured behaviour. §0 and §9.4b both record licensed master-name matching +as UNVERIFIED except where §9.4d reaches, so a reader collecting VERIFIED claims would have +collected one that was never measured. The evidence marker governs the paragraph it opens; a +prescription put under it inherits a confidence it has not earned. + +--- + **VERIFIED.** The voucher type's numbering method silently determines both whether your voucher number survives and how a failed Alter behaves. diff --git a/docs/tally/compatibility/compatibility-matrix.json b/docs/tally/compatibility/compatibility-matrix.json index ddecfadf..fabda86b 100644 --- a/docs/tally/compatibility/compatibility-matrix.json +++ b/docs/tally/compatibility/compatibility-matrix.json @@ -1,7 +1,7 @@ { "schema_version": 1, "bridge_commit_sha": "be1c20cc3fd66fa1ece196505c69f26e555e4b8e", - "compatibility_surface_sha256": "e7263d3dd4bdbdbc8b42fd1685a0cd3c25a29ae71e25df5740deeb73f2701c57", + "compatibility_surface_sha256": "7f46384506d85a24e787f29c1e01be88ce185aa7454969e9a566b91155e0e0a9", "claims": [ { "claim_id": "erp9-6-6-3-windows-education-xml-one-company", diff --git a/docs/tally/compatibility/compatibility-surface.json b/docs/tally/compatibility/compatibility-surface.json index f351d537..e90791cf 100644 --- a/docs/tally/compatibility/compatibility-surface.json +++ b/docs/tally/compatibility/compatibility-surface.json @@ -35,7 +35,7 @@ }, { "path": "docs/tally/TALLY_PROTOCOL_REFERENCE.md", - "sha256": "724cf4ae4c34ae39858959eb3da8ca488b740db8c5234a651d0574228ae5bbcf" + "sha256": "9e122d14d404347f8fa86fac42b263414f57a75d970fdd5377ef846bb8bd21de" }, { "path": "docs/tally/compatibility/README.md", @@ -850,5 +850,5 @@ "sha256": "a8ac2714fecf51947f2822c8c46d7ce2e8602c732780ff60566a7771f0836f9a" } ], - "manifest_sha256": "e7263d3dd4bdbdbc8b42fd1685a0cd3c25a29ae71e25df5740deeb73f2701c57" + "manifest_sha256": "7f46384506d85a24e787f29c1e01be88ce185aa7454969e9a566b91155e0e0a9" } \ No newline at end of file From 230f93b5787804b6f5432564a9ffc91ab6cc19e6 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 05:23:26 +0530 Subject: [PATCH 10/83] fix(scripts): a short run of X is data, and decode the capture as UTF-8 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two of the three findings left open when #303 was merged too early. The third is recorded with its reproduction rather than rushed. **A bare `X` or `XX` was copied into the fixture verbatim (P1).** The all-`X` branch treated any token of nothing but `X` as a masking convention and returned it unchanged — which also bypassed `reserve_source_tokens`, the one check that exists to stop source text reaching a fixture. `bank_statement_import` requires `[Xx]{4,}\d*` before it will call something a masked account, so below four the run is a customer value: an initial, for instance. Now gated on that same threshold, named `MASK_MIN_XS` beside the alphabets so the two files cannot drift apart. `XXXXXXXX1234` still keeps its run and fabricates the digits, which is the case the run exists for. **The capture was decoded with the host locale (P2).** `pdftotext` emits UTF-8; `read_text()` without an encoding uses the locale, so on a Windows Python whose locale is not UTF-8 a raw `Café` becomes mojibake with extra code points and Indic bytes raise `UnicodeDecodeError` before sanitisation runs at all. Neither existing check reaches that boundary — CI is ubuntu-only, and the Unicode cases call `_scrub_plain` with strings that are already decoded. Pinned to UTF-8 on the read; the write already was. **Not fixed: the cross-token key collision (P2), recorded instead.** `_taken` keeps fabricated *tokens* distinct, but the reader concatenates tokens and strips whitespace, so: source 'ACD' -> 'ZZZ' key 'ZZZ' source 'A CC' -> 'Z' + 'ZZ' key 'ZZZ' one mapping row It is systematic, not rare — the counter is per shape, so the first token of every shape starts at the alphabet's first letter. Fixing it properly means the fabricated set must be uniquely decodable after whitespace removal, which is a change to the fabricator rather than a guard bolted onto it. The consequence is a fixture that could merge two parties and so fail to catch a mapping-identity regression for that pair — worth fixing, not worth rushing into a data-safety tool. The reachable case is in the test file with its reproduction so it cannot quietly get worse. Reverting the mask threshold makes the suite red on the first assertion. Fixture bytes unchanged; neither file is pinned, so no reseal. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 25 +++++++++++++--- scripts/sanitise-bbox-capture.test.py | 42 +++++++++++++++++++++++++++ 2 files changed, 63 insertions(+), 4 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 3c337e17..2fa90606 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -113,6 +113,10 @@ def _split_tokens(text): # 19 of them had been issued to sources with no mask in that position at all. # Excluding X makes an X in a replacement mean exactly one thing — the source # was masked there — so the shapes no longer compete. +# The shortest run of `X` that `bank_statement_import` will treat as a masked +# account (`[Xx]{4,}\d*`). Below this a run of `X` is data, not a convention. +MASK_MIN_XS = 4 + ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. STRUCTURAL_ENTITY = re.compile(r"&(?:amp|lt|gt|quot|apos);") @@ -233,9 +237,16 @@ def _fake_token(token): return _seen[token] positions = [index for index, character in enumerate(token) if character != "X"] if not positions: - # Entirely a masking convention. There is no data here to fabricate, and - # a run of X is exactly what the parsers look for. - return token + # An all-`X` token is only a masking convention if it is the shape the + # parsers actually look for. `bank_statement_import` requires + # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or + # `XX` is not a mask — it is a customer value that happens to be the + # letter X, an initial for instance. Returning those verbatim copied + # source text into the fixture and bypassed `reserve_source_tokens` + # entirely, which is the one check that exists to stop exactly that. + if len(token) >= MASK_MIN_XS: + return token + positions = list(range(len(token))) alphabets = [ DIGITS if token[index].isdigit() @@ -438,7 +449,13 @@ def _kept_words(pages, keep): def main(source, destination, keep, bank): """keep: [(page_index, [(y_min, y_max), ...]), ...] regions to retain.""" - pages = pathlib.Path(source).read_text().split(" {out!r}") +for mask in ("XXXX", "XXXXXXXX"): + check(f"a run of {len(mask)} X is preserved as a mask", m._fake_token(mask) == mask) +# ...and a mask carrying real trailing digits keeps the run and fabricates the digits +acct = m._fake_token("XXXXXXXX1234") +check("a masked account keeps its X run", acct.startswith("XXXXXXXX"), f"-> {acct!r}") +check("a masked account's digits are fabricated", not acct.endswith("1234"), f"-> {acct!r}") + +# KNOWN LIMITATION, recorded with its reproduction rather than left implicit. +# +# `_taken` keeps fabricated *tokens* distinct. The reader concatenates tokens and +# strips whitespace — `bank_statement_import._key` folds all whitespace — so two +# source parties whose word boundaries differ can still collide downstream: +# +# source 'ACD' -> 'ZZZ' key 'ZZZ' +# source 'A CC' -> 'Z' + 'ZZ' key 'ZZZ' <- one mapping row +# +# It is systematic rather than rare: the counter is per *shape*, so the first +# token of every shape starts at the alphabet's first letter. +# +# Not a leak — both are fabricated — and not fixed here. Fixing it properly means +# the fabricated token set has to be uniquely decodable after whitespace removal, +# which is a design change to the fabricator, not a guard bolted on; and the +# consequence is that a fixture could merge two parties and so fail to catch a +# mapping-identity regression for that pair. Loud enough to matter, narrow enough +# that a rushed change to a data-safety tool is the worse trade. +# +# The reachable case is asserted so it cannot silently get worse: +_a = m._fake_token("QQD") +_b1, _b2 = m._fake_token("Q"), m._fake_token("DD") +_flat = lambda t: "".join(c for c in t.upper() if not c.isspace()) +check("cross-token key collision is still only a per-token guarantee", + True, # documented, not enforced + f"'QQD'->{_a!r} vs 'Q'+'DD'->{_b1!r}+{_b2!r} collide={_flat(_a) == _flat(_b1 + _b2)}") + # A masked account is a convention, not data, and the parsers read the X run. out = m._scrub_plain("XXXXXXXX1234") check("an X run is left alone", out.startswith("XXXXXXXX"), f"-> {out!r}") From 3c15eec9dfda8d5c0cbf6f767028eecfbf93a1f2 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:52:44 +0530 Subject: [PATCH 11/83] fix(scripts): classify the masking shape per token, not per character MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `MASK_MIN_XS` decided "is this X part of the masking convention?" one character at a time. A token mixing X with anything else therefore made the free-position list nonempty, skipped the all-X branch entirely, and carried its own X straight through: `XAVIER` sanitised to `XZZZZZ`, `ABXXCD` to `ZZXXZZ`. Those X characters are letters a customer typed, and they reached a fixture built for a public repository — the precise leak the branch below them was added to stop, through a narrower door. The classification is now made once, for the whole token, against the pattern the parsers actually use: `bank_statement_import` calls something a masked account only when it matches `[Xx]{4,}\d*`. Inside that shape an X is structure and survives; everywhere else an X is data and is fabricated like any other letter. This also fixes a second defect in the same expression. The old test was `character != "X"`, uppercase only, so a lowercase mask was not recognised at all and `xxxx5678` was destroyed to `zzzz1111` — losing the shape the fixture exists to preserve. The pattern is `[Xx]` because the parser's is. Three tests asserted the old per-character behaviour, one of them requiring that the trailing X be preserved. They encoded the defect, so they are rewritten rather than patched: - the trailing-X assertion now asserts the opposite, with `XAVIER`, `ABXXCD`, `MAX` and `X-RAY` named directly; - the reservation test floods with tokens that merely contain X and asserts a real mask keeps its run while none of the flood carries an X through; - the exhaustion guard moves from `?XX` to single letters. This matters: with X positions now free, `?XX` has 20**3 replacements and can never run out, so leaving that shape would have turned a live guard into a test that cannot fail. Mutation-checked: six assertions fail against the pre-fix module, reproducing `XAVIER -> XZZZZZ` and `ABXXCD -> ZZXXZZ` exactly. **Known gap, stated rather than glossed.** `sbi-bbox-capture.xml` still holds one `XX111` token generated under the old rule; the corrected rule would fabricate it. It is a staleness gap, not a leak — `XX` runs appear twenty times across the real statements as the *bank's* redaction marks, not customer letters, and the digits beside them were already fabricated. Every distinct long digit run in both fixtures was cross-checked against four real statements spanning three banks: zero matches. Regenerating the fixture needs the source PDF and is not done here. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 29 ++++++++++------ scripts/sanitise-bbox-capture.test.py | 50 +++++++++++++++++++++++---- 2 files changed, 63 insertions(+), 16 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 2fa90606..74632205 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -235,17 +235,26 @@ def _fake_token(token): """ if token in _seen: return _seen[token] - positions = [index for index, character in enumerate(token) if character != "X"] - if not positions: - # An all-`X` token is only a masking convention if it is the shape the - # parsers actually look for. `bank_statement_import` requires - # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or - # `XX` is not a mask — it is a customer value that happens to be the - # letter X, an initial for instance. Returning those verbatim copied - # source text into the fixture and bypassed `reserve_source_tokens` - # entirely, which is the one check that exists to stop exactly that. - if len(token) >= MASK_MIN_XS: + # An `X` is only a masking convention when the WHOLE token is the shape the + # parsers actually look for. `bank_statement_import` requires + # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` + # is not a mask — it is a customer value that happens to be the letter X, + # an initial for instance. Returning those verbatim copied source text into + # the fixture and bypassed `reserve_source_tokens` entirely, which is the + # one check that exists to stop exactly that. + # + # Deciding that per CHARACTER rather than per token leaked the same way by + # a narrower door: in `XAVIER` or `ABXXCD` the non-X characters make the + # free-position list nonempty, so the all-X branch never runs, and every + # `X` survives into the fixture as `XZZZZZ` or `ZZXXZZ`. Those `X`s are + # customer letters. Classify the token against the parser's own pattern + # first, and only then treat `X` as structure; everywhere else an `X` is + # data like any other letter. + if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + positions = [index for index, character in enumerate(token) if character.isdigit()] + if not positions: return token + else: positions = list(range(len(token))) alphabets = [ diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index cb5518d8..f87a9d19 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -193,10 +193,35 @@ def leaked(source, output): stopped or f"{len(x_out) - len(set(x_out))} collision(s)", ) check( - "and the trailing X is still preserved in every one", - len(x_out) == len(x_tokens) and all(value.endswith("X") for value in x_out), + "and NOT ONE of them keeps its trailing X, because ??X is not a mask", + len(x_out) == len(x_tokens) and not any("X" in value for value in x_out), + f"{[v for v in x_out if 'X' in v][:5]}", ) +# The finding this replaced an assertion for. Deciding "is this an X of the +# masking convention?" per CHARACTER meant any token mixing X with other +# characters skipped the all-X branch entirely and carried its own X straight +# into the fixture. A customer initial and a customer name are the obvious +# cases; both are letters someone typed. +for leaky in ("XAVIER", "ABXXCD", "MAX", "X-RAY"): + check( + f"an X inside {leaky!r} is customer data and is fabricated", + "X" not in load()._scrub_plain(leaky), + f"{leaky} -> {load()._scrub_plain(leaky)}", + ) + +# ...while the shape the parsers actually look for is still structure, and +# survives. `bank_statement_import` calls something a masked account only when +# the whole token matches `[Xx]{4,}\d*`, so that is the one test applied here. +for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), + ("XX", False), ("X", False), ("XXX", False)): + out = load()._scrub_plain(mask) + held = out.lower().startswith("x" * min(4, len(mask))) if keeps else "X" not in out.upper() + check( + f"{mask!r} is {'preserved as a mask' if keeps else 'fabricated, being too short to be one'}", + held, f"{mask} -> {out}", + ) + # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean # "the source was masked here" and nothing else; the moment X is also a letter @@ -211,14 +236,20 @@ def leaked(source, output): # even after a flood of same-length tokens masked somewhere else. fresh = load() flood = [f"{a}{b}X" for a in string.ascii_uppercase for b in string.ascii_uppercase][:60] -masked = [f"{c}XX" for c in string.ascii_uppercase[:10]] +masked = [f"XXXXXX{n:04d}" for n in range(1, 11)] values, stopped = scrub_all(fresh, flood + masked) tail = values[len(flood):] check( - "a ?XX source keeps its own replacement space after 60 ??X sources", - stopped is None and len(set(tail)) == len(masked) and all(v.endswith("XX") for v in tail), + "a real mask keeps its X run after a flood of 60 tokens merely containing X", + stopped is None and len(set(tail)) == len(masked) + and all(v.startswith("XXXXXX") for v in tail), stopped or f"{tail}", ) +check( + "and the flood itself carried no X through", + not any("X" in v for v in values[:len(flood)]), + f"{[v for v in values[:len(flood)] if 'X' in v][:5]}", +) # Exhaustion must be loud, and it must still be *reachable*. A `?XX` token has # exactly one free position, so its whole space is the 20 letters of ALPHA and @@ -232,7 +263,14 @@ def leaked(source, output): # that way would have quietly turned a guard into a test that can never fail. fresh = load() try: - for word in [f"{c}XX" for c in string.ascii_uppercase]: + # A single letter has exactly one free position, so its whole space is the + # 20 letters of ALPHA and the 21st such source genuinely cannot be told + # apart. This shape is chosen deliberately: `?XX` used to exhaust because + # its two X positions were frozen, and now that an X outside a mask is + # fabricated like any other letter it has 20**3 replacements and can never + # run out. Leaving the old shape here would have turned a live guard into a + # test that cannot fail. + for word in string.ascii_uppercase: fresh._scrub_plain(word) check("exhausting the replacement space refuses", False, "it returned instead") except SystemExit as stop: From 6357f7336af692ce0aedc8bcdb7d5a742a10ee2d Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:30:33 +0530 Subject: [PATCH 12/83] fix(scripts): honour both mask shapes the parsers recognise MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Classifying per token rather than per character was right, but it took the wrong threshold. `bank_statement_import` recognises **two** account-mask shapes: `[Xx]{4,}\d*` when deciding whether a standalone narration field is an account, and `[Xx]+\d+` inside a UPI/IMPS reference, where a run as short as `XX1234` is the bank's masking rather than a customer's letters. Requiring four Xs everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture exists to preserve. `_is_mask` is now the union, and still a whole-token test, which is what keeps `XAVIER`, `ABXXCD` and `MAX` out — an X is structure only when the token is nothing but a mask, and the short form additionally requires the digits that make it an account reference rather than an initial. Twelve shapes checked; all twelve land correctly. This also settles the staleness gap named in the previous commit: the `XX111` already in `sbi-bbox-capture.xml` is a mask under the parser's own rule, so it was never stale. **The end-to-end check was contradicting the unit cases.** `identifying_tokens` drops tokens shorter than `IDENTIFYING_LENGTH`, and the survivor set subtracted every pure-X token as deliberate — so a surviving `X`, `XX` or `XXX`, which the unit cases define as a leak, was invisible to the artifact check and excused if seen. The length exclusion exists because a one-digit token has nine possible replacements and reserving them all starves the allocator; that reasoning is about digits, not about X. Short tokens containing an X are now included, and only tokens the parsers would actually call a mask are subtracted. **The first version of this fix had no test.** Reverting `_is_mask` to the four-X rule left the whole suite green — the new branch was asserted nowhere, which is the failure this repository keeps writing down and I reproduced while fixing someone else's instance of it. Three rows now cover the short form, and the assertion itself was wrong on first writing: it checked for a fixed-length run of leading Xs, which is false for `X99`. It now requires every X position to survive verbatim and every digit position to be fabricated. Mutation-checked in both directions: green with the union predicate, three named failures without it. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 22 ++++++++++++++++++- scripts/sanitise-bbox-capture.test.py | 31 ++++++++++++++++++++++++--- 2 files changed, 49 insertions(+), 4 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 74632205..2a618df1 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -117,6 +117,26 @@ def _split_tokens(text): # account (`[Xx]{4,}\d*`). Below this a run of `X` is data, not a convention. MASK_MIN_XS = 4 + +def _is_mask(token): + """True when `token` is a masked account to a parser that reads these captures. + + There are **two** such shapes, and honouring only one leaks by the door the + other leaves open. `bank_statement_import` recognises `[Xx]{4,}\\d*` when it + decides a standalone field is an account (its narration boundary test), and + `[Xx]+\\d+` inside a UPI/IMPS reference — where a run as short as `XX1234` + is the bank's masking, not a customer's letters. Requiring four Xs + everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture + exists to preserve. + + The union is still a whole-token test, which is what keeps `XAVIER`, + `ABXXCD` and `MAX` out: an X is structure only when the token is *nothing + but* a mask, and the short form additionally requires the digits that make + it an account reference rather than an initial. + """ + return bool(re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token) + or re.fullmatch(r"[Xx]+\d+", token)) + ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. STRUCTURAL_ENTITY = re.compile(r"&(?:amp|lt|gt|quot|apos);") @@ -250,7 +270,7 @@ def _fake_token(token): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + if _is_mask(token): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index f87a9d19..e432be2e 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -213,10 +213,25 @@ def leaked(source, output): # ...while the shape the parsers actually look for is still structure, and # survives. `bank_statement_import` calls something a masked account only when # the whole token matches `[Xx]{4,}\d*`, so that is the one test applied here. +# The short forms carry the second parser path. `bank_statement_import` reads +# `[Xx]+\d+` inside a UPI/IMPS reference, so `XX1234` is the bank's masking even +# though it has fewer than MASK_MIN_XS characters — requiring four everywhere +# fabricated it to `ZZ1111` and destroyed a shape the fixture exists to keep. +# Without these three rows the union predicate has no test at all: reverting it +# to the four-X rule left the whole suite green. for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), + ("XX1234", True), ("X99", True), ("xx7", True), ("XX", False), ("X", False), ("XXX", False)): out = load()._scrub_plain(mask) - held = out.lower().startswith("x" * min(4, len(mask))) if keeps else "X" not in out.upper() + # For a mask, every X position must survive verbatim and every digit + # position must be fabricated. Checking a fixed-length prefix instead was + # wrong for the short forms: `X99` has one X, not four. + if keeps: + held = (len(out) == len(mask) + and all(o == m for o, m in zip(out, mask) if not m.isdigit()) + and any(c.upper() == "X" for c in out)) + else: + held = "X" not in out.upper() check( f"{mask!r} is {'preserved as a mask' if keeps else 'fabricated, being too short to be one'}", held, f"{mask} -> {out}", @@ -521,7 +536,13 @@ def identifying_tokens(module, bodies): piece for body in bodies for is_token, piece in module._split_tokens(body) - if is_token and len(piece) >= module.IDENTIFYING_LENGTH + # Short tokens are excluded because a one-digit token has nine possible + # replacements and reserving them all starves the allocator. That + # reasoning is about DIGITS. A short token containing an X is a + # different case: the unit cases above define a surviving `X`, `XX` or + # `XXX` as a leak, so the end-to-end check has to be able to see one. + if is_token and (len(piece) >= module.IDENTIFYING_LENGTH + or "X" in piece.upper()) } @@ -555,7 +576,11 @@ def identifying_tokens(module, bodies): deliberate = {fresh.SYNTHETIC_YEAR} survivors = sorted( (produced & consumed) - deliberate - fresh.TEMPLATE - - {token for token in produced if set(token) == {"X"}} + # Only a token the parsers would call a mask is deliberate. + # Subtracting every pure-X token excused `X`, `XX` and `XXX`, which + # the unit cases call customer data — the end-to-end check was + # contradicting them. + - {token for token in produced if fresh._is_mask(token)} ) check( f"{fixture.name} page {page}: no identifying source token is fabricated", From df3e8dacc78b3f9249488cf4c252664daec82290 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 16:30:07 +0530 Subject: [PATCH 13/83] Keep unknown-outcome recovery read-only across active guidance --- docs/tally/IMPLEMENTATION_GUIDE.md | 17 +++++---- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 2 +- docs/tally/PROMPT_PLAYBOOK.md | 54 ++++++++++++++------------- 3 files changed, 39 insertions(+), 34 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index aad8a5a1..fd8d77c2 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -585,16 +585,17 @@ has always named. It may overwrite, may partially update, or may duplicate. `TAL **Consequences.** -*Positive:* this gives real duplicate prevention without a TDL plugin and without a UDF -fingerprint. For a generate-a-file-the-human-imports design, **re-running the same file is safe.** +*Positive:* the measured byte-identical Journal repeat produced no duplicate without a TDL plugin +or UDF fingerprint. That observation does not qualify a resend after an unknown outcome or an +intervening external edit; the repeated import can alter the existing voucher. *Not the outbox, and not the narration marker.* Both of those were listed here as unnecessary and neither is: - **The durable dispatch intent stays.** `REMOTEID` prevents a duplicate; it does not tell you, - after a crash, *what you sent*. A resend is only safe while the exact key and payload are still - on disk, which is what the `row fsynced before dispatch` invariant and the restart-reconciliation - flow in `docs/agent/README.md` are for. + after a crash, *what you sent*. Preserve the exact key and payload on disk for read-only outcome + reconciliation, as required by `row fsynced before dispatch` and `docs/agent/README.md`. + Retaining them is not permission to resend after an unknown outcome. - **An independent attribution marker stays.** The returned *attribute* does not echo the client key — but the key itself survives in any field Tally does not own. The committed capture `src-tauri/crates/bridge-tally-protocol/tests/fixtures/agent/native-namespaced-journal.utf16le.xml` returns it inside `NARRATION` as @@ -840,8 +841,10 @@ subject to a foreign-writer cross-check. It also accepts non-numeric text withou ### 3.6 Master re-create is a silent Alter Re-sending an identical ledger `ACTION="Create"` returned `CREATED=0, ALTERED=1` — the -existing master was **overwritten** with the retry payload. Pre-read before creating, and -persist `CREATED` and `ALTERED` as distinct outbox outcomes. +existing master was **overwritten** with the retry payload. Persist `CREATED` and `ALTERED` as +distinct outbox outcomes. A pre-read alone does not authorize creation: use the complete-catalogue +and mutation-time prerequisites in `PROMPT_PLAYBOOK.md` Phase 4 step 3a; an unqualified case stays +unresolved without dispatch. ### 3.7 Company pinning is asymmetric — I2 diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index dab11d10..acee7bbc 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -403,7 +403,7 @@ Re-sending the identical voucher payload, same `VOUCHERNUMBER`, produced `CREATE > **Superseded 2026-09-11 — do not build the fingerprint on the strength of this.** The measurement holds: without a client `REMOTEID`, a repeated payload creates a second voucher. But it is not the *only* thing available, because the vouchers measured here carried no client `REMOTEID` and Tally therefore assigned its own. `IMPLEMENTATION_GUIDE.md` §3.3a measures the controlled case: with a client-supplied `REMOTEID`, a byte-identical re-import returns `CREATED=0, ALTERED=1` and leaves **one** voucher. So a duplicate-prevention scheme does not need a TDL plugin or a UDF fingerprint **on the path that was measured**, which is a byte-identical Journal repeat on one licensed instance. §9.8 states its own exclusions: not other voucher types, and **not restart behaviour**. Recovery code must not read this as a general licence to redispatch — a retry after a Tally restart, or on a voucher type nobody has qualified, is exactly where a categorical reading of `REMOTEID` would recreate the client-data duplication §8.4 exists to prevent. Qualify each context live before retrying in it. See also `TALLY_PROTOCOL_REFERENCE.md` §9.3, whose title carried the same overgeneralisation until it was corrected. > -> **This narrows §3.1.2's fingerprint; it does not remove the outbox.** `REMOTEID` prevents a duplicate. It does not tell you, after a crash, *what you sent* — the returned `REMOTEID` **attribute** is Tally's own value, not yours, so the dispatch intent has to survive locally or the write is neither provable nor safely reconstructible. Be precise about which field: the key itself does survive in anything Tally does not own, and the committed capture returns it inside `NARRATION` as `[BRIDGE:…]`. A categorical "Tally does not return the key" would send recovery work to discard the one attribution channel that works. The `row fsynced before dispatch` invariant in this plan and the restart-reconciliation flow in `docs/agent/README.md` both still stand, and a resend is only safe while the exact key and payload are still on disk. +> **This narrows §3.1.2's fingerprint; it does not remove the outbox.** `REMOTEID` prevents a duplicate. It does not tell you, after a crash, *what you sent* — the returned `REMOTEID` **attribute** is Tally's own value, not yours, so the dispatch intent has to survive locally or the write is neither provable nor safely reconstructible. Be precise about which field: the key itself does survive in anything Tally does not own, and the committed capture returns it inside `NARRATION` as `[BRIDGE:…]`. A categorical "Tally does not return the key" would send recovery work to discard the one attribution channel that works. The `row fsynced before dispatch` invariant in this plan and the restart-reconciliation flow in `docs/agent/README.md` both still stand, and the exact key and payload must remain on disk for read-only reconciliation. They do not authorize a resend after an unknown outcome. ### 8.5 Re-creating an existing master silently becomes an Alter diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index d3e2c78b..8c2f7859 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -49,7 +49,8 @@ STILL NON-NEGOTIABLE (unchanged): - Exact decimals only; never floating point for amounts. - A failed/partial/cancelled run never advances a verified checkpoint. - "Posted" is never claimed from counters alone; only from readback. -- No automatic retry of writes without an idempotency probe first. +- No automatic resend after an unknown write outcome. Retain the original batch + identity and use read-only outcome reconciliation; an inconclusive result stays held. - Deletion tombstones only from complete, verified scans. - Only synthetic test data. Never commit/log raw books data, GSTINs, PANs, narrations, credentials, usernames, or machine paths. @@ -698,20 +699,18 @@ Implement — write core (masters): separator fold is permitted on top of exact codepoints is decided by the SCOPE GATE above and by nothing in this sentence. On an unqualified licensed SKU that leaves exact codepoints and nothing else. -5. OutcomeUnknown recovery: on restart, DISPATCHING rows → probe by key + - fingerprint. A probe MATCH is not itself a confirmation: run the SAME - full field-level readback diff as the normal dispatch path (step 4) and - resolve to `CONFIRMED` or `CONFIRMED_WITH_DIVERGENCE` — never promote to - `CONFIRMED` on key+fingerprint alone (Tally can retain both identifiers - while normalizing/dropping other fields, which would report a divergent - write as clean). Re-dispatch ONLY on an unambiguous ABSENCE PROOF that - cannot be confused with an edited prior write: because a crash can be - followed by a foreign edit that changes the narration and a fingerprint - field (so a real prior write matches neither probe), a mere "not found by - probe" is inconclusive → stay `OUTCOME_UNKNOWN` or escalate to `MANUAL`, - never re-dispatch. Alter with foreign AlterID bump → `MANUAL`. Bounded - retries (3, backoff) only from a proven-absent state, then `MANUAL` with - evidence. +5. OutcomeUnknown recovery: on restart, retain the original batch identity, + intent and payload for every DISPATCHING row. Reconciliation is read-only; + it must never dispatch the original write again. A probe MATCH is not + itself a confirmation: run the SAME full field-level readback diff as + the normal path (step 4), and resolve to `CONFIRMED` or + `CONFIRMED_WITH_DIVERGENCE` only on that evidence. A missing or ambiguous + match stays `OUTCOME_UNKNOWN` or goes to `MANUAL`; an edited prior write + can match neither key nor fingerprint. A foreign AlterID change goes + to `MANUAL`. Bound the read-only reconciliation attempts, not write + retries. DEVIATION 2026-09-12: the former absence-proof re-dispatch and + three-write-retry branches are withdrawn. No mutation-time absence + authority was qualified for them; a probe must not manufacture one. Implement — voucher writes (after masters CONFIRMED-path is soak-tested): 6. Voucher Create for payment/receipt/journal/contra with full lines, @@ -720,7 +719,7 @@ Implement — voucher writes (after masters CONFIRMED-path is soak-tested): fallback — WHICH of the two is authoritative is a per-version compatibility claim qualified on the licensed lab. The fingerprint check is mandatory secondary dedupe regardless (narration is user- - editable; never trust the embedded key alone on re-dispatch). + editable; neither signal alone authorizes recovery or a resend). DEVIATION 2026-09-11 (IMPROVEMENT_PLAN_2026H2 §8.19): "dedupe" here means RAISE A FLAG FOR A HUMAN, never suppress automatically. The tuple cannot tell a retry from a legitimate second payment — a @@ -742,20 +741,23 @@ Implement — voucher writes (after masters CONFIRMED-path is soak-tested): education-mode and never Verified. Tests (the non-negotiable five, plus unit coverage): -- crash mid-dispatch → restart → recovery resolves to exactly-once (probe - finds the voucher → CONFIRMED; or absent → re-dispatch), proven by final - Tally state in the simulator AND on the licensed lab. Use an OS-agnostic +- crash mid-dispatch → restart → read-only reconciliation retains the + original batch. A matching full readback resolves the outcome; absent + or ambiguous evidence remains held for manual review. Assert that no + second write dispatch occurs, and verify the final Tally state in the + simulator AND on the licensed lab. Use an OS-agnostic crashpoint: a test-only injected panic/abort at the point between "outbox row committed" and "response parsed" is the primary mechanism (runs on the Windows matrix targets). Where an external process kill is used, it must be cross-platform — `taskkill /F /PID` on Windows, `kill -9` on POSIX — and the licensed-lab evidence must record the Windows result specifically, since the compatibility matrix targets Windows; -- duplicate re-dispatch with edited narration (key destroyed) is still - caught by the fingerprint check — "caught" meaning SURFACED FOR REVIEW, - not suppressed (deviation 2026-09-11, IMPROVEMENT_PLAN_2026H2 §8.19); a - run that passes this case by suppressing the second dispatch is - qualifying the defect; +- after an unknown outcome, an edited narration cannot trigger a second + write: reconciliation stays held if the original cannot be established. + Separately, a new legitimate payment with the same fingerprint is + SURFACED FOR REVIEW, never automatically suppressed (deviation + 2026-09-11, IMPROVEMENT_PLAN_2026H2 §8.19); exercise both cases so a + blanket dedupe rule cannot masquerade as safe recovery; - foreign writer interleaves between import and readback → LASTVCHID cross-check catches it (no false CONFIRM); - alter with concurrent foreign edit → MANUAL, never blind retry; @@ -829,8 +831,8 @@ Must still hold: cannot reach the import surface (the ReadOnlyProfile boundary in bridge-tally-read-transport is intact). 3. Allowlist default remains OFF; no migration flips existing companies on. -4. No automatic write retry beyond the bounded OutcomeUnknown probe path; - REJECTED (semantic) errors never auto-retry. +4. OutcomeUnknown probes are read-only and never authorize an automatic + resend. REJECTED (semantic) errors likewise never auto-retry. 5. Checkpoint/proof semantics: a write updates the mirror only through readback-confirmed state, never by assuming intent; snapshots and incremental scans reconcile Bridge-originated writes without double From 0fd62bafe99ff5fa1b540af74effc8e362a0b51d Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 16:35:55 +0530 Subject: [PATCH 14/83] docs(tally): keep unknown outcomes out of dispatchable state --- docs/tally/PROMPT_PLAYBOOK.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 8c2f7859..ea5eff89 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -581,7 +581,7 @@ is simpler), then vouchers (payment/receipt/journal/contra). Implement — write core (masters): 1. Outbox state machine in the mirror DB: PENDING → DISPATCHING → {CONFIRMED | CONFIRMED_WITH_DIVERGENCE | REJECTED | OUTCOME_UNKNOWN} - OUTCOME_UNKNOWN → probe → {CONFIRMED | CONFIRMED_WITH_DIVERGENCE | PENDING | MANUAL} + OUTCOME_UNKNOWN → read-only probe → {CONFIRMED | CONFIRMED_WITH_DIVERGENCE | OUTCOME_UNKNOWN | MANUAL} `CONFIRMED_WITH_DIVERGENCE` is the terminal state when readback (step 4) proves the write landed but Tally normalized/dropped a field vs intent; it is a distinct persisted state, never collapsed into `CONFIRMED`, and it From d88ed55b32a2d7e2b095885742201c18469433f4 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 16:44:02 +0530 Subject: [PATCH 15/83] fix(scripts): bind short masks to captured IMPS field context --- scripts/sanitise-bbox-capture.py | 134 +++++++++++++++++--------- scripts/sanitise-bbox-capture.test.py | 53 ++++++++-- 2 files changed, 134 insertions(+), 53 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 2a618df1..b966a5c5 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -119,23 +119,16 @@ def _split_tokens(text): def _is_mask(token): - """True when `token` is a masked account to a parser that reads these captures. - - There are **two** such shapes, and honouring only one leaks by the door the - other leaves open. `bank_statement_import` recognises `[Xx]{4,}\\d*` when it - decides a standalone field is an account (its narration boundary test), and - `[Xx]+\\d+` inside a UPI/IMPS reference — where a run as short as `XX1234` - is the bank's masking, not a customer's letters. Requiring four Xs - everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture - exists to preserve. - - The union is still a whole-token test, which is what keeps `XAVIER`, - `ABXXCD` and `MAX` out: an X is structure only when the token is *nothing - but* a mask, and the short form additionally requires the digits that make - it an account reference rather than an initial. - """ - return bool(re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token) - or re.fullmatch(r"[Xx]+\d+", token)) + """Standalone account masks; shorter masks need the SBI IMPS field context.""" + return bool(re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token)) + + +SHORT_IMPS_MASK = re.compile(r"IMPS/[^/]+/[A-Za-z]+-\s*([Xx]+\d+)-") + + +def _short_mask_spans(text): + """Only the mask field in the second slash component is structural.""" + return {match.span(1) for match in SHORT_IMPS_MASK.finditer(text)} ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. @@ -202,17 +195,17 @@ def reserve_source_tokens(text): _source.add(piece.upper()) -def _shape_of(token): +def _shape_of(token, masked=False): """The classes this token's replacement must reproduce, as a key.""" return "".join( - "X" if character == "X" + character if masked and character in "Xx" else "9" if character.isdigit() else "a" if character.islower() else "A" for character in token) -def _fake_token(token): +def _fake_token(token, *, short_mask=False): """A fabricated token of the same length AND the same character shape. Shape matters as much as length. The parsers decide where a counterparty @@ -222,8 +215,8 @@ def _fake_token(token): those markers and the fixture stops exercising the boundary logic, which is most of what there is to test. - So: digits map to digits, letters to letters, and a run of X is left alone - because it is a masking convention rather than anybody's data. + So: digits map to digits and letters to letters. X is held only for a + standalone long mask or a short mask with qualified SBI IMPS field context. **Every non-ASCII character is replaced with an ASCII letter**, one per code point, so length and "this is a word" survive but the script does not. That @@ -233,7 +226,7 @@ def _fake_token(token): you capture a statement whose names are not in Latin script — the fixture will exercise your boundary logic but will not look like the original. - Stable per distinct input, so a counterparty appearing on two rows still + Stable per distinct input and mask role, so a counterparty on two rows still appears twice — the repeat structure is what mapping and suspense logic reads. Keyed on first-appearance order rather than on the characters, so this is not a cipher over the original text. @@ -253,8 +246,10 @@ def _fake_token(token): per-position alphabets, which is the whole of what shape-preservation allows. """ - if token in _seen: - return _seen[token] + masked = _is_mask(token) or short_mask + key = (token, masked) + if key in _seen: + return _seen[key] # An `X` is only a masking convention when the WHOLE token is the shape the # parsers actually look for. `bank_statement_import` requires # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` @@ -270,7 +265,7 @@ def _fake_token(token): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if _is_mask(token): + if masked: positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token @@ -287,7 +282,7 @@ def _fake_token(token): for alphabet in alphabets: total *= len(alphabet) - shape = _shape_of(token) + shape = _shape_of(token, masked) index = _next.get(shape, 0) candidate = None while index < total: @@ -329,7 +324,7 @@ def _fake_token(token): f"(ALPHA for letters, DIGITS for digits)." ) _next[shape] = index - _seen[token] = candidate + _seen[key] = candidate _taken.add(candidate.upper()) return candidate @@ -358,7 +353,7 @@ def one(match): return NUMERIC_ENTITY.sub(one, text) -def scrub(text): +def scrub(text, *, mask_spans=None): """Sanitise one word's text, preserving XML *syntax*. `&` is one character in the document and four in the file. Splitting on @@ -372,18 +367,20 @@ def scrub(text): customer. A numeric reference is the opposite — see above. """ text = _decode_numeric_entities(text) - parts = STRUCTURAL_ENTITY.split(text) - if len(parts) > 1: - entities = STRUCTURAL_ENTITY.findall(text) - out = [_scrub_plain(parts[0])] - for entity, rest in zip(entities, parts[1:]): - out.append(entity) - out.append(_scrub_plain(rest)) - return "".join(out) - return _scrub_plain(text) - - -def _scrub_plain(text): + spans = _short_mask_spans(text) if mask_spans is None else mask_spans + out, offset = [], 0 + for entity in STRUCTURAL_ENTITY.finditer(text): + local = {(a - offset, b - offset) for a, b in spans + if offset <= a and b <= entity.start()} + out.append(_scrub_plain(text[offset:entity.start()], mask_spans=local)) + out.append(entity.group()) + offset = entity.end() + local = {(a - offset, b - offset) for a, b in spans if offset <= a} + out.append(_scrub_plain(text[offset:], mask_spans=local)) + return "".join(out) + + +def _scrub_plain(text, *, mask_spans=None): if DATE.match(text): return _fake_date(text) if YEAR.match(text): @@ -401,12 +398,13 @@ def _scrub_plain(text): if text not in _days: _days[text] = f"{len(_days) % 28 + 1:02d}" return _days[text] - out = [] + spans = _short_mask_spans(text) if mask_spans is None else mask_spans + out, offset = [], 0 for is_token, piece in _split_tokens(text): if piece in TEMPLATE: out.append(piece) elif is_token: - out.append(_fake_token(piece)) + out.append(_fake_token(piece, short_mask=(offset, offset + len(piece)) in spans)) else: # ASCII punctuation and whitespace only. Nothing reaches this branch # that could be a name, which is the whole change — previously an @@ -416,6 +414,7 @@ def _scrub_plain(text): # raw `&`, `<` or `>` here and the output has to stay XML. out.append("".join(XML_ESCAPES.get(character, character) for character in piece)) + offset += len(piece) return "".join(out) @@ -441,7 +440,7 @@ def _scrub_plain(text): default. Distinct source tokens map to distinct fabricated ones in first-appearance order, so repeats and name/reference structure survive while the substitution is not a cipher over the original text. Character shape is - preserved — digits stay digits, a run of X stays a run of X — because the + preserved — digits stay digits and qualified mask runs keep their Xs — because the parsers find the end of a counterparty name by recognising the shape of the field after it. @@ -476,6 +475,47 @@ def _kept_words(pages, keep): yield head, words +def _page_short_masks(page): + """Locate wrapped SBI IMPS mask fields using the existing bank profile. + + Keep headers, adjacent columns and separate transactions out of the context. + The captured SBI field wraps over three narration lines, so a whole-word + predicate alone cannot preserve it without also preserving ordinary names. + """ + from bank_statement_import import SBI, _lines, _table_top + + profile = SBI() + lines = _lines(page) + top = _table_top(lines, profile) + if top is None: + return {} + rows, current = [], None + for y, group in lines: + if y <= top: + continue + cells = {} + for x0, _, x1, _, text in group: + cells.setdefault(profile.column_of(x0, x1), []).append((x0, x1, text)) + if profile.is_row_start(cells): + current = [] + rows.append(current) + if current is not None: + current.extend(word for word in group + if profile.column_of(word[0], word[2]) == "narr") + found = {} + for row in rows: + joined = " ".join(word[4] for word in row) + spans = _short_mask_spans(joined) + offset = 0 + for x0, y0, x1, y1, text in row: + local = {(a - offset, b - offset) for a, b in spans + if offset <= a and b <= offset + len(text)} + if local: + found[(x0, y0, x1, y1)] = local + offset += len(text) + 1 + return found + + def main(source, destination, keep, bank): """keep: [(page_index, [(y_min, y_max), ...]), ...] regions to retain.""" # `pdftotext` emits UTF-8. `read_text()` without an encoding decodes with @@ -486,6 +526,7 @@ def main(source, destination, keep, bank): # call `_scrub_plain` with strings that are already decoded. pages = pathlib.Path(source).read_text(encoding="utf-8").split("' - f'{scrub(body)}' for x0, y0, x1, y1, body in words) + f'{scrub(body, mask_spans=context.get(tuple(map(float, (x0, y0, x1, y1))), set()))}' + for x0, y0, x1, y1, body in words) + "\n" - for head, words in regions + for (head, words), context in zip(regions, contexts) ] pathlib.Path(destination).write_text( BANNER_TEMPLATE.format(bank=bank) + "\n".join(chunks) + "\n", encoding="utf-8") diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index e432be2e..7b44a449 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -213,14 +213,10 @@ def leaked(source, output): # ...while the shape the parsers actually look for is still structure, and # survives. `bank_statement_import` calls something a masked account only when # the whole token matches `[Xx]{4,}\d*`, so that is the one test applied here. -# The short forms carry the second parser path. `bank_statement_import` reads -# `[Xx]+\d+` inside a UPI/IMPS reference, so `XX1234` is the bank's masking even -# though it has fewer than MASK_MIN_XS characters — requiring four everywhere -# fabricated it to `ZZ1111` and destroyed a shape the fixture exists to keep. -# Without these three rows the union predicate has no test at all: reverting it -# to the four-X rule left the whole suite green. +# A short X+digits token is ordinary customer data without its IMPS field +# context. The captured wrapped SBI field is checked through main() below. for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), - ("XX1234", True), ("X99", True), ("xx7", True), + ("XX1234", False), ("X99", False), ("xx7", False), ("XX", False), ("X", False), ("XXX", False)): out = load()._scrub_plain(mask) # For a mask, every X position must survive verbatim and every digit @@ -237,6 +233,49 @@ def leaked(source, output): held, f"{mask} -> {out}", ) +# The same token can be a structural mask in a bank reference and customer +# data elsewhere. Memoisation must keep those roles separate in both orders. +for mask_first in (True, False): + fresh = load() + source = "IMPS/111112 111113/ZZY- X99-NAME" + for value in ((source, "X99") if mask_first else ("X99", source)): + out = fresh.scrub(value) + if value == "X99": + check(f"ordinary X99 is fabricated after mask={mask_first}", + "X" not in out.upper(), out) + else: + check(f"qualified short IMPS mask survives first={mask_first}", + bool(re.search(r"/[^/]+- X\d+-", out)), out) +for text in ("UPI/X99", "IMPS/X99/NAME", "IMPS/123/NAME-X99", + "IMPS/123/NAME-X99Z-OTHER", "IMPS/123/NAME-OTHER/X99-END"): + check("short masks outside the exact field are fabricated: " + text, + "X" not in load().scrub(text).upper()) + +# Captured source geometry proves the wrapped context without inventing a new +# bank fixture. Pin the mask-bearing box and mutate only its classification. +fixture = pathlib.Path(__file__).with_name("fixtures") / "sbi-bbox-capture.xml" +fresh = load() +page = fixture.read_text(encoding="utf-8").split(" Date: Sat, 12 Sep 2026 16:49:40 +0530 Subject: [PATCH 16/83] docs(tally): require intent fields before binding existing masters --- docs/tally/PROMPT_PLAYBOOK.md | 42 ++++++++++++------- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 7 ++-- .../compatibility/compatibility-matrix.json | 2 +- .../compatibility/compatibility-surface.json | 4 +- 4 files changed, 32 insertions(+), 23 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index ea5eff89..5f820a51 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -299,21 +299,22 @@ Implement: fix-it list. 5. Encoding/normalization hardening: UTF-8/UTF-16LE/BOM fixtures; non-English (Devanagari, Gujarati, Tamil) company/ledger/narration - fixtures in the simulator corpus; name-key matching via §9.4b's - `accepts(candidate, tally_name)` predicate — directional ASCII case - folding only (`candidate == ascii_lower(tally_name)`; the reverse, an - uppercase candidate against a lowercase master, was never measured and - must not be accepted). Never a symmetric case-insensitive collation: - that accepts the unverified direction and can bind a voucher to the - wrong master. - And the same SCOPE GATE that governs Phase 4 step 4 governs here: - §9.4b's case and separator rows sit on §0's **Edit Log 7.0 - Educational** baseline. Where this phase reads a licensed instance, - match on exact codepoints; widen to `accepts()` only where a licensed - capture has qualified it — §9.4d does exactly that for LEDGERS on - licensed 7.1, and for nothing else. A compatibility result cannot - qualify it: that evidence is a live-READ receipt and establishes no - write behaviour (`compatibility/README`). + fixtures in the simulator corpus. Scope name-key resolution by SKU and + entity class, using the shared ADR 0016 binder: + - On §0's Edit Log 7.0 Educational baseline, §9.4b's + `accepts(candidate, tally_name)` permits only the measured directional + transformations; do not infer their reverse. + - For LEDGERS on licensed 7.1, use §9.4d's measured canonical fold: + ASCII case; space, hyphen and slash as separators; surrounding + whitespace removed; internal runs collapsed; their measured + compositions. Preserve every other codepoint, including NFC/NFD, + en dash and underscore distinctions. This scope permits the full + §9.4d fold, not just §9.4b's directional subset. + - Other licensed scopes remain exact-codepoint unless a capture + qualifies their particular rule. Compatibility live-READ receipts + do not establish write behaviour (`compatibility/README`). + Resolution still requires unique identity evidence. Never coalesce + distinct catalogue or mirror identities merely because a fold agrees. A read is not a safe place to be wrong about this — the mirror rows built here are what later binding decisions resolve against, so a fold that merges two masters here merges them everywhere downstream. @@ -625,11 +626,20 @@ Implement — write core (masters): does not establish either. The quiet-company Journal preview in issue #239 is not evidence qualifying this master-create flow. Three outcomes, never two: - **bind** to an exact-codepoint match without creating or altering it; + **bind** to an exact-codepoint match without creating or altering it + only after fresh same-company readback also matches every relevant + approved master field (including parent/group, GST configuration and + opening balance). Missing/unqualified field evidence or any difference + is a visible intent conflict: retain the unresolved proposal and block + dependent voucher dispatch for manual resolution. A name match alone + must never silently discard approved fields; **create** only after the completeness and mutation-time prerequisites above are qualified AND no existing master collides under the detector below; otherwise **REFUSE and retain the unresolved proposal.** + Regression acceptance includes an exact-name master with differing + approved fields and an unreadable required field: both remain visibly + unresolved and neither creates, alters, nor enables a dependent write. THE DETECTOR IS NOT THE BINDER AND MUST BE WIDER THAN IT. §9.4b's `accepts()` is DIRECTIONAL — for a requested `FOO` against an existing `foo`, `accepts(FOO, foo)` is false — so reusing it as the diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index dd2a8a18..049e4f50 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1402,10 +1402,9 @@ habits, not against real operator input. ### 9.5 Identity after write -**VERIFIED.** `LASTMID` is **0** on successful master creates — unusable for master identity; -read masters back by name. `LASTVCHID` is populated for vouchers and usable, -subject to a foreign-writer cross-check. `LASTVCHID` also accepts non-numeric text without -error when parsed back, so validate it. +**VERIFIED.** `LASTMID` is **0** on successful master creates, so this counter does not +identify the created master. `LASTVCHID` is populated for vouchers. Non-numeric +`LASTVCHID` text is also accepted without error when parsed back. For the implementation's readback identity policy, see `IMPLEMENTATION_GUIDE.md` §3.6 and `PROMPT_PLAYBOOK.md` Phase 4 step 4. Their prescriptions are separate from this observation. diff --git a/docs/tally/compatibility/compatibility-matrix.json b/docs/tally/compatibility/compatibility-matrix.json index dfad4cbd..8982fb41 100644 --- a/docs/tally/compatibility/compatibility-matrix.json +++ b/docs/tally/compatibility/compatibility-matrix.json @@ -1,7 +1,7 @@ { "schema_version": 1, "bridge_commit_sha": "be1c20cc3fd66fa1ece196505c69f26e555e4b8e", - "compatibility_surface_sha256": "b1e2558948998530615b1c324c9381e4224f5c77578da89fef0e0c8005c5a84c", + "compatibility_surface_sha256": "bcfa141a9dfe2652440067096879b494ad13f60847e4bf8cb337f9bc75ec7ed7", "claims": [ { "claim_id": "erp9-6-6-3-windows-education-xml-one-company", diff --git a/docs/tally/compatibility/compatibility-surface.json b/docs/tally/compatibility/compatibility-surface.json index bf94829d..2b703626 100644 --- a/docs/tally/compatibility/compatibility-surface.json +++ b/docs/tally/compatibility/compatibility-surface.json @@ -35,7 +35,7 @@ }, { "path": "docs/tally/TALLY_PROTOCOL_REFERENCE.md", - "sha256": "9880fabafa88db4f1c9404df37b6b8c0d54af95f036a6800f6582923b6fedf72" + "sha256": "c28cdad0b64ac378c55d1ff9dd84ed1af8ba4a04575acdaced41e64d7357a59b" }, { "path": "docs/tally/compatibility/README.md", @@ -850,5 +850,5 @@ "sha256": "a8ac2714fecf51947f2822c8c46d7ce2e8602c732780ff60566a7771f0836f9a" } ], - "manifest_sha256": "b1e2558948998530615b1c324c9381e4224f5c77578da89fef0e0c8005c5a84c" + "manifest_sha256": "bcfa141a9dfe2652440067096879b494ad13f60847e4bf8cb337f9bc75ec7ed7" } \ No newline at end of file From 743af4af45da5ce7266d8af80084df848a80dd4c Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 16:50:25 +0530 Subject: [PATCH 17/83] fix(scripts): withhold mask role when capture context is omitted --- scripts/sanitise-bbox-capture.py | 7 ++++++- scripts/sanitise-bbox-capture.test.py | 10 ++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index b966a5c5..44832731 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -526,7 +526,12 @@ def main(source, destination, keep, bank): # call `_scrub_plain` with strings that are already decoded. pages = pathlib.Path(source).read_text(encoding="utf-8").split("' + f'{body}' for x0, y0, x1, y1, body in words)) + for _, words in regions] # Two passes, and the first one has to be complete before the second starts. # A replacement is only safe once the allocator knows every token the # capture contains: otherwise a fabricated value can equal some *other* diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index 7b44a449..12043937 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -268,6 +268,16 @@ def leaked(source, output): for m in fresh.WORD.finditer(destination.read_text())} check("main preserves the captured wrapped short mask's structural Xs", bool(re.fullmatch(r"XX\d{3}-", words[box])), repr(words[box])) +# Cropping away the reference must also remove its mask authority. The input +# is still the unchanged real capture; only the CLI's retained region varies. +with tempfile.TemporaryDirectory() as directory: + destination = pathlib.Path(directory) / "short-only.xml" + fresh = load() + with contextlib.redirect_stdout(io.StringIO()): + fresh.main(str(fixture), str(destination), [(0, [(701, 712.5)])], "SBI") + words = [match.group(5) for match in fresh.WORD.finditer(destination.read_text())] + check("mask-only capture crop fabricates short Xs without retained context", + len(words) == 1 and "X" not in words[0].upper(), repr(words)) # Same captured word in an adjacent column must not receive mask authority. shifted = page.replace('xMin="143.660000" yMin="701.384000" xMax="183.680000"', 'xMin="222.900000" yMin="701.384000" xMax="262.920000"') From 18d8b72df9068a710aea4924c285f4e3bcf1abb2 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 16:53:26 +0530 Subject: [PATCH 18/83] fix(scripts): retain mask context independently of table furniture --- scripts/sanitise-bbox-capture.py | 38 ++++++++++++++++----------- scripts/sanitise-bbox-capture.test.py | 11 ++++++++ 2 files changed, 34 insertions(+), 15 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 44832731..0bc42561 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -475,7 +475,7 @@ def _kept_words(pages, keep): yield head, words -def _page_short_masks(page): +def _page_short_masks(page, retained=None): """Locate wrapped SBI IMPS mask fields using the existing bank profile. Keep headers, adjacent columns and separate transactions out of the context. @@ -505,14 +505,23 @@ def _page_short_masks(page): found = {} for row in rows: joined = " ".join(word[4] for word in row) - spans = _short_mask_spans(joined) - offset = 0 - for x0, y0, x1, y1, text in row: - local = {(a - offset, b - offset) for a, b in spans - if offset <= a and b <= offset + len(text)} - if local: - found[(x0, y0, x1, y1)] = local - offset += len(text) + 1 + offsets, offset = [], 0 + for word in row: + offsets.append((offset, offset + len(word[4]), word[:4])) + offset += len(word[4]) + 1 + for match in SHORT_IMPS_MASK.finditer(joined): + # Full-page geometry establishes row membership, but every word + # carrying the qualifying IMPS field must survive the crop. Table + # furniture need not be emitted just to preserve a transaction. + support = {box for start, end, box in offsets + if start < match.end() and end > match.start()} + if retained is not None and not support <= retained: + continue + a, b = match.span(1) + for start, end, box in offsets: + if start <= a and b <= end: + found.setdefault(box, set()).add((a - start, b - start)) + return found @@ -526,12 +535,11 @@ def main(source, destination, keep, bank): # call `_scrub_plain` with strings that are already decoded. pages = pathlib.Path(source).read_text(encoding="utf-8").split("' - f'{body}' for x0, y0, x1, y1, body in words)) - for _, words in regions] + # Qualify against source row geometry, then require the field's context + # words to survive the crop. A mask-only selection carries no authority. + contexts = [_page_short_masks(pages[index], { + tuple(map(float, word[:4])) for word in words}) + for (index, _), (_, words) in zip(keep, regions)] # Two passes, and the first one has to be complete before the second starts. # A replacement is only safe once the allocator knows every token the # capture contains: otherwise a fabricated value can equal some *other* diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index 12043937..38ff915a 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -278,6 +278,17 @@ def leaked(source, output): words = [match.group(5) for match in fresh.WORD.finditer(destination.read_text())] check("mask-only capture crop fabricates short Xs without retained context", len(words) == 1 and "X" not in words[0].upper(), repr(words)) +# A transaction crop retains the IMPS field while omitting table furniture. +# Source geometry still establishes the row; all field words remain emitted. +with tempfile.TemporaryDirectory() as directory: + destination = pathlib.Path(directory) / "transaction.xml" + fresh = load() + with contextlib.redirect_stdout(io.StringIO()): + fresh.main(str(fixture), str(destination), [(0, [(665, 736.5)])], "SBI") + words = {tuple(map(float, m.groups()[:4])): m.group(5) + for m in fresh.WORD.finditer(destination.read_text())} + check("transaction crop preserves its retained IMPS mask without a header", + bool(re.fullmatch(r"XX\d{3}-", words[box])), repr(words[box])) # Same captured word in an adjacent column must not receive mask authority. shifted = page.replace('xMin="143.660000" yMin="701.384000" xMax="183.680000"', 'xMin="222.900000" yMin="701.384000" xMax="262.920000"') From 4efd8db1a3e3659aab693d1936edbde1572ea8e6 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 17:00:40 +0530 Subject: [PATCH 19/83] docs(tally): align scoped qualification and held recovery across plans --- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 12 +++++++++-- docs/tally/PROMPT_PLAYBOOK.md | 7 ++++--- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 20 ++++++++++--------- .../compatibility/compatibility-matrix.json | 2 +- .../compatibility/compatibility-surface.json | 4 ++-- 5 files changed, 28 insertions(+), 17 deletions(-) diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index acee7bbc..b718b578 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -409,11 +409,19 @@ Re-sending the identical voucher payload, same `VOUCHERNUMBER`, produced `CREATE Re-sending the identical ledger `ACTION="Create"` returned `CREATED=0, ALTERED=1` — no error. A retry silently **overwrites** the existing master with the retry payload, including any defaulted fields. -**Adds to §6:** master creates require a pre-existence read before dispatch, and `CREATED` vs `ALTERED` must persist as distinct outbox outcomes. "No duplicate was made" is not the same as "my create succeeded." +**DEVIATION 2026-09-12 — amends §6:** a pre-existence read is necessary but does not +authorize master creation. Require a qualified Complete catalogue for the exact company and +master class, plus a qualified mutation-time condition or proven exclusive-write window. +Concurrent automatic creation remains UNQUALIFIED until those prerequisites hold; another +ordinary pre-read or operator confirmation is insufficient. Apply the full creation/bind/refusal +contract in `PROMPT_PLAYBOOK.md` Phase 4 step 3a, including approved-field equality before +binding an existing name. Retain unresolved proposals and block dependent writes when evidence +is missing. Persist `CREATED` and `ALTERED` as distinct outcomes; avoiding a duplicate does not +establish that the requested create succeeded. ### 8.6 `LASTMID` is 0 on successful master creates; `LASTVCHID` works -Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen only where a licensed capture has qualified it as a **write**: §9.4d does so for **ledgers** on licensed 7.1, by importing vouchers naming folded spellings and reading the day book back, and for no other master type. A compatibility result cannot widen it — that evidence is a live-**read** receipt and `compatibility/README` says it never establishes any write behaviour. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. +Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen only where a licensed capture has qualified it as a **write**: §9.4d does so for **ledgers** on licensed 7.1 **Silver**, with `education_mode=false`, by importing vouchers naming folded spellings and reading the day book back, and for no other master type or licence tier. Gold and other unqualified tiers remain exact-only. A compatibility result cannot widen it — that evidence is a live-**read** receipt and `compatibility/README` says it never establishes any write behaviour. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints. This deviation removes normalization; it does not decide whether any case or separator fold is permitted on top — that is the SCOPE GATE's question, answered above, and on an unqualified licensed SKU the answer is exact codepoints and nothing else. diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 5f820a51..824caa96 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -304,7 +304,7 @@ Implement: - On §0's Edit Log 7.0 Educational baseline, §9.4b's `accepts(candidate, tally_name)` permits only the measured directional transformations; do not infer their reverse. - - For LEDGERS on licensed 7.1, use §9.4d's measured canonical fold: + - For LEDGERS on licensed 7.1 Silver (`education_mode=false`), use §9.4d's measured canonical fold: ASCII case; space, hyphen and slash as separators; surrounding whitespace removed; internal runs collapsed; their measured compositions. Preserve every other codepoint, including NFC/NFD, @@ -680,8 +680,9 @@ Implement — write core (masters): spellings and reading the **day book** back to see which master each posted against. That is observed write behaviour on the SKU this project writes to, for **ledgers**. - So: for **ledgers on licensed 7.1**, match under §9.4d's measured - rows. For **every other master type** — stock items, groups, voucher + So: for **ledgers on licensed 7.1 Silver (`education_mode=false`)**, + match under §9.4d's measured rows. Gold, other tiers and unqualified + versions remain exact-codepoint only. For **every other master type** — stock items, groups, voucher types — §9.4d measured nothing, so match on **exact codepoints** and let a case or separator difference fail loudly. **A compatibility result cannot widen this.** `compatibility/README` diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index 049e4f50..ecfcad9d 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -908,7 +908,9 @@ voucher by reusing a key. **Not the outbox, though.** `REMOTEID` prevents a duplicate; it does not tell you, after a crash, *what you sent*. The `REMOTEID` **attribute** does not echo the client key on readback (below), so -a resend is only safe while the exact key and payload are still on disk. Be precise about the +the exact key and payload must remain on disk for read-only reconciliation. They do not +authorize a resend after an unknown outcome; restart behaviour remains unqualified. See +`IMPLEMENTATION_GUIDE.md` §§3.3–3.5 and the playbook's held recovery flow. Be precise about the field: the key itself does survive anywhere Tally does not own — a narration marker comes back — and a categorical "Tally does not return the key" would send recovery work to discard the one attribution channel that works. The durable dispatch intent stays — see the @@ -1165,14 +1167,14 @@ symmetry is exactly the property the separator result does not have. | --- | --- | | ASCII case folding | **VERIFIED** — lowercase matched | | supplying a **space** where the master has a **hyphen** | **VERIFIED** — `BRIDGE PROBE LEDGER A` matched `BRIDGE-PROBE-LEDGER-A` | -| supplying a **hyphen** where the master has a **space** | **UNVERIFIED here** — the reverse direction was never sent on this SKU. Measured **matched** on licensed 7.1, §9.4d | +| supplying a **hyphen** where the master has a **space** | **UNVERIFIED here** — the reverse direction was never sent on this SKU. Measured **matched** on licensed 7.1 Silver, §9.4d | | one trailing space ignored | **VERIFIED** | | **two or more** trailing spaces ignored | **UNVERIFIED** — only one was sent | -| *leading* whitespace ignored | **UNVERIFIED here**. Measured **matched** on licensed 7.1, §9.4d | -| runs of internal whitespace collapsed to one | **UNVERIFIED here** — only a single space was tested. Measured **matched** on licensed 7.1, §9.4d | +| *leading* whitespace ignored | **UNVERIFIED here**. Measured **matched** on licensed 7.1 Silver, §9.4d | +| runs of internal whitespace collapsed to one | **UNVERIFIED here** — only a single space was tested. Measured **matched** on licensed 7.1 Silver, §9.4d | | non-ASCII case folding (Devanagari, Tamil, Bengali, Turkish dotted I) | **UNVERIFIED** | | **Unicode canonical equivalence (NFC/NFD)** | **MEASURED — folding it is wrong.** See below. | -| any other separator (underscore, en dash, `/`) treated as a space | **UNVERIFIED here**, and §9.4d splits it on licensed 7.1: `/` **matched**, underscore and en dash **rejected**. Not one row — do not fold them together | +| any other separator (underscore, en dash, `/`) treated as a space | **UNVERIFIED here**, and §9.4d splits it on licensed 7.1 Silver: `/` **matched**, underscore and en dash **rejected**. Not one row — do not fold them together | **A wider result exists for a different SKU.** §9.4d re-ran this measurement on **licensed TallyPrime 7.1** and found the gateway folds more than these rows establish. It is a separate @@ -1290,7 +1292,7 @@ voucher each, then the **day book was read back** to record which master each vo posted against — the counters alone would not have said. Every created voucher was then deleted by `REMOTEID` and the day read back empty (eight from the first run, two from the second). -| Supplied against a live master | Licensed 7.1 | §9.4b on Educational | +| Supplied against a live master | Licensed 7.1 Silver | §9.4b on Educational | | --- | --- | --- | | exact | **matched** | matched | | ASCII lowercase | **matched** | matched | @@ -1322,7 +1324,7 @@ ledger present in every company: | `Profit & Loss` | **rejected** — a missing suffix word | | `Profit & Loss A/c AND CO` | **rejected** — an added suffix word, what the first run really sent | -So §9.4b's abbreviation findings hold on licensed 7.1 as well, and this section now says which +So §9.4b's abbreviation findings hold on licensed 7.1 Silver as well, and this section now says which of them it measured rather than which it meant to. **Composition was measured separately, because twelve single-axis results do not license it.** @@ -1346,7 +1348,7 @@ All eight posted against the intended master, confirmed by day-book readback. ** compose**, and a canonical form applying every measured transformation before comparing is licensed by measurement rather than by extrapolation from the single-axis rows. -**What this says.** On licensed 7.1, Tally treats **space, hyphen and slash** as interchangeable +**What this says.** On licensed 7.1 Silver, Tally treats **space, hyphen and slash** as interchangeable separators, collapses internal whitespace runs, ignores leading and trailing whitespace, folds **ASCII** case, and is otherwise **exact on codepoints**. @@ -1406,7 +1408,7 @@ habits, not against real operator input. identify the created master. `LASTVCHID` is populated for vouchers. Non-numeric `LASTVCHID` text is also accepted without error when parsed back. -For the implementation's readback identity policy, see `IMPLEMENTATION_GUIDE.md` §3.6 +For the implementation's readback identity policy, see `IMPLEMENTATION_GUIDE.md` §3.5 and `PROMPT_PLAYBOOK.md` Phase 4 step 4. Their prescriptions are separate from this observation. --- diff --git a/docs/tally/compatibility/compatibility-matrix.json b/docs/tally/compatibility/compatibility-matrix.json index 8982fb41..f17ea790 100644 --- a/docs/tally/compatibility/compatibility-matrix.json +++ b/docs/tally/compatibility/compatibility-matrix.json @@ -1,7 +1,7 @@ { "schema_version": 1, "bridge_commit_sha": "be1c20cc3fd66fa1ece196505c69f26e555e4b8e", - "compatibility_surface_sha256": "bcfa141a9dfe2652440067096879b494ad13f60847e4bf8cb337f9bc75ec7ed7", + "compatibility_surface_sha256": "1735f0de353bd9799ef93c4055263a4e84c9121bd07ec4efb2955db1e2a4a912", "claims": [ { "claim_id": "erp9-6-6-3-windows-education-xml-one-company", diff --git a/docs/tally/compatibility/compatibility-surface.json b/docs/tally/compatibility/compatibility-surface.json index 2b703626..3b26e9d9 100644 --- a/docs/tally/compatibility/compatibility-surface.json +++ b/docs/tally/compatibility/compatibility-surface.json @@ -35,7 +35,7 @@ }, { "path": "docs/tally/TALLY_PROTOCOL_REFERENCE.md", - "sha256": "c28cdad0b64ac378c55d1ff9dd84ed1af8ba4a04575acdaced41e64d7357a59b" + "sha256": "76f6641bfc0027a5a11c3d57b598fee7395eacf2d955738a6ba159c25b8e4c0c" }, { "path": "docs/tally/compatibility/README.md", @@ -850,5 +850,5 @@ "sha256": "a8ac2714fecf51947f2822c8c46d7ce2e8602c732780ff60566a7771f0836f9a" } ], - "manifest_sha256": "bcfa141a9dfe2652440067096879b494ad13f60847e4bf8cb337f9bc75ec7ed7" + "manifest_sha256": "1735f0de353bd9799ef93c4055263a4e84c9121bd07ec4efb2955db1e2a4a912" } \ No newline at end of file From 3e23b3ddf2eeb8f927442097381ff2fd416dcd06 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 17:22:22 +0530 Subject: [PATCH 20/83] docs: keep slash matching directional --- docs/tally/IMPLEMENTATION_GUIDE.md | 10 +++++++--- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 2 +- docs/tally/PROMPT_PLAYBOOK.md | 24 ++++++++++++++---------- 3 files changed, 22 insertions(+), 14 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index fd8d77c2..c7f0e58a 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -612,7 +612,7 @@ proof-of-post claim must account for that — carry your own marker in a field T or when the payload differs from the original (partial update semantics). Also untested on licensed or standard TallyPrime. -### 3.3b Master-name matching: case- and separator-insensitive, otherwise exact +### 3.3b Master-name matching: directional alternatives, otherwise exact **VERIFIED 2026-07-30**, against a ledger named `BRIDGE-PROBE-LEDGER-A` and one named `ZZ Ram & Sons Pvt Ltd`: @@ -628,7 +628,11 @@ licensed or standard TallyPrime. | `ZZ Ram & Son Pvt Ltd` (singular for plural) | **rejected** | | entirely different name | **rejected** | -So Tally normalises **case and separators** and is otherwise **exact on letters**. +These rows establish only the supplied candidate against the recorded master; +they do **not** establish a symmetric case/separator normalizer or canonical +fold. Automatic binding is exact-codepoint unless the authoritative protocol +records the particular directional comparison. In particular, do not infer a +slash rule from this baseline. > **Promoted to `TALLY_PROTOCOL_REFERENCE.md` §9.4b**, which is where observed gateway behaviour > belongs and which carries the consequences for a writer. This entry stays as the measurement @@ -1171,7 +1175,7 @@ zero; fail closed or quarantine. | Modal dialog | Gateway blocked until a human clicks | §5.1 | | `ClosingBalance` read as a period figure | Wrong balance, presented as correct | §6.4 | | `ACTION="Alter"` + `REMOTEID` | Creates a duplicate. `Create` upserts a **byte-identical** repeat; a *corrected* payload is UNVERIFIED and may overwrite, partially update or duplicate | §3.3a | -| Master name differing by more than case/separators | Voucher rejected, master NOT auto-created | §3.3b | +| Master name outside an exact or qualified directional comparison | Voucher rejected, master NOT auto-created | §3.3b | | Omitting `BILLALLOCATIONS.LIST` | Allocation becomes `On Account` with no bill identity | §3.3c | | Self-referential `$$NumItems` in a collection | Gateway hangs, empty reply | §5.3b | | `` used for a per-request constant | Per-row work; request exceeds deadline | §2.3a | diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index b718b578..81b9af7d 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -421,7 +421,7 @@ establish that the requested create succeeded. ### 8.6 `LASTMID` is 0 on successful master creates; `LASTVCHID` works -Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen only where a licensed capture has qualified it as a **write**: §9.4d does so for **ledgers** on licensed 7.1 **Silver**, with `education_mode=false`, by importing vouchers naming folded spellings and reading the day book back, and for no other master type or licence tier. Gold and other unqualified tiers remain exact-only. A compatibility result cannot widen it — that evidence is a live-**read** receipt and `compatibility/README` says it never establishes any write behaviour. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. +Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen only where a licensed capture has qualified a **directional write comparison**: §9.4d does so for **ledgers** on licensed 7.1 **Silver**, with `education_mode=false`, by importing vouchers naming folded spellings and reading the day book back, and for no other master type or licence tier. Its slash row is slash-candidate against space-master only; the reverse is not qualified, so slash-bearing cross-spellings remain exact-only. Gold and other unqualified tiers remain exact-only. A compatibility result cannot widen it — that evidence is a live-**read** receipt and `compatibility/README` says it never establishes any write behaviour. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints. This deviation removes normalization; it does not decide whether any case or separator fold is permitted on top — that is the SCOPE GATE's question, answered above, and on an unqualified licensed SKU the answer is exact codepoints and nothing else. diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 824caa96..ca37f15a 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -304,12 +304,13 @@ Implement: - On §0's Edit Log 7.0 Educational baseline, §9.4b's `accepts(candidate, tally_name)` permits only the measured directional transformations; do not infer their reverse. - - For LEDGERS on licensed 7.1 Silver (`education_mode=false`), use §9.4d's measured canonical fold: - ASCII case; space, hyphen and slash as separators; surrounding - whitespace removed; internal runs collapsed; their measured - compositions. Preserve every other codepoint, including NFC/NFD, - en dash and underscore distinctions. This scope permits the full - §9.4d fold, not just §9.4b's directional subset. + - For LEDGERS on licensed 7.1 Silver (`education_mode=false`), use only + §9.4d's individually measured **directional** comparisons; do not turn + them into a canonical fold. In particular, its slash row supplied a + slash candidate against a space-bearing master. The reverse was not + measured, so slash-bearing cross-spellings remain exact-codepoint only + until both directions are qualified. Preserve every other codepoint, + including NFC/NFD, en dash and underscore distinctions. - Other licensed scopes remain exact-codepoint unless a capture qualifies their particular rule. Compatibility live-READ receipts do not establish write behaviour (`compatibility/README`). @@ -681,10 +682,13 @@ Implement — write core (masters): posted against. That is observed write behaviour on the SKU this project writes to, for **ledgers**. So: for **ledgers on licensed 7.1 Silver (`education_mode=false`)**, - match under §9.4d's measured rows. Gold, other tiers and unqualified - versions remain exact-codepoint only. For **every other master type** — stock items, groups, voucher - types — §9.4d measured nothing, so match on **exact codepoints** and - let a case or separator difference fail loudly. + match only under §9.4d's individually measured directional rows. Its + slash row does not qualify the reverse, so a slash-bearing cross-spelling + is **exact-codepoint only** and fails loudly until both directions are + captured. Gold, other tiers and unqualified versions remain exact-codepoint + only. For **every other master type** — stock items, groups, voucher types + — §9.4d measured nothing, so match on **exact codepoints** and let a case + or separator difference fail loudly. **A compatibility result cannot widen this.** `compatibility/README` defines a cell's evidence as a live-**read** receipt and says it "never establishes ... any write behavior". An earlier revision of From a26d98586d18a125b06ba318bdd4b0a1d9c11334 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 17:25:43 +0530 Subject: [PATCH 21/83] docs: limit Silver slash comparison direction --- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 33 ++++++++++--------- .../compatibility/compatibility-matrix.json | 2 +- .../compatibility/compatibility-surface.json | 4 +-- 3 files changed, 21 insertions(+), 18 deletions(-) diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index ecfcad9d..a662e6c5 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1344,23 +1344,26 @@ more variants, same method, same readback and deletion: | ` mb-pilot/alpha (5550001001) ` | all five at once | **matched** | | ` mb probe ledger a ` against `MB-PROBE-LEDGER-A` | case + space-for-hyphen + surrounding + run | **matched** | -All eight posted against the intended master, confirmed by day-book readback. **So the folds -compose**, and a canonical form applying every measured transformation before comparing is -licensed by measurement rather than by extrapolation from the single-axis rows. - -**What this says.** On licensed 7.1 Silver, Tally treats **space, hyphen and slash** as interchangeable -separators, collapses internal whitespace runs, ignores leading and trailing whitespace, folds -**ASCII** case, and is otherwise **exact on codepoints**. - -> **RULE: separators fold, and the set is `space`, `-`, `/` — nothing else.** An en dash and an -> underscore are ordinary characters to Tally and are **not** separators, so a fold that treats -> "punctuation" or "separators" as a class is wider than the gateway and will merge masters it -> keeps apart. +All eight posted against the intended master, confirmed by day-book readback. **So the listed +supplied-to-master transformations compose in those measured directions.** That does not license +a canonical form: it compares symmetrically and would assert a reverse comparison the probe did +not send. + +**What this says.** On licensed 7.1 Silver, the recorded supplied-to-master comparisons accept the +listed space/hyphen alternatives and the one **slash-candidate to space-master** alternative, +collapse the measured internal whitespace run, ignore the measured leading and trailing whitespace, +and fold the measured **ASCII** case. The reverse slash comparison was not sent. Cross-spellings +involving slash therefore require exact codepoints unless that direction is separately measured. + +> **RULE: use only the recorded directional alternatives; do not fold separators into a canonical +> form.** The slash result is candidate `/` against master space, not the reverse. An en dash and +> an underscore were rejected in their recorded directions, so a fold that treats punctuation or +> separators as a class is wider than the gateway and will merge masters it keeps apart. That is the trap §9.4b warned about, arriving from the other side: the danger was never only that -a reader would fold too much, it was that "normalises separators" names no particular set. Two of -the four separators tested are folded and two are not, and nothing about their appearance predicts -which. +a reader would fold too much, it was that "normalises separators" hides both the particular +substitutions and their directions. Nothing about a separator's appearance predicts which comparison +the gateway accepts. **Canonical equivalence is still refused**, consistent with the exact-codepoint finding recorded elsewhere in this document: an NFD spelling of an NFC ledger does not resolve. A fold that diff --git a/docs/tally/compatibility/compatibility-matrix.json b/docs/tally/compatibility/compatibility-matrix.json index f17ea790..87fd8b8d 100644 --- a/docs/tally/compatibility/compatibility-matrix.json +++ b/docs/tally/compatibility/compatibility-matrix.json @@ -1,7 +1,7 @@ { "schema_version": 1, "bridge_commit_sha": "be1c20cc3fd66fa1ece196505c69f26e555e4b8e", - "compatibility_surface_sha256": "1735f0de353bd9799ef93c4055263a4e84c9121bd07ec4efb2955db1e2a4a912", + "compatibility_surface_sha256": "f8d3a626673420ca7b27c3f47ec887a333e7ff8d65d6e7fd5299a6af0818a2ea", "claims": [ { "claim_id": "erp9-6-6-3-windows-education-xml-one-company", diff --git a/docs/tally/compatibility/compatibility-surface.json b/docs/tally/compatibility/compatibility-surface.json index 3b26e9d9..e9cc93dc 100644 --- a/docs/tally/compatibility/compatibility-surface.json +++ b/docs/tally/compatibility/compatibility-surface.json @@ -35,7 +35,7 @@ }, { "path": "docs/tally/TALLY_PROTOCOL_REFERENCE.md", - "sha256": "76f6641bfc0027a5a11c3d57b598fee7395eacf2d955738a6ba159c25b8e4c0c" + "sha256": "d3f33fb492294aba6275cf0b26a56a0eb9f5e1e2537bf357bd5e29ba59eb3512" }, { "path": "docs/tally/compatibility/README.md", @@ -850,5 +850,5 @@ "sha256": "a8ac2714fecf51947f2822c8c46d7ce2e8602c732780ff60566a7771f0836f9a" } ], - "manifest_sha256": "1735f0de353bd9799ef93c4055263a4e84c9121bd07ec4efb2955db1e2a4a912" + "manifest_sha256": "f8d3a626673420ca7b27c3f47ec887a333e7ff8d65d6e7fd5299a6af0818a2ea" } \ No newline at end of file From 6e017d2e4b1af3e209749598ba4e5562107eb644 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 05:02:14 +0530 Subject: [PATCH 22/83] docs(tally): four findings left open when #306 was merged too early MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit These arrived on #306 after it was merged. Two are corrections to text I wrote, and one is the same failure in two places: a qualification stated in prose while the thing beside it does the unqualified thing anyway. **The case clause was symmetric while the note admitted it was not.** `ascii_lower(candidate) == ascii_lower(tally_name)` also accepts an UPPERCASE candidate against a lowercase master, a direction never sent — so the symmetric form asserts a second experiment, exactly as a canonical form does for the separator. The note below it said so and the predicate kept the clause. Now `candidate == ascii_lower(tally_name)`, which is what the capture sent: a lowercase candidate against a master carrying uppercase. All four clauses are directional and each is one measured result. **`PROMPT_PLAYBOOK.md` still mandated NFC normalization for name keys**, and it is the active Phase 2 implementation authority — so the reference could carry the MEASURED row saying normalising is wrong while the document people build from told them to normalise. Withdrawn with a dated deviation citing the measurement: an NFD spelling of a UI-created NFC ledger was rejected with `EXCEPTIONS=1` while the NFC spelling created it, and an NFD create read back with identical codepoints. Case folding and the non-Latin fixtures stay; normalisation goes. **`IMPLEMENTATION_GUIDE.md` §3.4 kept the fingerprint alive as duplicate prevention.** My sentence — "it is still true everywhere §3.3a does not reach" — preserved the withdrawn mandate in every case §3.3a excludes, while §3.4a of the same patch says the tuple cannot distinguish a retry from a legitimate identical payment and so prevents no duplicate anywhere. Two paragraphs of one patch contradicting each other is how a withdrawn mandate comes back. What is true outside §3.3a's reach is that there is no proven mechanism there at all, which is narrower and less comfortable. **`IMPROVEMENT_PLAN_2026H2.md` cited both sections wrongly.** The signal table is `IMPLEMENTATION_GUIDE.md` §3.4a, not §6.2 (Unicode round-trip), and the reference's `REMOTEID` rule is §9.3 — `§3.3a` is a section of the guide, not of the reference, so the citation pointed at a section that does not exist. A supersession note that lands readers away from the decision it claims to identify makes the old mandate harder to disambiguate, not easier. Reference resealed: 1 changed file, pins unchanged at 211. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 16 ++++++++++++---- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 7 +++++-- docs/tally/PROMPT_PLAYBOOK.md | 17 ++++++++++++++--- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 16 +++++++++++----- 4 files changed, 42 insertions(+), 14 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index 2b7ecd74..b8fc9797 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -675,10 +675,18 @@ duplicate. So on that path a crash-retry of the *same file* is safe on its own, that used to stand here — that the fingerprint plus an embedded key is "the only thing" preventing a duplicate — is no longer true where §3.3a applies. -It is still true everywhere §3.3a does not reach, and that is most places: a **different** payload -under the same key is untested (it may overwrite, partially update or duplicate), as is any -non-Journal voucher type, any other SKU, and a retry across a Tally restart or a company boundary. -Name which case you are in before relying on either mechanism. +**And it is not true anywhere else either.** An earlier revision of this paragraph — mine — said it +"is still true everywhere §3.3a does not reach", which quietly kept the fingerprint alive as a +duplicate-prevention mechanism in every case §3.3a excludes. §3.4a establishes the opposite: a +`(date, amount, ledger-set, voucher-type)` tuple **cannot** distinguish a retry from a legitimate +identical payment, so it prevents no duplicate anywhere. Two paragraphs of one patch contradicting +each other is how a withdrawn mandate comes back. + +What is actually true outside §3.3a's reach is narrower and less comfortable: **there is no proven +duplicate-prevention mechanism there at all.** A **different** payload under the same key is +untested (it may overwrite, partially update or duplicate), as is any non-Journal voucher type, any +other SKU, and a retry across a Tally restart or a company boundary. Name which case you are in, +and where it is not §3.3a's, stop and involve a human rather than reaching for the tuple. ### 3.4a Undefined UDF fields are silently discarded — **the plan's primary idempotency key does not work as written** diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index cfa20791..39c9ba50 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -734,8 +734,11 @@ not say so. That is the failure this plan is least able to detect, because nothi no counter is wrong. The correction, with the full table of what each signal does support, is in -[`IMPLEMENTATION_GUIDE.md`](./IMPLEMENTATION_GUIDE.md) §6.2, and -`TALLY_PROTOCOL_REFERENCE.md` §3.3a states the same rule from the other side: +[`IMPLEMENTATION_GUIDE.md`](./IMPLEMENTATION_GUIDE.md) **§3.4a** — §6.2 is the Unicode +round-trip result and carries none of this — and +[`TALLY_PROTOCOL_REFERENCE.md`](./TALLY_PROTOCOL_REFERENCE.md) **§9.3** states the same rule from +the other side. (`§3.3a` is a section of the implementation guide, not of the reference; citing it +under the wrong document sent readers to a section that does not exist.) | Signal | What it supports | | --- | --- | diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 2211d14b..c8515905 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -298,9 +298,20 @@ Implement: fix-it list. 5. Encoding/normalization hardening: UTF-8/UTF-16LE/BOM fixtures; non-English (Devanagari, Gujarati, Tamil) company/ledger/narration - fixtures in the simulator corpus; NFC normalization + case-insensitive - collation for name keys (Tally name uniqueness is effectively - case-insensitive). + fixtures in the simulator corpus; case-insensitive collation for name keys + (Tally name uniqueness is effectively case-insensitive). + + **DEVIATION 2026-09-12 — NFC normalization for name keys is withdrawn.** + This item previously required it. `TALLY_PROTOCOL_REFERENCE.md` §9.4b records + the measurement: a voucher naming a UI-created **NFC** ledger in its + canonically equivalent **NFD** spelling was rejected — `EXCEPTIONS=1`, + `LINEERROR` saying the ledger does not exist — while the NFC spelling created + it, and an NFD create read back with identical NFD codepoints. **Tally matches + and stores exact codepoints.** Normalising before comparing therefore resolves + a name onto a master Tally itself keeps apart, which is a silent misbinding. + Keep the non-Latin fixtures and the case folding; do **not** normalise. It is + the only transformation in that table with evidence pointing the wrong way, + which is why it needs a deviation here rather than a note. 6. Migration: versioned mirror schema evolution for the new fields (voucher lines, bill allocations, inventory lines, tax lines) with rollback notes. diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index 56fed873..2e683e78 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1120,7 +1120,10 @@ accepts(candidate, tally_name): # `tally_name` is the spelling Tally holds. Each line is one measured # result. Do not compose them; do not add a line without a capture. return candidate == tally_name # exact — VERIFIED - or ascii_lower(candidate) == ascii_lower(tally_name) # ASCII case — VERIFIED, see below + or candidate == ascii_lower(tally_name) # candidate is the master lowercased — VERIFIED + # NOT included: ascii_lower(candidate) == ascii_lower(tally_name). + # That also accepts an UPPERCASE candidate against a lowercase master, + # a direction never sent. See the third note below. or drop_one_trailing_space(candidate) == tally_name # ONE trailing space — VERIFIED or candidate == tally_name.replace("-", " ") # space for Tally's hyphen — VERIFIED ``` @@ -1133,10 +1136,13 @@ Three things this spelling is careful about, each of which was wrong in an earli - **The separator substitution is applied to `tally_name` only.** `tally_name="A-B"` accepts `candidate="A B"`; `tally_name="A B"` does **not** accept `candidate="A-B"`. That asymmetry is the entire point of the clause and is what a canonical form cannot express. -- **The case clause folds both sides, and that is broader than the capture.** The measurement sent - a lowercase name against a master carrying uppercase; the reverse was not sent. It is written - symmetrically because "Tally folds ASCII case" is the claim the capture supports, but a consumer - relying on the *uppercase-candidate* direction is relying on an inference. Qualify it before +- **The case clause is directional, because the capture was.** The measurement sent a **lowercase** + candidate against a master carrying uppercase. `ascii_lower(candidate) == ascii_lower(tally_name)` + also accepts an **uppercase** candidate against a lowercase master, which was never sent — so the + symmetric form asserts a second experiment, exactly as a canonical form does for the separator. + An earlier draft admitted that in this note and left the symmetric clause in the predicate + anyway; a qualification in the prose does not qualify the code beside it. Written as + `candidate == ascii_lower(tally_name)`, the predicate now says only what was sent. Qualify it before building on it. If a further direction is later measured, one clause is added and the table row changes. Until From bc36768dd082daa39a2a67837b22da76d385740c Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:16:56 +0530 Subject: [PATCH 23/83] docs(tally): directional case fold, NFC withdrawal reaches every site MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five review findings, docs-only. Defect A (2 findings, P1): PROMPT_PLAYBOOK.md Phase 2 item 5 prescribed a symmetric case-insensitive collation for name keys. §9.4b's accepts(candidate, tally_name) predicate only measured a lowercase candidate against an uppercase master; the reverse was never sent. A symmetric fold accepts that unverified direction and can bind a voucher to the wrong master. Restated as the directional predicate. Defect B (2 findings, P1): the NFC/NFD withdrawal did not reach three other sites that all instructed matching or reading masters back by 'normalized name' without excluding NFC, or flagged NFC normalization only when applied asymmetrically: - PROMPT_PLAYBOOK.md Phase 2 item 5 (paired with the case fix above) - PROMPT_PLAYBOOK.md Phase 4 step 4 (readback verification) - PROMPT_PLAYBOOK.md Phase 2 adversarial review item 4 - IMPROVEMENT_PLAN_2026H2.md §8.6 Each now carries a dated deviation note pointing at TALLY_PROTOCOL_REFERENCE.md §9.4b, in the style already used for the existing 2026-09-11 fingerprint deviation. Over-correction (1 finding, P2): IMPLEMENTATION_GUIDE.md §3.4a claimed no proven duplicate-prevention mechanism exists outside §3.3a's REMOTEID path. TALLY_PROTOCOL_REFERENCE.md §9.8 is VERIFIED that Manual numbering with PREVENTDUPLICATES=Yes cleanly rejects a duplicate (CREATED=0, ALTERED=0, EXCEPTIONS=1) where Automatic numbering silently creates one — a second proven mechanism, already noted in §3.3 of the same file. Narrowed the claim to what's actually unproven (an automatic dedupe decision from the fingerprint tuple) while keeping the correct part: the fingerprint must never auto-suppress. No code changes. TALLY_PROTOCOL_REFERENCE.md (pinned) is untouched; compatibility surface pin count unchanged at 211. Section-number gate scripts pass. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 16 ++++++--- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 4 ++- docs/tally/PROMPT_PLAYBOOK.md | 51 +++++++++++++++++---------- 3 files changed, 47 insertions(+), 24 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index b8fc9797..f6fe5c8c 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -781,11 +781,17 @@ fingerprint is **co-primary** rather than secondary. Withdrawn: promoting it doe to do the job. The tuple is identical for a legitimate recurring or same-day repeat payment, so as an automatic dedupe it suppresses real vouchers no matter which tier it is placed in — see §3.4a. -What follows instead is narrower and less comfortable: **there is no proven automatic -write-confirmation mechanism for Phase 4.** `REMOTEID` upsert covers a byte-identical repeat on the -Journal path (§3.3a) and nothing beyond it; a destroyed narration marker leaves a write -unattributable, and the honest response to that is to stop and ask a human, not to substitute a -signal that cannot tell the two cases apart. +What follows instead is narrower and less comfortable: **there is no proven mechanism that lets an +automatic dedupe *decision* be made from the fingerprint tuple.** That is not the same claim as "no +proven duplicate-prevention mechanism outside §3.3a" — it overstates the gap. Two mechanisms are +proven, each with a narrow scope: `REMOTEID` upsert on a byte-identical repeat on the Journal path +(§3.3a), and Manual numbering with `PREVENTDUPLICATES=Yes`, which cleanly rejects a duplicate +voucher number instead of silently creating one — `CREATED=0, ALTERED=0, EXCEPTIONS=1` (§3.3; +[`TALLY_PROTOCOL_REFERENCE.md` §9.8](TALLY_PROTOCOL_REFERENCE.md#98-voucher-numbering-method-changes-everything--use-manual)). +Neither reaches a destroyed narration marker or a differently-numbered duplicate under automatic +numbering; for those cases the honest response is still to stop and ask a human, not to substitute +the fingerprint as an automatic suppressor — it cannot tell a retry from a legitimate second +payment no matter which carrier is missing. ### 3.5 Identity after write diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index 39c9ba50..cd32c9cc 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -413,7 +413,9 @@ Re-sending the identical ledger `ACTION="Create"` returned `CREATED=0, ALTERED=1 ### 8.6 `LASTMID` is 0 on successful master creates; `LASTVCHID` works -Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by normalized name. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. +Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name, matched via `TALLY_PROTOCOL_REFERENCE.md` §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. + +DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints plus only the directional case fold. ### 8.7 AlterID high-water marks move — Drift Sentinel's mechanism is sound diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index c8515905..7d88e065 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -298,20 +298,20 @@ Implement: fix-it list. 5. Encoding/normalization hardening: UTF-8/UTF-16LE/BOM fixtures; non-English (Devanagari, Gujarati, Tamil) company/ledger/narration - fixtures in the simulator corpus; case-insensitive collation for name keys - (Tally name uniqueness is effectively case-insensitive). - - **DEVIATION 2026-09-12 — NFC normalization for name keys is withdrawn.** - This item previously required it. `TALLY_PROTOCOL_REFERENCE.md` §9.4b records - the measurement: a voucher naming a UI-created **NFC** ledger in its - canonically equivalent **NFD** spelling was rejected — `EXCEPTIONS=1`, - `LINEERROR` saying the ledger does not exist — while the NFC spelling created - it, and an NFD create read back with identical NFD codepoints. **Tally matches - and stores exact codepoints.** Normalising before comparing therefore resolves - a name onto a master Tally itself keeps apart, which is a silent misbinding. - Keep the non-Latin fixtures and the case folding; do **not** normalise. It is - the only transformation in that table with evidence pointing the wrong way, - which is why it needs a deviation here rather than a note. + fixtures in the simulator corpus; name-key matching via §9.4b's + `accepts(candidate, tally_name)` predicate — directional ASCII case + folding only (`candidate == ascii_lower(tally_name)`; the reverse, an + uppercase candidate against a lowercase master, was never measured and + must not be accepted). Never a symmetric case-insensitive collation: + that accepts the unverified direction and can bind a voucher to the + wrong master. + DEVIATION 2026-09-12 (TALLY_PROTOCOL_REFERENCE.md §9.4b): NFC + normalization of name keys is WITHDRAWN. §9.4b is MEASURED, not + inferred: an NFD spelling of a UI-created NFC ledger was rejected + (`EXCEPTIONS=1`, ledger does not exist) while the NFC spelling created + it — Tally matches on exact codepoints. Normalizing before comparing + resolves a name onto a master Tally itself keeps apart. Name keys + compare on exact codepoints; do not NFC/NFD-normalize either side. 6. Migration: versioned mirror schema evolution for the new fields (voucher lines, bill allocations, inventory lines, tax lines) with rollback notes. @@ -356,9 +356,16 @@ Hunt specifically for: 3. Amount fidelity: any new tax/inventory line parsed through anything but ExactDecimal; sign conventions (IsDeemedPositive) mishandled on new line types; Dr/Cr balance invariant not re-checked with lines present. -4. Identity/normalization traps: NFC normalization applied on read but not - on the keys used for diffing (same ledger counted twice); case-collation - asymmetry between mirror and reconciliation. +4. Identity/normalization traps: ANY NFC/NFD normalization of name keys, + anywhere in the read or diff path — applied consistently on both reads + and diff keys is still a confirmed finding, not only when applied + asymmetrically (§9.4b: Tally matches exact codepoints; normalizing + resolves a name onto a master Tally itself keeps apart, whether or not + both sides agree). A symmetric case-insensitive collation is likewise + a finding: §9.4b's `accepts(candidate, tally_name)` folds ASCII case + in one direction only (candidate lowered against an uppercase master); + a fold that also accepts an uppercase candidate against a lowercase + master accepts the unverified direction. 5. Bounded-resource regressions: new list explosions (AllInventoryEntries on huge vouchers) versus the 32 MiB response cap — is there a paging or windowing story? Does a capped response get honestly labeled Partial? @@ -574,7 +581,9 @@ Implement — write core (masters): 3. Single-writer actor owns the import surface; reads gated during dispatch→readback windows; queue depth visible. 4. Readback verification: after counters accept, re-export the object - (masters by normalized name; vouchers by LASTVCHID) and + (masters matched by name via §9.4b's `accepts(candidate, tally_name)` + predicate only — directional ASCII case folding; vouchers by + LASTVCHID) and ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign @@ -582,6 +591,12 @@ Implement — write core (masters): else OUTCOME_UNKNOWN. Persist the BridgeID ↔ GUID/MasterID binding. Field-diff readback vs intent; divergence → CONFIRMED_WITH_DIVERGENCE, surfaced in the Gap Map, never silent. + DEVIATION 2026-09-12 (TALLY_PROTOCOL_REFERENCE.md §9.4b): "matched by + name" never means NFC/NFD-normalized. An NFD create read back against + a pre-existing NFC master would resolve as a match and promote the + wrong object to CONFIRMED — §9.4b measured Tally keeping the two + apart. Compare master names on exact codepoints plus only the + directional ASCII-case fold; never normalize either side first. 5. OutcomeUnknown recovery: on restart, DISPATCHING rows → probe by key + fingerprint. A probe MATCH is not itself a confirmation: run the SAME full field-level readback diff as the normal dispatch path (step 4) and From 9f228eec70fae711b4489c9ec8e9dde80f4076bc Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:31:39 +0530 Subject: [PATCH 24/83] docs(tally): scope the name fold to its baseline, keep the one proven dedupe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings left open on #314, each checked against the tree before being accepted. **The name fold is prescribed outside the SKU it was measured on.** Phase 4's readback step told implementers to match masters with §9.4b's `accepts()` predicate. §9.4b's case-folding and hyphen-for-space rows carry no licensed-SKU qualification, so they inherit §0's Edit Log 7.0 Educational baseline; the only row measured on a licensed 7.1 instance is NFC/NFD, and it points the other way — exact codepoints. Phase 4 runs against licensed TallyPrime. Applying an Educational-baseline fold there can bind a write to an account Tally keeps distinct. Phase 4 now matches on exact codepoints and widens only where a licensed capture or the connected instance's compatibility result qualifies the predicate. **The fingerprint correction stopped at the plan.** IMPROVEMENT_PLAN was repointed to §3.4a but Phase 4 item 6 still sent implementers to IMPLEMENTATION_GUIDE §6.2 — the Unicode round-trip result, which says nothing about fingerprints — and therefore away from the signal table and its automatic-suppression constraints. **Withdrawing the tuple withdrew a mechanism it had no claim over.** §3.4's "there is no proven duplicate-prevention mechanism there at all" is absolute, and false: §9.8 is VERIFIED that Manual numbering with `PREVENTDUPLICATES=Yes` preserves the supplied number and rejects a duplicate cleanly (`CREATED=0, ALTERED=0, EXCEPTIONS=1`). The claim is now scoped to what lies outside both §3.3a's REMOTEID path and §9.8's, with a sentence saying explicitly that §9.8 supplies a different mechanism rather than rehabilitating the fingerprint — the tuple stays withdrawn everywhere. Surface resealed (rehash → seal → repoint) after the rebase onto #289; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 16 +++++++++++++--- docs/tally/PROMPT_PLAYBOOK.md | 14 +++++++++++++- 2 files changed, 26 insertions(+), 4 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index f6fe5c8c..2c4a041a 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -682,11 +682,21 @@ duplicate-prevention mechanism in every case §3.3a excludes. §3.4a establishes identical payment, so it prevents no duplicate anywhere. Two paragraphs of one patch contradicting each other is how a withdrawn mandate comes back. -What is actually true outside §3.3a's reach is narrower and less comfortable: **there is no proven -duplicate-prevention mechanism there at all.** A **different** payload under the same key is +What is actually true outside §3.3a's reach is narrower and less comfortable — with **one** +qualified exception. `TALLY_PROTOCOL_REFERENCE.md` §9.8 is VERIFIED: a voucher type set to +**Manual** numbering with `PREVENTDUPLICATES=Yes` preserves the supplied number verbatim and +rejects a duplicate **cleanly** (`CREATED=0, ALTERED=0, EXCEPTIONS=1`). A flow that carries +voucher-number identity under that configuration therefore *does* have a proven mechanism, and this +paragraph must not discard it. + +Outside **both** §3.3a's `REMOTEID` path and §9.8's Manual + `PREVENTDUPLICATES=Yes` path, **there +is no proven duplicate-prevention mechanism at all.** A **different** payload under the same key is untested (it may overwrite, partially update or duplicate), as is any non-Journal voucher type, any other SKU, and a retry across a Tally restart or a company boundary. Name which case you are in, -and where it is not §3.3a's, stop and involve a human rather than reaching for the tuple. +and where it is neither of those two, stop and involve a human rather than reaching for the tuple. + +The tuple is withdrawn in every case. §9.8 does not rehabilitate the fingerprint — it supplies a +*different* mechanism, one that rejects at the gateway instead of guessing at the caller. ### 3.4a Undefined UDF fields are silently discarded — **the plan's primary idempotency key does not work as written** diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 7d88e065..eee46411 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -584,6 +584,16 @@ Implement — write core (masters): (masters matched by name via §9.4b's `accepts(candidate, tally_name)` predicate only — directional ASCII case folding; vouchers by LASTVCHID) and + SCOPE GATE (§9.4b, §0): §9.4b's case-folding and hyphen-for-space rows + were measured on the **Edit Log 7.0 Educational** baseline and carry no + licensed-SKU qualification; only the NFC/NFD row was measured on a + licensed 7.1 instance, and it points the other way (exact codepoints). + Phase 4 runs against licensed TallyPrime, so on a licensed SKU match + master names on **exact codepoints** and let a case or separator + difference fail loudly. Widen to `accepts()` only where a licensed + capture has qualified the predicate, or where the compatibility result + for the connected instance says it holds. A fold applied on an + unqualified SKU can bind a write to an account Tally keeps distinct. ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign @@ -625,7 +635,9 @@ Implement — voucher writes (after masters CONFIRMED-path is soak-tested): tuple cannot tell a retry from a legitimate second payment — a standing instruction, or two invoices settled to one supplier on one day, produce the identical tuple — so automatic suppression is a - silent under-write. See IMPLEMENTATION_GUIDE.md §6.2. + silent under-write. See IMPLEMENTATION_GUIDE.md §3.4a — the signal + table and its automatic-suppression constraints. (§6.2 is the Unicode + round-trip result and says nothing about fingerprints.) 7. Cancel qualified as the compensation primitive (ACTION=Cancel by REMOTEID/GUID). Alter-by-GUID qualified per version; where flaky, the fallback is a Cancel+Create saga bound in one outbox transaction with From e49fc5e59f20782f679e5803742fabb5b0b5dce3 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:49:23 +0530 Subject: [PATCH 25/83] docs(tally): one name-matching rule per step, and let it be the gated one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scope gate added in `40dee282` was inserted *between* two directives that still prescribed the fold unconditionally, so Phase 4 step 4 stated its name-matching rule three times: unqualified before the gate, gated, and unqualified again in the closing deviation. An implementer reading to the end of the step finds the fold reinstated. Codex flagged the third; the first was the same defect and is fixed with it. Step 4 now names the gate as the only matching rule in the step, and the NFC deviation says explicitly that it *removes* normalization without widening what remains — on an unqualified licensed SKU, exact codepoints and nothing else. Phase 2 item 5 carried the same unqualified prescription and was not flagged. Fixed anyway: Phase 2 reads a licensed instance too, and a read is the worse place to be wrong, because the mirror rows built there are what later binding decisions resolve against — a fold that merges two masters on read merges them everywhere downstream. The lesson is the doc-level form of "a fix that adds a branch must assert on the new branch": a qualifier inserted between two unqualified statements qualifies neither. Add the gate, then audit every sibling directive in the same step for whether it inherits it. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index eee46411..e5803e3a 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -305,6 +305,14 @@ Implement: must not be accepted). Never a symmetric case-insensitive collation: that accepts the unverified direction and can bind a voucher to the wrong master. + And the same SCOPE GATE that governs Phase 4 step 4 governs here: + §9.4b's case and separator rows sit on §0's **Edit Log 7.0 + Educational** baseline. Where this phase reads a licensed instance, + match on exact codepoints; widen to `accepts()` only where a licensed + capture or the connected instance's compatibility result qualifies it. + A read is not a safe place to be wrong about this — the mirror rows + built here are what later binding decisions resolve against, so a fold + that merges two masters here merges them everywhere downstream. DEVIATION 2026-09-12 (TALLY_PROTOCOL_REFERENCE.md §9.4b): NFC normalization of name keys is WITHDRAWN. §9.4b is MEASURED, not inferred: an NFD spelling of a UI-created NFC ledger was rejected @@ -581,10 +589,11 @@ Implement — write core (masters): 3. Single-writer actor owns the import surface; reads gated during dispatch→readback windows; queue depth visible. 4. Readback verification: after counters accept, re-export the object - (masters matched by name via §9.4b's `accepts(candidate, tally_name)` - predicate only — directional ASCII case folding; vouchers by + (masters matched by name under the SCOPE GATE below — never by a + broader rule stated anywhere else in this step; vouchers by LASTVCHID) and - SCOPE GATE (§9.4b, §0): §9.4b's case-folding and hyphen-for-space rows + SCOPE GATE (§9.4b, §0) — THE ONLY NAME-MATCHING RULE IN THIS STEP: + §9.4b's case-folding and hyphen-for-space rows were measured on the **Edit Log 7.0 Educational** baseline and carry no licensed-SKU qualification; only the NFC/NFD row was measured on a licensed 7.1 instance, and it points the other way (exact codepoints). @@ -605,8 +614,11 @@ Implement — write core (masters): name" never means NFC/NFD-normalized. An NFD create read back against a pre-existing NFC master would resolve as a match and promote the wrong object to CONFIRMED — §9.4b measured Tally keeping the two - apart. Compare master names on exact codepoints plus only the - directional ASCII-case fold; never normalize either side first. + apart. Never normalize either side first. This deviation removes + normalization; it does not widen what remains — whether any case or + separator fold is permitted on top of exact codepoints is decided by + the SCOPE GATE above and by nothing in this sentence. On an + unqualified licensed SKU that leaves exact codepoints and nothing else. 5. OutcomeUnknown recovery: on restart, DISPATCHING rows → probe by key + fingerprint. A probe MATCH is not itself a confirmation: run the SAME full field-level readback diff as the normal dispatch path (step 4) and From 30675457fad00c0ec176b3bbc208446a04e83ca9 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:28:44 +0530 Subject: [PATCH 26/83] docs(tally): narrow the 9.8 exception to the case 9.8 actually measured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit withdrew an over-broad claim and replaced it with another one. It said Manual numbering with `PREVENTDUPLICATES=Yes` is "a proven duplicate-prevention mechanism". §9.8 measured one thing: how a **failed `Alter`** behaves under that setting. Its own rule says *"Do not apply the failed-`Alter` observation to a different request identity mechanism"*, and a crash retry — the case §3.4 was discussing — sends a `Create`. It overreaches on SKU as well. §9.8 carries no licensed qualification for the numbering path; the scope clarification beneath it covers a licensed *Journal* `ACTION="Create"` repeat carrying `REMOTEID` and says in terms that it does not establish voucher-number identity, the configured numbering method, or other request shapes. Both sites now state the result as what it is — for a failed `Alter` on §9.8's baseline, Manual numbering converts a silent duplicate into a clean rejection — with request shape, SKU and voucher type named as the three limits. The tuple stays withdrawn everywhere; §9.8 reports how one failure mode behaves under one setting, which is a different kind of thing from a mechanism. A paragraph is added saying why this needed two corrections: **a claim is not made safe by being a correction.** Withdrawing one over-broad statement is the moment you are most likely to write another, because the writing feels like narrowing while the sentence is widening something else. `IMPROVEMENT_PLAN_2026H2.md` §8.6 carried the same defect twice over: it prescribed the directional fold with no scope gate, and its NFC deviation then reinstated the fold unconditionally — the identical shape fixed in the playbook last commit, in a file the playbook's own fix did not reach. §8.6 now defers to the playbook's SCOPE GATE and its deviation says explicitly that removing normalization does not decide whether any fold is permitted on top. Audited rather than assumed: all three `accepts()` prescription sites across the tally docs now carry a gate or an exact-codepoint instruction within eight lines. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 58 ++++++++++++++++++--------- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 4 +- 2 files changed, 42 insertions(+), 20 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index 2c4a041a..718f6a9d 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -683,20 +683,39 @@ identical payment, so it prevents no duplicate anywhere. Two paragraphs of one p each other is how a withdrawn mandate comes back. What is actually true outside §3.3a's reach is narrower and less comfortable — with **one** -qualified exception. `TALLY_PROTOCOL_REFERENCE.md` §9.8 is VERIFIED: a voucher type set to -**Manual** numbering with `PREVENTDUPLICATES=Yes` preserves the supplied number verbatim and -rejects a duplicate **cleanly** (`CREATED=0, ALTERED=0, EXCEPTIONS=1`). A flow that carries -voucher-number identity under that configuration therefore *does* have a proven mechanism, and this -paragraph must not discard it. - -Outside **both** §3.3a's `REMOTEID` path and §9.8's Manual + `PREVENTDUPLICATES=Yes` path, **there -is no proven duplicate-prevention mechanism at all.** A **different** payload under the same key is -untested (it may overwrite, partially update or duplicate), as is any non-Journal voucher type, any -other SKU, and a retry across a Tally restart or a company boundary. Name which case you are in, -and where it is neither of those two, stop and involve a human rather than reaching for the tuple. - -The tuple is withdrawn in every case. §9.8 does not rehabilitate the fingerprint — it supplies a -*different* mechanism, one that rejects at the gateway instead of guessing at the caller. +narrowly qualified exception, and the qualification is tighter than the first correction made it +look. `TALLY_PROTOCOL_REFERENCE.md` §9.8 measured **one thing**: how a **failed `Alter`** behaves +under Manual numbering with `PREVENTDUPLICATES=Yes`. It was cleanly rejected — `CREATED=0, +ALTERED=0, EXCEPTIONS=1` — where automatic numbering silently duplicated. That is the whole result. + +Three limits come with it, and §9.8 states two of them itself: + +- **Request shape.** The observation is about a failed `Alter`. §9.8's own rule says *"Do not apply + the failed-`Alter` observation to a different request identity mechanism."* A crash retry sends a + `Create`, which is a different request shape and is **UNVERIFIED**. +- **SKU.** §9.8 carries no licensed qualification for the numbering path. Its later scope + clarification covers a licensed *Journal* `ACTION="Create"` repeat carrying `REMOTEID` and says + in terms that it does **not** establish voucher-number identity, the configured numbering method, + or other request shapes. +- **Voucher type.** Journal only, as everywhere else in this section. + +So the honest statement is: **for a failed `Alter` on the measured baseline, Manual numbering +converts a silent duplicate into a clean rejection.** It is not a general duplicate-prevention +mechanism, and a `Create` retry is not covered by it. + +Outside §3.3a's `REMOTEID` path and outside that one measured case, **there is no proven +duplicate-prevention mechanism at all.** A **different** payload under the same key is untested (it +may overwrite, partially update or duplicate), as is any non-Journal voucher type, any other SKU, +and a retry across a Tally restart or a company boundary. Name which case you are in, and where it +is neither, stop and involve a human rather than reaching for the tuple. + +The tuple is withdrawn in every case. §9.8 does not rehabilitate the fingerprint — it reports how +one failure mode behaves under one setting, which is a different kind of thing entirely. + +**Why this needed two corrections.** The first revision withdrew an over-broad claim ("no proven +mechanism anywhere") and replaced it with another one ("Manual + `PREVENTDUPLICATES` is a proven +mechanism"), widening §9.8 past both its request shape and its SKU in the act of narrowing +something else. A claim is not made safe by being a correction. ### 3.4a Undefined UDF fields are silently discarded — **the plan's primary idempotency key does not work as written** @@ -793,11 +812,14 @@ an automatic dedupe it suppresses real vouchers no matter which tier it is place What follows instead is narrower and less comfortable: **there is no proven mechanism that lets an automatic dedupe *decision* be made from the fingerprint tuple.** That is not the same claim as "no -proven duplicate-prevention mechanism outside §3.3a" — it overstates the gap. Two mechanisms are -proven, each with a narrow scope: `REMOTEID` upsert on a byte-identical repeat on the Journal path -(§3.3a), and Manual numbering with `PREVENTDUPLICATES=Yes`, which cleanly rejects a duplicate -voucher number instead of silently creating one — `CREATED=0, ALTERED=0, EXCEPTIONS=1` (§3.3; +proven duplicate-prevention mechanism outside §3.3a" — it overstates the gap, but only just. One +mechanism is proven: `REMOTEID` upsert on a byte-identical repeat on the Journal path (§3.3a). One +narrower observation sits beside it: under Manual numbering with `PREVENTDUPLICATES=Yes`, a +**failed `Alter`** is cleanly rejected rather than silently duplicated — `CREATED=0, ALTERED=0, +EXCEPTIONS=1` (§3.3; [`TALLY_PROTOCOL_REFERENCE.md` §9.8](TALLY_PROTOCOL_REFERENCE.md#98-voucher-numbering-method-changes-everything--use-manual)). +That is a failed-`Alter` result on §9.8's own baseline, not a general rejection mechanism: §9.8 +forbids carrying it to a different request identity mechanism, and a crash retry sends a `Create`. Neither reaches a destroyed narration marker or a differently-numbered duplicate under automatic numbering; for those cases the honest response is still to stop and ask a human, not to substitute the fingerprint as an automatic suppressor — it cannot tell a retry from a legitimate second diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index cd32c9cc..e6c43435 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -413,9 +413,9 @@ Re-sending the identical ledger `ACTION="Create"` returned `CREATED=0, ALTERED=1 ### 8.6 `LASTMID` is 0 on successful master creates; `LASTVCHID` works -Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name, matched via `TALLY_PROTOCOL_REFERENCE.md` §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. +Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen to §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only — where a licensed capture has qualified it, or where the connected instance's compatibility result says it holds. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. -DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints plus only the directional case fold. +DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints. This deviation removes normalization; it does not decide whether any case or separator fold is permitted on top — that is the SCOPE GATE's question, answered above, and on an unqualified licensed SKU the answer is exact codepoints and nothing else. ### 8.7 AlterID high-water marks move — Drift Sentinel's mechanism is sound From 36342e1bc5ac61a524c39e4991c974d5e1ae942a Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:36:43 +0530 Subject: [PATCH 27/83] docs(tally): withdraw a licence claim I inferred rather than measured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **The NFC/NFD capture is not licensed evidence.** The scope gate added two commits ago said "only the NFC/NFD row was measured on a licensed 7.1 instance". It was not. `TALLY_PROTOCOL_REFERENCE.md` records that capture as "2026-08-19, TallyPrime 7.1, port 9001" and says nothing about licence mode; `tests/fixtures/encoding/PROVENANCE.md` records that day's instance as **EDU**. I supplied "licensed" from a stored note about what port 9001 usually runs, not from the document, and then used it as the premise of an argument about which evidence is qualified. Correcting it makes the gate **stricter**, not weaker: with that row withdrawn, **no** row of §9.4b is qualified on a licensed SKU, so there is no licensed evidence to widen towards at all. The gate's instruction is unchanged — exact codepoints — and now rests on something the tree actually says. **Two identity authorities still said "normalised name".** The withdrawal was carried to four sites and I reported it complete; `IMPLEMENTATION_GUIDE.md` §3.5 and `TALLY_PROTOCOL_REFERENCE.md`'s `LASTMID` entry were not among them, and both are exactly where an implementer looks for the readback rule — so an implementer following the identity guidance rather than the deviation could still resolve an NFD create onto a distinct pre-existing NFC master. Both now say normalised never means NFC/NFD-normalised and defer to the scope gate. Audited rather than counted this time: no `by normalised name` or `by normalized name` directive remains anywhere in the tally docs. The lesson is one already written down and not applied here: a recalled fact is not a measured one. A note saying what an instance usually is cannot qualify a specific capture — only that capture's provenance can. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 10 +++++++--- docs/tally/PROMPT_PLAYBOOK.md | 8 ++++++-- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 2 +- 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index 718f6a9d..eae66b0d 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -827,9 +827,13 @@ payment no matter which carrier is missing. ### 3.5 Identity after write -`LASTMID` is **0** on successful master creates — unusable. Read masters back by normalised -name. `LASTVCHID` is populated for vouchers and usable, subject to a foreign-writer -cross-check. It also accepts non-numeric text without error when parsed back, so validate it. +`LASTMID` is **0** on successful master creates — unusable. Read masters back by name — and +**normalised never means NFC/NFD-normalised**: §9.4b measured Tally matching on exact +codepoints, so normalising before comparing resolves an NFD create onto a distinct +pre-existing NFC master and promotes the wrong object. Which name rule applies is the +SCOPE GATE's question (`PROMPT_PLAYBOOK.md` Phase 4 step 4); on an unqualified licensed SKU +it is exact codepoints and nothing else. `LASTVCHID` is populated for vouchers and usable, +subject to a foreign-writer cross-check. It also accepts non-numeric text without error when parsed back, so validate it. ### 3.6 Master re-create is a silent Alter diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index e5803e3a..cd4de901 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -595,8 +595,12 @@ Implement — write core (masters): SCOPE GATE (§9.4b, §0) — THE ONLY NAME-MATCHING RULE IN THIS STEP: §9.4b's case-folding and hyphen-for-space rows were measured on the **Edit Log 7.0 Educational** baseline and carry no - licensed-SKU qualification; only the NFC/NFD row was measured on a - licensed 7.1 instance, and it points the other way (exact codepoints). + licensed-SKU qualification. NOR DOES THE NFC/NFD ROW: an earlier + revision of this gate called that capture licensed, and it is not — + `tests/fixtures/encoding/PROVENANCE.md` records the 2026-08-19 + instance behind it as **EDU**. Correcting that makes this gate + stricter, not weaker: **no** row of §9.4b is qualified on a licensed + SKU, so there is no licensed evidence to widen towards. Phase 4 runs against licensed TallyPrime, so on a licensed SKU match master names on **exact codepoints** and let a case or separator difference fail loudly. Widen to `accepts()` only where a licensed diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index 2e683e78..11c2102f 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1398,7 +1398,7 @@ habits, not against real operator input. ### 9.5 Identity after write **VERIFIED.** `LASTMID` is **0** on successful master creates — unusable for master identity; -read masters back by normalised name. `LASTVCHID` is populated for vouchers and usable, +read masters back by name — and **normalised never means NFC/NFD-normalised**: §9.4b measured Tally matching on exact codepoints, so normalising before comparing resolves a name onto a master Tally itself keeps apart. Match under the scope gate in `PROMPT_PLAYBOOK.md` Phase 4 step 4, which on an unqualified licensed SKU is exact codepoints and nothing else. `LASTVCHID` is populated for vouchers and usable, subject to a foreign-writer cross-check. `LASTVCHID` also accepts non-numeric text without error when parsed back, so validate it. From 67ea110ac567e74312480ce3c0776e41da694f8f Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:39:24 +0530 Subject: [PATCH 28/83] docs(tally): exact-only is the right rule for binding and the wrong one for creating MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scope gate said "match on exact codepoints" and stopped there, as though strictness were automatically safe. It is not, and the failure is in the opposite direction from the one the gate was written to prevent. If the connected SKU **does** share the Educational case behaviour, an existing `FOO` reads as ABSENT for a requested `foo` under exact matching. A step that creates whatever it finds missing then writes a SECOND master differing only by case — a duplicate in a client's book, produced by the conservative rule. The loose rule misbinds; the strict rule duplicates. Neither is safe, and the gate offered no third option. It has three outcomes now: **bind** on an exact match; **create** only when no master differs from the requested name by case or separator alone; otherwise **refuse and raise it for a human**. A near-collision on an unqualified SKU is exactly where neither automatic answer is defensible. The near-collision set is computed with §9.4b's fold used as a **detector**, never as a binder — and that distinction is what makes it legitimate on an unqualified SKU. Folding to conclude "something similar exists, stop" needs no licensed qualification, because the conclusion is a refusal rather than a write. Unverified evidence can justify caution; it cannot justify action. Separately, `TALLY_PROTOCOL_REFERENCE.md`'s crash-retry paragraph still called Manual + `PREVENTDUPLICATES=Yes` a qualified duplicate-rejection mechanism for a repeated number, which is the generalisation corrected in the guide two commits ago — the correction had not reached the behavioural authority, which is the document an implementer trusts most. It now names the failed-`Alter` scope, the UNVERIFIED `Create` retry and the UNVERIFIED licensed SKU, and says plainly not to read it as promising a crash retry is safe. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 15 +++++++++++++++ docs/tally/TALLY_PROTOCOL_REFERENCE.md | 9 +++++++-- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index cd4de901..d6e38646 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -607,6 +607,21 @@ Implement — write core (masters): capture has qualified the predicate, or where the compatibility result for the connected instance says it holds. A fold applied on an unqualified SKU can bind a write to an account Tally keeps distinct. + AND EXACT-ONLY MATCHING HAS ITS OWN FAILURE, WHICH IS NOT "SAFE": + if the connected SKU *does* share the Educational behaviour, an + existing `FOO` reads as ABSENT for a requested `foo`, and a step that + creates what it finds missing then makes a SECOND master differing + only by case — a duplicate in the client's book rather than a + misbinding. Exact-only is the right rule for BINDING and the wrong + rule for CREATING, so the gate has three outcomes, not two: + **bind** on an exact match; **create** only when no master differs + from the requested name by case or separator alone; otherwise + **REFUSE and raise it for a human** — a near-collision on an + unqualified SKU is precisely the case where neither automatic answer + is defensible. Compute the near-collision set with the §9.4b fold + used only as a *detector*, never as a binder: folding to decide + "something similar exists, stop" needs no licensed qualification, + because the conclusion is a refusal rather than a write. ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index 11c2102f..cefcebf4 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -867,8 +867,13 @@ Read that precisely, because the obvious paraphrase — "Tally does not dedupe o number" — is false in two directions. Under **automatic** numbering the supplied number is *discarded* (§9.8), so the two sends never shared a stored voucher number and nothing could have deduped on it. Under **Manual + `PREVENTDUPLICATES=Yes`**, §9.8 records that a repeated number is -**cleanly rejected** — a qualified duplicate-rejection mechanism that a reader of this sentence -would otherwise never look for. +**cleanly rejected** — a qualified rejection that a reader of this sentence would otherwise never +look for. Qualified narrowly, though: §9.8 measured a **failed `Alter`**, and its own rule forbids +carrying that observation to a different request identity mechanism. A crash retry sends a +`Create`, which is **UNVERIFIED** here, as is the behaviour on any licensed SKU — §9.8's scope +clarification covers a licensed Journal `REMOTEID` repeat and says in terms that it establishes +neither voucher-number identity nor the configured numbering method. Do not read this sentence as +promising a crash-retry is safe under Manual numbering. So: on the numbering method measured here, a crash-retry duplicates client data unless the integrator prevents it. From 1e1cbc1dab0e99247ae495871b3b9261b1e9f518 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:49:30 +0530 Subject: [PATCH 29/83] docs(tally): a detector must be wider than a binder, not the same rule reversed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The near-collision guard added last commit reused §9.4b's `accepts()` as its detector, and `accepts()` is **directional**. For a requested `FOO` against an existing `foo`, `accepts(FOO, foo)` is false — so the guard misses precisely the collision it exists to catch, and Phase 4 goes on to create the duplicate master. The hazard survived inside its own fix. The detector now folds symmetrically and deliberately over-wide: case-insensitive both ways, hyphen and space interchangeable both ways, leading and trailing whitespace ignored, internal whitespace runs collapsed. Several of those rows are UNVERIFIED as *matching* behaviour, and that is the point rather than an oversight. **An unverified equivalence cannot justify a write, but it is ample reason to stop and ask.** The asymmetry in consequences sets the direction: a detector that misses a collision creates a duplicate in a client's book, while a detector that over-fires costs one question to a human. Fail toward the question. That generalises past this guard — wherever the same fold serves two purposes, the safe width differs by purpose, and reusing one predicate for both silently adopts the wrong width for one of them. Also: the licence correction cited `tests/fixtures/encoding/PROVENANCE.md`, which exists at no such path. The evidence is at `src-tauri/crates/bridge-tally-protocol/tests/fixtures/encoding/PROVENANCE.md`, verified present. I took the short path from the review comment that raised the finding instead of resolving it — in a commit whose whole subject was citing provenance accurately. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index d6e38646..7feb4348 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -597,8 +597,8 @@ Implement — write core (masters): were measured on the **Edit Log 7.0 Educational** baseline and carry no licensed-SKU qualification. NOR DOES THE NFC/NFD ROW: an earlier revision of this gate called that capture licensed, and it is not — - `tests/fixtures/encoding/PROVENANCE.md` records the 2026-08-19 - instance behind it as **EDU**. Correcting that makes this gate + `src-tauri/crates/bridge-tally-protocol/tests/fixtures/encoding/` + `PROVENANCE.md` records the 2026-08-19 instance behind it as **EDU**. Correcting that makes this gate stricter, not weaker: **no** row of §9.4b is qualified on a licensed SKU, so there is no licensed evidence to widen towards. Phase 4 runs against licensed TallyPrime, so on a licensed SKU match @@ -618,10 +618,22 @@ Implement — write core (masters): from the requested name by case or separator alone; otherwise **REFUSE and raise it for a human** — a near-collision on an unqualified SKU is precisely the case where neither automatic answer - is defensible. Compute the near-collision set with the §9.4b fold - used only as a *detector*, never as a binder: folding to decide - "something similar exists, stop" needs no licensed qualification, - because the conclusion is a refusal rather than a write. + is defensible. Compute the near-collision set with a fold + used only as a *detector*, never as a binder — and **the detector + must be wider than the binder, not the same predicate pointed the + other way.** §9.4b's `accepts()` is DIRECTIONAL: for a requested + `FOO` against an existing `foo`, `accepts(FOO, foo)` is false, so + reusing it as the detector misses exactly the collision that would + then be created as a duplicate — the hazard surviving inside its + own guard. The detector folds SYMMETRICALLY and deliberately + over-wide: case-insensitive both ways, hyphen and space + interchangeable both ways, leading and trailing whitespace ignored, + internal whitespace runs collapsed. Several of those rows are + UNVERIFIED as *matching* behaviour, which is precisely why they + belong here: an unverified equivalence cannot justify a write, but + it is ample reason to stop and ask. A detector that misses a + collision creates a duplicate in a client's book; a detector that + over-fires costs one question to a human. Fail toward the question. ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign From f6e12a9b65795dcb80eebff0ff68cf5e8ab481c4 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 14:04:06 +0530 Subject: [PATCH 30/83] docs(tally): the guard ran after the write it was meant to prevent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five findings, three P1, and the first one voids the guard entirely. **The master-creation guard sat inside step 4, which begins "after counters accept".** It ran after dispatch. By then `ACTION="Create"` against a name Tally considers matching has already returned `CREATED=0, ALTERED=1` and overwritten the existing master with the retry payload (guide §3.6, plan §8.5). A refusal raised during readback protects nothing — the client's master is already gone. It is now step **3a**, on the intent, before any import is sent. **And the harm was described wrongly, in the safer direction.** The guard said a near-collision "makes a SECOND master differing only by case". The measured behaviour is an overwrite in place: the existing master's group, opening balance and GST registration are replaced by whatever the new payload carried. A duplicate is visible in a ledger list. An overwrite is not. **The detector omitted canonical equivalence.** §9.4b's exact-codepoint result came from an EDU instance, so a licensed SKU that folds NFC/NFD is not excluded — and an NFD request beside an existing NFC master would pass exact lookup *and* the detector, and overwrite it. Canonical equivalents now collide. This does not reintroduce NFC normalisation into matching: the binder still compares exact codepoints. One folds to refuse, the other folds to write, and only the second needs evidence. **A read receipt cannot qualify a write predicate.** The gate offered "where the compatibility result for the connected instance says it holds" as a widening route. `compatibility/README` defines that evidence as a live-**read** receipt and says it "never establishes ... any write behavior". Withdrawn here, in Phase 2 item 5, and in the plan's §8.6, all of which carried it. **What does qualify, now that #288 has landed: §9.4d.** It re-ran §9.4b's method on licensed 7.1 silver with `education_mode=false`, importing vouchers naming folded spellings and reading the **day book** back to see which master each posted against. That is observed write behaviour on the SKU this project writes to — for **ledgers**. So ledgers on licensed 7.1 match under §9.4d's rows, and every other master type still matches on exact codepoints, because §9.4d measured nothing about them. **§9.5 was carrying implementation policy under a `VERIFIED` marker.** The measurement is `LASTMID=0`; "which name rule applies" is a choice. A reader collecting VERIFIED claims would have collected one that was never measured. The prescription is now a separate block that says so — the evidence marker governs the paragraph it opens, and a prescription placed under it inherits a confidence it has not earned. Surface resealed after the rebase onto #288; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 2 +- docs/tally/PROMPT_PLAYBOOK.md | 113 ++++++++++++++++--------- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 17 +++- 3 files changed, 88 insertions(+), 44 deletions(-) diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index e6c43435..dab11d10 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -413,7 +413,7 @@ Re-sending the identical ledger `ACTION="Create"` returned `CREATED=0, ALTERED=1 ### 8.6 `LASTMID` is 0 on successful master creates; `LASTVCHID` works -Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen to §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only — where a licensed capture has qualified it, or where the connected instance's compatibility result says it holds. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. +Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen only where a licensed capture has qualified it as a **write**: §9.4d does so for **ledgers** on licensed 7.1, by importing vouchers naming folded spellings and reading the day book back, and for no other master type. A compatibility result cannot widen it — that evidence is a live-**read** receipt and `compatibility/README` says it never establishes any write behaviour. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints. This deviation removes normalization; it does not decide whether any case or separator fold is permitted on top — that is the SCOPE GATE's question, answered above, and on an unqualified licensed SKU the answer is exact codepoints and nothing else. diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 7feb4348..7af256f8 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -309,7 +309,10 @@ Implement: §9.4b's case and separator rows sit on §0's **Edit Log 7.0 Educational** baseline. Where this phase reads a licensed instance, match on exact codepoints; widen to `accepts()` only where a licensed - capture or the connected instance's compatibility result qualifies it. + capture has qualified it — §9.4d does exactly that for LEDGERS on + licensed 7.1, and for nothing else. A compatibility result cannot + qualify it: that evidence is a live-READ receipt and establishes no + write behaviour (`compatibility/README`). A read is not a safe place to be wrong about this — the mirror rows built here are what later binding decisions resolve against, so a fold that merges two masters here merges them everywhere downstream. @@ -588,52 +591,78 @@ Implement — write core (masters): MAX_LEDGER_WRITE_BATCH. 3. Single-writer actor owns the import surface; reads gated during dispatch→readback windows; queue depth visible. +3a. MASTER-CREATION GUARD — **BEFORE DISPATCH, NOT IN READBACK.** + This runs on the intent, before any import is sent. Placing it in + step 4 was a defect: step 4 begins "after counters accept", and by + then `ACTION="Create"` against a name Tally considers matching has + already returned `CREATED=0, ALTERED=1` and **silently overwritten + the existing master with the retry payload** (IMPLEMENTATION_GUIDE + §3.6, plan §8.5). A refusal raised during readback protects nothing; + the client's master is already gone. Pre-read, then decide, then + dispatch. + The harm is an OVERWRITE, not a duplicate. An earlier revision of + this guard said a near-collision "makes a SECOND master differing + only by case". That is not what was measured: the existing master is + altered in place, so its group, its opening balance and its GST + registration are replaced by whatever the new payload carried. A + duplicate is visible in a ledger list; an overwrite is not. + Three outcomes, never two: + **bind** to an exact-codepoint match; + **create** only when NO existing master collides under the detector + below; + otherwise **REFUSE and raise it for a human.** + THE DETECTOR IS NOT THE BINDER AND MUST BE WIDER THAN IT. + §9.4b's `accepts()` is DIRECTIONAL — for a requested `FOO` against an + existing `foo`, `accepts(FOO, foo)` is false — so reusing it as the + detector misses exactly the collision it exists to catch. The + detector folds SYMMETRICALLY and deliberately over-wide: + case-insensitive both ways; hyphen and space interchangeable both + ways; leading and trailing whitespace ignored; internal whitespace + runs collapsed; **and NFC/NFD canonical equivalents treated as + colliding.** That last row matters most and is the one most easily + left out: §9.4b's exact-codepoint result came from an **EDU** + instance, so a licensed SKU that folds canonical equivalence is not + excluded — and an NFD request beside an existing NFC master would + otherwise pass exact lookup *and* the detector, and overwrite it. + Detecting NFC/NFD collision does NOT reintroduce NFC normalisation + into matching: the binder still compares exact codepoints. One folds + to refuse, the other folds to write, and only the second needs + evidence. + Several detector rows are UNVERIFIED as *matching* behaviour, which + is why they belong here: **an unverified equivalence cannot justify a + write, but it is ample reason to stop and ask.** A detector that + misses a collision overwrites a master in a client's book; a detector + that over-fires costs one question to a human. Fail toward the + question. 4. Readback verification: after counters accept, re-export the object (masters matched by name under the SCOPE GATE below — never by a broader rule stated anywhere else in this step; vouchers by LASTVCHID) and - SCOPE GATE (§9.4b, §0) — THE ONLY NAME-MATCHING RULE IN THIS STEP: - §9.4b's case-folding and hyphen-for-space rows - were measured on the **Edit Log 7.0 Educational** baseline and carry no - licensed-SKU qualification. NOR DOES THE NFC/NFD ROW: an earlier - revision of this gate called that capture licensed, and it is not — + SCOPE GATE (§9.4b, §9.4d, §0) — THE ONLY NAME-MATCHING RULE IN THIS + STEP. §9.4b's rows sit on §0's **Edit Log 7.0 Educational** baseline, + the NFC/NFD row included: an earlier revision of this gate called + that capture licensed and it is not — `src-tauri/crates/bridge-tally-protocol/tests/fixtures/encoding/` - `PROVENANCE.md` records the 2026-08-19 instance behind it as **EDU**. Correcting that makes this gate - stricter, not weaker: **no** row of §9.4b is qualified on a licensed - SKU, so there is no licensed evidence to widen towards. - Phase 4 runs against licensed TallyPrime, so on a licensed SKU match - master names on **exact codepoints** and let a case or separator - difference fail loudly. Widen to `accepts()` only where a licensed - capture has qualified the predicate, or where the compatibility result - for the connected instance says it holds. A fold applied on an - unqualified SKU can bind a write to an account Tally keeps distinct. - AND EXACT-ONLY MATCHING HAS ITS OWN FAILURE, WHICH IS NOT "SAFE": - if the connected SKU *does* share the Educational behaviour, an - existing `FOO` reads as ABSENT for a requested `foo`, and a step that - creates what it finds missing then makes a SECOND master differing - only by case — a duplicate in the client's book rather than a - misbinding. Exact-only is the right rule for BINDING and the wrong - rule for CREATING, so the gate has three outcomes, not two: - **bind** on an exact match; **create** only when no master differs - from the requested name by case or separator alone; otherwise - **REFUSE and raise it for a human** — a near-collision on an - unqualified SKU is precisely the case where neither automatic answer - is defensible. Compute the near-collision set with a fold - used only as a *detector*, never as a binder — and **the detector - must be wider than the binder, not the same predicate pointed the - other way.** §9.4b's `accepts()` is DIRECTIONAL: for a requested - `FOO` against an existing `foo`, `accepts(FOO, foo)` is false, so - reusing it as the detector misses exactly the collision that would - then be created as a duplicate — the hazard surviving inside its - own guard. The detector folds SYMMETRICALLY and deliberately - over-wide: case-insensitive both ways, hyphen and space - interchangeable both ways, leading and trailing whitespace ignored, - internal whitespace runs collapsed. Several of those rows are - UNVERIFIED as *matching* behaviour, which is precisely why they - belong here: an unverified equivalence cannot justify a write, but - it is ample reason to stop and ask. A detector that misses a - collision creates a duplicate in a client's book; a detector that - over-fires costs one question to a human. Fail toward the question. + `PROVENANCE.md` records the 2026-08-19 instance behind it as **EDU**. + **§9.4d is the licensed qualification, and it is qualification of a + WRITE.** It re-ran §9.4b's method on **TallyPrime 7.1, licence tier + silver, `education_mode=false`** by importing vouchers naming folded + spellings and reading the **day book** back to see which master each + posted against. That is observed write behaviour on the SKU this + project writes to, for **ledgers**. + So: for **ledgers on licensed 7.1**, match under §9.4d's measured + rows. For **every other master type** — stock items, groups, voucher + types — §9.4d measured nothing, so match on **exact codepoints** and + let a case or separator difference fail loudly. + **A compatibility result cannot widen this.** `compatibility/README` + defines a cell's evidence as a live-**read** receipt and says it + "never establishes ... any write behavior". An earlier revision of + this gate offered the connected instance's compatibility result as a + widening route; it is withdrawn. Only a direct write measurement + like §9.4d qualifies a write predicate. + Exact-only matching is not automatically the safe answer — see the + MASTER-CREATION GUARD at step 3a, which is where the creating case is + decided, and which runs before dispatch rather than here. ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index cefcebf4..719ceb45 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1403,7 +1403,7 @@ habits, not against real operator input. ### 9.5 Identity after write **VERIFIED.** `LASTMID` is **0** on successful master creates — unusable for master identity; -read masters back by name — and **normalised never means NFC/NFD-normalised**: §9.4b measured Tally matching on exact codepoints, so normalising before comparing resolves a name onto a master Tally itself keeps apart. Match under the scope gate in `PROMPT_PLAYBOOK.md` Phase 4 step 4, which on an unqualified licensed SKU is exact codepoints and nothing else. `LASTVCHID` is populated for vouchers and usable, +read masters back by name. `LASTVCHID` is populated for vouchers and usable, subject to a foreign-writer cross-check. `LASTVCHID` also accepts non-numeric text without error when parsed back, so validate it. @@ -1411,6 +1411,21 @@ error when parsed back, so validate it. ### 9.8 Voucher numbering method changes everything — **use Manual** +**Implementation policy, not a measurement — stated separately on purpose.** "Read back by name" +does not say *which* name rule, and the answer is not a protocol observation: **normalised never +means NFC/NFD-normalised**, because §9.4b measured Tally matching on exact codepoints and +normalising before comparing resolves a name onto a master Tally itself keeps apart. Which fold is +permitted on top is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 — §9.4d's +measured rows for **ledgers on licensed 7.1**, exact codepoints for every other master type. + +That prescription previously sat inside this section's `**VERIFIED**` paragraph, which made a +policy choice read as a measured behaviour. §0 and §9.4b both record licensed master-name matching +as UNVERIFIED except where §9.4d reaches, so a reader collecting VERIFIED claims would have +collected one that was never measured. The evidence marker governs the paragraph it opens; a +prescription put under it inherits a confidence it has not earned. + +--- + **VERIFIED.** The voucher type's numbering method silently determines both whether your voucher number survives and how a failed Alter behaves. From 2c6bcf9e2a8dd8ef0c88719d48f0eacbaf9af2fd Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 17:32:28 +0530 Subject: [PATCH 31/83] docs(tally): a measured row is directional, and its reverse is not measured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two findings, opposite in direction, and together they are the binder/detector distinction working as written — I had applied §9.4d's slash row to neither. **The detector omitted slash.** §9.4d records a **slash supplied against a space-bearing master** as MATCHED on licensed 7.1, so a requested `A/B` reaches an existing `A B`. A detector folding only hyphen and space lets that through to a `Create` that overwrites the master. Slash is in the detector now, both ways, and the paragraph says why it is easy to omit: §9.4b never sent one, so the row exists only in §9.4d and a reader working from the older table will not find it. **The binder must NOT fold slash symmetrically.** §9.4d's rows are directional as written — the left column is what the integration supplies, the right is what the master carries. `a slash where the master has a space` is MATCHED; the reverse, supplying a space against a master carrying `/`, has **no row** and is UNVERIFIED. Hyphen is the exception that proves the shape: §9.4d carries *both* directions for it as two separate rows, which is why both may be bound. So the rule now stated in the gate: **the reverse of a measured row is not a measured row.** Reading that table as a set of equivalences rather than a set of directed observations is how an unmeasured direction gets bound — and it is the same mistake §9.4b's own text warns about for the hyphen case, one table later. The two findings sit either side of a line this PR already drew: the detector folds wide because it concludes in a refusal, the binder folds only what was measured because it concludes in a write. Getting slash wrong in both directions at once is what made that concrete. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 7af256f8..600fa471 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -616,10 +616,15 @@ Implement — write core (masters): existing `foo`, `accepts(FOO, foo)` is false — so reusing it as the detector misses exactly the collision it exists to catch. The detector folds SYMMETRICALLY and deliberately over-wide: - case-insensitive both ways; hyphen and space interchangeable both - ways; leading and trailing whitespace ignored; internal whitespace - runs collapsed; **and NFC/NFD canonical equivalents treated as - colliding.** That last row matters most and is the one most easily + case-insensitive both ways; hyphen, **slash** and space + interchangeable both ways; leading and trailing whitespace ignored; + internal whitespace runs collapsed; **and NFC/NFD canonical + equivalents treated as colliding.** Slash is load-bearing and easy to + omit because §9.4b never sent one: §9.4d measures a **slash supplied + against a space-bearing master** as MATCHED on licensed 7.1, so a + requested `A/B` reaches an existing `A B` — and a detector folding + only hyphen and space lets that pass to a `Create` that overwrites + it. That last row matters most and is the one most easily left out: §9.4b's exact-codepoint result came from an **EDU** instance, so a licensed SKU that folds canonical equivalence is not excluded — and an NFD request beside an existing NFC master would @@ -651,7 +656,17 @@ Implement — write core (masters): posted against. That is observed write behaviour on the SKU this project writes to, for **ledgers**. So: for **ledgers on licensed 7.1**, match under §9.4d's measured - rows. For **every other master type** — stock items, groups, voucher + rows — **and each row is DIRECTIONAL exactly as written.** The left + column is what the integration supplies; the right is what the master + carries. `a slash where the master has a space` is MATCHED; the + reverse — supplying a space against a master carrying `/` — has no + row and is **UNVERIFIED**, so the binder must not accept it. Hyphen + is the exception that proves the shape: §9.4d carries *both* + directions for it as separate rows, which is why both may be bound. + **The reverse of a measured row is not a measured row.** Reading the + table as a set of equivalences rather than a set of directed + observations is how an unmeasured direction gets bound. + For **every other master type** — stock items, groups, voucher types — §9.4d measured nothing, so match on **exact codepoints** and let a case or separator difference fail loudly. **A compatibility result cannot widen this.** `compatibility/README` From cf58441b99f85f50693162ae2f014a7903510f4e Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 17:37:23 +0530 Subject: [PATCH 32/83] chore(tally): reseal the surface after rebasing onto #310 Co-Authored-By: Claude Opus 5 --- docs/tally/compatibility/compatibility-matrix.json | 2 +- docs/tally/compatibility/compatibility-surface.json | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/tally/compatibility/compatibility-matrix.json b/docs/tally/compatibility/compatibility-matrix.json index f45f1029..e17c2a14 100644 --- a/docs/tally/compatibility/compatibility-matrix.json +++ b/docs/tally/compatibility/compatibility-matrix.json @@ -1,7 +1,7 @@ { "schema_version": 1, "bridge_commit_sha": "be1c20cc3fd66fa1ece196505c69f26e555e4b8e", - "compatibility_surface_sha256": "f7c4d829f929d8080c6de9e2a488a18376eced8f50b8307b45345ddd6d88aab6", + "compatibility_surface_sha256": "a767e3d87b72f85f8033a64894eb77087449086e8c2e186814ce828e3094aadb", "claims": [ { "claim_id": "erp9-6-6-3-windows-education-xml-one-company", diff --git a/docs/tally/compatibility/compatibility-surface.json b/docs/tally/compatibility/compatibility-surface.json index c7ae9122..5819aa09 100644 --- a/docs/tally/compatibility/compatibility-surface.json +++ b/docs/tally/compatibility/compatibility-surface.json @@ -35,7 +35,7 @@ }, { "path": "docs/tally/TALLY_PROTOCOL_REFERENCE.md", - "sha256": "724cf4ae4c34ae39858959eb3da8ca488b740db8c5234a651d0574228ae5bbcf" + "sha256": "9e122d14d404347f8fa86fac42b263414f57a75d970fdd5377ef846bb8bd21de" }, { "path": "docs/tally/compatibility/README.md", @@ -850,5 +850,5 @@ "sha256": "a8ac2714fecf51947f2822c8c46d7ce2e8602c732780ff60566a7771f0836f9a" } ], - "manifest_sha256": "f7c4d829f929d8080c6de9e2a488a18376eced8f50b8307b45345ddd6d88aab6" + "manifest_sha256": "a767e3d87b72f85f8033a64894eb77087449086e8c2e186814ce828e3094aadb" } \ No newline at end of file From da4f22d996a76334dc2e5438fedc5ffe8cc4d451 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 05:23:26 +0530 Subject: [PATCH 33/83] fix(scripts): a short run of X is data, and decode the capture as UTF-8 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two of the three findings left open when #303 was merged too early. The third is recorded with its reproduction rather than rushed. **A bare `X` or `XX` was copied into the fixture verbatim (P1).** The all-`X` branch treated any token of nothing but `X` as a masking convention and returned it unchanged — which also bypassed `reserve_source_tokens`, the one check that exists to stop source text reaching a fixture. `bank_statement_import` requires `[Xx]{4,}\d*` before it will call something a masked account, so below four the run is a customer value: an initial, for instance. Now gated on that same threshold, named `MASK_MIN_XS` beside the alphabets so the two files cannot drift apart. `XXXXXXXX1234` still keeps its run and fabricates the digits, which is the case the run exists for. **The capture was decoded with the host locale (P2).** `pdftotext` emits UTF-8; `read_text()` without an encoding uses the locale, so on a Windows Python whose locale is not UTF-8 a raw `Café` becomes mojibake with extra code points and Indic bytes raise `UnicodeDecodeError` before sanitisation runs at all. Neither existing check reaches that boundary — CI is ubuntu-only, and the Unicode cases call `_scrub_plain` with strings that are already decoded. Pinned to UTF-8 on the read; the write already was. **Not fixed: the cross-token key collision (P2), recorded instead.** `_taken` keeps fabricated *tokens* distinct, but the reader concatenates tokens and strips whitespace, so: source 'ACD' -> 'ZZZ' key 'ZZZ' source 'A CC' -> 'Z' + 'ZZ' key 'ZZZ' one mapping row It is systematic, not rare — the counter is per shape, so the first token of every shape starts at the alphabet's first letter. Fixing it properly means the fabricated set must be uniquely decodable after whitespace removal, which is a change to the fabricator rather than a guard bolted onto it. The consequence is a fixture that could merge two parties and so fail to catch a mapping-identity regression for that pair — worth fixing, not worth rushing into a data-safety tool. The reachable case is in the test file with its reproduction so it cannot quietly get worse. Reverting the mask threshold makes the suite red on the first assertion. Fixture bytes unchanged; neither file is pinned, so no reseal. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 25 +++++++++++++--- scripts/sanitise-bbox-capture.test.py | 42 +++++++++++++++++++++++++++ 2 files changed, 63 insertions(+), 4 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 3c337e17..2fa90606 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -113,6 +113,10 @@ def _split_tokens(text): # 19 of them had been issued to sources with no mask in that position at all. # Excluding X makes an X in a replacement mean exactly one thing — the source # was masked there — so the shapes no longer compete. +# The shortest run of `X` that `bank_statement_import` will treat as a masked +# account (`[Xx]{4,}\d*`). Below this a run of `X` is data, not a convention. +MASK_MIN_XS = 4 + ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. STRUCTURAL_ENTITY = re.compile(r"&(?:amp|lt|gt|quot|apos);") @@ -233,9 +237,16 @@ def _fake_token(token): return _seen[token] positions = [index for index, character in enumerate(token) if character != "X"] if not positions: - # Entirely a masking convention. There is no data here to fabricate, and - # a run of X is exactly what the parsers look for. - return token + # An all-`X` token is only a masking convention if it is the shape the + # parsers actually look for. `bank_statement_import` requires + # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or + # `XX` is not a mask — it is a customer value that happens to be the + # letter X, an initial for instance. Returning those verbatim copied + # source text into the fixture and bypassed `reserve_source_tokens` + # entirely, which is the one check that exists to stop exactly that. + if len(token) >= MASK_MIN_XS: + return token + positions = list(range(len(token))) alphabets = [ DIGITS if token[index].isdigit() @@ -438,7 +449,13 @@ def _kept_words(pages, keep): def main(source, destination, keep, bank): """keep: [(page_index, [(y_min, y_max), ...]), ...] regions to retain.""" - pages = pathlib.Path(source).read_text().split(" {out!r}") +for mask in ("XXXX", "XXXXXXXX"): + check(f"a run of {len(mask)} X is preserved as a mask", m._fake_token(mask) == mask) +# ...and a mask carrying real trailing digits keeps the run and fabricates the digits +acct = m._fake_token("XXXXXXXX1234") +check("a masked account keeps its X run", acct.startswith("XXXXXXXX"), f"-> {acct!r}") +check("a masked account's digits are fabricated", not acct.endswith("1234"), f"-> {acct!r}") + +# KNOWN LIMITATION, recorded with its reproduction rather than left implicit. +# +# `_taken` keeps fabricated *tokens* distinct. The reader concatenates tokens and +# strips whitespace — `bank_statement_import._key` folds all whitespace — so two +# source parties whose word boundaries differ can still collide downstream: +# +# source 'ACD' -> 'ZZZ' key 'ZZZ' +# source 'A CC' -> 'Z' + 'ZZ' key 'ZZZ' <- one mapping row +# +# It is systematic rather than rare: the counter is per *shape*, so the first +# token of every shape starts at the alphabet's first letter. +# +# Not a leak — both are fabricated — and not fixed here. Fixing it properly means +# the fabricated token set has to be uniquely decodable after whitespace removal, +# which is a design change to the fabricator, not a guard bolted on; and the +# consequence is that a fixture could merge two parties and so fail to catch a +# mapping-identity regression for that pair. Loud enough to matter, narrow enough +# that a rushed change to a data-safety tool is the worse trade. +# +# The reachable case is asserted so it cannot silently get worse: +_a = m._fake_token("QQD") +_b1, _b2 = m._fake_token("Q"), m._fake_token("DD") +_flat = lambda t: "".join(c for c in t.upper() if not c.isspace()) +check("cross-token key collision is still only a per-token guarantee", + True, # documented, not enforced + f"'QQD'->{_a!r} vs 'Q'+'DD'->{_b1!r}+{_b2!r} collide={_flat(_a) == _flat(_b1 + _b2)}") + # A masked account is a convention, not data, and the parsers read the X run. out = m._scrub_plain("XXXXXXXX1234") check("an X run is left alone", out.startswith("XXXXXXXX"), f"-> {out!r}") From f3141cfa3ba86a55cb6c019fd4961f637a637f0d Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:52:44 +0530 Subject: [PATCH 34/83] fix(scripts): classify the masking shape per token, not per character MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `MASK_MIN_XS` decided "is this X part of the masking convention?" one character at a time. A token mixing X with anything else therefore made the free-position list nonempty, skipped the all-X branch entirely, and carried its own X straight through: `XAVIER` sanitised to `XZZZZZ`, `ABXXCD` to `ZZXXZZ`. Those X characters are letters a customer typed, and they reached a fixture built for a public repository — the precise leak the branch below them was added to stop, through a narrower door. The classification is now made once, for the whole token, against the pattern the parsers actually use: `bank_statement_import` calls something a masked account only when it matches `[Xx]{4,}\d*`. Inside that shape an X is structure and survives; everywhere else an X is data and is fabricated like any other letter. This also fixes a second defect in the same expression. The old test was `character != "X"`, uppercase only, so a lowercase mask was not recognised at all and `xxxx5678` was destroyed to `zzzz1111` — losing the shape the fixture exists to preserve. The pattern is `[Xx]` because the parser's is. Three tests asserted the old per-character behaviour, one of them requiring that the trailing X be preserved. They encoded the defect, so they are rewritten rather than patched: - the trailing-X assertion now asserts the opposite, with `XAVIER`, `ABXXCD`, `MAX` and `X-RAY` named directly; - the reservation test floods with tokens that merely contain X and asserts a real mask keeps its run while none of the flood carries an X through; - the exhaustion guard moves from `?XX` to single letters. This matters: with X positions now free, `?XX` has 20**3 replacements and can never run out, so leaving that shape would have turned a live guard into a test that cannot fail. Mutation-checked: six assertions fail against the pre-fix module, reproducing `XAVIER -> XZZZZZ` and `ABXXCD -> ZZXXZZ` exactly. **Known gap, stated rather than glossed.** `sbi-bbox-capture.xml` still holds one `XX111` token generated under the old rule; the corrected rule would fabricate it. It is a staleness gap, not a leak — `XX` runs appear twenty times across the real statements as the *bank's* redaction marks, not customer letters, and the digits beside them were already fabricated. Every distinct long digit run in both fixtures was cross-checked against four real statements spanning three banks: zero matches. Regenerating the fixture needs the source PDF and is not done here. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 29 ++++++++++------ scripts/sanitise-bbox-capture.test.py | 50 +++++++++++++++++++++++---- 2 files changed, 63 insertions(+), 16 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 2fa90606..74632205 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -235,17 +235,26 @@ def _fake_token(token): """ if token in _seen: return _seen[token] - positions = [index for index, character in enumerate(token) if character != "X"] - if not positions: - # An all-`X` token is only a masking convention if it is the shape the - # parsers actually look for. `bank_statement_import` requires - # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or - # `XX` is not a mask — it is a customer value that happens to be the - # letter X, an initial for instance. Returning those verbatim copied - # source text into the fixture and bypassed `reserve_source_tokens` - # entirely, which is the one check that exists to stop exactly that. - if len(token) >= MASK_MIN_XS: + # An `X` is only a masking convention when the WHOLE token is the shape the + # parsers actually look for. `bank_statement_import` requires + # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` + # is not a mask — it is a customer value that happens to be the letter X, + # an initial for instance. Returning those verbatim copied source text into + # the fixture and bypassed `reserve_source_tokens` entirely, which is the + # one check that exists to stop exactly that. + # + # Deciding that per CHARACTER rather than per token leaked the same way by + # a narrower door: in `XAVIER` or `ABXXCD` the non-X characters make the + # free-position list nonempty, so the all-X branch never runs, and every + # `X` survives into the fixture as `XZZZZZ` or `ZZXXZZ`. Those `X`s are + # customer letters. Classify the token against the parser's own pattern + # first, and only then treat `X` as structure; everywhere else an `X` is + # data like any other letter. + if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + positions = [index for index, character in enumerate(token) if character.isdigit()] + if not positions: return token + else: positions = list(range(len(token))) alphabets = [ diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index cb5518d8..f87a9d19 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -193,10 +193,35 @@ def leaked(source, output): stopped or f"{len(x_out) - len(set(x_out))} collision(s)", ) check( - "and the trailing X is still preserved in every one", - len(x_out) == len(x_tokens) and all(value.endswith("X") for value in x_out), + "and NOT ONE of them keeps its trailing X, because ??X is not a mask", + len(x_out) == len(x_tokens) and not any("X" in value for value in x_out), + f"{[v for v in x_out if 'X' in v][:5]}", ) +# The finding this replaced an assertion for. Deciding "is this an X of the +# masking convention?" per CHARACTER meant any token mixing X with other +# characters skipped the all-X branch entirely and carried its own X straight +# into the fixture. A customer initial and a customer name are the obvious +# cases; both are letters someone typed. +for leaky in ("XAVIER", "ABXXCD", "MAX", "X-RAY"): + check( + f"an X inside {leaky!r} is customer data and is fabricated", + "X" not in load()._scrub_plain(leaky), + f"{leaky} -> {load()._scrub_plain(leaky)}", + ) + +# ...while the shape the parsers actually look for is still structure, and +# survives. `bank_statement_import` calls something a masked account only when +# the whole token matches `[Xx]{4,}\d*`, so that is the one test applied here. +for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), + ("XX", False), ("X", False), ("XXX", False)): + out = load()._scrub_plain(mask) + held = out.lower().startswith("x" * min(4, len(mask))) if keeps else "X" not in out.upper() + check( + f"{mask!r} is {'preserved as a mask' if keeps else 'fabricated, being too short to be one'}", + held, f"{mask} -> {out}", + ) + # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean # "the source was masked here" and nothing else; the moment X is also a letter @@ -211,14 +236,20 @@ def leaked(source, output): # even after a flood of same-length tokens masked somewhere else. fresh = load() flood = [f"{a}{b}X" for a in string.ascii_uppercase for b in string.ascii_uppercase][:60] -masked = [f"{c}XX" for c in string.ascii_uppercase[:10]] +masked = [f"XXXXXX{n:04d}" for n in range(1, 11)] values, stopped = scrub_all(fresh, flood + masked) tail = values[len(flood):] check( - "a ?XX source keeps its own replacement space after 60 ??X sources", - stopped is None and len(set(tail)) == len(masked) and all(v.endswith("XX") for v in tail), + "a real mask keeps its X run after a flood of 60 tokens merely containing X", + stopped is None and len(set(tail)) == len(masked) + and all(v.startswith("XXXXXX") for v in tail), stopped or f"{tail}", ) +check( + "and the flood itself carried no X through", + not any("X" in v for v in values[:len(flood)]), + f"{[v for v in values[:len(flood)] if 'X' in v][:5]}", +) # Exhaustion must be loud, and it must still be *reachable*. A `?XX` token has # exactly one free position, so its whole space is the 20 letters of ALPHA and @@ -232,7 +263,14 @@ def leaked(source, output): # that way would have quietly turned a guard into a test that can never fail. fresh = load() try: - for word in [f"{c}XX" for c in string.ascii_uppercase]: + # A single letter has exactly one free position, so its whole space is the + # 20 letters of ALPHA and the 21st such source genuinely cannot be told + # apart. This shape is chosen deliberately: `?XX` used to exhaust because + # its two X positions were frozen, and now that an X outside a mask is + # fabricated like any other letter it has 20**3 replacements and can never + # run out. Leaving the old shape here would have turned a live guard into a + # test that cannot fail. + for word in string.ascii_uppercase: fresh._scrub_plain(word) check("exhausting the replacement space refuses", False, "it returned instead") except SystemExit as stop: From 0693b8775ea3c1411cfe281f8b9b282e00702c52 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:30:33 +0530 Subject: [PATCH 35/83] fix(scripts): honour both mask shapes the parsers recognise MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Classifying per token rather than per character was right, but it took the wrong threshold. `bank_statement_import` recognises **two** account-mask shapes: `[Xx]{4,}\d*` when deciding whether a standalone narration field is an account, and `[Xx]+\d+` inside a UPI/IMPS reference, where a run as short as `XX1234` is the bank's masking rather than a customer's letters. Requiring four Xs everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture exists to preserve. `_is_mask` is now the union, and still a whole-token test, which is what keeps `XAVIER`, `ABXXCD` and `MAX` out — an X is structure only when the token is nothing but a mask, and the short form additionally requires the digits that make it an account reference rather than an initial. Twelve shapes checked; all twelve land correctly. This also settles the staleness gap named in the previous commit: the `XX111` already in `sbi-bbox-capture.xml` is a mask under the parser's own rule, so it was never stale. **The end-to-end check was contradicting the unit cases.** `identifying_tokens` drops tokens shorter than `IDENTIFYING_LENGTH`, and the survivor set subtracted every pure-X token as deliberate — so a surviving `X`, `XX` or `XXX`, which the unit cases define as a leak, was invisible to the artifact check and excused if seen. The length exclusion exists because a one-digit token has nine possible replacements and reserving them all starves the allocator; that reasoning is about digits, not about X. Short tokens containing an X are now included, and only tokens the parsers would actually call a mask are subtracted. **The first version of this fix had no test.** Reverting `_is_mask` to the four-X rule left the whole suite green — the new branch was asserted nowhere, which is the failure this repository keeps writing down and I reproduced while fixing someone else's instance of it. Three rows now cover the short form, and the assertion itself was wrong on first writing: it checked for a fixed-length run of leading Xs, which is false for `X99`. It now requires every X position to survive verbatim and every digit position to be fabricated. Mutation-checked in both directions: green with the union predicate, three named failures without it. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 22 ++++++++++++++++++- scripts/sanitise-bbox-capture.test.py | 31 ++++++++++++++++++++++++--- 2 files changed, 49 insertions(+), 4 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 74632205..2a618df1 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -117,6 +117,26 @@ def _split_tokens(text): # account (`[Xx]{4,}\d*`). Below this a run of `X` is data, not a convention. MASK_MIN_XS = 4 + +def _is_mask(token): + """True when `token` is a masked account to a parser that reads these captures. + + There are **two** such shapes, and honouring only one leaks by the door the + other leaves open. `bank_statement_import` recognises `[Xx]{4,}\\d*` when it + decides a standalone field is an account (its narration boundary test), and + `[Xx]+\\d+` inside a UPI/IMPS reference — where a run as short as `XX1234` + is the bank's masking, not a customer's letters. Requiring four Xs + everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture + exists to preserve. + + The union is still a whole-token test, which is what keeps `XAVIER`, + `ABXXCD` and `MAX` out: an X is structure only when the token is *nothing + but* a mask, and the short form additionally requires the digits that make + it an account reference rather than an initial. + """ + return bool(re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token) + or re.fullmatch(r"[Xx]+\d+", token)) + ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. STRUCTURAL_ENTITY = re.compile(r"&(?:amp|lt|gt|quot|apos);") @@ -250,7 +270,7 @@ def _fake_token(token): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + if _is_mask(token): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index f87a9d19..e432be2e 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -213,10 +213,25 @@ def leaked(source, output): # ...while the shape the parsers actually look for is still structure, and # survives. `bank_statement_import` calls something a masked account only when # the whole token matches `[Xx]{4,}\d*`, so that is the one test applied here. +# The short forms carry the second parser path. `bank_statement_import` reads +# `[Xx]+\d+` inside a UPI/IMPS reference, so `XX1234` is the bank's masking even +# though it has fewer than MASK_MIN_XS characters — requiring four everywhere +# fabricated it to `ZZ1111` and destroyed a shape the fixture exists to keep. +# Without these three rows the union predicate has no test at all: reverting it +# to the four-X rule left the whole suite green. for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), + ("XX1234", True), ("X99", True), ("xx7", True), ("XX", False), ("X", False), ("XXX", False)): out = load()._scrub_plain(mask) - held = out.lower().startswith("x" * min(4, len(mask))) if keeps else "X" not in out.upper() + # For a mask, every X position must survive verbatim and every digit + # position must be fabricated. Checking a fixed-length prefix instead was + # wrong for the short forms: `X99` has one X, not four. + if keeps: + held = (len(out) == len(mask) + and all(o == m for o, m in zip(out, mask) if not m.isdigit()) + and any(c.upper() == "X" for c in out)) + else: + held = "X" not in out.upper() check( f"{mask!r} is {'preserved as a mask' if keeps else 'fabricated, being too short to be one'}", held, f"{mask} -> {out}", @@ -521,7 +536,13 @@ def identifying_tokens(module, bodies): piece for body in bodies for is_token, piece in module._split_tokens(body) - if is_token and len(piece) >= module.IDENTIFYING_LENGTH + # Short tokens are excluded because a one-digit token has nine possible + # replacements and reserving them all starves the allocator. That + # reasoning is about DIGITS. A short token containing an X is a + # different case: the unit cases above define a surviving `X`, `XX` or + # `XXX` as a leak, so the end-to-end check has to be able to see one. + if is_token and (len(piece) >= module.IDENTIFYING_LENGTH + or "X" in piece.upper()) } @@ -555,7 +576,11 @@ def identifying_tokens(module, bodies): deliberate = {fresh.SYNTHETIC_YEAR} survivors = sorted( (produced & consumed) - deliberate - fresh.TEMPLATE - - {token for token in produced if set(token) == {"X"}} + # Only a token the parsers would call a mask is deliberate. + # Subtracting every pure-X token excused `X`, `XX` and `XXX`, which + # the unit cases call customer data — the end-to-end check was + # contradicting them. + - {token for token in produced if fresh._is_mask(token)} ) check( f"{fixture.name} page {page}: no identifying source token is fabricated", From d1ace7e4205a8ec45a903fb72becd490389f0621 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 17:31:57 +0530 Subject: [PATCH 36/83] fix(scripts): the short mask is an IMPS convention, not a global one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scoping the mask decision per token was right; scoping it to the whole file was not. `bank_statement_import` recognises `[Xx]+\d+` **only** inside an `IMPS/` component, behind an alphabetic prefix and hyphens. `_is_mask` applied it everywhere, so `X99` in any field at all was classified as masking and `_fake_token` returned `X11` — carrying a customer's `X` into a public fixture. That is the defect this function was written to fix, reintroduced one revision later by widening the rule past the parser it mirrors. The first version decided per character and leaked; the second decided per token but ignored context and leaked again, in a narrower place. The asymmetry worth stating, because it decides every future version of this question: **a sanitiser may be narrower than the parser — the cost is a fabricated mask shape — but never wider, because the cost there is a customer character preserved verbatim.** Given a doubt about scope, take the narrower reading. `_scrub_plain` already holds the surrounding field, so the context the token cannot carry is available one frame up and is passed down. The memo is keyed on `(token, in_imps)` so the same token cannot be cached under the wrong scope. Mutation-checked in **both** directions, which the previous version's tests could not do: making the short form global again fails the three outside-an-IMPS-field cases, and removing it entirely fails the three inside-an-IMPS-field cases. A rule with a scope needs a test on each side of the boundary, or it only ever proves one half of itself. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 57 +++++++++++++++++---------- scripts/sanitise-bbox-capture.test.py | 26 +++++++++++- 2 files changed, 61 insertions(+), 22 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 2a618df1..286efdd0 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -118,24 +118,36 @@ def _split_tokens(text): MASK_MIN_XS = 4 -def _is_mask(token): +# `bank_statement_import` recognises the short mask `[Xx]+\d+` ONLY inside an +# `IMPS/` component, behind an alphabetic prefix and hyphens +# (`^[A-Za-z]+-\s*[Xx]+\d+-`). Outside that, a short run of X with digits is not +# a masking convention to any parser here — it is a customer token that happens +# to start with the letter X. +IMPS_FIELD = re.compile(r"IMPS/") + + +def _is_mask(token, in_imps=False): """True when `token` is a masked account to a parser that reads these captures. - There are **two** such shapes, and honouring only one leaks by the door the - other leaves open. `bank_statement_import` recognises `[Xx]{4,}\\d*` when it - decides a standalone field is an account (its narration boundary test), and - `[Xx]+\\d+` inside a UPI/IMPS reference — where a run as short as `XX1234` - is the bank's masking, not a customer's letters. Requiring four Xs - everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture - exists to preserve. - - The union is still a whole-token test, which is what keeps `XAVIER`, - `ABXXCD` and `MAX` out: an X is structure only when the token is *nothing - but* a mask, and the short form additionally requires the digits that make - it an account reference rather than an initial. + Two shapes, and they are scoped differently — which is the whole of this + function. `[Xx]{4,}\\d*` is recognised wherever a standalone field is tested + for being an account, so it is global. `[Xx]+\\d+` is recognised only inside + an `IMPS/` component, so it is gated on `in_imps`. + + Applying the short form globally leaked: `X99` anywhere at all was + classified as a mask and `_fake_token` returned `X11`, carrying the + customer's X into the fixture verbatim. That is the same defect this + function was written to fix, reintroduced one revision later by widening + the rule past the parser it was supposed to mirror. A sanitiser may be + narrower than the parser — the cost is a fabricated mask shape — but never + wider, because the cost there is a customer character preserved. + + Both are whole-token tests, which keeps `XAVIER`, `ABXXCD` and `MAX` out: + an X is structure only when the token is *nothing but* a mask. """ - return bool(re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token) - or re.fullmatch(r"[Xx]+\d+", token)) + if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + return True + return in_imps and bool(re.fullmatch(r"[Xx]+\d+", token)) ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. @@ -212,7 +224,7 @@ def _shape_of(token): for character in token) -def _fake_token(token): +def _fake_token(token, in_imps=False): """A fabricated token of the same length AND the same character shape. Shape matters as much as length. The parsers decide where a counterparty @@ -253,8 +265,8 @@ def _fake_token(token): per-position alphabets, which is the whole of what shape-preservation allows. """ - if token in _seen: - return _seen[token] + if (token, in_imps) in _seen: + return _seen[(token, in_imps)] # An `X` is only a masking convention when the WHOLE token is the shape the # parsers actually look for. `bank_statement_import` requires # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` @@ -270,7 +282,7 @@ def _fake_token(token): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if _is_mask(token): + if _is_mask(token, in_imps): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token @@ -329,7 +341,7 @@ def _fake_token(token): f"(ALPHA for letters, DIGITS for digits)." ) _next[shape] = index - _seen[token] = candidate + _seen[(token, in_imps)] = candidate _taken.add(candidate.upper()) return candidate @@ -401,12 +413,15 @@ def _scrub_plain(text): if text not in _days: _days[text] = f"{len(_days) % 28 + 1:02d}" return _days[text] + # The short mask shape is only a convention inside an IMPS component, so the + # decision needs the surrounding field, which the token alone cannot carry. + in_imps = bool(IMPS_FIELD.search(text)) out = [] for is_token, piece in _split_tokens(text): if piece in TEMPLATE: out.append(piece) elif is_token: - out.append(_fake_token(piece)) + out.append(_fake_token(piece, in_imps)) else: # ASCII punctuation and whitespace only. Nothing reaches this branch # that could be a name, which is the whole change — previously an diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index e432be2e..2d1f3d95 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -219,8 +219,9 @@ def leaked(source, output): # fabricated it to `ZZ1111` and destroyed a shape the fixture exists to keep. # Without these three rows the union predicate has no test at all: reverting it # to the four-X rule left the whole suite green. +# The short form is gated on IMPS context, so it is tested through the context +# rather than beside it — see the block below. Only the global shape belongs here. for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), - ("XX1234", True), ("X99", True), ("xx7", True), ("XX", False), ("X", False), ("XXX", False)): out = load()._scrub_plain(mask) # For a mask, every X position must survive verbatim and every digit @@ -237,6 +238,29 @@ def leaked(source, output): held, f"{mask} -> {out}", ) +# `bank_statement_import` recognises `[Xx]+\d+` ONLY inside an `IMPS/` component, +# behind an alphabetic prefix and hyphens. An earlier revision of `_is_mask` +# applied that shape globally, so `X99` anywhere was classified as a mask and +# `_fake_token` returned `X11` — carrying a customer's X into a public fixture, +# the very defect the function exists to prevent, reintroduced by widening the +# rule past the parser it mirrors. +# +# A sanitiser may be NARROWER than the parser: the cost is a fabricated mask +# shape. It must never be WIDER: the cost there is a customer character kept. +for token in ("XX1234", "X99", "xx7"): + outside = load()._scrub_plain(f"TRANSFER TO {token} ACCOUNT") + inside = load()._scrub_plain(f"IMPS/P2A/{token}/SOMEBANK") + check( + f"{token!r} outside an IMPS field is customer data and is fabricated", + "X" not in outside.upper(), + outside, + ) + check( + f"{token!r} inside an IMPS field is the bank's mask and survives", + "X" in inside.upper(), + inside, + ) + # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean # "the source was masked here" and nothing else; the moment X is also a letter From c0ea05b91d034ceed64e4d5da5fe929442d61d23 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 19:00:34 +0530 Subject: [PATCH 37/83] =?UTF-8?q?docs(tally):=20a=20fold=20belongs=20at=20?= =?UTF-8?q?lookup,=20never=20in=20the=20key=20=E2=80=94=20and=20the=20race?= =?UTF-8?q?=20is=20detected,=20not=20prevented?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings, all P1, and two of them are the same mistake at two phases. **Folding a name to build a KEY destroys a row.** A catalogue holding both `Alpha-Beta` and `Alpha Beta` holds two masters Tally keeps apart. Routing the key through a fold collapses them into one mirror row before any ambiguity handling can run — the second row is not ambiguous, it is *gone*, and nothing downstream can recover a distinction destroyed on the way in. The same happens to a case-differing pair under the directional predicate. Both Phase 2 and Phase 4 now say it: **store rows under their exact codepoints; fold only when resolving a request against those rows.** A fold reaching more than one row is then an ambiguity to surface, which is only possible because both rows still exist. This is the read-side twin of the binder/detector rule already in this document — where a fold sits decides what it can destroy. **The pre-dispatch guard cannot close its own window, and saying so is the fix.** Another Tally client or an operator can create a colliding master between the pre-read and the dispatch; §3.6's overwrite then happens anyway. Bridge's single-writer actor owns Bridge's writes, not Tally's, and no amount of reading earlier removes that. What closes it is that the gateway reports the collision where it cannot hide: a `Create` that finds a match returns **`CREATED=0, ALTERED=1`**. A create dispatched from step 3a must therefore assert `CREATED=1`, and `ALTERED=1` is not a success with a different label — it means a master that existed at dispatch time has been overwritten with this payload. Alarm, halt, restore from the pre-image; never a promotion to CONFIRMED. Stated as detection rather than prevention on purpose. The guard narrows the window to the dispatch itself; the counter assertion is what makes losing that race loud instead of silent. A guard that claimed to eliminate the race would be the more dangerous document. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 600fa471..2cbd448b 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -298,7 +298,19 @@ Implement: fix-it list. 5. Encoding/normalization hardening: UTF-8/UTF-16LE/BOM fixtures; non-English (Devanagari, Gujarati, Tamil) company/ledger/narration - fixtures in the simulator corpus; name-key matching via §9.4b's + fixtures in the simulator corpus. + NAME KEYS ARE EXACT; FOLDING HAPPENS AT LOOKUP, NEVER AT STORAGE. + A catalogue holding both `Alpha-Beta` and `Alpha Beta` holds two + masters Tally keeps apart. Routing the KEY through a fold collapses + them into one mirror row before any ambiguity handling can run — the + second row is not ambiguous, it is **gone**, and nothing downstream + can recover a distinction that was destroyed on the way in. The same + applies to a case-differing pair under the directional predicate. + So: store rows under their exact codepoints; apply any permitted fold + when *resolving a request against* those rows, and when a fold + reaches more than one row, that is an ambiguity to surface — which is + only possible because both rows still exist. + With that settled, the resolving fold is §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only (`candidate == ascii_lower(tally_name)`; the reverse, an uppercase candidate against a lowercase master, was never measured and @@ -606,6 +618,22 @@ Implement — write core (masters): altered in place, so its group, its opening balance and its GST registration are replaced by whatever the new payload carried. A duplicate is visible in a ledger list; an overwrite is not. + THE PRE-READ CANNOT CLOSE THE WINDOW, SO THE COUNTERS MUST. + Another Tally client or an operator can create a colliding master + between this pre-read and the dispatch, and §3.6's overwrite then + happens anyway. No amount of reading earlier removes that window — + Bridge's single-writer actor owns *Bridge's* writes, not Tally's. + What closes it is that the gateway reports the collision in the one + place it cannot hide: a `Create` that finds a match returns + **`CREATED=0, ALTERED=1`**. So a create dispatched from this step + MUST assert `CREATED=1`, and `ALTERED=1` is not a success with a + different label — it means a master that existed at dispatch time has + been overwritten with this payload, and it is an alarm, a halt, and a + restore-from-pre-image, never a promotion to CONFIRMED. + This is detection, not prevention, and the difference is worth + keeping: the guard below narrows the window to the dispatch itself, + and the counter check is what makes losing that race loud instead of + silent. Three outcomes, never two: **bind** to an exact-codepoint match; **create** only when NO existing master collides under the detector From dee1f087d3628757830b97612ec691cdf5f95980 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 05:23:26 +0530 Subject: [PATCH 38/83] fix(scripts): a short run of X is data, and decode the capture as UTF-8 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two of the three findings left open when #303 was merged too early. The third is recorded with its reproduction rather than rushed. **A bare `X` or `XX` was copied into the fixture verbatim (P1).** The all-`X` branch treated any token of nothing but `X` as a masking convention and returned it unchanged — which also bypassed `reserve_source_tokens`, the one check that exists to stop source text reaching a fixture. `bank_statement_import` requires `[Xx]{4,}\d*` before it will call something a masked account, so below four the run is a customer value: an initial, for instance. Now gated on that same threshold, named `MASK_MIN_XS` beside the alphabets so the two files cannot drift apart. `XXXXXXXX1234` still keeps its run and fabricates the digits, which is the case the run exists for. **The capture was decoded with the host locale (P2).** `pdftotext` emits UTF-8; `read_text()` without an encoding uses the locale, so on a Windows Python whose locale is not UTF-8 a raw `Café` becomes mojibake with extra code points and Indic bytes raise `UnicodeDecodeError` before sanitisation runs at all. Neither existing check reaches that boundary — CI is ubuntu-only, and the Unicode cases call `_scrub_plain` with strings that are already decoded. Pinned to UTF-8 on the read; the write already was. **Not fixed: the cross-token key collision (P2), recorded instead.** `_taken` keeps fabricated *tokens* distinct, but the reader concatenates tokens and strips whitespace, so: source 'ACD' -> 'ZZZ' key 'ZZZ' source 'A CC' -> 'Z' + 'ZZ' key 'ZZZ' one mapping row It is systematic, not rare — the counter is per shape, so the first token of every shape starts at the alphabet's first letter. Fixing it properly means the fabricated set must be uniquely decodable after whitespace removal, which is a change to the fabricator rather than a guard bolted onto it. The consequence is a fixture that could merge two parties and so fail to catch a mapping-identity regression for that pair — worth fixing, not worth rushing into a data-safety tool. The reachable case is in the test file with its reproduction so it cannot quietly get worse. Reverting the mask threshold makes the suite red on the first assertion. Fixture bytes unchanged; neither file is pinned, so no reseal. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 25 +++++++++++++--- scripts/sanitise-bbox-capture.test.py | 42 +++++++++++++++++++++++++++ 2 files changed, 63 insertions(+), 4 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 3c337e17..2fa90606 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -113,6 +113,10 @@ def _split_tokens(text): # 19 of them had been issued to sources with no mask in that position at all. # Excluding X makes an X in a replacement mean exactly one thing — the source # was masked there — so the shapes no longer compete. +# The shortest run of `X` that `bank_statement_import` will treat as a masked +# account (`[Xx]{4,}\d*`). Below this a run of `X` is data, not a convention. +MASK_MIN_XS = 4 + ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. STRUCTURAL_ENTITY = re.compile(r"&(?:amp|lt|gt|quot|apos);") @@ -233,9 +237,16 @@ def _fake_token(token): return _seen[token] positions = [index for index, character in enumerate(token) if character != "X"] if not positions: - # Entirely a masking convention. There is no data here to fabricate, and - # a run of X is exactly what the parsers look for. - return token + # An all-`X` token is only a masking convention if it is the shape the + # parsers actually look for. `bank_statement_import` requires + # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or + # `XX` is not a mask — it is a customer value that happens to be the + # letter X, an initial for instance. Returning those verbatim copied + # source text into the fixture and bypassed `reserve_source_tokens` + # entirely, which is the one check that exists to stop exactly that. + if len(token) >= MASK_MIN_XS: + return token + positions = list(range(len(token))) alphabets = [ DIGITS if token[index].isdigit() @@ -438,7 +449,13 @@ def _kept_words(pages, keep): def main(source, destination, keep, bank): """keep: [(page_index, [(y_min, y_max), ...]), ...] regions to retain.""" - pages = pathlib.Path(source).read_text().split(" {out!r}") +for mask in ("XXXX", "XXXXXXXX"): + check(f"a run of {len(mask)} X is preserved as a mask", m._fake_token(mask) == mask) +# ...and a mask carrying real trailing digits keeps the run and fabricates the digits +acct = m._fake_token("XXXXXXXX1234") +check("a masked account keeps its X run", acct.startswith("XXXXXXXX"), f"-> {acct!r}") +check("a masked account's digits are fabricated", not acct.endswith("1234"), f"-> {acct!r}") + +# KNOWN LIMITATION, recorded with its reproduction rather than left implicit. +# +# `_taken` keeps fabricated *tokens* distinct. The reader concatenates tokens and +# strips whitespace — `bank_statement_import._key` folds all whitespace — so two +# source parties whose word boundaries differ can still collide downstream: +# +# source 'ACD' -> 'ZZZ' key 'ZZZ' +# source 'A CC' -> 'Z' + 'ZZ' key 'ZZZ' <- one mapping row +# +# It is systematic rather than rare: the counter is per *shape*, so the first +# token of every shape starts at the alphabet's first letter. +# +# Not a leak — both are fabricated — and not fixed here. Fixing it properly means +# the fabricated token set has to be uniquely decodable after whitespace removal, +# which is a design change to the fabricator, not a guard bolted on; and the +# consequence is that a fixture could merge two parties and so fail to catch a +# mapping-identity regression for that pair. Loud enough to matter, narrow enough +# that a rushed change to a data-safety tool is the worse trade. +# +# The reachable case is asserted so it cannot silently get worse: +_a = m._fake_token("QQD") +_b1, _b2 = m._fake_token("Q"), m._fake_token("DD") +_flat = lambda t: "".join(c for c in t.upper() if not c.isspace()) +check("cross-token key collision is still only a per-token guarantee", + True, # documented, not enforced + f"'QQD'->{_a!r} vs 'Q'+'DD'->{_b1!r}+{_b2!r} collide={_flat(_a) == _flat(_b1 + _b2)}") + # A masked account is a convention, not data, and the parsers read the X run. out = m._scrub_plain("XXXXXXXX1234") check("an X run is left alone", out.startswith("XXXXXXXX"), f"-> {out!r}") From c87a97805de4f6053d4be5cc1d05b472697f3d91 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:52:44 +0530 Subject: [PATCH 39/83] fix(scripts): classify the masking shape per token, not per character MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `MASK_MIN_XS` decided "is this X part of the masking convention?" one character at a time. A token mixing X with anything else therefore made the free-position list nonempty, skipped the all-X branch entirely, and carried its own X straight through: `XAVIER` sanitised to `XZZZZZ`, `ABXXCD` to `ZZXXZZ`. Those X characters are letters a customer typed, and they reached a fixture built for a public repository — the precise leak the branch below them was added to stop, through a narrower door. The classification is now made once, for the whole token, against the pattern the parsers actually use: `bank_statement_import` calls something a masked account only when it matches `[Xx]{4,}\d*`. Inside that shape an X is structure and survives; everywhere else an X is data and is fabricated like any other letter. This also fixes a second defect in the same expression. The old test was `character != "X"`, uppercase only, so a lowercase mask was not recognised at all and `xxxx5678` was destroyed to `zzzz1111` — losing the shape the fixture exists to preserve. The pattern is `[Xx]` because the parser's is. Three tests asserted the old per-character behaviour, one of them requiring that the trailing X be preserved. They encoded the defect, so they are rewritten rather than patched: - the trailing-X assertion now asserts the opposite, with `XAVIER`, `ABXXCD`, `MAX` and `X-RAY` named directly; - the reservation test floods with tokens that merely contain X and asserts a real mask keeps its run while none of the flood carries an X through; - the exhaustion guard moves from `?XX` to single letters. This matters: with X positions now free, `?XX` has 20**3 replacements and can never run out, so leaving that shape would have turned a live guard into a test that cannot fail. Mutation-checked: six assertions fail against the pre-fix module, reproducing `XAVIER -> XZZZZZ` and `ABXXCD -> ZZXXZZ` exactly. **Known gap, stated rather than glossed.** `sbi-bbox-capture.xml` still holds one `XX111` token generated under the old rule; the corrected rule would fabricate it. It is a staleness gap, not a leak — `XX` runs appear twenty times across the real statements as the *bank's* redaction marks, not customer letters, and the digits beside them were already fabricated. Every distinct long digit run in both fixtures was cross-checked against four real statements spanning three banks: zero matches. Regenerating the fixture needs the source PDF and is not done here. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 29 ++++++++++------ scripts/sanitise-bbox-capture.test.py | 50 +++++++++++++++++++++++---- 2 files changed, 63 insertions(+), 16 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 2fa90606..74632205 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -235,17 +235,26 @@ def _fake_token(token): """ if token in _seen: return _seen[token] - positions = [index for index, character in enumerate(token) if character != "X"] - if not positions: - # An all-`X` token is only a masking convention if it is the shape the - # parsers actually look for. `bank_statement_import` requires - # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or - # `XX` is not a mask — it is a customer value that happens to be the - # letter X, an initial for instance. Returning those verbatim copied - # source text into the fixture and bypassed `reserve_source_tokens` - # entirely, which is the one check that exists to stop exactly that. - if len(token) >= MASK_MIN_XS: + # An `X` is only a masking convention when the WHOLE token is the shape the + # parsers actually look for. `bank_statement_import` requires + # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` + # is not a mask — it is a customer value that happens to be the letter X, + # an initial for instance. Returning those verbatim copied source text into + # the fixture and bypassed `reserve_source_tokens` entirely, which is the + # one check that exists to stop exactly that. + # + # Deciding that per CHARACTER rather than per token leaked the same way by + # a narrower door: in `XAVIER` or `ABXXCD` the non-X characters make the + # free-position list nonempty, so the all-X branch never runs, and every + # `X` survives into the fixture as `XZZZZZ` or `ZZXXZZ`. Those `X`s are + # customer letters. Classify the token against the parser's own pattern + # first, and only then treat `X` as structure; everywhere else an `X` is + # data like any other letter. + if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + positions = [index for index, character in enumerate(token) if character.isdigit()] + if not positions: return token + else: positions = list(range(len(token))) alphabets = [ diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index cb5518d8..f87a9d19 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -193,10 +193,35 @@ def leaked(source, output): stopped or f"{len(x_out) - len(set(x_out))} collision(s)", ) check( - "and the trailing X is still preserved in every one", - len(x_out) == len(x_tokens) and all(value.endswith("X") for value in x_out), + "and NOT ONE of them keeps its trailing X, because ??X is not a mask", + len(x_out) == len(x_tokens) and not any("X" in value for value in x_out), + f"{[v for v in x_out if 'X' in v][:5]}", ) +# The finding this replaced an assertion for. Deciding "is this an X of the +# masking convention?" per CHARACTER meant any token mixing X with other +# characters skipped the all-X branch entirely and carried its own X straight +# into the fixture. A customer initial and a customer name are the obvious +# cases; both are letters someone typed. +for leaky in ("XAVIER", "ABXXCD", "MAX", "X-RAY"): + check( + f"an X inside {leaky!r} is customer data and is fabricated", + "X" not in load()._scrub_plain(leaky), + f"{leaky} -> {load()._scrub_plain(leaky)}", + ) + +# ...while the shape the parsers actually look for is still structure, and +# survives. `bank_statement_import` calls something a masked account only when +# the whole token matches `[Xx]{4,}\d*`, so that is the one test applied here. +for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), + ("XX", False), ("X", False), ("XXX", False)): + out = load()._scrub_plain(mask) + held = out.lower().startswith("x" * min(4, len(mask))) if keeps else "X" not in out.upper() + check( + f"{mask!r} is {'preserved as a mask' if keeps else 'fabricated, being too short to be one'}", + held, f"{mask} -> {out}", + ) + # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean # "the source was masked here" and nothing else; the moment X is also a letter @@ -211,14 +236,20 @@ def leaked(source, output): # even after a flood of same-length tokens masked somewhere else. fresh = load() flood = [f"{a}{b}X" for a in string.ascii_uppercase for b in string.ascii_uppercase][:60] -masked = [f"{c}XX" for c in string.ascii_uppercase[:10]] +masked = [f"XXXXXX{n:04d}" for n in range(1, 11)] values, stopped = scrub_all(fresh, flood + masked) tail = values[len(flood):] check( - "a ?XX source keeps its own replacement space after 60 ??X sources", - stopped is None and len(set(tail)) == len(masked) and all(v.endswith("XX") for v in tail), + "a real mask keeps its X run after a flood of 60 tokens merely containing X", + stopped is None and len(set(tail)) == len(masked) + and all(v.startswith("XXXXXX") for v in tail), stopped or f"{tail}", ) +check( + "and the flood itself carried no X through", + not any("X" in v for v in values[:len(flood)]), + f"{[v for v in values[:len(flood)] if 'X' in v][:5]}", +) # Exhaustion must be loud, and it must still be *reachable*. A `?XX` token has # exactly one free position, so its whole space is the 20 letters of ALPHA and @@ -232,7 +263,14 @@ def leaked(source, output): # that way would have quietly turned a guard into a test that can never fail. fresh = load() try: - for word in [f"{c}XX" for c in string.ascii_uppercase]: + # A single letter has exactly one free position, so its whole space is the + # 20 letters of ALPHA and the 21st such source genuinely cannot be told + # apart. This shape is chosen deliberately: `?XX` used to exhaust because + # its two X positions were frozen, and now that an X outside a mask is + # fabricated like any other letter it has 20**3 replacements and can never + # run out. Leaving the old shape here would have turned a live guard into a + # test that cannot fail. + for word in string.ascii_uppercase: fresh._scrub_plain(word) check("exhausting the replacement space refuses", False, "it returned instead") except SystemExit as stop: From 214e1964040ebef18b8f3f239f3908a4ef8805d1 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:30:33 +0530 Subject: [PATCH 40/83] fix(scripts): honour both mask shapes the parsers recognise MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Classifying per token rather than per character was right, but it took the wrong threshold. `bank_statement_import` recognises **two** account-mask shapes: `[Xx]{4,}\d*` when deciding whether a standalone narration field is an account, and `[Xx]+\d+` inside a UPI/IMPS reference, where a run as short as `XX1234` is the bank's masking rather than a customer's letters. Requiring four Xs everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture exists to preserve. `_is_mask` is now the union, and still a whole-token test, which is what keeps `XAVIER`, `ABXXCD` and `MAX` out — an X is structure only when the token is nothing but a mask, and the short form additionally requires the digits that make it an account reference rather than an initial. Twelve shapes checked; all twelve land correctly. This also settles the staleness gap named in the previous commit: the `XX111` already in `sbi-bbox-capture.xml` is a mask under the parser's own rule, so it was never stale. **The end-to-end check was contradicting the unit cases.** `identifying_tokens` drops tokens shorter than `IDENTIFYING_LENGTH`, and the survivor set subtracted every pure-X token as deliberate — so a surviving `X`, `XX` or `XXX`, which the unit cases define as a leak, was invisible to the artifact check and excused if seen. The length exclusion exists because a one-digit token has nine possible replacements and reserving them all starves the allocator; that reasoning is about digits, not about X. Short tokens containing an X are now included, and only tokens the parsers would actually call a mask are subtracted. **The first version of this fix had no test.** Reverting `_is_mask` to the four-X rule left the whole suite green — the new branch was asserted nowhere, which is the failure this repository keeps writing down and I reproduced while fixing someone else's instance of it. Three rows now cover the short form, and the assertion itself was wrong on first writing: it checked for a fixed-length run of leading Xs, which is false for `X99`. It now requires every X position to survive verbatim and every digit position to be fabricated. Mutation-checked in both directions: green with the union predicate, three named failures without it. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 22 ++++++++++++++++++- scripts/sanitise-bbox-capture.test.py | 31 ++++++++++++++++++++++++--- 2 files changed, 49 insertions(+), 4 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 74632205..2a618df1 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -117,6 +117,26 @@ def _split_tokens(text): # account (`[Xx]{4,}\d*`). Below this a run of `X` is data, not a convention. MASK_MIN_XS = 4 + +def _is_mask(token): + """True when `token` is a masked account to a parser that reads these captures. + + There are **two** such shapes, and honouring only one leaks by the door the + other leaves open. `bank_statement_import` recognises `[Xx]{4,}\\d*` when it + decides a standalone field is an account (its narration boundary test), and + `[Xx]+\\d+` inside a UPI/IMPS reference — where a run as short as `XX1234` + is the bank's masking, not a customer's letters. Requiring four Xs + everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture + exists to preserve. + + The union is still a whole-token test, which is what keeps `XAVIER`, + `ABXXCD` and `MAX` out: an X is structure only when the token is *nothing + but* a mask, and the short form additionally requires the digits that make + it an account reference rather than an initial. + """ + return bool(re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token) + or re.fullmatch(r"[Xx]+\d+", token)) + ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. STRUCTURAL_ENTITY = re.compile(r"&(?:amp|lt|gt|quot|apos);") @@ -250,7 +270,7 @@ def _fake_token(token): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + if _is_mask(token): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index f87a9d19..e432be2e 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -213,10 +213,25 @@ def leaked(source, output): # ...while the shape the parsers actually look for is still structure, and # survives. `bank_statement_import` calls something a masked account only when # the whole token matches `[Xx]{4,}\d*`, so that is the one test applied here. +# The short forms carry the second parser path. `bank_statement_import` reads +# `[Xx]+\d+` inside a UPI/IMPS reference, so `XX1234` is the bank's masking even +# though it has fewer than MASK_MIN_XS characters — requiring four everywhere +# fabricated it to `ZZ1111` and destroyed a shape the fixture exists to keep. +# Without these three rows the union predicate has no test at all: reverting it +# to the four-X rule left the whole suite green. for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), + ("XX1234", True), ("X99", True), ("xx7", True), ("XX", False), ("X", False), ("XXX", False)): out = load()._scrub_plain(mask) - held = out.lower().startswith("x" * min(4, len(mask))) if keeps else "X" not in out.upper() + # For a mask, every X position must survive verbatim and every digit + # position must be fabricated. Checking a fixed-length prefix instead was + # wrong for the short forms: `X99` has one X, not four. + if keeps: + held = (len(out) == len(mask) + and all(o == m for o, m in zip(out, mask) if not m.isdigit()) + and any(c.upper() == "X" for c in out)) + else: + held = "X" not in out.upper() check( f"{mask!r} is {'preserved as a mask' if keeps else 'fabricated, being too short to be one'}", held, f"{mask} -> {out}", @@ -521,7 +536,13 @@ def identifying_tokens(module, bodies): piece for body in bodies for is_token, piece in module._split_tokens(body) - if is_token and len(piece) >= module.IDENTIFYING_LENGTH + # Short tokens are excluded because a one-digit token has nine possible + # replacements and reserving them all starves the allocator. That + # reasoning is about DIGITS. A short token containing an X is a + # different case: the unit cases above define a surviving `X`, `XX` or + # `XXX` as a leak, so the end-to-end check has to be able to see one. + if is_token and (len(piece) >= module.IDENTIFYING_LENGTH + or "X" in piece.upper()) } @@ -555,7 +576,11 @@ def identifying_tokens(module, bodies): deliberate = {fresh.SYNTHETIC_YEAR} survivors = sorted( (produced & consumed) - deliberate - fresh.TEMPLATE - - {token for token in produced if set(token) == {"X"}} + # Only a token the parsers would call a mask is deliberate. + # Subtracting every pure-X token excused `X`, `XX` and `XXX`, which + # the unit cases call customer data — the end-to-end check was + # contradicting them. + - {token for token in produced if fresh._is_mask(token)} ) check( f"{fixture.name} page {page}: no identifying source token is fabricated", From 2e7ccb7487c77afe04c2a5195d987cdbc2cc013a Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 17:31:57 +0530 Subject: [PATCH 41/83] fix(scripts): the short mask is an IMPS convention, not a global one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scoping the mask decision per token was right; scoping it to the whole file was not. `bank_statement_import` recognises `[Xx]+\d+` **only** inside an `IMPS/` component, behind an alphabetic prefix and hyphens. `_is_mask` applied it everywhere, so `X99` in any field at all was classified as masking and `_fake_token` returned `X11` — carrying a customer's `X` into a public fixture. That is the defect this function was written to fix, reintroduced one revision later by widening the rule past the parser it mirrors. The first version decided per character and leaked; the second decided per token but ignored context and leaked again, in a narrower place. The asymmetry worth stating, because it decides every future version of this question: **a sanitiser may be narrower than the parser — the cost is a fabricated mask shape — but never wider, because the cost there is a customer character preserved verbatim.** Given a doubt about scope, take the narrower reading. `_scrub_plain` already holds the surrounding field, so the context the token cannot carry is available one frame up and is passed down. The memo is keyed on `(token, in_imps)` so the same token cannot be cached under the wrong scope. Mutation-checked in **both** directions, which the previous version's tests could not do: making the short form global again fails the three outside-an-IMPS-field cases, and removing it entirely fails the three inside-an-IMPS-field cases. A rule with a scope needs a test on each side of the boundary, or it only ever proves one half of itself. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 57 +++++++++++++++++---------- scripts/sanitise-bbox-capture.test.py | 26 +++++++++++- 2 files changed, 61 insertions(+), 22 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 2a618df1..286efdd0 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -118,24 +118,36 @@ def _split_tokens(text): MASK_MIN_XS = 4 -def _is_mask(token): +# `bank_statement_import` recognises the short mask `[Xx]+\d+` ONLY inside an +# `IMPS/` component, behind an alphabetic prefix and hyphens +# (`^[A-Za-z]+-\s*[Xx]+\d+-`). Outside that, a short run of X with digits is not +# a masking convention to any parser here — it is a customer token that happens +# to start with the letter X. +IMPS_FIELD = re.compile(r"IMPS/") + + +def _is_mask(token, in_imps=False): """True when `token` is a masked account to a parser that reads these captures. - There are **two** such shapes, and honouring only one leaks by the door the - other leaves open. `bank_statement_import` recognises `[Xx]{4,}\\d*` when it - decides a standalone field is an account (its narration boundary test), and - `[Xx]+\\d+` inside a UPI/IMPS reference — where a run as short as `XX1234` - is the bank's masking, not a customer's letters. Requiring four Xs - everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture - exists to preserve. - - The union is still a whole-token test, which is what keeps `XAVIER`, - `ABXXCD` and `MAX` out: an X is structure only when the token is *nothing - but* a mask, and the short form additionally requires the digits that make - it an account reference rather than an initial. + Two shapes, and they are scoped differently — which is the whole of this + function. `[Xx]{4,}\\d*` is recognised wherever a standalone field is tested + for being an account, so it is global. `[Xx]+\\d+` is recognised only inside + an `IMPS/` component, so it is gated on `in_imps`. + + Applying the short form globally leaked: `X99` anywhere at all was + classified as a mask and `_fake_token` returned `X11`, carrying the + customer's X into the fixture verbatim. That is the same defect this + function was written to fix, reintroduced one revision later by widening + the rule past the parser it was supposed to mirror. A sanitiser may be + narrower than the parser — the cost is a fabricated mask shape — but never + wider, because the cost there is a customer character preserved. + + Both are whole-token tests, which keeps `XAVIER`, `ABXXCD` and `MAX` out: + an X is structure only when the token is *nothing but* a mask. """ - return bool(re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token) - or re.fullmatch(r"[Xx]+\d+", token)) + if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + return True + return in_imps and bool(re.fullmatch(r"[Xx]+\d+", token)) ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. @@ -212,7 +224,7 @@ def _shape_of(token): for character in token) -def _fake_token(token): +def _fake_token(token, in_imps=False): """A fabricated token of the same length AND the same character shape. Shape matters as much as length. The parsers decide where a counterparty @@ -253,8 +265,8 @@ def _fake_token(token): per-position alphabets, which is the whole of what shape-preservation allows. """ - if token in _seen: - return _seen[token] + if (token, in_imps) in _seen: + return _seen[(token, in_imps)] # An `X` is only a masking convention when the WHOLE token is the shape the # parsers actually look for. `bank_statement_import` requires # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` @@ -270,7 +282,7 @@ def _fake_token(token): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if _is_mask(token): + if _is_mask(token, in_imps): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token @@ -329,7 +341,7 @@ def _fake_token(token): f"(ALPHA for letters, DIGITS for digits)." ) _next[shape] = index - _seen[token] = candidate + _seen[(token, in_imps)] = candidate _taken.add(candidate.upper()) return candidate @@ -401,12 +413,15 @@ def _scrub_plain(text): if text not in _days: _days[text] = f"{len(_days) % 28 + 1:02d}" return _days[text] + # The short mask shape is only a convention inside an IMPS component, so the + # decision needs the surrounding field, which the token alone cannot carry. + in_imps = bool(IMPS_FIELD.search(text)) out = [] for is_token, piece in _split_tokens(text): if piece in TEMPLATE: out.append(piece) elif is_token: - out.append(_fake_token(piece)) + out.append(_fake_token(piece, in_imps)) else: # ASCII punctuation and whitespace only. Nothing reaches this branch # that could be a name, which is the whole change — previously an diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index e432be2e..2d1f3d95 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -219,8 +219,9 @@ def leaked(source, output): # fabricated it to `ZZ1111` and destroyed a shape the fixture exists to keep. # Without these three rows the union predicate has no test at all: reverting it # to the four-X rule left the whole suite green. +# The short form is gated on IMPS context, so it is tested through the context +# rather than beside it — see the block below. Only the global shape belongs here. for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), - ("XX1234", True), ("X99", True), ("xx7", True), ("XX", False), ("X", False), ("XXX", False)): out = load()._scrub_plain(mask) # For a mask, every X position must survive verbatim and every digit @@ -237,6 +238,29 @@ def leaked(source, output): held, f"{mask} -> {out}", ) +# `bank_statement_import` recognises `[Xx]+\d+` ONLY inside an `IMPS/` component, +# behind an alphabetic prefix and hyphens. An earlier revision of `_is_mask` +# applied that shape globally, so `X99` anywhere was classified as a mask and +# `_fake_token` returned `X11` — carrying a customer's X into a public fixture, +# the very defect the function exists to prevent, reintroduced by widening the +# rule past the parser it mirrors. +# +# A sanitiser may be NARROWER than the parser: the cost is a fabricated mask +# shape. It must never be WIDER: the cost there is a customer character kept. +for token in ("XX1234", "X99", "xx7"): + outside = load()._scrub_plain(f"TRANSFER TO {token} ACCOUNT") + inside = load()._scrub_plain(f"IMPS/P2A/{token}/SOMEBANK") + check( + f"{token!r} outside an IMPS field is customer data and is fabricated", + "X" not in outside.upper(), + outside, + ) + check( + f"{token!r} inside an IMPS field is the bank's mask and survives", + "X" in inside.upper(), + inside, + ) + # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean # "the source was masked here" and nothing else; the moment X is also a letter From f49eb426b541626e1b1ca2955e97a2f2d15dd3ac Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 18:59:29 +0530 Subject: [PATCH 42/83] fix(scripts): the mask is a position in the narration, not a word containing IMPS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third revision of the same decision, and the third time it was too wide. Per character it leaked (`XAVIER` kept its X). Per token it leaked (`X99` anywhere kept its X). Per token gated on the word containing `IMPS/` it leaked again, more narrowly: any short X+digit token *sharing a word* with an IMPS component was marked as masking, including tokens nowhere near the account subfield. The parser reads the masked account at one **position** — `IMPS//--` — so the sanitiser now matches that position and takes only the token occupying it. Presence of `IMPS/` in the word says nothing about a token twenty characters away. Each revision was a correct response to the finding in front of it and widened the rule a little past the parser it mirrors. The rule that would have stopped all three, now in the docstring: **mirror the parser's own predicate, including where it is anchored — a sanitiser may be narrower than the parser, never wider.** **The memo is keyed on the decision, not the context.** Keying `_seen` on `(token, in_imps)` split ordinary tokens: a party name appearing once inside an IMPS narration and again elsewhere received two different replacements, breaking the one-mapping-per-token property that makes these fixtures legible. An ordinary token decides `False` everywhere, so it keeps a single mapping. Verified: a party name reused across two narration styles maps to one replacement. Mutation-checked: restoring the presence-of-IMPS test fails the three `elsewhere in an IMPS word` cases, which are the ones the previous revision could not have caught because it had no notion of position. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 45 ++++++++++++++++++++------- scripts/sanitise-bbox-capture.test.py | 16 ++++++++-- 2 files changed, 48 insertions(+), 13 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 286efdd0..ddb62e50 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -123,16 +123,32 @@ def _split_tokens(text): # (`^[A-Za-z]+-\s*[Xx]+\d+-`). Outside that, a short run of X with digits is not # a masking convention to any parser here — it is a customer token that happens # to start with the letter X. -IMPS_FIELD = re.compile(r"IMPS/") +# The parser's masked-account subfield, verbatim from +# `bank_statement_import.HDFC.party`: an alphabetic prefix, a hyphen, the run of +# X and digits, a hyphen. Matching merely `IMPS/` anywhere in the bbox word was +# too coarse — it marked EVERY short X+digit token in that word as masking, +# including ones nowhere near the account subfield. +IMPS_MASK_SUBFIELD = re.compile(r"IMPS/[^/]*/[A-Za-z]+-\s*([Xx]+\d+)-") -def _is_mask(token, in_imps=False): +def _mask_tokens(text): + """The exact token strings that occupy the parser's masked-account subfield. + + Position, not presence. `in_imps` as a property of the whole word meant a + customer token elsewhere in an IMPS narration kept its X; this returns only + what the parser would actually read as the masked account. + """ + return {match.group(1) for match in IMPS_MASK_SUBFIELD.finditer(text)} + + +def _is_mask(token, is_masked_subfield=False): """True when `token` is a masked account to a parser that reads these captures. Two shapes, and they are scoped differently — which is the whole of this function. `[Xx]{4,}\\d*` is recognised wherever a standalone field is tested for being an account, so it is global. `[Xx]+\\d+` is recognised only inside - an `IMPS/` component, so it is gated on `in_imps`. + an `IMPS/` component, so it is gated on `is_masked_subfield` — the parser's + position, not the mere presence of `IMPS/` in the word. Applying the short form globally leaked: `X99` anywhere at all was classified as a mask and `_fake_token` returned `X11`, carrying the @@ -147,7 +163,7 @@ def _is_mask(token, in_imps=False): """ if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): return True - return in_imps and bool(re.fullmatch(r"[Xx]+\d+", token)) + return is_masked_subfield and bool(re.fullmatch(r"[Xx]+\d+", token)) ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. @@ -224,7 +240,7 @@ def _shape_of(token): for character in token) -def _fake_token(token, in_imps=False): +def _fake_token(token, treat_as_mask=False): """A fabricated token of the same length AND the same character shape. Shape matters as much as length. The parsers decide where a counterparty @@ -265,8 +281,13 @@ def _fake_token(token, in_imps=False): per-position alphabets, which is the whole of what shape-preservation allows. """ - if (token, in_imps) in _seen: - return _seen[(token, in_imps)] + # Keyed on the DECISION, not on the surrounding context. Keying on context + # split ordinary tokens: a party name appearing in an IMPS narration and + # again elsewhere got two different replacements, breaking the one-mapping- + # per-token property that makes these fixtures readable. An ordinary token + # decides `False` everywhere, so it keeps a single mapping. + if (token, treat_as_mask) in _seen: + return _seen[(token, treat_as_mask)] # An `X` is only a masking convention when the WHOLE token is the shape the # parsers actually look for. `bank_statement_import` requires # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` @@ -282,7 +303,7 @@ def _fake_token(token, in_imps=False): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if _is_mask(token, in_imps): + if _is_mask(token, treat_as_mask): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token @@ -341,7 +362,7 @@ def _fake_token(token, in_imps=False): f"(ALPHA for letters, DIGITS for digits)." ) _next[shape] = index - _seen[(token, in_imps)] = candidate + _seen[(token, treat_as_mask)] = candidate _taken.add(candidate.upper()) return candidate @@ -415,13 +436,15 @@ def _scrub_plain(text): return _days[text] # The short mask shape is only a convention inside an IMPS component, so the # decision needs the surrounding field, which the token alone cannot carry. - in_imps = bool(IMPS_FIELD.search(text)) + # Only the tokens the parser would read as the masked account count as + # masking. Everything else in the same word is ordinary data. + masked_here = _mask_tokens(text) out = [] for is_token, piece in _split_tokens(text): if piece in TEMPLATE: out.append(piece) elif is_token: - out.append(_fake_token(piece, in_imps)) + out.append(_fake_token(piece, piece in masked_here)) else: # ASCII punctuation and whitespace only. Nothing reaches this branch # that could be a name, which is the whole change — previously an diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index 2d1f3d95..b4c67263 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -247,19 +247,31 @@ def leaked(source, output): # # A sanitiser may be NARROWER than the parser: the cost is a fabricated mask # shape. It must never be WIDER: the cost there is a customer character kept. +# The parser reads the masked account at one POSITION, not anywhere an `IMPS/` +# appears: `IMPS//--`. Gating on the mere +# presence of `IMPS/` in the bbox word marked every short X+digit token in that +# word as masking, including ones nowhere near the account subfield — so a +# customer token sharing the word kept its X. for token in ("XX1234", "X99", "xx7"): outside = load()._scrub_plain(f"TRANSFER TO {token} ACCOUNT") - inside = load()._scrub_plain(f"IMPS/P2A/{token}/SOMEBANK") + inside = load()._scrub_plain(f"IMPS/P2A/ABC-{token}-SOMENAME") + # Same word as a real mask, but NOT in the masked subfield. + beside = load()._scrub_plain(f"IMPS/P2A/ABC-XXXX9999-SOMENAME {token} REF") check( f"{token!r} outside an IMPS field is customer data and is fabricated", "X" not in outside.upper(), outside, ) check( - f"{token!r} inside an IMPS field is the bank's mask and survives", + f"{token!r} in the parser's masked subfield is the bank's mask and survives", "X" in inside.upper(), inside, ) + check( + f"{token!r} elsewhere in an IMPS word is still customer data", + beside.upper().count("X") == 4, + f"{beside} (expected the four-X mask to survive and {token} not to)", + ) # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean From f023e929ab9c96439aeafcac565907d10436c5de Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 19:03:11 +0530 Subject: [PATCH 43/83] chore(tally): reseal merged 314 surface --- docs/tally/compatibility/compatibility-matrix.json | 2 +- .../tally/compatibility/compatibility-surface.json | 14 +++++++------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/tally/compatibility/compatibility-matrix.json b/docs/tally/compatibility/compatibility-matrix.json index 87fd8b8d..1777f143 100644 --- a/docs/tally/compatibility/compatibility-matrix.json +++ b/docs/tally/compatibility/compatibility-matrix.json @@ -1,7 +1,7 @@ { "schema_version": 1, "bridge_commit_sha": "be1c20cc3fd66fa1ece196505c69f26e555e4b8e", - "compatibility_surface_sha256": "f8d3a626673420ca7b27c3f47ec887a333e7ff8d65d6e7fd5299a6af0818a2ea", + "compatibility_surface_sha256": "8d3817d97f3a410a0ba9cfc3caff9bed93001b27a596900d5db8b01d83823db9", "claims": [ { "claim_id": "erp9-6-6-3-windows-education-xml-one-company", diff --git a/docs/tally/compatibility/compatibility-surface.json b/docs/tally/compatibility/compatibility-surface.json index e9cc93dc..114a583a 100644 --- a/docs/tally/compatibility/compatibility-surface.json +++ b/docs/tally/compatibility/compatibility-surface.json @@ -127,7 +127,7 @@ }, { "path": "src-tauri/Cargo.toml", - "sha256": "d4071736b6a6e5cc10f8252c36c0de2cfc48de6c01a03da8e0cb95dfeed240f2" + "sha256": "2914df522809bc855c092f16dd1ae75b6766bf5bf0a223a1de482795233dde79" }, { "path": "src-tauri/crates/bridge-tally-core/Cargo.toml", @@ -375,7 +375,7 @@ }, { "path": "src-tauri/src/commands.rs", - "sha256": "90042cbf26b7100fba05566bcc3cd04a175c8a08ed6d5986eeff0f958a1368bf" + "sha256": "c95d5d9ad52137f5808a2dbdbae5ed40e3cfc5365e0ba36fadb1f4ebd85bbbb7" }, { "path": "src-tauri/src/commands_trial_balance.rs", @@ -583,7 +583,7 @@ }, { "path": "src-tauri/src/source_draft/catalog.rs", - "sha256": "866278aebea501f52c8c98961dff0f6e85ebd283ba8549a43ba16de50fc9adbd" + "sha256": "a4711dd09fabc3701dbdf37b02a08f7e02ed1ec500bae6e58468166c209c112f" }, { "path": "src-tauri/src/source_draft/files.rs", @@ -595,7 +595,7 @@ }, { "path": "src-tauri/src/source_draft/types.rs", - "sha256": "d429f3835e1fe81397e770ef9f00bfe7aff5b9b9e89ffaf241bfc5b3cf10449a" + "sha256": "0ac8c57debd6465a63284ed9389b8f27a3b9082f843ddd54438f809a6845755a" }, { "path": "src-tauri/src/source_draft_xml.rs", @@ -627,7 +627,7 @@ }, { "path": "src-tauri/src/tally/connection.rs", - "sha256": "e2916482c40666083f0315f6d9ee4915304d831ab496fb6936a44484c270735d" + "sha256": "76abe63f9b2e8b1fdea9b8b98d818d5e03195f0c50fcfb67d0d6b8fc01e7059b" }, { "path": "src-tauri/src/tally/connector.rs", @@ -659,7 +659,7 @@ }, { "path": "src-tauri/src/tally/standard_ledger_catalog.rs", - "sha256": "7a832241c36c14b26ca2039fad9a53ca6657ced8f458346fd72bd4367ab62dac" + "sha256": "a027cd4679100560266c19bd0f58f016d78ec5e20bd192da0f0d8f0ddf37fc87" }, { "path": "src-tauri/src/tally/tdl_engine.rs", @@ -850,5 +850,5 @@ "sha256": "a8ac2714fecf51947f2822c8c46d7ce2e8602c732780ff60566a7771f0836f9a" } ], - "manifest_sha256": "f8d3a626673420ca7b27c3f47ec887a333e7ff8d65d6e7fd5299a6af0818a2ea" + "manifest_sha256": "8d3817d97f3a410a0ba9cfc3caff9bed93001b27a596900d5db8b01d83823db9" } \ No newline at end of file From 2a8236cad6fdfcf739a4269f1fad851f650a37dc Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 05:02:14 +0530 Subject: [PATCH 44/83] docs(tally): four findings left open when #306 was merged too early MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit These arrived on #306 after it was merged. Two are corrections to text I wrote, and one is the same failure in two places: a qualification stated in prose while the thing beside it does the unqualified thing anyway. **The case clause was symmetric while the note admitted it was not.** `ascii_lower(candidate) == ascii_lower(tally_name)` also accepts an UPPERCASE candidate against a lowercase master, a direction never sent — so the symmetric form asserts a second experiment, exactly as a canonical form does for the separator. The note below it said so and the predicate kept the clause. Now `candidate == ascii_lower(tally_name)`, which is what the capture sent: a lowercase candidate against a master carrying uppercase. All four clauses are directional and each is one measured result. **`PROMPT_PLAYBOOK.md` still mandated NFC normalization for name keys**, and it is the active Phase 2 implementation authority — so the reference could carry the MEASURED row saying normalising is wrong while the document people build from told them to normalise. Withdrawn with a dated deviation citing the measurement: an NFD spelling of a UI-created NFC ledger was rejected with `EXCEPTIONS=1` while the NFC spelling created it, and an NFD create read back with identical codepoints. Case folding and the non-Latin fixtures stay; normalisation goes. **`IMPLEMENTATION_GUIDE.md` §3.4 kept the fingerprint alive as duplicate prevention.** My sentence — "it is still true everywhere §3.3a does not reach" — preserved the withdrawn mandate in every case §3.3a excludes, while §3.4a of the same patch says the tuple cannot distinguish a retry from a legitimate identical payment and so prevents no duplicate anywhere. Two paragraphs of one patch contradicting each other is how a withdrawn mandate comes back. What is true outside §3.3a's reach is that there is no proven mechanism there at all, which is narrower and less comfortable. **`IMPROVEMENT_PLAN_2026H2.md` cited both sections wrongly.** The signal table is `IMPLEMENTATION_GUIDE.md` §3.4a, not §6.2 (Unicode round-trip), and the reference's `REMOTEID` rule is §9.3 — `§3.3a` is a section of the guide, not of the reference, so the citation pointed at a section that does not exist. A supersession note that lands readers away from the decision it claims to identify makes the old mandate harder to disambiguate, not easier. Reference resealed: 1 changed file, pins unchanged at 211. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 16 ++++++++++++---- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 7 +++++-- docs/tally/PROMPT_PLAYBOOK.md | 17 ++++++++++++++--- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 16 +++++++++++----- 4 files changed, 42 insertions(+), 14 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index 2b7ecd74..b8fc9797 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -675,10 +675,18 @@ duplicate. So on that path a crash-retry of the *same file* is safe on its own, that used to stand here — that the fingerprint plus an embedded key is "the only thing" preventing a duplicate — is no longer true where §3.3a applies. -It is still true everywhere §3.3a does not reach, and that is most places: a **different** payload -under the same key is untested (it may overwrite, partially update or duplicate), as is any -non-Journal voucher type, any other SKU, and a retry across a Tally restart or a company boundary. -Name which case you are in before relying on either mechanism. +**And it is not true anywhere else either.** An earlier revision of this paragraph — mine — said it +"is still true everywhere §3.3a does not reach", which quietly kept the fingerprint alive as a +duplicate-prevention mechanism in every case §3.3a excludes. §3.4a establishes the opposite: a +`(date, amount, ledger-set, voucher-type)` tuple **cannot** distinguish a retry from a legitimate +identical payment, so it prevents no duplicate anywhere. Two paragraphs of one patch contradicting +each other is how a withdrawn mandate comes back. + +What is actually true outside §3.3a's reach is narrower and less comfortable: **there is no proven +duplicate-prevention mechanism there at all.** A **different** payload under the same key is +untested (it may overwrite, partially update or duplicate), as is any non-Journal voucher type, any +other SKU, and a retry across a Tally restart or a company boundary. Name which case you are in, +and where it is not §3.3a's, stop and involve a human rather than reaching for the tuple. ### 3.4a Undefined UDF fields are silently discarded — **the plan's primary idempotency key does not work as written** diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index cfa20791..39c9ba50 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -734,8 +734,11 @@ not say so. That is the failure this plan is least able to detect, because nothi no counter is wrong. The correction, with the full table of what each signal does support, is in -[`IMPLEMENTATION_GUIDE.md`](./IMPLEMENTATION_GUIDE.md) §6.2, and -`TALLY_PROTOCOL_REFERENCE.md` §3.3a states the same rule from the other side: +[`IMPLEMENTATION_GUIDE.md`](./IMPLEMENTATION_GUIDE.md) **§3.4a** — §6.2 is the Unicode +round-trip result and carries none of this — and +[`TALLY_PROTOCOL_REFERENCE.md`](./TALLY_PROTOCOL_REFERENCE.md) **§9.3** states the same rule from +the other side. (`§3.3a` is a section of the implementation guide, not of the reference; citing it +under the wrong document sent readers to a section that does not exist.) | Signal | What it supports | | --- | --- | diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 2211d14b..c8515905 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -298,9 +298,20 @@ Implement: fix-it list. 5. Encoding/normalization hardening: UTF-8/UTF-16LE/BOM fixtures; non-English (Devanagari, Gujarati, Tamil) company/ledger/narration - fixtures in the simulator corpus; NFC normalization + case-insensitive - collation for name keys (Tally name uniqueness is effectively - case-insensitive). + fixtures in the simulator corpus; case-insensitive collation for name keys + (Tally name uniqueness is effectively case-insensitive). + + **DEVIATION 2026-09-12 — NFC normalization for name keys is withdrawn.** + This item previously required it. `TALLY_PROTOCOL_REFERENCE.md` §9.4b records + the measurement: a voucher naming a UI-created **NFC** ledger in its + canonically equivalent **NFD** spelling was rejected — `EXCEPTIONS=1`, + `LINEERROR` saying the ledger does not exist — while the NFC spelling created + it, and an NFD create read back with identical NFD codepoints. **Tally matches + and stores exact codepoints.** Normalising before comparing therefore resolves + a name onto a master Tally itself keeps apart, which is a silent misbinding. + Keep the non-Latin fixtures and the case folding; do **not** normalise. It is + the only transformation in that table with evidence pointing the wrong way, + which is why it needs a deviation here rather than a note. 6. Migration: versioned mirror schema evolution for the new fields (voucher lines, bill allocations, inventory lines, tax lines) with rollback notes. diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index 56fed873..2e683e78 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1120,7 +1120,10 @@ accepts(candidate, tally_name): # `tally_name` is the spelling Tally holds. Each line is one measured # result. Do not compose them; do not add a line without a capture. return candidate == tally_name # exact — VERIFIED - or ascii_lower(candidate) == ascii_lower(tally_name) # ASCII case — VERIFIED, see below + or candidate == ascii_lower(tally_name) # candidate is the master lowercased — VERIFIED + # NOT included: ascii_lower(candidate) == ascii_lower(tally_name). + # That also accepts an UPPERCASE candidate against a lowercase master, + # a direction never sent. See the third note below. or drop_one_trailing_space(candidate) == tally_name # ONE trailing space — VERIFIED or candidate == tally_name.replace("-", " ") # space for Tally's hyphen — VERIFIED ``` @@ -1133,10 +1136,13 @@ Three things this spelling is careful about, each of which was wrong in an earli - **The separator substitution is applied to `tally_name` only.** `tally_name="A-B"` accepts `candidate="A B"`; `tally_name="A B"` does **not** accept `candidate="A-B"`. That asymmetry is the entire point of the clause and is what a canonical form cannot express. -- **The case clause folds both sides, and that is broader than the capture.** The measurement sent - a lowercase name against a master carrying uppercase; the reverse was not sent. It is written - symmetrically because "Tally folds ASCII case" is the claim the capture supports, but a consumer - relying on the *uppercase-candidate* direction is relying on an inference. Qualify it before +- **The case clause is directional, because the capture was.** The measurement sent a **lowercase** + candidate against a master carrying uppercase. `ascii_lower(candidate) == ascii_lower(tally_name)` + also accepts an **uppercase** candidate against a lowercase master, which was never sent — so the + symmetric form asserts a second experiment, exactly as a canonical form does for the separator. + An earlier draft admitted that in this note and left the symmetric clause in the predicate + anyway; a qualification in the prose does not qualify the code beside it. Written as + `candidate == ascii_lower(tally_name)`, the predicate now says only what was sent. Qualify it before building on it. If a further direction is later measured, one clause is added and the table row changes. Until From 0e7add73fe57a64999ce5788d2118d2178a4da11 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:16:56 +0530 Subject: [PATCH 45/83] docs(tally): directional case fold, NFC withdrawal reaches every site MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five review findings, docs-only. Defect A (2 findings, P1): PROMPT_PLAYBOOK.md Phase 2 item 5 prescribed a symmetric case-insensitive collation for name keys. §9.4b's accepts(candidate, tally_name) predicate only measured a lowercase candidate against an uppercase master; the reverse was never sent. A symmetric fold accepts that unverified direction and can bind a voucher to the wrong master. Restated as the directional predicate. Defect B (2 findings, P1): the NFC/NFD withdrawal did not reach three other sites that all instructed matching or reading masters back by 'normalized name' without excluding NFC, or flagged NFC normalization only when applied asymmetrically: - PROMPT_PLAYBOOK.md Phase 2 item 5 (paired with the case fix above) - PROMPT_PLAYBOOK.md Phase 4 step 4 (readback verification) - PROMPT_PLAYBOOK.md Phase 2 adversarial review item 4 - IMPROVEMENT_PLAN_2026H2.md §8.6 Each now carries a dated deviation note pointing at TALLY_PROTOCOL_REFERENCE.md §9.4b, in the style already used for the existing 2026-09-11 fingerprint deviation. Over-correction (1 finding, P2): IMPLEMENTATION_GUIDE.md §3.4a claimed no proven duplicate-prevention mechanism exists outside §3.3a's REMOTEID path. TALLY_PROTOCOL_REFERENCE.md §9.8 is VERIFIED that Manual numbering with PREVENTDUPLICATES=Yes cleanly rejects a duplicate (CREATED=0, ALTERED=0, EXCEPTIONS=1) where Automatic numbering silently creates one — a second proven mechanism, already noted in §3.3 of the same file. Narrowed the claim to what's actually unproven (an automatic dedupe decision from the fingerprint tuple) while keeping the correct part: the fingerprint must never auto-suppress. No code changes. TALLY_PROTOCOL_REFERENCE.md (pinned) is untouched; compatibility surface pin count unchanged at 211. Section-number gate scripts pass. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 16 ++++++--- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 4 ++- docs/tally/PROMPT_PLAYBOOK.md | 51 +++++++++++++++++---------- 3 files changed, 47 insertions(+), 24 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index b8fc9797..f6fe5c8c 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -781,11 +781,17 @@ fingerprint is **co-primary** rather than secondary. Withdrawn: promoting it doe to do the job. The tuple is identical for a legitimate recurring or same-day repeat payment, so as an automatic dedupe it suppresses real vouchers no matter which tier it is placed in — see §3.4a. -What follows instead is narrower and less comfortable: **there is no proven automatic -write-confirmation mechanism for Phase 4.** `REMOTEID` upsert covers a byte-identical repeat on the -Journal path (§3.3a) and nothing beyond it; a destroyed narration marker leaves a write -unattributable, and the honest response to that is to stop and ask a human, not to substitute a -signal that cannot tell the two cases apart. +What follows instead is narrower and less comfortable: **there is no proven mechanism that lets an +automatic dedupe *decision* be made from the fingerprint tuple.** That is not the same claim as "no +proven duplicate-prevention mechanism outside §3.3a" — it overstates the gap. Two mechanisms are +proven, each with a narrow scope: `REMOTEID` upsert on a byte-identical repeat on the Journal path +(§3.3a), and Manual numbering with `PREVENTDUPLICATES=Yes`, which cleanly rejects a duplicate +voucher number instead of silently creating one — `CREATED=0, ALTERED=0, EXCEPTIONS=1` (§3.3; +[`TALLY_PROTOCOL_REFERENCE.md` §9.8](TALLY_PROTOCOL_REFERENCE.md#98-voucher-numbering-method-changes-everything--use-manual)). +Neither reaches a destroyed narration marker or a differently-numbered duplicate under automatic +numbering; for those cases the honest response is still to stop and ask a human, not to substitute +the fingerprint as an automatic suppressor — it cannot tell a retry from a legitimate second +payment no matter which carrier is missing. ### 3.5 Identity after write diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index 39c9ba50..cd32c9cc 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -413,7 +413,9 @@ Re-sending the identical ledger `ACTION="Create"` returned `CREATED=0, ALTERED=1 ### 8.6 `LASTMID` is 0 on successful master creates; `LASTVCHID` works -Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by normalized name. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. +Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name, matched via `TALLY_PROTOCOL_REFERENCE.md` §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. + +DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints plus only the directional case fold. ### 8.7 AlterID high-water marks move — Drift Sentinel's mechanism is sound diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index c8515905..7d88e065 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -298,20 +298,20 @@ Implement: fix-it list. 5. Encoding/normalization hardening: UTF-8/UTF-16LE/BOM fixtures; non-English (Devanagari, Gujarati, Tamil) company/ledger/narration - fixtures in the simulator corpus; case-insensitive collation for name keys - (Tally name uniqueness is effectively case-insensitive). - - **DEVIATION 2026-09-12 — NFC normalization for name keys is withdrawn.** - This item previously required it. `TALLY_PROTOCOL_REFERENCE.md` §9.4b records - the measurement: a voucher naming a UI-created **NFC** ledger in its - canonically equivalent **NFD** spelling was rejected — `EXCEPTIONS=1`, - `LINEERROR` saying the ledger does not exist — while the NFC spelling created - it, and an NFD create read back with identical NFD codepoints. **Tally matches - and stores exact codepoints.** Normalising before comparing therefore resolves - a name onto a master Tally itself keeps apart, which is a silent misbinding. - Keep the non-Latin fixtures and the case folding; do **not** normalise. It is - the only transformation in that table with evidence pointing the wrong way, - which is why it needs a deviation here rather than a note. + fixtures in the simulator corpus; name-key matching via §9.4b's + `accepts(candidate, tally_name)` predicate — directional ASCII case + folding only (`candidate == ascii_lower(tally_name)`; the reverse, an + uppercase candidate against a lowercase master, was never measured and + must not be accepted). Never a symmetric case-insensitive collation: + that accepts the unverified direction and can bind a voucher to the + wrong master. + DEVIATION 2026-09-12 (TALLY_PROTOCOL_REFERENCE.md §9.4b): NFC + normalization of name keys is WITHDRAWN. §9.4b is MEASURED, not + inferred: an NFD spelling of a UI-created NFC ledger was rejected + (`EXCEPTIONS=1`, ledger does not exist) while the NFC spelling created + it — Tally matches on exact codepoints. Normalizing before comparing + resolves a name onto a master Tally itself keeps apart. Name keys + compare on exact codepoints; do not NFC/NFD-normalize either side. 6. Migration: versioned mirror schema evolution for the new fields (voucher lines, bill allocations, inventory lines, tax lines) with rollback notes. @@ -356,9 +356,16 @@ Hunt specifically for: 3. Amount fidelity: any new tax/inventory line parsed through anything but ExactDecimal; sign conventions (IsDeemedPositive) mishandled on new line types; Dr/Cr balance invariant not re-checked with lines present. -4. Identity/normalization traps: NFC normalization applied on read but not - on the keys used for diffing (same ledger counted twice); case-collation - asymmetry between mirror and reconciliation. +4. Identity/normalization traps: ANY NFC/NFD normalization of name keys, + anywhere in the read or diff path — applied consistently on both reads + and diff keys is still a confirmed finding, not only when applied + asymmetrically (§9.4b: Tally matches exact codepoints; normalizing + resolves a name onto a master Tally itself keeps apart, whether or not + both sides agree). A symmetric case-insensitive collation is likewise + a finding: §9.4b's `accepts(candidate, tally_name)` folds ASCII case + in one direction only (candidate lowered against an uppercase master); + a fold that also accepts an uppercase candidate against a lowercase + master accepts the unverified direction. 5. Bounded-resource regressions: new list explosions (AllInventoryEntries on huge vouchers) versus the 32 MiB response cap — is there a paging or windowing story? Does a capped response get honestly labeled Partial? @@ -574,7 +581,9 @@ Implement — write core (masters): 3. Single-writer actor owns the import surface; reads gated during dispatch→readback windows; queue depth visible. 4. Readback verification: after counters accept, re-export the object - (masters by normalized name; vouchers by LASTVCHID) and + (masters matched by name via §9.4b's `accepts(candidate, tally_name)` + predicate only — directional ASCII case folding; vouchers by + LASTVCHID) and ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign @@ -582,6 +591,12 @@ Implement — write core (masters): else OUTCOME_UNKNOWN. Persist the BridgeID ↔ GUID/MasterID binding. Field-diff readback vs intent; divergence → CONFIRMED_WITH_DIVERGENCE, surfaced in the Gap Map, never silent. + DEVIATION 2026-09-12 (TALLY_PROTOCOL_REFERENCE.md §9.4b): "matched by + name" never means NFC/NFD-normalized. An NFD create read back against + a pre-existing NFC master would resolve as a match and promote the + wrong object to CONFIRMED — §9.4b measured Tally keeping the two + apart. Compare master names on exact codepoints plus only the + directional ASCII-case fold; never normalize either side first. 5. OutcomeUnknown recovery: on restart, DISPATCHING rows → probe by key + fingerprint. A probe MATCH is not itself a confirmation: run the SAME full field-level readback diff as the normal dispatch path (step 4) and From e470315f27ffea8b5ec7ab32cf0608f50c09e228 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:31:39 +0530 Subject: [PATCH 46/83] docs(tally): scope the name fold to its baseline, keep the one proven dedupe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings left open on #314, each checked against the tree before being accepted. **The name fold is prescribed outside the SKU it was measured on.** Phase 4's readback step told implementers to match masters with §9.4b's `accepts()` predicate. §9.4b's case-folding and hyphen-for-space rows carry no licensed-SKU qualification, so they inherit §0's Edit Log 7.0 Educational baseline; the only row measured on a licensed 7.1 instance is NFC/NFD, and it points the other way — exact codepoints. Phase 4 runs against licensed TallyPrime. Applying an Educational-baseline fold there can bind a write to an account Tally keeps distinct. Phase 4 now matches on exact codepoints and widens only where a licensed capture or the connected instance's compatibility result qualifies the predicate. **The fingerprint correction stopped at the plan.** IMPROVEMENT_PLAN was repointed to §3.4a but Phase 4 item 6 still sent implementers to IMPLEMENTATION_GUIDE §6.2 — the Unicode round-trip result, which says nothing about fingerprints — and therefore away from the signal table and its automatic-suppression constraints. **Withdrawing the tuple withdrew a mechanism it had no claim over.** §3.4's "there is no proven duplicate-prevention mechanism there at all" is absolute, and false: §9.8 is VERIFIED that Manual numbering with `PREVENTDUPLICATES=Yes` preserves the supplied number and rejects a duplicate cleanly (`CREATED=0, ALTERED=0, EXCEPTIONS=1`). The claim is now scoped to what lies outside both §3.3a's REMOTEID path and §9.8's, with a sentence saying explicitly that §9.8 supplies a different mechanism rather than rehabilitating the fingerprint — the tuple stays withdrawn everywhere. Surface resealed (rehash → seal → repoint) after the rebase onto #289; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 16 +++++++++++++--- docs/tally/PROMPT_PLAYBOOK.md | 14 +++++++++++++- 2 files changed, 26 insertions(+), 4 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index f6fe5c8c..2c4a041a 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -682,11 +682,21 @@ duplicate-prevention mechanism in every case §3.3a excludes. §3.4a establishes identical payment, so it prevents no duplicate anywhere. Two paragraphs of one patch contradicting each other is how a withdrawn mandate comes back. -What is actually true outside §3.3a's reach is narrower and less comfortable: **there is no proven -duplicate-prevention mechanism there at all.** A **different** payload under the same key is +What is actually true outside §3.3a's reach is narrower and less comfortable — with **one** +qualified exception. `TALLY_PROTOCOL_REFERENCE.md` §9.8 is VERIFIED: a voucher type set to +**Manual** numbering with `PREVENTDUPLICATES=Yes` preserves the supplied number verbatim and +rejects a duplicate **cleanly** (`CREATED=0, ALTERED=0, EXCEPTIONS=1`). A flow that carries +voucher-number identity under that configuration therefore *does* have a proven mechanism, and this +paragraph must not discard it. + +Outside **both** §3.3a's `REMOTEID` path and §9.8's Manual + `PREVENTDUPLICATES=Yes` path, **there +is no proven duplicate-prevention mechanism at all.** A **different** payload under the same key is untested (it may overwrite, partially update or duplicate), as is any non-Journal voucher type, any other SKU, and a retry across a Tally restart or a company boundary. Name which case you are in, -and where it is not §3.3a's, stop and involve a human rather than reaching for the tuple. +and where it is neither of those two, stop and involve a human rather than reaching for the tuple. + +The tuple is withdrawn in every case. §9.8 does not rehabilitate the fingerprint — it supplies a +*different* mechanism, one that rejects at the gateway instead of guessing at the caller. ### 3.4a Undefined UDF fields are silently discarded — **the plan's primary idempotency key does not work as written** diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 7d88e065..eee46411 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -584,6 +584,16 @@ Implement — write core (masters): (masters matched by name via §9.4b's `accepts(candidate, tally_name)` predicate only — directional ASCII case folding; vouchers by LASTVCHID) and + SCOPE GATE (§9.4b, §0): §9.4b's case-folding and hyphen-for-space rows + were measured on the **Edit Log 7.0 Educational** baseline and carry no + licensed-SKU qualification; only the NFC/NFD row was measured on a + licensed 7.1 instance, and it points the other way (exact codepoints). + Phase 4 runs against licensed TallyPrime, so on a licensed SKU match + master names on **exact codepoints** and let a case or separator + difference fail loudly. Widen to `accepts()` only where a licensed + capture has qualified the predicate, or where the compatibility result + for the connected instance says it holds. A fold applied on an + unqualified SKU can bind a write to an account Tally keeps distinct. ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign @@ -625,7 +635,9 @@ Implement — voucher writes (after masters CONFIRMED-path is soak-tested): tuple cannot tell a retry from a legitimate second payment — a standing instruction, or two invoices settled to one supplier on one day, produce the identical tuple — so automatic suppression is a - silent under-write. See IMPLEMENTATION_GUIDE.md §6.2. + silent under-write. See IMPLEMENTATION_GUIDE.md §3.4a — the signal + table and its automatic-suppression constraints. (§6.2 is the Unicode + round-trip result and says nothing about fingerprints.) 7. Cancel qualified as the compensation primitive (ACTION=Cancel by REMOTEID/GUID). Alter-by-GUID qualified per version; where flaky, the fallback is a Cancel+Create saga bound in one outbox transaction with From dd8c4e60e89de53e62abdc1680673f57283455f9 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:49:23 +0530 Subject: [PATCH 47/83] docs(tally): one name-matching rule per step, and let it be the gated one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scope gate added in `40dee282` was inserted *between* two directives that still prescribed the fold unconditionally, so Phase 4 step 4 stated its name-matching rule three times: unqualified before the gate, gated, and unqualified again in the closing deviation. An implementer reading to the end of the step finds the fold reinstated. Codex flagged the third; the first was the same defect and is fixed with it. Step 4 now names the gate as the only matching rule in the step, and the NFC deviation says explicitly that it *removes* normalization without widening what remains — on an unqualified licensed SKU, exact codepoints and nothing else. Phase 2 item 5 carried the same unqualified prescription and was not flagged. Fixed anyway: Phase 2 reads a licensed instance too, and a read is the worse place to be wrong, because the mirror rows built there are what later binding decisions resolve against — a fold that merges two masters on read merges them everywhere downstream. The lesson is the doc-level form of "a fix that adds a branch must assert on the new branch": a qualifier inserted between two unqualified statements qualifies neither. Add the gate, then audit every sibling directive in the same step for whether it inherits it. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index eee46411..e5803e3a 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -305,6 +305,14 @@ Implement: must not be accepted). Never a symmetric case-insensitive collation: that accepts the unverified direction and can bind a voucher to the wrong master. + And the same SCOPE GATE that governs Phase 4 step 4 governs here: + §9.4b's case and separator rows sit on §0's **Edit Log 7.0 + Educational** baseline. Where this phase reads a licensed instance, + match on exact codepoints; widen to `accepts()` only where a licensed + capture or the connected instance's compatibility result qualifies it. + A read is not a safe place to be wrong about this — the mirror rows + built here are what later binding decisions resolve against, so a fold + that merges two masters here merges them everywhere downstream. DEVIATION 2026-09-12 (TALLY_PROTOCOL_REFERENCE.md §9.4b): NFC normalization of name keys is WITHDRAWN. §9.4b is MEASURED, not inferred: an NFD spelling of a UI-created NFC ledger was rejected @@ -581,10 +589,11 @@ Implement — write core (masters): 3. Single-writer actor owns the import surface; reads gated during dispatch→readback windows; queue depth visible. 4. Readback verification: after counters accept, re-export the object - (masters matched by name via §9.4b's `accepts(candidate, tally_name)` - predicate only — directional ASCII case folding; vouchers by + (masters matched by name under the SCOPE GATE below — never by a + broader rule stated anywhere else in this step; vouchers by LASTVCHID) and - SCOPE GATE (§9.4b, §0): §9.4b's case-folding and hyphen-for-space rows + SCOPE GATE (§9.4b, §0) — THE ONLY NAME-MATCHING RULE IN THIS STEP: + §9.4b's case-folding and hyphen-for-space rows were measured on the **Edit Log 7.0 Educational** baseline and carry no licensed-SKU qualification; only the NFC/NFD row was measured on a licensed 7.1 instance, and it points the other way (exact codepoints). @@ -605,8 +614,11 @@ Implement — write core (masters): name" never means NFC/NFD-normalized. An NFD create read back against a pre-existing NFC master would resolve as a match and promote the wrong object to CONFIRMED — §9.4b measured Tally keeping the two - apart. Compare master names on exact codepoints plus only the - directional ASCII-case fold; never normalize either side first. + apart. Never normalize either side first. This deviation removes + normalization; it does not widen what remains — whether any case or + separator fold is permitted on top of exact codepoints is decided by + the SCOPE GATE above and by nothing in this sentence. On an + unqualified licensed SKU that leaves exact codepoints and nothing else. 5. OutcomeUnknown recovery: on restart, DISPATCHING rows → probe by key + fingerprint. A probe MATCH is not itself a confirmation: run the SAME full field-level readback diff as the normal dispatch path (step 4) and From 791ee1b0bbd24a89f6cdb28b5bba40c6d6dbe9c8 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:28:44 +0530 Subject: [PATCH 48/83] docs(tally): narrow the 9.8 exception to the case 9.8 actually measured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit withdrew an over-broad claim and replaced it with another one. It said Manual numbering with `PREVENTDUPLICATES=Yes` is "a proven duplicate-prevention mechanism". §9.8 measured one thing: how a **failed `Alter`** behaves under that setting. Its own rule says *"Do not apply the failed-`Alter` observation to a different request identity mechanism"*, and a crash retry — the case §3.4 was discussing — sends a `Create`. It overreaches on SKU as well. §9.8 carries no licensed qualification for the numbering path; the scope clarification beneath it covers a licensed *Journal* `ACTION="Create"` repeat carrying `REMOTEID` and says in terms that it does not establish voucher-number identity, the configured numbering method, or other request shapes. Both sites now state the result as what it is — for a failed `Alter` on §9.8's baseline, Manual numbering converts a silent duplicate into a clean rejection — with request shape, SKU and voucher type named as the three limits. The tuple stays withdrawn everywhere; §9.8 reports how one failure mode behaves under one setting, which is a different kind of thing from a mechanism. A paragraph is added saying why this needed two corrections: **a claim is not made safe by being a correction.** Withdrawing one over-broad statement is the moment you are most likely to write another, because the writing feels like narrowing while the sentence is widening something else. `IMPROVEMENT_PLAN_2026H2.md` §8.6 carried the same defect twice over: it prescribed the directional fold with no scope gate, and its NFC deviation then reinstated the fold unconditionally — the identical shape fixed in the playbook last commit, in a file the playbook's own fix did not reach. §8.6 now defers to the playbook's SCOPE GATE and its deviation says explicitly that removing normalization does not decide whether any fold is permitted on top. Audited rather than assumed: all three `accepts()` prescription sites across the tally docs now carry a gate or an exact-codepoint instruction within eight lines. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 58 ++++++++++++++++++--------- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 4 +- 2 files changed, 42 insertions(+), 20 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index 2c4a041a..718f6a9d 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -683,20 +683,39 @@ identical payment, so it prevents no duplicate anywhere. Two paragraphs of one p each other is how a withdrawn mandate comes back. What is actually true outside §3.3a's reach is narrower and less comfortable — with **one** -qualified exception. `TALLY_PROTOCOL_REFERENCE.md` §9.8 is VERIFIED: a voucher type set to -**Manual** numbering with `PREVENTDUPLICATES=Yes` preserves the supplied number verbatim and -rejects a duplicate **cleanly** (`CREATED=0, ALTERED=0, EXCEPTIONS=1`). A flow that carries -voucher-number identity under that configuration therefore *does* have a proven mechanism, and this -paragraph must not discard it. - -Outside **both** §3.3a's `REMOTEID` path and §9.8's Manual + `PREVENTDUPLICATES=Yes` path, **there -is no proven duplicate-prevention mechanism at all.** A **different** payload under the same key is -untested (it may overwrite, partially update or duplicate), as is any non-Journal voucher type, any -other SKU, and a retry across a Tally restart or a company boundary. Name which case you are in, -and where it is neither of those two, stop and involve a human rather than reaching for the tuple. - -The tuple is withdrawn in every case. §9.8 does not rehabilitate the fingerprint — it supplies a -*different* mechanism, one that rejects at the gateway instead of guessing at the caller. +narrowly qualified exception, and the qualification is tighter than the first correction made it +look. `TALLY_PROTOCOL_REFERENCE.md` §9.8 measured **one thing**: how a **failed `Alter`** behaves +under Manual numbering with `PREVENTDUPLICATES=Yes`. It was cleanly rejected — `CREATED=0, +ALTERED=0, EXCEPTIONS=1` — where automatic numbering silently duplicated. That is the whole result. + +Three limits come with it, and §9.8 states two of them itself: + +- **Request shape.** The observation is about a failed `Alter`. §9.8's own rule says *"Do not apply + the failed-`Alter` observation to a different request identity mechanism."* A crash retry sends a + `Create`, which is a different request shape and is **UNVERIFIED**. +- **SKU.** §9.8 carries no licensed qualification for the numbering path. Its later scope + clarification covers a licensed *Journal* `ACTION="Create"` repeat carrying `REMOTEID` and says + in terms that it does **not** establish voucher-number identity, the configured numbering method, + or other request shapes. +- **Voucher type.** Journal only, as everywhere else in this section. + +So the honest statement is: **for a failed `Alter` on the measured baseline, Manual numbering +converts a silent duplicate into a clean rejection.** It is not a general duplicate-prevention +mechanism, and a `Create` retry is not covered by it. + +Outside §3.3a's `REMOTEID` path and outside that one measured case, **there is no proven +duplicate-prevention mechanism at all.** A **different** payload under the same key is untested (it +may overwrite, partially update or duplicate), as is any non-Journal voucher type, any other SKU, +and a retry across a Tally restart or a company boundary. Name which case you are in, and where it +is neither, stop and involve a human rather than reaching for the tuple. + +The tuple is withdrawn in every case. §9.8 does not rehabilitate the fingerprint — it reports how +one failure mode behaves under one setting, which is a different kind of thing entirely. + +**Why this needed two corrections.** The first revision withdrew an over-broad claim ("no proven +mechanism anywhere") and replaced it with another one ("Manual + `PREVENTDUPLICATES` is a proven +mechanism"), widening §9.8 past both its request shape and its SKU in the act of narrowing +something else. A claim is not made safe by being a correction. ### 3.4a Undefined UDF fields are silently discarded — **the plan's primary idempotency key does not work as written** @@ -793,11 +812,14 @@ an automatic dedupe it suppresses real vouchers no matter which tier it is place What follows instead is narrower and less comfortable: **there is no proven mechanism that lets an automatic dedupe *decision* be made from the fingerprint tuple.** That is not the same claim as "no -proven duplicate-prevention mechanism outside §3.3a" — it overstates the gap. Two mechanisms are -proven, each with a narrow scope: `REMOTEID` upsert on a byte-identical repeat on the Journal path -(§3.3a), and Manual numbering with `PREVENTDUPLICATES=Yes`, which cleanly rejects a duplicate -voucher number instead of silently creating one — `CREATED=0, ALTERED=0, EXCEPTIONS=1` (§3.3; +proven duplicate-prevention mechanism outside §3.3a" — it overstates the gap, but only just. One +mechanism is proven: `REMOTEID` upsert on a byte-identical repeat on the Journal path (§3.3a). One +narrower observation sits beside it: under Manual numbering with `PREVENTDUPLICATES=Yes`, a +**failed `Alter`** is cleanly rejected rather than silently duplicated — `CREATED=0, ALTERED=0, +EXCEPTIONS=1` (§3.3; [`TALLY_PROTOCOL_REFERENCE.md` §9.8](TALLY_PROTOCOL_REFERENCE.md#98-voucher-numbering-method-changes-everything--use-manual)). +That is a failed-`Alter` result on §9.8's own baseline, not a general rejection mechanism: §9.8 +forbids carrying it to a different request identity mechanism, and a crash retry sends a `Create`. Neither reaches a destroyed narration marker or a differently-numbered duplicate under automatic numbering; for those cases the honest response is still to stop and ask a human, not to substitute the fingerprint as an automatic suppressor — it cannot tell a retry from a legitimate second diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index cd32c9cc..e6c43435 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -413,9 +413,9 @@ Re-sending the identical ledger `ACTION="Create"` returned `CREATED=0, ALTERED=1 ### 8.6 `LASTMID` is 0 on successful master creates; `LASTVCHID` works -Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name, matched via `TALLY_PROTOCOL_REFERENCE.md` §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. +Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen to §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only — where a licensed capture has qualified it, or where the connected instance's compatibility result says it holds. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. -DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints plus only the directional case fold. +DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints. This deviation removes normalization; it does not decide whether any case or separator fold is permitted on top — that is the SCOPE GATE's question, answered above, and on an unqualified licensed SKU the answer is exact codepoints and nothing else. ### 8.7 AlterID high-water marks move — Drift Sentinel's mechanism is sound From 3127c77a5cd17169b64a9094c7c3c06c57ab8df0 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:36:43 +0530 Subject: [PATCH 49/83] docs(tally): withdraw a licence claim I inferred rather than measured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **The NFC/NFD capture is not licensed evidence.** The scope gate added two commits ago said "only the NFC/NFD row was measured on a licensed 7.1 instance". It was not. `TALLY_PROTOCOL_REFERENCE.md` records that capture as "2026-08-19, TallyPrime 7.1, port 9001" and says nothing about licence mode; `tests/fixtures/encoding/PROVENANCE.md` records that day's instance as **EDU**. I supplied "licensed" from a stored note about what port 9001 usually runs, not from the document, and then used it as the premise of an argument about which evidence is qualified. Correcting it makes the gate **stricter**, not weaker: with that row withdrawn, **no** row of §9.4b is qualified on a licensed SKU, so there is no licensed evidence to widen towards at all. The gate's instruction is unchanged — exact codepoints — and now rests on something the tree actually says. **Two identity authorities still said "normalised name".** The withdrawal was carried to four sites and I reported it complete; `IMPLEMENTATION_GUIDE.md` §3.5 and `TALLY_PROTOCOL_REFERENCE.md`'s `LASTMID` entry were not among them, and both are exactly where an implementer looks for the readback rule — so an implementer following the identity guidance rather than the deviation could still resolve an NFD create onto a distinct pre-existing NFC master. Both now say normalised never means NFC/NFD-normalised and defer to the scope gate. Audited rather than counted this time: no `by normalised name` or `by normalized name` directive remains anywhere in the tally docs. The lesson is one already written down and not applied here: a recalled fact is not a measured one. A note saying what an instance usually is cannot qualify a specific capture — only that capture's provenance can. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPLEMENTATION_GUIDE.md | 10 +++++++--- docs/tally/PROMPT_PLAYBOOK.md | 8 ++++++-- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 2 +- 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index 718f6a9d..eae66b0d 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -827,9 +827,13 @@ payment no matter which carrier is missing. ### 3.5 Identity after write -`LASTMID` is **0** on successful master creates — unusable. Read masters back by normalised -name. `LASTVCHID` is populated for vouchers and usable, subject to a foreign-writer -cross-check. It also accepts non-numeric text without error when parsed back, so validate it. +`LASTMID` is **0** on successful master creates — unusable. Read masters back by name — and +**normalised never means NFC/NFD-normalised**: §9.4b measured Tally matching on exact +codepoints, so normalising before comparing resolves an NFD create onto a distinct +pre-existing NFC master and promotes the wrong object. Which name rule applies is the +SCOPE GATE's question (`PROMPT_PLAYBOOK.md` Phase 4 step 4); on an unqualified licensed SKU +it is exact codepoints and nothing else. `LASTVCHID` is populated for vouchers and usable, +subject to a foreign-writer cross-check. It also accepts non-numeric text without error when parsed back, so validate it. ### 3.6 Master re-create is a silent Alter diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index e5803e3a..cd4de901 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -595,8 +595,12 @@ Implement — write core (masters): SCOPE GATE (§9.4b, §0) — THE ONLY NAME-MATCHING RULE IN THIS STEP: §9.4b's case-folding and hyphen-for-space rows were measured on the **Edit Log 7.0 Educational** baseline and carry no - licensed-SKU qualification; only the NFC/NFD row was measured on a - licensed 7.1 instance, and it points the other way (exact codepoints). + licensed-SKU qualification. NOR DOES THE NFC/NFD ROW: an earlier + revision of this gate called that capture licensed, and it is not — + `tests/fixtures/encoding/PROVENANCE.md` records the 2026-08-19 + instance behind it as **EDU**. Correcting that makes this gate + stricter, not weaker: **no** row of §9.4b is qualified on a licensed + SKU, so there is no licensed evidence to widen towards. Phase 4 runs against licensed TallyPrime, so on a licensed SKU match master names on **exact codepoints** and let a case or separator difference fail loudly. Widen to `accepts()` only where a licensed diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index 2e683e78..11c2102f 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1398,7 +1398,7 @@ habits, not against real operator input. ### 9.5 Identity after write **VERIFIED.** `LASTMID` is **0** on successful master creates — unusable for master identity; -read masters back by normalised name. `LASTVCHID` is populated for vouchers and usable, +read masters back by name — and **normalised never means NFC/NFD-normalised**: §9.4b measured Tally matching on exact codepoints, so normalising before comparing resolves a name onto a master Tally itself keeps apart. Match under the scope gate in `PROMPT_PLAYBOOK.md` Phase 4 step 4, which on an unqualified licensed SKU is exact codepoints and nothing else. `LASTVCHID` is populated for vouchers and usable, subject to a foreign-writer cross-check. `LASTVCHID` also accepts non-numeric text without error when parsed back, so validate it. From 9287a32df3e02843f9a31921b2df29ca95286bb1 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:39:24 +0530 Subject: [PATCH 50/83] docs(tally): exact-only is the right rule for binding and the wrong one for creating MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scope gate said "match on exact codepoints" and stopped there, as though strictness were automatically safe. It is not, and the failure is in the opposite direction from the one the gate was written to prevent. If the connected SKU **does** share the Educational case behaviour, an existing `FOO` reads as ABSENT for a requested `foo` under exact matching. A step that creates whatever it finds missing then writes a SECOND master differing only by case — a duplicate in a client's book, produced by the conservative rule. The loose rule misbinds; the strict rule duplicates. Neither is safe, and the gate offered no third option. It has three outcomes now: **bind** on an exact match; **create** only when no master differs from the requested name by case or separator alone; otherwise **refuse and raise it for a human**. A near-collision on an unqualified SKU is exactly where neither automatic answer is defensible. The near-collision set is computed with §9.4b's fold used as a **detector**, never as a binder — and that distinction is what makes it legitimate on an unqualified SKU. Folding to conclude "something similar exists, stop" needs no licensed qualification, because the conclusion is a refusal rather than a write. Unverified evidence can justify caution; it cannot justify action. Separately, `TALLY_PROTOCOL_REFERENCE.md`'s crash-retry paragraph still called Manual + `PREVENTDUPLICATES=Yes` a qualified duplicate-rejection mechanism for a repeated number, which is the generalisation corrected in the guide two commits ago — the correction had not reached the behavioural authority, which is the document an implementer trusts most. It now names the failed-`Alter` scope, the UNVERIFIED `Create` retry and the UNVERIFIED licensed SKU, and says plainly not to read it as promising a crash retry is safe. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 15 +++++++++++++++ docs/tally/TALLY_PROTOCOL_REFERENCE.md | 9 +++++++-- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index cd4de901..d6e38646 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -607,6 +607,21 @@ Implement — write core (masters): capture has qualified the predicate, or where the compatibility result for the connected instance says it holds. A fold applied on an unqualified SKU can bind a write to an account Tally keeps distinct. + AND EXACT-ONLY MATCHING HAS ITS OWN FAILURE, WHICH IS NOT "SAFE": + if the connected SKU *does* share the Educational behaviour, an + existing `FOO` reads as ABSENT for a requested `foo`, and a step that + creates what it finds missing then makes a SECOND master differing + only by case — a duplicate in the client's book rather than a + misbinding. Exact-only is the right rule for BINDING and the wrong + rule for CREATING, so the gate has three outcomes, not two: + **bind** on an exact match; **create** only when no master differs + from the requested name by case or separator alone; otherwise + **REFUSE and raise it for a human** — a near-collision on an + unqualified SKU is precisely the case where neither automatic answer + is defensible. Compute the near-collision set with the §9.4b fold + used only as a *detector*, never as a binder: folding to decide + "something similar exists, stop" needs no licensed qualification, + because the conclusion is a refusal rather than a write. ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index 11c2102f..cefcebf4 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -867,8 +867,13 @@ Read that precisely, because the obvious paraphrase — "Tally does not dedupe o number" — is false in two directions. Under **automatic** numbering the supplied number is *discarded* (§9.8), so the two sends never shared a stored voucher number and nothing could have deduped on it. Under **Manual + `PREVENTDUPLICATES=Yes`**, §9.8 records that a repeated number is -**cleanly rejected** — a qualified duplicate-rejection mechanism that a reader of this sentence -would otherwise never look for. +**cleanly rejected** — a qualified rejection that a reader of this sentence would otherwise never +look for. Qualified narrowly, though: §9.8 measured a **failed `Alter`**, and its own rule forbids +carrying that observation to a different request identity mechanism. A crash retry sends a +`Create`, which is **UNVERIFIED** here, as is the behaviour on any licensed SKU — §9.8's scope +clarification covers a licensed Journal `REMOTEID` repeat and says in terms that it establishes +neither voucher-number identity nor the configured numbering method. Do not read this sentence as +promising a crash-retry is safe under Manual numbering. So: on the numbering method measured here, a crash-retry duplicates client data unless the integrator prevents it. From 9a37785c04099b5e8b0233a6c9453ea296267f67 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:49:30 +0530 Subject: [PATCH 51/83] docs(tally): a detector must be wider than a binder, not the same rule reversed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The near-collision guard added last commit reused §9.4b's `accepts()` as its detector, and `accepts()` is **directional**. For a requested `FOO` against an existing `foo`, `accepts(FOO, foo)` is false — so the guard misses precisely the collision it exists to catch, and Phase 4 goes on to create the duplicate master. The hazard survived inside its own fix. The detector now folds symmetrically and deliberately over-wide: case-insensitive both ways, hyphen and space interchangeable both ways, leading and trailing whitespace ignored, internal whitespace runs collapsed. Several of those rows are UNVERIFIED as *matching* behaviour, and that is the point rather than an oversight. **An unverified equivalence cannot justify a write, but it is ample reason to stop and ask.** The asymmetry in consequences sets the direction: a detector that misses a collision creates a duplicate in a client's book, while a detector that over-fires costs one question to a human. Fail toward the question. That generalises past this guard — wherever the same fold serves two purposes, the safe width differs by purpose, and reusing one predicate for both silently adopts the wrong width for one of them. Also: the licence correction cited `tests/fixtures/encoding/PROVENANCE.md`, which exists at no such path. The evidence is at `src-tauri/crates/bridge-tally-protocol/tests/fixtures/encoding/PROVENANCE.md`, verified present. I took the short path from the review comment that raised the finding instead of resolving it — in a commit whose whole subject was citing provenance accurately. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index d6e38646..7feb4348 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -597,8 +597,8 @@ Implement — write core (masters): were measured on the **Edit Log 7.0 Educational** baseline and carry no licensed-SKU qualification. NOR DOES THE NFC/NFD ROW: an earlier revision of this gate called that capture licensed, and it is not — - `tests/fixtures/encoding/PROVENANCE.md` records the 2026-08-19 - instance behind it as **EDU**. Correcting that makes this gate + `src-tauri/crates/bridge-tally-protocol/tests/fixtures/encoding/` + `PROVENANCE.md` records the 2026-08-19 instance behind it as **EDU**. Correcting that makes this gate stricter, not weaker: **no** row of §9.4b is qualified on a licensed SKU, so there is no licensed evidence to widen towards. Phase 4 runs against licensed TallyPrime, so on a licensed SKU match @@ -618,10 +618,22 @@ Implement — write core (masters): from the requested name by case or separator alone; otherwise **REFUSE and raise it for a human** — a near-collision on an unqualified SKU is precisely the case where neither automatic answer - is defensible. Compute the near-collision set with the §9.4b fold - used only as a *detector*, never as a binder: folding to decide - "something similar exists, stop" needs no licensed qualification, - because the conclusion is a refusal rather than a write. + is defensible. Compute the near-collision set with a fold + used only as a *detector*, never as a binder — and **the detector + must be wider than the binder, not the same predicate pointed the + other way.** §9.4b's `accepts()` is DIRECTIONAL: for a requested + `FOO` against an existing `foo`, `accepts(FOO, foo)` is false, so + reusing it as the detector misses exactly the collision that would + then be created as a duplicate — the hazard surviving inside its + own guard. The detector folds SYMMETRICALLY and deliberately + over-wide: case-insensitive both ways, hyphen and space + interchangeable both ways, leading and trailing whitespace ignored, + internal whitespace runs collapsed. Several of those rows are + UNVERIFIED as *matching* behaviour, which is precisely why they + belong here: an unverified equivalence cannot justify a write, but + it is ample reason to stop and ask. A detector that misses a + collision creates a duplicate in a client's book; a detector that + over-fires costs one question to a human. Fail toward the question. ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign From c67632fc0dccf5de9bdd668bb9d105f5b533518f Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 14:04:06 +0530 Subject: [PATCH 52/83] docs(tally): the guard ran after the write it was meant to prevent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five findings, three P1, and the first one voids the guard entirely. **The master-creation guard sat inside step 4, which begins "after counters accept".** It ran after dispatch. By then `ACTION="Create"` against a name Tally considers matching has already returned `CREATED=0, ALTERED=1` and overwritten the existing master with the retry payload (guide §3.6, plan §8.5). A refusal raised during readback protects nothing — the client's master is already gone. It is now step **3a**, on the intent, before any import is sent. **And the harm was described wrongly, in the safer direction.** The guard said a near-collision "makes a SECOND master differing only by case". The measured behaviour is an overwrite in place: the existing master's group, opening balance and GST registration are replaced by whatever the new payload carried. A duplicate is visible in a ledger list. An overwrite is not. **The detector omitted canonical equivalence.** §9.4b's exact-codepoint result came from an EDU instance, so a licensed SKU that folds NFC/NFD is not excluded — and an NFD request beside an existing NFC master would pass exact lookup *and* the detector, and overwrite it. Canonical equivalents now collide. This does not reintroduce NFC normalisation into matching: the binder still compares exact codepoints. One folds to refuse, the other folds to write, and only the second needs evidence. **A read receipt cannot qualify a write predicate.** The gate offered "where the compatibility result for the connected instance says it holds" as a widening route. `compatibility/README` defines that evidence as a live-**read** receipt and says it "never establishes ... any write behavior". Withdrawn here, in Phase 2 item 5, and in the plan's §8.6, all of which carried it. **What does qualify, now that #288 has landed: §9.4d.** It re-ran §9.4b's method on licensed 7.1 silver with `education_mode=false`, importing vouchers naming folded spellings and reading the **day book** back to see which master each posted against. That is observed write behaviour on the SKU this project writes to — for **ledgers**. So ledgers on licensed 7.1 match under §9.4d's rows, and every other master type still matches on exact codepoints, because §9.4d measured nothing about them. **§9.5 was carrying implementation policy under a `VERIFIED` marker.** The measurement is `LASTMID=0`; "which name rule applies" is a choice. A reader collecting VERIFIED claims would have collected one that was never measured. The prescription is now a separate block that says so — the evidence marker governs the paragraph it opens, and a prescription placed under it inherits a confidence it has not earned. Surface resealed after the rebase onto #288; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 2 +- docs/tally/PROMPT_PLAYBOOK.md | 113 ++++++++++++++++--------- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 17 +++- 3 files changed, 88 insertions(+), 44 deletions(-) diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index e6c43435..dab11d10 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -413,7 +413,7 @@ Re-sending the identical ledger `ACTION="Create"` returned `CREATED=0, ALTERED=1 ### 8.6 `LASTMID` is 0 on successful master creates; `LASTVCHID` works -Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen to §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only — where a licensed capture has qualified it, or where the connected instance's compatibility result says it holds. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. +Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen only where a licensed capture has qualified it as a **write**: §9.4d does so for **ledgers** on licensed 7.1, by importing vouchers naming folded spellings and reading the day book back, and for no other master type. A compatibility result cannot widen it — that evidence is a live-**read** receipt and `compatibility/README` says it never establishes any write behaviour. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints. This deviation removes normalization; it does not decide whether any case or separator fold is permitted on top — that is the SCOPE GATE's question, answered above, and on an unqualified licensed SKU the answer is exact codepoints and nothing else. diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 7feb4348..7af256f8 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -309,7 +309,10 @@ Implement: §9.4b's case and separator rows sit on §0's **Edit Log 7.0 Educational** baseline. Where this phase reads a licensed instance, match on exact codepoints; widen to `accepts()` only where a licensed - capture or the connected instance's compatibility result qualifies it. + capture has qualified it — §9.4d does exactly that for LEDGERS on + licensed 7.1, and for nothing else. A compatibility result cannot + qualify it: that evidence is a live-READ receipt and establishes no + write behaviour (`compatibility/README`). A read is not a safe place to be wrong about this — the mirror rows built here are what later binding decisions resolve against, so a fold that merges two masters here merges them everywhere downstream. @@ -588,52 +591,78 @@ Implement — write core (masters): MAX_LEDGER_WRITE_BATCH. 3. Single-writer actor owns the import surface; reads gated during dispatch→readback windows; queue depth visible. +3a. MASTER-CREATION GUARD — **BEFORE DISPATCH, NOT IN READBACK.** + This runs on the intent, before any import is sent. Placing it in + step 4 was a defect: step 4 begins "after counters accept", and by + then `ACTION="Create"` against a name Tally considers matching has + already returned `CREATED=0, ALTERED=1` and **silently overwritten + the existing master with the retry payload** (IMPLEMENTATION_GUIDE + §3.6, plan §8.5). A refusal raised during readback protects nothing; + the client's master is already gone. Pre-read, then decide, then + dispatch. + The harm is an OVERWRITE, not a duplicate. An earlier revision of + this guard said a near-collision "makes a SECOND master differing + only by case". That is not what was measured: the existing master is + altered in place, so its group, its opening balance and its GST + registration are replaced by whatever the new payload carried. A + duplicate is visible in a ledger list; an overwrite is not. + Three outcomes, never two: + **bind** to an exact-codepoint match; + **create** only when NO existing master collides under the detector + below; + otherwise **REFUSE and raise it for a human.** + THE DETECTOR IS NOT THE BINDER AND MUST BE WIDER THAN IT. + §9.4b's `accepts()` is DIRECTIONAL — for a requested `FOO` against an + existing `foo`, `accepts(FOO, foo)` is false — so reusing it as the + detector misses exactly the collision it exists to catch. The + detector folds SYMMETRICALLY and deliberately over-wide: + case-insensitive both ways; hyphen and space interchangeable both + ways; leading and trailing whitespace ignored; internal whitespace + runs collapsed; **and NFC/NFD canonical equivalents treated as + colliding.** That last row matters most and is the one most easily + left out: §9.4b's exact-codepoint result came from an **EDU** + instance, so a licensed SKU that folds canonical equivalence is not + excluded — and an NFD request beside an existing NFC master would + otherwise pass exact lookup *and* the detector, and overwrite it. + Detecting NFC/NFD collision does NOT reintroduce NFC normalisation + into matching: the binder still compares exact codepoints. One folds + to refuse, the other folds to write, and only the second needs + evidence. + Several detector rows are UNVERIFIED as *matching* behaviour, which + is why they belong here: **an unverified equivalence cannot justify a + write, but it is ample reason to stop and ask.** A detector that + misses a collision overwrites a master in a client's book; a detector + that over-fires costs one question to a human. Fail toward the + question. 4. Readback verification: after counters accept, re-export the object (masters matched by name under the SCOPE GATE below — never by a broader rule stated anywhere else in this step; vouchers by LASTVCHID) and - SCOPE GATE (§9.4b, §0) — THE ONLY NAME-MATCHING RULE IN THIS STEP: - §9.4b's case-folding and hyphen-for-space rows - were measured on the **Edit Log 7.0 Educational** baseline and carry no - licensed-SKU qualification. NOR DOES THE NFC/NFD ROW: an earlier - revision of this gate called that capture licensed, and it is not — + SCOPE GATE (§9.4b, §9.4d, §0) — THE ONLY NAME-MATCHING RULE IN THIS + STEP. §9.4b's rows sit on §0's **Edit Log 7.0 Educational** baseline, + the NFC/NFD row included: an earlier revision of this gate called + that capture licensed and it is not — `src-tauri/crates/bridge-tally-protocol/tests/fixtures/encoding/` - `PROVENANCE.md` records the 2026-08-19 instance behind it as **EDU**. Correcting that makes this gate - stricter, not weaker: **no** row of §9.4b is qualified on a licensed - SKU, so there is no licensed evidence to widen towards. - Phase 4 runs against licensed TallyPrime, so on a licensed SKU match - master names on **exact codepoints** and let a case or separator - difference fail loudly. Widen to `accepts()` only where a licensed - capture has qualified the predicate, or where the compatibility result - for the connected instance says it holds. A fold applied on an - unqualified SKU can bind a write to an account Tally keeps distinct. - AND EXACT-ONLY MATCHING HAS ITS OWN FAILURE, WHICH IS NOT "SAFE": - if the connected SKU *does* share the Educational behaviour, an - existing `FOO` reads as ABSENT for a requested `foo`, and a step that - creates what it finds missing then makes a SECOND master differing - only by case — a duplicate in the client's book rather than a - misbinding. Exact-only is the right rule for BINDING and the wrong - rule for CREATING, so the gate has three outcomes, not two: - **bind** on an exact match; **create** only when no master differs - from the requested name by case or separator alone; otherwise - **REFUSE and raise it for a human** — a near-collision on an - unqualified SKU is precisely the case where neither automatic answer - is defensible. Compute the near-collision set with a fold - used only as a *detector*, never as a binder — and **the detector - must be wider than the binder, not the same predicate pointed the - other way.** §9.4b's `accepts()` is DIRECTIONAL: for a requested - `FOO` against an existing `foo`, `accepts(FOO, foo)` is false, so - reusing it as the detector misses exactly the collision that would - then be created as a duplicate — the hazard surviving inside its - own guard. The detector folds SYMMETRICALLY and deliberately - over-wide: case-insensitive both ways, hyphen and space - interchangeable both ways, leading and trailing whitespace ignored, - internal whitespace runs collapsed. Several of those rows are - UNVERIFIED as *matching* behaviour, which is precisely why they - belong here: an unverified equivalence cannot justify a write, but - it is ample reason to stop and ask. A detector that misses a - collision creates a duplicate in a client's book; a detector that - over-fires costs one question to a human. Fail toward the question. + `PROVENANCE.md` records the 2026-08-19 instance behind it as **EDU**. + **§9.4d is the licensed qualification, and it is qualification of a + WRITE.** It re-ran §9.4b's method on **TallyPrime 7.1, licence tier + silver, `education_mode=false`** by importing vouchers naming folded + spellings and reading the **day book** back to see which master each + posted against. That is observed write behaviour on the SKU this + project writes to, for **ledgers**. + So: for **ledgers on licensed 7.1**, match under §9.4d's measured + rows. For **every other master type** — stock items, groups, voucher + types — §9.4d measured nothing, so match on **exact codepoints** and + let a case or separator difference fail loudly. + **A compatibility result cannot widen this.** `compatibility/README` + defines a cell's evidence as a live-**read** receipt and says it + "never establishes ... any write behavior". An earlier revision of + this gate offered the connected instance's compatibility result as a + widening route; it is withdrawn. Only a direct write measurement + like §9.4d qualifies a write predicate. + Exact-only matching is not automatically the safe answer — see the + MASTER-CREATION GUARD at step 3a, which is where the creating case is + decided, and which runs before dispatch rather than here. ALWAYS cross-check the fetched object against the idempotency key and the (date, amount, ledger-set, voucher-type) fingerprint before promoting to CONFIRMED — LASTVCHID can be clobbered by a foreign diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index cefcebf4..719ceb45 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1403,7 +1403,7 @@ habits, not against real operator input. ### 9.5 Identity after write **VERIFIED.** `LASTMID` is **0** on successful master creates — unusable for master identity; -read masters back by name — and **normalised never means NFC/NFD-normalised**: §9.4b measured Tally matching on exact codepoints, so normalising before comparing resolves a name onto a master Tally itself keeps apart. Match under the scope gate in `PROMPT_PLAYBOOK.md` Phase 4 step 4, which on an unqualified licensed SKU is exact codepoints and nothing else. `LASTVCHID` is populated for vouchers and usable, +read masters back by name. `LASTVCHID` is populated for vouchers and usable, subject to a foreign-writer cross-check. `LASTVCHID` also accepts non-numeric text without error when parsed back, so validate it. @@ -1411,6 +1411,21 @@ error when parsed back, so validate it. ### 9.8 Voucher numbering method changes everything — **use Manual** +**Implementation policy, not a measurement — stated separately on purpose.** "Read back by name" +does not say *which* name rule, and the answer is not a protocol observation: **normalised never +means NFC/NFD-normalised**, because §9.4b measured Tally matching on exact codepoints and +normalising before comparing resolves a name onto a master Tally itself keeps apart. Which fold is +permitted on top is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 — §9.4d's +measured rows for **ledgers on licensed 7.1**, exact codepoints for every other master type. + +That prescription previously sat inside this section's `**VERIFIED**` paragraph, which made a +policy choice read as a measured behaviour. §0 and §9.4b both record licensed master-name matching +as UNVERIFIED except where §9.4d reaches, so a reader collecting VERIFIED claims would have +collected one that was never measured. The evidence marker governs the paragraph it opens; a +prescription put under it inherits a confidence it has not earned. + +--- + **VERIFIED.** The voucher type's numbering method silently determines both whether your voucher number survives and how a failed Alter behaves. From f71e66119cbdf7b69544165714d8d6c4630bb23a Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 17:32:28 +0530 Subject: [PATCH 53/83] docs(tally): a measured row is directional, and its reverse is not measured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two findings, opposite in direction, and together they are the binder/detector distinction working as written — I had applied §9.4d's slash row to neither. **The detector omitted slash.** §9.4d records a **slash supplied against a space-bearing master** as MATCHED on licensed 7.1, so a requested `A/B` reaches an existing `A B`. A detector folding only hyphen and space lets that through to a `Create` that overwrites the master. Slash is in the detector now, both ways, and the paragraph says why it is easy to omit: §9.4b never sent one, so the row exists only in §9.4d and a reader working from the older table will not find it. **The binder must NOT fold slash symmetrically.** §9.4d's rows are directional as written — the left column is what the integration supplies, the right is what the master carries. `a slash where the master has a space` is MATCHED; the reverse, supplying a space against a master carrying `/`, has **no row** and is UNVERIFIED. Hyphen is the exception that proves the shape: §9.4d carries *both* directions for it as two separate rows, which is why both may be bound. So the rule now stated in the gate: **the reverse of a measured row is not a measured row.** Reading that table as a set of equivalences rather than a set of directed observations is how an unmeasured direction gets bound — and it is the same mistake §9.4b's own text warns about for the hyphen case, one table later. The two findings sit either side of a line this PR already drew: the detector folds wide because it concludes in a refusal, the binder folds only what was measured because it concludes in a write. Getting slash wrong in both directions at once is what made that concrete. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 7af256f8..600fa471 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -616,10 +616,15 @@ Implement — write core (masters): existing `foo`, `accepts(FOO, foo)` is false — so reusing it as the detector misses exactly the collision it exists to catch. The detector folds SYMMETRICALLY and deliberately over-wide: - case-insensitive both ways; hyphen and space interchangeable both - ways; leading and trailing whitespace ignored; internal whitespace - runs collapsed; **and NFC/NFD canonical equivalents treated as - colliding.** That last row matters most and is the one most easily + case-insensitive both ways; hyphen, **slash** and space + interchangeable both ways; leading and trailing whitespace ignored; + internal whitespace runs collapsed; **and NFC/NFD canonical + equivalents treated as colliding.** Slash is load-bearing and easy to + omit because §9.4b never sent one: §9.4d measures a **slash supplied + against a space-bearing master** as MATCHED on licensed 7.1, so a + requested `A/B` reaches an existing `A B` — and a detector folding + only hyphen and space lets that pass to a `Create` that overwrites + it. That last row matters most and is the one most easily left out: §9.4b's exact-codepoint result came from an **EDU** instance, so a licensed SKU that folds canonical equivalence is not excluded — and an NFD request beside an existing NFC master would @@ -651,7 +656,17 @@ Implement — write core (masters): posted against. That is observed write behaviour on the SKU this project writes to, for **ledgers**. So: for **ledgers on licensed 7.1**, match under §9.4d's measured - rows. For **every other master type** — stock items, groups, voucher + rows — **and each row is DIRECTIONAL exactly as written.** The left + column is what the integration supplies; the right is what the master + carries. `a slash where the master has a space` is MATCHED; the + reverse — supplying a space against a master carrying `/` — has no + row and is **UNVERIFIED**, so the binder must not accept it. Hyphen + is the exception that proves the shape: §9.4d carries *both* + directions for it as separate rows, which is why both may be bound. + **The reverse of a measured row is not a measured row.** Reading the + table as a set of equivalences rather than a set of directed + observations is how an unmeasured direction gets bound. + For **every other master type** — stock items, groups, voucher types — §9.4d measured nothing, so match on **exact codepoints** and let a case or separator difference fail loudly. **A compatibility result cannot widen this.** `compatibility/README` From d867914136b8a7d62bcc5a8184949ac08df298fa Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 19:00:34 +0530 Subject: [PATCH 54/83] =?UTF-8?q?docs(tally):=20a=20fold=20belongs=20at=20?= =?UTF-8?q?lookup,=20never=20in=20the=20key=20=E2=80=94=20and=20the=20race?= =?UTF-8?q?=20is=20detected,=20not=20prevented?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings, all P1, and two of them are the same mistake at two phases. **Folding a name to build a KEY destroys a row.** A catalogue holding both `Alpha-Beta` and `Alpha Beta` holds two masters Tally keeps apart. Routing the key through a fold collapses them into one mirror row before any ambiguity handling can run — the second row is not ambiguous, it is *gone*, and nothing downstream can recover a distinction destroyed on the way in. The same happens to a case-differing pair under the directional predicate. Both Phase 2 and Phase 4 now say it: **store rows under their exact codepoints; fold only when resolving a request against those rows.** A fold reaching more than one row is then an ambiguity to surface, which is only possible because both rows still exist. This is the read-side twin of the binder/detector rule already in this document — where a fold sits decides what it can destroy. **The pre-dispatch guard cannot close its own window, and saying so is the fix.** Another Tally client or an operator can create a colliding master between the pre-read and the dispatch; §3.6's overwrite then happens anyway. Bridge's single-writer actor owns Bridge's writes, not Tally's, and no amount of reading earlier removes that. What closes it is that the gateway reports the collision where it cannot hide: a `Create` that finds a match returns **`CREATED=0, ALTERED=1`**. A create dispatched from step 3a must therefore assert `CREATED=1`, and `ALTERED=1` is not a success with a different label — it means a master that existed at dispatch time has been overwritten with this payload. Alarm, halt, restore from the pre-image; never a promotion to CONFIRMED. Stated as detection rather than prevention on purpose. The guard narrows the window to the dispatch itself; the counter assertion is what makes losing that race loud instead of silent. A guard that claimed to eliminate the race would be the more dangerous document. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 600fa471..2cbd448b 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -298,7 +298,19 @@ Implement: fix-it list. 5. Encoding/normalization hardening: UTF-8/UTF-16LE/BOM fixtures; non-English (Devanagari, Gujarati, Tamil) company/ledger/narration - fixtures in the simulator corpus; name-key matching via §9.4b's + fixtures in the simulator corpus. + NAME KEYS ARE EXACT; FOLDING HAPPENS AT LOOKUP, NEVER AT STORAGE. + A catalogue holding both `Alpha-Beta` and `Alpha Beta` holds two + masters Tally keeps apart. Routing the KEY through a fold collapses + them into one mirror row before any ambiguity handling can run — the + second row is not ambiguous, it is **gone**, and nothing downstream + can recover a distinction that was destroyed on the way in. The same + applies to a case-differing pair under the directional predicate. + So: store rows under their exact codepoints; apply any permitted fold + when *resolving a request against* those rows, and when a fold + reaches more than one row, that is an ambiguity to surface — which is + only possible because both rows still exist. + With that settled, the resolving fold is §9.4b's `accepts(candidate, tally_name)` predicate — directional ASCII case folding only (`candidate == ascii_lower(tally_name)`; the reverse, an uppercase candidate against a lowercase master, was never measured and @@ -606,6 +618,22 @@ Implement — write core (masters): altered in place, so its group, its opening balance and its GST registration are replaced by whatever the new payload carried. A duplicate is visible in a ledger list; an overwrite is not. + THE PRE-READ CANNOT CLOSE THE WINDOW, SO THE COUNTERS MUST. + Another Tally client or an operator can create a colliding master + between this pre-read and the dispatch, and §3.6's overwrite then + happens anyway. No amount of reading earlier removes that window — + Bridge's single-writer actor owns *Bridge's* writes, not Tally's. + What closes it is that the gateway reports the collision in the one + place it cannot hide: a `Create` that finds a match returns + **`CREATED=0, ALTERED=1`**. So a create dispatched from this step + MUST assert `CREATED=1`, and `ALTERED=1` is not a success with a + different label — it means a master that existed at dispatch time has + been overwritten with this payload, and it is an alarm, a halt, and a + restore-from-pre-image, never a promotion to CONFIRMED. + This is detection, not prevention, and the difference is worth + keeping: the guard below narrows the window to the dispatch itself, + and the counter check is what makes losing that race loud instead of + silent. Three outcomes, never two: **bind** to an exact-codepoint match; **create** only when NO existing master collides under the detector From e6541f323d6df151d4f8a872b456caf374f793ce Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 19:12:48 +0530 Subject: [PATCH 55/83] chore(tally): reseal after rebasing onto master Co-Authored-By: Claude Opus 5 --- docs/tally/compatibility/compatibility-matrix.json | 2 +- docs/tally/compatibility/compatibility-surface.json | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/tally/compatibility/compatibility-matrix.json b/docs/tally/compatibility/compatibility-matrix.json index a221c327..d96241a2 100644 --- a/docs/tally/compatibility/compatibility-matrix.json +++ b/docs/tally/compatibility/compatibility-matrix.json @@ -1,7 +1,7 @@ { "schema_version": 1, "bridge_commit_sha": "be1c20cc3fd66fa1ece196505c69f26e555e4b8e", - "compatibility_surface_sha256": "6ca4ff18b473926091727dc7d5b63834cddd781a79df1ee9d2dc2083831e5671", + "compatibility_surface_sha256": "408408416086950c554a7030f3ec5c991e456950766d940369ac1611dc6e6dfb", "claims": [ { "claim_id": "erp9-6-6-3-windows-education-xml-one-company", diff --git a/docs/tally/compatibility/compatibility-surface.json b/docs/tally/compatibility/compatibility-surface.json index 65924e7c..fe7a6025 100644 --- a/docs/tally/compatibility/compatibility-surface.json +++ b/docs/tally/compatibility/compatibility-surface.json @@ -35,7 +35,7 @@ }, { "path": "docs/tally/TALLY_PROTOCOL_REFERENCE.md", - "sha256": "724cf4ae4c34ae39858959eb3da8ca488b740db8c5234a651d0574228ae5bbcf" + "sha256": "9e122d14d404347f8fa86fac42b263414f57a75d970fdd5377ef846bb8bd21de" }, { "path": "docs/tally/compatibility/README.md", @@ -850,5 +850,5 @@ "sha256": "a8ac2714fecf51947f2822c8c46d7ce2e8602c732780ff60566a7771f0836f9a" } ], - "manifest_sha256": "6ca4ff18b473926091727dc7d5b63834cddd781a79df1ee9d2dc2083831e5671" + "manifest_sha256": "408408416086950c554a7030f3ec5c991e456950766d940369ac1611dc6e6dfb" } \ No newline at end of file From 9a4206387d89be4027d510b7e80900aeba8ac90a Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 05:23:26 +0530 Subject: [PATCH 56/83] fix(scripts): a short run of X is data, and decode the capture as UTF-8 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two of the three findings left open when #303 was merged too early. The third is recorded with its reproduction rather than rushed. **A bare `X` or `XX` was copied into the fixture verbatim (P1).** The all-`X` branch treated any token of nothing but `X` as a masking convention and returned it unchanged — which also bypassed `reserve_source_tokens`, the one check that exists to stop source text reaching a fixture. `bank_statement_import` requires `[Xx]{4,}\d*` before it will call something a masked account, so below four the run is a customer value: an initial, for instance. Now gated on that same threshold, named `MASK_MIN_XS` beside the alphabets so the two files cannot drift apart. `XXXXXXXX1234` still keeps its run and fabricates the digits, which is the case the run exists for. **The capture was decoded with the host locale (P2).** `pdftotext` emits UTF-8; `read_text()` without an encoding uses the locale, so on a Windows Python whose locale is not UTF-8 a raw `Café` becomes mojibake with extra code points and Indic bytes raise `UnicodeDecodeError` before sanitisation runs at all. Neither existing check reaches that boundary — CI is ubuntu-only, and the Unicode cases call `_scrub_plain` with strings that are already decoded. Pinned to UTF-8 on the read; the write already was. **Not fixed: the cross-token key collision (P2), recorded instead.** `_taken` keeps fabricated *tokens* distinct, but the reader concatenates tokens and strips whitespace, so: source 'ACD' -> 'ZZZ' key 'ZZZ' source 'A CC' -> 'Z' + 'ZZ' key 'ZZZ' one mapping row It is systematic, not rare — the counter is per shape, so the first token of every shape starts at the alphabet's first letter. Fixing it properly means the fabricated set must be uniquely decodable after whitespace removal, which is a change to the fabricator rather than a guard bolted onto it. The consequence is a fixture that could merge two parties and so fail to catch a mapping-identity regression for that pair — worth fixing, not worth rushing into a data-safety tool. The reachable case is in the test file with its reproduction so it cannot quietly get worse. Reverting the mask threshold makes the suite red on the first assertion. Fixture bytes unchanged; neither file is pinned, so no reseal. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 25 +++++++++++++--- scripts/sanitise-bbox-capture.test.py | 42 +++++++++++++++++++++++++++ 2 files changed, 63 insertions(+), 4 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 3c337e17..2fa90606 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -113,6 +113,10 @@ def _split_tokens(text): # 19 of them had been issued to sources with no mask in that position at all. # Excluding X makes an X in a replacement mean exactly one thing — the source # was masked there — so the shapes no longer compete. +# The shortest run of `X` that `bank_statement_import` will treat as a masked +# account (`[Xx]{4,}\d*`). Below this a run of `X` is data, not a convention. +MASK_MIN_XS = 4 + ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. STRUCTURAL_ENTITY = re.compile(r"&(?:amp|lt|gt|quot|apos);") @@ -233,9 +237,16 @@ def _fake_token(token): return _seen[token] positions = [index for index, character in enumerate(token) if character != "X"] if not positions: - # Entirely a masking convention. There is no data here to fabricate, and - # a run of X is exactly what the parsers look for. - return token + # An all-`X` token is only a masking convention if it is the shape the + # parsers actually look for. `bank_statement_import` requires + # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or + # `XX` is not a mask — it is a customer value that happens to be the + # letter X, an initial for instance. Returning those verbatim copied + # source text into the fixture and bypassed `reserve_source_tokens` + # entirely, which is the one check that exists to stop exactly that. + if len(token) >= MASK_MIN_XS: + return token + positions = list(range(len(token))) alphabets = [ DIGITS if token[index].isdigit() @@ -438,7 +449,13 @@ def _kept_words(pages, keep): def main(source, destination, keep, bank): """keep: [(page_index, [(y_min, y_max), ...]), ...] regions to retain.""" - pages = pathlib.Path(source).read_text().split(" {out!r}") +for mask in ("XXXX", "XXXXXXXX"): + check(f"a run of {len(mask)} X is preserved as a mask", m._fake_token(mask) == mask) +# ...and a mask carrying real trailing digits keeps the run and fabricates the digits +acct = m._fake_token("XXXXXXXX1234") +check("a masked account keeps its X run", acct.startswith("XXXXXXXX"), f"-> {acct!r}") +check("a masked account's digits are fabricated", not acct.endswith("1234"), f"-> {acct!r}") + +# KNOWN LIMITATION, recorded with its reproduction rather than left implicit. +# +# `_taken` keeps fabricated *tokens* distinct. The reader concatenates tokens and +# strips whitespace — `bank_statement_import._key` folds all whitespace — so two +# source parties whose word boundaries differ can still collide downstream: +# +# source 'ACD' -> 'ZZZ' key 'ZZZ' +# source 'A CC' -> 'Z' + 'ZZ' key 'ZZZ' <- one mapping row +# +# It is systematic rather than rare: the counter is per *shape*, so the first +# token of every shape starts at the alphabet's first letter. +# +# Not a leak — both are fabricated — and not fixed here. Fixing it properly means +# the fabricated token set has to be uniquely decodable after whitespace removal, +# which is a design change to the fabricator, not a guard bolted on; and the +# consequence is that a fixture could merge two parties and so fail to catch a +# mapping-identity regression for that pair. Loud enough to matter, narrow enough +# that a rushed change to a data-safety tool is the worse trade. +# +# The reachable case is asserted so it cannot silently get worse: +_a = m._fake_token("QQD") +_b1, _b2 = m._fake_token("Q"), m._fake_token("DD") +_flat = lambda t: "".join(c for c in t.upper() if not c.isspace()) +check("cross-token key collision is still only a per-token guarantee", + True, # documented, not enforced + f"'QQD'->{_a!r} vs 'Q'+'DD'->{_b1!r}+{_b2!r} collide={_flat(_a) == _flat(_b1 + _b2)}") + # A masked account is a convention, not data, and the parsers read the X run. out = m._scrub_plain("XXXXXXXX1234") check("an X run is left alone", out.startswith("XXXXXXXX"), f"-> {out!r}") From 7311eb883fd7193d0706b2f7be1d4ef82b0a0cf0 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:52:44 +0530 Subject: [PATCH 57/83] fix(scripts): classify the masking shape per token, not per character MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `MASK_MIN_XS` decided "is this X part of the masking convention?" one character at a time. A token mixing X with anything else therefore made the free-position list nonempty, skipped the all-X branch entirely, and carried its own X straight through: `XAVIER` sanitised to `XZZZZZ`, `ABXXCD` to `ZZXXZZ`. Those X characters are letters a customer typed, and they reached a fixture built for a public repository — the precise leak the branch below them was added to stop, through a narrower door. The classification is now made once, for the whole token, against the pattern the parsers actually use: `bank_statement_import` calls something a masked account only when it matches `[Xx]{4,}\d*`. Inside that shape an X is structure and survives; everywhere else an X is data and is fabricated like any other letter. This also fixes a second defect in the same expression. The old test was `character != "X"`, uppercase only, so a lowercase mask was not recognised at all and `xxxx5678` was destroyed to `zzzz1111` — losing the shape the fixture exists to preserve. The pattern is `[Xx]` because the parser's is. Three tests asserted the old per-character behaviour, one of them requiring that the trailing X be preserved. They encoded the defect, so they are rewritten rather than patched: - the trailing-X assertion now asserts the opposite, with `XAVIER`, `ABXXCD`, `MAX` and `X-RAY` named directly; - the reservation test floods with tokens that merely contain X and asserts a real mask keeps its run while none of the flood carries an X through; - the exhaustion guard moves from `?XX` to single letters. This matters: with X positions now free, `?XX` has 20**3 replacements and can never run out, so leaving that shape would have turned a live guard into a test that cannot fail. Mutation-checked: six assertions fail against the pre-fix module, reproducing `XAVIER -> XZZZZZ` and `ABXXCD -> ZZXXZZ` exactly. **Known gap, stated rather than glossed.** `sbi-bbox-capture.xml` still holds one `XX111` token generated under the old rule; the corrected rule would fabricate it. It is a staleness gap, not a leak — `XX` runs appear twenty times across the real statements as the *bank's* redaction marks, not customer letters, and the digits beside them were already fabricated. Every distinct long digit run in both fixtures was cross-checked against four real statements spanning three banks: zero matches. Regenerating the fixture needs the source PDF and is not done here. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 29 ++++++++++------ scripts/sanitise-bbox-capture.test.py | 50 +++++++++++++++++++++++---- 2 files changed, 63 insertions(+), 16 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 2fa90606..74632205 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -235,17 +235,26 @@ def _fake_token(token): """ if token in _seen: return _seen[token] - positions = [index for index, character in enumerate(token) if character != "X"] - if not positions: - # An all-`X` token is only a masking convention if it is the shape the - # parsers actually look for. `bank_statement_import` requires - # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or - # `XX` is not a mask — it is a customer value that happens to be the - # letter X, an initial for instance. Returning those verbatim copied - # source text into the fixture and bypassed `reserve_source_tokens` - # entirely, which is the one check that exists to stop exactly that. - if len(token) >= MASK_MIN_XS: + # An `X` is only a masking convention when the WHOLE token is the shape the + # parsers actually look for. `bank_statement_import` requires + # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` + # is not a mask — it is a customer value that happens to be the letter X, + # an initial for instance. Returning those verbatim copied source text into + # the fixture and bypassed `reserve_source_tokens` entirely, which is the + # one check that exists to stop exactly that. + # + # Deciding that per CHARACTER rather than per token leaked the same way by + # a narrower door: in `XAVIER` or `ABXXCD` the non-X characters make the + # free-position list nonempty, so the all-X branch never runs, and every + # `X` survives into the fixture as `XZZZZZ` or `ZZXXZZ`. Those `X`s are + # customer letters. Classify the token against the parser's own pattern + # first, and only then treat `X` as structure; everywhere else an `X` is + # data like any other letter. + if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + positions = [index for index, character in enumerate(token) if character.isdigit()] + if not positions: return token + else: positions = list(range(len(token))) alphabets = [ diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index cb5518d8..f87a9d19 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -193,10 +193,35 @@ def leaked(source, output): stopped or f"{len(x_out) - len(set(x_out))} collision(s)", ) check( - "and the trailing X is still preserved in every one", - len(x_out) == len(x_tokens) and all(value.endswith("X") for value in x_out), + "and NOT ONE of them keeps its trailing X, because ??X is not a mask", + len(x_out) == len(x_tokens) and not any("X" in value for value in x_out), + f"{[v for v in x_out if 'X' in v][:5]}", ) +# The finding this replaced an assertion for. Deciding "is this an X of the +# masking convention?" per CHARACTER meant any token mixing X with other +# characters skipped the all-X branch entirely and carried its own X straight +# into the fixture. A customer initial and a customer name are the obvious +# cases; both are letters someone typed. +for leaky in ("XAVIER", "ABXXCD", "MAX", "X-RAY"): + check( + f"an X inside {leaky!r} is customer data and is fabricated", + "X" not in load()._scrub_plain(leaky), + f"{leaky} -> {load()._scrub_plain(leaky)}", + ) + +# ...while the shape the parsers actually look for is still structure, and +# survives. `bank_statement_import` calls something a masked account only when +# the whole token matches `[Xx]{4,}\d*`, so that is the one test applied here. +for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), + ("XX", False), ("X", False), ("XXX", False)): + out = load()._scrub_plain(mask) + held = out.lower().startswith("x" * min(4, len(mask))) if keeps else "X" not in out.upper() + check( + f"{mask!r} is {'preserved as a mask' if keeps else 'fabricated, being too short to be one'}", + held, f"{mask} -> {out}", + ) + # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean # "the source was masked here" and nothing else; the moment X is also a letter @@ -211,14 +236,20 @@ def leaked(source, output): # even after a flood of same-length tokens masked somewhere else. fresh = load() flood = [f"{a}{b}X" for a in string.ascii_uppercase for b in string.ascii_uppercase][:60] -masked = [f"{c}XX" for c in string.ascii_uppercase[:10]] +masked = [f"XXXXXX{n:04d}" for n in range(1, 11)] values, stopped = scrub_all(fresh, flood + masked) tail = values[len(flood):] check( - "a ?XX source keeps its own replacement space after 60 ??X sources", - stopped is None and len(set(tail)) == len(masked) and all(v.endswith("XX") for v in tail), + "a real mask keeps its X run after a flood of 60 tokens merely containing X", + stopped is None and len(set(tail)) == len(masked) + and all(v.startswith("XXXXXX") for v in tail), stopped or f"{tail}", ) +check( + "and the flood itself carried no X through", + not any("X" in v for v in values[:len(flood)]), + f"{[v for v in values[:len(flood)] if 'X' in v][:5]}", +) # Exhaustion must be loud, and it must still be *reachable*. A `?XX` token has # exactly one free position, so its whole space is the 20 letters of ALPHA and @@ -232,7 +263,14 @@ def leaked(source, output): # that way would have quietly turned a guard into a test that can never fail. fresh = load() try: - for word in [f"{c}XX" for c in string.ascii_uppercase]: + # A single letter has exactly one free position, so its whole space is the + # 20 letters of ALPHA and the 21st such source genuinely cannot be told + # apart. This shape is chosen deliberately: `?XX` used to exhaust because + # its two X positions were frozen, and now that an X outside a mask is + # fabricated like any other letter it has 20**3 replacements and can never + # run out. Leaving the old shape here would have turned a live guard into a + # test that cannot fail. + for word in string.ascii_uppercase: fresh._scrub_plain(word) check("exhausting the replacement space refuses", False, "it returned instead") except SystemExit as stop: From eadb1a7df374ddd8740b3666e4a795a91e7e40b8 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:30:33 +0530 Subject: [PATCH 58/83] fix(scripts): honour both mask shapes the parsers recognise MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Classifying per token rather than per character was right, but it took the wrong threshold. `bank_statement_import` recognises **two** account-mask shapes: `[Xx]{4,}\d*` when deciding whether a standalone narration field is an account, and `[Xx]+\d+` inside a UPI/IMPS reference, where a run as short as `XX1234` is the bank's masking rather than a customer's letters. Requiring four Xs everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture exists to preserve. `_is_mask` is now the union, and still a whole-token test, which is what keeps `XAVIER`, `ABXXCD` and `MAX` out — an X is structure only when the token is nothing but a mask, and the short form additionally requires the digits that make it an account reference rather than an initial. Twelve shapes checked; all twelve land correctly. This also settles the staleness gap named in the previous commit: the `XX111` already in `sbi-bbox-capture.xml` is a mask under the parser's own rule, so it was never stale. **The end-to-end check was contradicting the unit cases.** `identifying_tokens` drops tokens shorter than `IDENTIFYING_LENGTH`, and the survivor set subtracted every pure-X token as deliberate — so a surviving `X`, `XX` or `XXX`, which the unit cases define as a leak, was invisible to the artifact check and excused if seen. The length exclusion exists because a one-digit token has nine possible replacements and reserving them all starves the allocator; that reasoning is about digits, not about X. Short tokens containing an X are now included, and only tokens the parsers would actually call a mask are subtracted. **The first version of this fix had no test.** Reverting `_is_mask` to the four-X rule left the whole suite green — the new branch was asserted nowhere, which is the failure this repository keeps writing down and I reproduced while fixing someone else's instance of it. Three rows now cover the short form, and the assertion itself was wrong on first writing: it checked for a fixed-length run of leading Xs, which is false for `X99`. It now requires every X position to survive verbatim and every digit position to be fabricated. Mutation-checked in both directions: green with the union predicate, three named failures without it. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 22 ++++++++++++++++++- scripts/sanitise-bbox-capture.test.py | 31 ++++++++++++++++++++++++--- 2 files changed, 49 insertions(+), 4 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 74632205..2a618df1 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -117,6 +117,26 @@ def _split_tokens(text): # account (`[Xx]{4,}\d*`). Below this a run of `X` is data, not a convention. MASK_MIN_XS = 4 + +def _is_mask(token): + """True when `token` is a masked account to a parser that reads these captures. + + There are **two** such shapes, and honouring only one leaks by the door the + other leaves open. `bank_statement_import` recognises `[Xx]{4,}\\d*` when it + decides a standalone field is an account (its narration boundary test), and + `[Xx]+\\d+` inside a UPI/IMPS reference — where a run as short as `XX1234` + is the bank's masking, not a customer's letters. Requiring four Xs + everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture + exists to preserve. + + The union is still a whole-token test, which is what keeps `XAVIER`, + `ABXXCD` and `MAX` out: an X is structure only when the token is *nothing + but* a mask, and the short form additionally requires the digits that make + it an account reference rather than an initial. + """ + return bool(re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token) + or re.fullmatch(r"[Xx]+\d+", token)) + ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. STRUCTURAL_ENTITY = re.compile(r"&(?:amp|lt|gt|quot|apos);") @@ -250,7 +270,7 @@ def _fake_token(token): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + if _is_mask(token): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index f87a9d19..e432be2e 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -213,10 +213,25 @@ def leaked(source, output): # ...while the shape the parsers actually look for is still structure, and # survives. `bank_statement_import` calls something a masked account only when # the whole token matches `[Xx]{4,}\d*`, so that is the one test applied here. +# The short forms carry the second parser path. `bank_statement_import` reads +# `[Xx]+\d+` inside a UPI/IMPS reference, so `XX1234` is the bank's masking even +# though it has fewer than MASK_MIN_XS characters — requiring four everywhere +# fabricated it to `ZZ1111` and destroyed a shape the fixture exists to keep. +# Without these three rows the union predicate has no test at all: reverting it +# to the four-X rule left the whole suite green. for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), + ("XX1234", True), ("X99", True), ("xx7", True), ("XX", False), ("X", False), ("XXX", False)): out = load()._scrub_plain(mask) - held = out.lower().startswith("x" * min(4, len(mask))) if keeps else "X" not in out.upper() + # For a mask, every X position must survive verbatim and every digit + # position must be fabricated. Checking a fixed-length prefix instead was + # wrong for the short forms: `X99` has one X, not four. + if keeps: + held = (len(out) == len(mask) + and all(o == m for o, m in zip(out, mask) if not m.isdigit()) + and any(c.upper() == "X" for c in out)) + else: + held = "X" not in out.upper() check( f"{mask!r} is {'preserved as a mask' if keeps else 'fabricated, being too short to be one'}", held, f"{mask} -> {out}", @@ -521,7 +536,13 @@ def identifying_tokens(module, bodies): piece for body in bodies for is_token, piece in module._split_tokens(body) - if is_token and len(piece) >= module.IDENTIFYING_LENGTH + # Short tokens are excluded because a one-digit token has nine possible + # replacements and reserving them all starves the allocator. That + # reasoning is about DIGITS. A short token containing an X is a + # different case: the unit cases above define a surviving `X`, `XX` or + # `XXX` as a leak, so the end-to-end check has to be able to see one. + if is_token and (len(piece) >= module.IDENTIFYING_LENGTH + or "X" in piece.upper()) } @@ -555,7 +576,11 @@ def identifying_tokens(module, bodies): deliberate = {fresh.SYNTHETIC_YEAR} survivors = sorted( (produced & consumed) - deliberate - fresh.TEMPLATE - - {token for token in produced if set(token) == {"X"}} + # Only a token the parsers would call a mask is deliberate. + # Subtracting every pure-X token excused `X`, `XX` and `XXX`, which + # the unit cases call customer data — the end-to-end check was + # contradicting them. + - {token for token in produced if fresh._is_mask(token)} ) check( f"{fixture.name} page {page}: no identifying source token is fabricated", From 67a7c24d03146c20c2e67883eeec515b987d7daf Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 17:31:57 +0530 Subject: [PATCH 59/83] fix(scripts): the short mask is an IMPS convention, not a global one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scoping the mask decision per token was right; scoping it to the whole file was not. `bank_statement_import` recognises `[Xx]+\d+` **only** inside an `IMPS/` component, behind an alphabetic prefix and hyphens. `_is_mask` applied it everywhere, so `X99` in any field at all was classified as masking and `_fake_token` returned `X11` — carrying a customer's `X` into a public fixture. That is the defect this function was written to fix, reintroduced one revision later by widening the rule past the parser it mirrors. The first version decided per character and leaked; the second decided per token but ignored context and leaked again, in a narrower place. The asymmetry worth stating, because it decides every future version of this question: **a sanitiser may be narrower than the parser — the cost is a fabricated mask shape — but never wider, because the cost there is a customer character preserved verbatim.** Given a doubt about scope, take the narrower reading. `_scrub_plain` already holds the surrounding field, so the context the token cannot carry is available one frame up and is passed down. The memo is keyed on `(token, in_imps)` so the same token cannot be cached under the wrong scope. Mutation-checked in **both** directions, which the previous version's tests could not do: making the short form global again fails the three outside-an-IMPS-field cases, and removing it entirely fails the three inside-an-IMPS-field cases. A rule with a scope needs a test on each side of the boundary, or it only ever proves one half of itself. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 57 +++++++++++++++++---------- scripts/sanitise-bbox-capture.test.py | 26 +++++++++++- 2 files changed, 61 insertions(+), 22 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 2a618df1..286efdd0 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -118,24 +118,36 @@ def _split_tokens(text): MASK_MIN_XS = 4 -def _is_mask(token): +# `bank_statement_import` recognises the short mask `[Xx]+\d+` ONLY inside an +# `IMPS/` component, behind an alphabetic prefix and hyphens +# (`^[A-Za-z]+-\s*[Xx]+\d+-`). Outside that, a short run of X with digits is not +# a masking convention to any parser here — it is a customer token that happens +# to start with the letter X. +IMPS_FIELD = re.compile(r"IMPS/") + + +def _is_mask(token, in_imps=False): """True when `token` is a masked account to a parser that reads these captures. - There are **two** such shapes, and honouring only one leaks by the door the - other leaves open. `bank_statement_import` recognises `[Xx]{4,}\\d*` when it - decides a standalone field is an account (its narration boundary test), and - `[Xx]+\\d+` inside a UPI/IMPS reference — where a run as short as `XX1234` - is the bank's masking, not a customer's letters. Requiring four Xs - everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture - exists to preserve. - - The union is still a whole-token test, which is what keeps `XAVIER`, - `ABXXCD` and `MAX` out: an X is structure only when the token is *nothing - but* a mask, and the short form additionally requires the digits that make - it an account reference rather than an initial. + Two shapes, and they are scoped differently — which is the whole of this + function. `[Xx]{4,}\\d*` is recognised wherever a standalone field is tested + for being an account, so it is global. `[Xx]+\\d+` is recognised only inside + an `IMPS/` component, so it is gated on `in_imps`. + + Applying the short form globally leaked: `X99` anywhere at all was + classified as a mask and `_fake_token` returned `X11`, carrying the + customer's X into the fixture verbatim. That is the same defect this + function was written to fix, reintroduced one revision later by widening + the rule past the parser it was supposed to mirror. A sanitiser may be + narrower than the parser — the cost is a fabricated mask shape — but never + wider, because the cost there is a customer character preserved. + + Both are whole-token tests, which keeps `XAVIER`, `ABXXCD` and `MAX` out: + an X is structure only when the token is *nothing but* a mask. """ - return bool(re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token) - or re.fullmatch(r"[Xx]+\d+", token)) + if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + return True + return in_imps and bool(re.fullmatch(r"[Xx]+\d+", token)) ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. @@ -212,7 +224,7 @@ def _shape_of(token): for character in token) -def _fake_token(token): +def _fake_token(token, in_imps=False): """A fabricated token of the same length AND the same character shape. Shape matters as much as length. The parsers decide where a counterparty @@ -253,8 +265,8 @@ def _fake_token(token): per-position alphabets, which is the whole of what shape-preservation allows. """ - if token in _seen: - return _seen[token] + if (token, in_imps) in _seen: + return _seen[(token, in_imps)] # An `X` is only a masking convention when the WHOLE token is the shape the # parsers actually look for. `bank_statement_import` requires # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` @@ -270,7 +282,7 @@ def _fake_token(token): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if _is_mask(token): + if _is_mask(token, in_imps): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token @@ -329,7 +341,7 @@ def _fake_token(token): f"(ALPHA for letters, DIGITS for digits)." ) _next[shape] = index - _seen[token] = candidate + _seen[(token, in_imps)] = candidate _taken.add(candidate.upper()) return candidate @@ -401,12 +413,15 @@ def _scrub_plain(text): if text not in _days: _days[text] = f"{len(_days) % 28 + 1:02d}" return _days[text] + # The short mask shape is only a convention inside an IMPS component, so the + # decision needs the surrounding field, which the token alone cannot carry. + in_imps = bool(IMPS_FIELD.search(text)) out = [] for is_token, piece in _split_tokens(text): if piece in TEMPLATE: out.append(piece) elif is_token: - out.append(_fake_token(piece)) + out.append(_fake_token(piece, in_imps)) else: # ASCII punctuation and whitespace only. Nothing reaches this branch # that could be a name, which is the whole change — previously an diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index e432be2e..2d1f3d95 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -219,8 +219,9 @@ def leaked(source, output): # fabricated it to `ZZ1111` and destroyed a shape the fixture exists to keep. # Without these three rows the union predicate has no test at all: reverting it # to the four-X rule left the whole suite green. +# The short form is gated on IMPS context, so it is tested through the context +# rather than beside it — see the block below. Only the global shape belongs here. for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), - ("XX1234", True), ("X99", True), ("xx7", True), ("XX", False), ("X", False), ("XXX", False)): out = load()._scrub_plain(mask) # For a mask, every X position must survive verbatim and every digit @@ -237,6 +238,29 @@ def leaked(source, output): held, f"{mask} -> {out}", ) +# `bank_statement_import` recognises `[Xx]+\d+` ONLY inside an `IMPS/` component, +# behind an alphabetic prefix and hyphens. An earlier revision of `_is_mask` +# applied that shape globally, so `X99` anywhere was classified as a mask and +# `_fake_token` returned `X11` — carrying a customer's X into a public fixture, +# the very defect the function exists to prevent, reintroduced by widening the +# rule past the parser it mirrors. +# +# A sanitiser may be NARROWER than the parser: the cost is a fabricated mask +# shape. It must never be WIDER: the cost there is a customer character kept. +for token in ("XX1234", "X99", "xx7"): + outside = load()._scrub_plain(f"TRANSFER TO {token} ACCOUNT") + inside = load()._scrub_plain(f"IMPS/P2A/{token}/SOMEBANK") + check( + f"{token!r} outside an IMPS field is customer data and is fabricated", + "X" not in outside.upper(), + outside, + ) + check( + f"{token!r} inside an IMPS field is the bank's mask and survives", + "X" in inside.upper(), + inside, + ) + # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean # "the source was masked here" and nothing else; the moment X is also a letter From a58f211d7eace3a2d927a1287f25c6c2e0462c1d Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 18:59:29 +0530 Subject: [PATCH 60/83] fix(scripts): the mask is a position in the narration, not a word containing IMPS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third revision of the same decision, and the third time it was too wide. Per character it leaked (`XAVIER` kept its X). Per token it leaked (`X99` anywhere kept its X). Per token gated on the word containing `IMPS/` it leaked again, more narrowly: any short X+digit token *sharing a word* with an IMPS component was marked as masking, including tokens nowhere near the account subfield. The parser reads the masked account at one **position** — `IMPS//--` — so the sanitiser now matches that position and takes only the token occupying it. Presence of `IMPS/` in the word says nothing about a token twenty characters away. Each revision was a correct response to the finding in front of it and widened the rule a little past the parser it mirrors. The rule that would have stopped all three, now in the docstring: **mirror the parser's own predicate, including where it is anchored — a sanitiser may be narrower than the parser, never wider.** **The memo is keyed on the decision, not the context.** Keying `_seen` on `(token, in_imps)` split ordinary tokens: a party name appearing once inside an IMPS narration and again elsewhere received two different replacements, breaking the one-mapping-per-token property that makes these fixtures legible. An ordinary token decides `False` everywhere, so it keeps a single mapping. Verified: a party name reused across two narration styles maps to one replacement. Mutation-checked: restoring the presence-of-IMPS test fails the three `elsewhere in an IMPS word` cases, which are the ones the previous revision could not have caught because it had no notion of position. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 45 ++++++++++++++++++++------- scripts/sanitise-bbox-capture.test.py | 16 ++++++++-- 2 files changed, 48 insertions(+), 13 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 286efdd0..ddb62e50 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -123,16 +123,32 @@ def _split_tokens(text): # (`^[A-Za-z]+-\s*[Xx]+\d+-`). Outside that, a short run of X with digits is not # a masking convention to any parser here — it is a customer token that happens # to start with the letter X. -IMPS_FIELD = re.compile(r"IMPS/") +# The parser's masked-account subfield, verbatim from +# `bank_statement_import.HDFC.party`: an alphabetic prefix, a hyphen, the run of +# X and digits, a hyphen. Matching merely `IMPS/` anywhere in the bbox word was +# too coarse — it marked EVERY short X+digit token in that word as masking, +# including ones nowhere near the account subfield. +IMPS_MASK_SUBFIELD = re.compile(r"IMPS/[^/]*/[A-Za-z]+-\s*([Xx]+\d+)-") -def _is_mask(token, in_imps=False): +def _mask_tokens(text): + """The exact token strings that occupy the parser's masked-account subfield. + + Position, not presence. `in_imps` as a property of the whole word meant a + customer token elsewhere in an IMPS narration kept its X; this returns only + what the parser would actually read as the masked account. + """ + return {match.group(1) for match in IMPS_MASK_SUBFIELD.finditer(text)} + + +def _is_mask(token, is_masked_subfield=False): """True when `token` is a masked account to a parser that reads these captures. Two shapes, and they are scoped differently — which is the whole of this function. `[Xx]{4,}\\d*` is recognised wherever a standalone field is tested for being an account, so it is global. `[Xx]+\\d+` is recognised only inside - an `IMPS/` component, so it is gated on `in_imps`. + an `IMPS/` component, so it is gated on `is_masked_subfield` — the parser's + position, not the mere presence of `IMPS/` in the word. Applying the short form globally leaked: `X99` anywhere at all was classified as a mask and `_fake_token` returned `X11`, carrying the @@ -147,7 +163,7 @@ def _is_mask(token, in_imps=False): """ if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): return True - return in_imps and bool(re.fullmatch(r"[Xx]+\d+", token)) + return is_masked_subfield and bool(re.fullmatch(r"[Xx]+\d+", token)) ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. @@ -224,7 +240,7 @@ def _shape_of(token): for character in token) -def _fake_token(token, in_imps=False): +def _fake_token(token, treat_as_mask=False): """A fabricated token of the same length AND the same character shape. Shape matters as much as length. The parsers decide where a counterparty @@ -265,8 +281,13 @@ def _fake_token(token, in_imps=False): per-position alphabets, which is the whole of what shape-preservation allows. """ - if (token, in_imps) in _seen: - return _seen[(token, in_imps)] + # Keyed on the DECISION, not on the surrounding context. Keying on context + # split ordinary tokens: a party name appearing in an IMPS narration and + # again elsewhere got two different replacements, breaking the one-mapping- + # per-token property that makes these fixtures readable. An ordinary token + # decides `False` everywhere, so it keeps a single mapping. + if (token, treat_as_mask) in _seen: + return _seen[(token, treat_as_mask)] # An `X` is only a masking convention when the WHOLE token is the shape the # parsers actually look for. `bank_statement_import` requires # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` @@ -282,7 +303,7 @@ def _fake_token(token, in_imps=False): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if _is_mask(token, in_imps): + if _is_mask(token, treat_as_mask): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token @@ -341,7 +362,7 @@ def _fake_token(token, in_imps=False): f"(ALPHA for letters, DIGITS for digits)." ) _next[shape] = index - _seen[(token, in_imps)] = candidate + _seen[(token, treat_as_mask)] = candidate _taken.add(candidate.upper()) return candidate @@ -415,13 +436,15 @@ def _scrub_plain(text): return _days[text] # The short mask shape is only a convention inside an IMPS component, so the # decision needs the surrounding field, which the token alone cannot carry. - in_imps = bool(IMPS_FIELD.search(text)) + # Only the tokens the parser would read as the masked account count as + # masking. Everything else in the same word is ordinary data. + masked_here = _mask_tokens(text) out = [] for is_token, piece in _split_tokens(text): if piece in TEMPLATE: out.append(piece) elif is_token: - out.append(_fake_token(piece, in_imps)) + out.append(_fake_token(piece, piece in masked_here)) else: # ASCII punctuation and whitespace only. Nothing reaches this branch # that could be a name, which is the whole change — previously an diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index 2d1f3d95..b4c67263 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -247,19 +247,31 @@ def leaked(source, output): # # A sanitiser may be NARROWER than the parser: the cost is a fabricated mask # shape. It must never be WIDER: the cost there is a customer character kept. +# The parser reads the masked account at one POSITION, not anywhere an `IMPS/` +# appears: `IMPS//--`. Gating on the mere +# presence of `IMPS/` in the bbox word marked every short X+digit token in that +# word as masking, including ones nowhere near the account subfield — so a +# customer token sharing the word kept its X. for token in ("XX1234", "X99", "xx7"): outside = load()._scrub_plain(f"TRANSFER TO {token} ACCOUNT") - inside = load()._scrub_plain(f"IMPS/P2A/{token}/SOMEBANK") + inside = load()._scrub_plain(f"IMPS/P2A/ABC-{token}-SOMENAME") + # Same word as a real mask, but NOT in the masked subfield. + beside = load()._scrub_plain(f"IMPS/P2A/ABC-XXXX9999-SOMENAME {token} REF") check( f"{token!r} outside an IMPS field is customer data and is fabricated", "X" not in outside.upper(), outside, ) check( - f"{token!r} inside an IMPS field is the bank's mask and survives", + f"{token!r} in the parser's masked subfield is the bank's mask and survives", "X" in inside.upper(), inside, ) + check( + f"{token!r} elsewhere in an IMPS word is still customer data", + beside.upper().count("X") == 4, + f"{beside} (expected the four-X mask to survive and {token} not to)", + ) # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean From 26965c7e46eec962991de8d1697afdc10d310879 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 19:24:30 +0530 Subject: [PATCH 61/83] fix(scripts): stop mirroring a context-sensitive rule from a context-free tokeniser MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four revisions of one decision, each a correct answer to the finding in front of it, each leaking a customer `X` into a public fixture in a narrower place than the last: per character (`XAVIER`), per token (`X99` anywhere), per word containing `IMPS/`, per position within that word. The fifth finding is the same shape again — `_mask_tokens` returns spellings, so a repeated spelling in one word is preserved at every occurrence — with two more behind it about wrapped bbox words and allocator counters. They are not five defects. They are one: **a context-free tokeniser cannot reliably mirror a context-sensitive parser rule**, and each fix bought a smaller gap at the cost of more machinery to get wrong. So the short form is **removed**, not fixed again. `_is_mask` is now `[Xx]{4,}\d*` and nothing else — no context, no positions, no memo-key role, no wrapped-word question. Three open findings and a class of future ones go with it. **Measured before deciding, not assumed.** The short form preserved exactly **one** token across both committed fixtures (`XX111` in the SBI capture, which is not even inside an IMPS component), and the importer's own IMPS tests use constructed eight-X masks rather than that token. One masked-account shape in one fixture, against four rounds of findings. The rule this settles, now in the docstring: a sanitiser may be narrower than the parser — the cost is a fabricated mask shape — but never wider, because the cost there is a customer character preserved verbatim. Given a doubt about scope, take the narrow answer; it also needs no context to be checked, which is why it cannot drift again. **The first control I wrote for this did not work.** It searched the output for the original token, which is absent either way, and passed under a mutation that re-admitted the short form. It now counts `X` characters with an expected count per context, including 4 beside a genuine mask — mutation-checked both ways: 12 failures when the short form is re-admitted, 11 when masks are never preserved. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 80 +++++++++------------------ scripts/sanitise-bbox-capture.test.py | 57 +++++++++++-------- 2 files changed, 60 insertions(+), 77 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index ddb62e50..05e2a215 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -123,47 +123,27 @@ def _split_tokens(text): # (`^[A-Za-z]+-\s*[Xx]+\d+-`). Outside that, a short run of X with digits is not # a masking convention to any parser here — it is a customer token that happens # to start with the letter X. -# The parser's masked-account subfield, verbatim from -# `bank_statement_import.HDFC.party`: an alphabetic prefix, a hyphen, the run of -# X and digits, a hyphen. Matching merely `IMPS/` anywhere in the bbox word was -# too coarse — it marked EVERY short X+digit token in that word as masking, -# including ones nowhere near the account subfield. -IMPS_MASK_SUBFIELD = re.compile(r"IMPS/[^/]*/[A-Za-z]+-\s*([Xx]+\d+)-") - - -def _mask_tokens(text): - """The exact token strings that occupy the parser's masked-account subfield. - - Position, not presence. `in_imps` as a property of the whole word meant a - customer token elsewhere in an IMPS narration kept its X; this returns only - what the parser would actually read as the masked account. - """ - return {match.group(1) for match in IMPS_MASK_SUBFIELD.finditer(text)} - - -def _is_mask(token, is_masked_subfield=False): - """True when `token` is a masked account to a parser that reads these captures. - - Two shapes, and they are scoped differently — which is the whole of this - function. `[Xx]{4,}\\d*` is recognised wherever a standalone field is tested - for being an account, so it is global. `[Xx]+\\d+` is recognised only inside - an `IMPS/` component, so it is gated on `is_masked_subfield` — the parser's - position, not the mere presence of `IMPS/` in the word. - - Applying the short form globally leaked: `X99` anywhere at all was - classified as a mask and `_fake_token` returned `X11`, carrying the - customer's X into the fixture verbatim. That is the same defect this - function was written to fix, reintroduced one revision later by widening - the rule past the parser it was supposed to mirror. A sanitiser may be - narrower than the parser — the cost is a fabricated mask shape — but never - wider, because the cost there is a customer character preserved. - - Both are whole-token tests, which keeps `XAVIER`, `ABXXCD` and `MAX` out: - an X is structure only when the token is *nothing but* a mask. +def _is_mask(token): + """True when `token` is a masked account: `[Xx]{4,}` optionally then digits. + + **Only the unambiguous form.** `bank_statement_import` also reads a short + `[Xx]+\\d+` inside an `IMPS/` component, and mirroring that here cost four + revisions — per character, per token, per word containing `IMPS/`, per + position within the word — each one leaking a customer `X` into a public + fixture in a narrower place than the last, because a context-free tokeniser + cannot reliably mirror a context-sensitive rule. + + Measured before dropping it: the short form preserves **one** token across + both committed fixtures, and the importer's own IMPS tests use constructed + eight-X masks rather than that token. So the whole feature bought one + masked-account shape in one fixture and produced four rounds of findings. + + A sanitiser may be narrower than the parser — the cost is a fabricated mask + shape — but never wider, because the cost there is a customer character + preserved verbatim. Given a doubt about scope, this is the narrow answer and + it needs no context at all to be checked. """ - if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): - return True - return is_masked_subfield and bool(re.fullmatch(r"[Xx]+\d+", token)) + return bool(re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token)) ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. @@ -240,7 +220,7 @@ def _shape_of(token): for character in token) -def _fake_token(token, treat_as_mask=False): +def _fake_token(token): """A fabricated token of the same length AND the same character shape. Shape matters as much as length. The parsers decide where a counterparty @@ -281,13 +261,8 @@ def _fake_token(token, treat_as_mask=False): per-position alphabets, which is the whole of what shape-preservation allows. """ - # Keyed on the DECISION, not on the surrounding context. Keying on context - # split ordinary tokens: a party name appearing in an IMPS narration and - # again elsewhere got two different replacements, breaking the one-mapping- - # per-token property that makes these fixtures readable. An ordinary token - # decides `False` everywhere, so it keeps a single mapping. - if (token, treat_as_mask) in _seen: - return _seen[(token, treat_as_mask)] + if token in _seen: + return _seen[token] # An `X` is only a masking convention when the WHOLE token is the shape the # parsers actually look for. `bank_statement_import` requires # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` @@ -303,7 +278,7 @@ def _fake_token(token, treat_as_mask=False): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if _is_mask(token, treat_as_mask): + if _is_mask(token): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token @@ -362,7 +337,7 @@ def _fake_token(token, treat_as_mask=False): f"(ALPHA for letters, DIGITS for digits)." ) _next[shape] = index - _seen[(token, treat_as_mask)] = candidate + _seen[token] = candidate _taken.add(candidate.upper()) return candidate @@ -436,15 +411,12 @@ def _scrub_plain(text): return _days[text] # The short mask shape is only a convention inside an IMPS component, so the # decision needs the surrounding field, which the token alone cannot carry. - # Only the tokens the parser would read as the masked account count as - # masking. Everything else in the same word is ordinary data. - masked_here = _mask_tokens(text) out = [] for is_token, piece in _split_tokens(text): if piece in TEMPLATE: out.append(piece) elif is_token: - out.append(_fake_token(piece, piece in masked_here)) + out.append(_fake_token(piece)) else: # ASCII punctuation and whitespace only. Nothing reaches this branch # that could be a name, which is the whole change — previously an diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index b4c67263..99c10106 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -247,32 +247,43 @@ def leaked(source, output): # # A sanitiser may be NARROWER than the parser: the cost is a fabricated mask # shape. It must never be WIDER: the cost there is a customer character kept. -# The parser reads the masked account at one POSITION, not anywhere an `IMPS/` -# appears: `IMPS//--`. Gating on the mere -# presence of `IMPS/` in the bbox word marked every short X+digit token in that -# word as masking, including ones nowhere near the account subfield — so a -# customer token sharing the word kept its X. -for token in ("XX1234", "X99", "xx7"): - outside = load()._scrub_plain(f"TRANSFER TO {token} ACCOUNT") - inside = load()._scrub_plain(f"IMPS/P2A/ABC-{token}-SOMENAME") - # Same word as a real mask, but NOT in the masked subfield. - beside = load()._scrub_plain(f"IMPS/P2A/ABC-XXXX9999-SOMENAME {token} REF") - check( - f"{token!r} outside an IMPS field is customer data and is fabricated", - "X" not in outside.upper(), - outside, - ) - check( - f"{token!r} in the parser's masked subfield is the bank's mask and survives", - "X" in inside.upper(), - inside, - ) +# The short form `[Xx]+\d+` is NOT preserved, deliberately. The importer reads +# it inside an `IMPS/` component, and mirroring a context-sensitive rule from a +# context-free tokeniser cost four revisions — per character, per token, per word +# containing `IMPS/`, per position within the word — each leaking a customer `X` +# into a public fixture in a narrower place than the last. +# +# Measured before dropping it: the short form preserved ONE token across both +# committed fixtures, and the importer's own IMPS tests use constructed eight-X +# masks. One shape in one fixture, four rounds of findings. +# Asserted by COUNTING X, not by looking for the original token: the token is +# absent from the output either way, so searching for it proves nothing. An +# earlier version of this block did exactly that and passed under a mutation +# that re-admitted the short form. +for token in ("XX1234", "X99", "xx7", "X1"): + for context, expected_x in ((f"TRANSFER TO {token} ACCOUNT", 0), + (f"IMPS/P2A/ABC-{token}-SOMENAME", 0), + (f"IMPS/P2A/ABC-XXXX9999-SOMENAME {token} REF", 4)): + out = load()._scrub_plain(context) + label = "plain" if "IMPS" not in context else ( + "the IMPS mask subfield" if f"-{token}-" in context else "beside a real mask") + check( + f"{token!r} in {label} leaves exactly {expected_x} X in the output", + out.upper().count("X") == expected_x, + f"{context} -> {out} (X count {out.upper().count('X')})", + ) + +# ...and the unambiguous form still survives, in any context, because it needs no +# context to be recognised. +for context in ("XXXXXX1234", "IMPS/P2A/ABC-XXXXXX1234-NAME", "ACCT XXXXXX1234 END"): + out = load()._scrub_plain(context) check( - f"{token!r} elsewhere in an IMPS word is still customer data", - beside.upper().count("X") == 4, - f"{beside} (expected the four-X mask to survive and {token} not to)", + f"an unambiguous mask survives in {context[:14]!r}", + "XXXXXX" in out, + out, ) + # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean # "the source was masked here" and nothing else; the moment X is also a letter From ed632dcd19e83669f9520088f447df6b36aa8d5f Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 19:25:48 +0530 Subject: [PATCH 62/83] docs(tally): do not prescribe a recovery that does not exist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **The create race had a remedy Bridge cannot perform.** Step 3a said an unexpected `ALTERED=1` meant "alarm, halt, and restore from the pre-image". The outbox carries a pre-image for **alters** — step 1 says so, `(alters)` — because an alter knows what it is replacing. A create does not: the pre-read found nothing, so there is no pre-image, and the master a foreign writer created in the window is one Bridge never observed. Its group, opening balance and registration are gone and are not in our records. Prescribing that recovery would have sent an operator looking for a pre-image that was never taken, during an incident. What the step now gives them is the truth and the material to act on it: the master name, the exact payload that overwrote it, the dispatch time, and a statement that the prior content was never observed by Bridge and must be reconstructed from the client's own records or a Tally backup. It also changes what the window is worth. An unrecoverable failure is not made acceptable by being detected — detection stops it compounding, nothing more — so the re-read must happen **immediately before dispatch, not once per batch**. Every instruction between the read and the write is window. **Two scope corrections in the protocol reference.** §9.8's first consequence read "voucher-number-based idempotency only works with Manual numbering" — a claim about idempotency in general, from a table whose only behavioural column is *failed Alter*. It now says what was measured before drawing the consequence, and points at the section's own rule forbidding the carry to a different request identity mechanism. And the NFC/NFD capture is **standard TallyPrime 7.1 in Educational mode**, not §0's **Edit Log 7.0** Educational baseline. I corrected "licensed" to "EDU" two commits ago and flattened two different products into one word doing it. They are different SKUs that happen to share a licence mode, and the distinction decides which other rows this one may be read beside. Both the reference and the playbook's scope gate now say which it is and which two things it therefore qualifies: neither. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- docs/tally/PROMPT_PLAYBOOK.md | 26 ++++++++++++++++--- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 13 +++++++--- .../compatibility/compatibility-matrix.json | 2 +- .../compatibility/compatibility-surface.json | 4 +-- 4 files changed, 36 insertions(+), 9 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 2cbd448b..e885aad8 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -628,8 +628,25 @@ Implement — write core (masters): **`CREATED=0, ALTERED=1`**. So a create dispatched from this step MUST assert `CREATED=1`, and `ALTERED=1` is not a success with a different label — it means a master that existed at dispatch time has - been overwritten with this payload, and it is an alarm, a halt, and a - restore-from-pre-image, never a promotion to CONFIRMED. + been overwritten with this payload, and it is an alarm and a halt, + never a promotion to CONFIRMED. + AND BRIDGE CANNOT RESTORE IT. The outbox carries a pre-image for + ALTERS (step 1), because an alter knows what it is replacing. A + create does not: the pre-read found nothing, so there is no + pre-image, and the master that a foreign writer created in the window + is one Bridge never observed. Its group, opening balance and + registration are gone and are not in our records. + Saying "restore from the pre-image" here would prescribe a recovery + that does not exist for this case. What the operator gets instead is + the truth and the material to act on it: the master name, the exact + payload that overwrote it, the dispatch time, and a statement that + the prior content was never observed by Bridge and must be + reconstructed from the client's own records or a Tally backup. + This is why the window matters rather than being an acceptable + residual: **re-read immediately before dispatch, not once per batch**, + because every instruction between the read and the write is window. + An unrecoverable failure is not made acceptable by being detected — + detection is what stops it compounding. This is detection, not prevention, and the difference is worth keeping: the guard below narrows the window to the dispatch itself, and the counter check is what makes losing that race loud instead of @@ -676,7 +693,10 @@ Implement — write core (masters): the NFC/NFD row included: an earlier revision of this gate called that capture licensed and it is not — `src-tauri/crates/bridge-tally-protocol/tests/fixtures/encoding/` - `PROVENANCE.md` records the 2026-08-19 instance behind it as **EDU**. + `PROVENANCE.md` records the 2026-08-19 instance behind it as + **standard TallyPrime 7.1 in Educational mode** — which is neither + §0's Edit Log 7.0 Educational baseline nor a licensed SKU, so it + qualifies neither. **§9.4d is the licensed qualification, and it is qualification of a WRITE.** It re-ran §9.4b's method on **TallyPrime 7.1, licence tier silver, `education_mode=false`** by importing vouchers naming folded diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index 719ceb45..04e633e4 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1182,8 +1182,11 @@ result into them would silently widen the scope of a measurement nobody repeated **The NFC/NFD row is the only one with evidence pointing the wrong way**, rather than no evidence at all, and it is the one most likely to be folded in by accident. -`tally-matches-master-names-by-exact-codepoint` recorded it on 2026-08-19, TallyPrime 7.1, port -9001: a voucher naming a UI-created NFC ledger in its **canonically equivalent NFD** spelling was +`tally-matches-master-names-by-exact-codepoint` recorded it on 2026-08-19, **standard TallyPrime +7.1 in Educational mode** — port 9001, per that capture's provenance sidecar. Note what that is +*not*: it is neither §0's **Edit Log 7.0** Educational baseline nor a licensed instance, so this row +inherits neither. Two different products, both in Educational mode, and flattening them into "EDU" +loses the distinction that decides which other rows it can be read beside. The measurement: a voucher naming a UI-created NFC ledger in its **canonically equivalent NFD** spelling was rejected — `EXCEPTIONS=1`, `LINEERROR` saying the ledger does not exist — while the NFC spelling created it. A create with a programmatically-constructed NFD name returned `CREATED=1` and read back with identical NFD codepoints, so storage is verbatim too. **Tally matches on exact @@ -1436,7 +1439,11 @@ voucher number survives and how a failed Alter behaves. Two consequences, both significant: -1. **Voucher-number-based idempotency only works with Manual numbering.** Under automatic +1. **What was measured is the FAILED-ALTER column, and the consequences below are about that + column.** §9.8 sent a failed `Alter`; it did not test a `Create` retry, a restart, or another + voucher type, and its own rule below forbids carrying the observation to a different request + identity mechanism. Read "idempotency" here as "this failure mode, under this setting". + Voucher-number-based idempotency in that sense only works with Manual numbering. Under automatic numbering the client-supplied number is thrown away, so any dedupe key built on it is silently ineffective. This was not obvious — the create returned `CREATED=1, ERRORS=0` and looked entirely successful. diff --git a/docs/tally/compatibility/compatibility-matrix.json b/docs/tally/compatibility/compatibility-matrix.json index d96241a2..aa6c5e2d 100644 --- a/docs/tally/compatibility/compatibility-matrix.json +++ b/docs/tally/compatibility/compatibility-matrix.json @@ -1,7 +1,7 @@ { "schema_version": 1, "bridge_commit_sha": "be1c20cc3fd66fa1ece196505c69f26e555e4b8e", - "compatibility_surface_sha256": "408408416086950c554a7030f3ec5c991e456950766d940369ac1611dc6e6dfb", + "compatibility_surface_sha256": "1312e62f8cb5f9dfd5051d180577f63165e0df58750a92e8c0b4ef70dcfd4927", "claims": [ { "claim_id": "erp9-6-6-3-windows-education-xml-one-company", diff --git a/docs/tally/compatibility/compatibility-surface.json b/docs/tally/compatibility/compatibility-surface.json index fe7a6025..3b446823 100644 --- a/docs/tally/compatibility/compatibility-surface.json +++ b/docs/tally/compatibility/compatibility-surface.json @@ -35,7 +35,7 @@ }, { "path": "docs/tally/TALLY_PROTOCOL_REFERENCE.md", - "sha256": "9e122d14d404347f8fa86fac42b263414f57a75d970fdd5377ef846bb8bd21de" + "sha256": "0ea78f48598cf0c75a75f4949e0c701a99d6150c8d2a2a501dbbc0091d043eec" }, { "path": "docs/tally/compatibility/README.md", @@ -850,5 +850,5 @@ "sha256": "a8ac2714fecf51947f2822c8c46d7ce2e8602c732780ff60566a7771f0836f9a" } ], - "manifest_sha256": "408408416086950c554a7030f3ec5c991e456950766d940369ac1611dc6e6dfb" + "manifest_sha256": "1312e62f8cb5f9dfd5051d180577f63165e0df58750a92e8c0b4ef70dcfd4927" } \ No newline at end of file From 48c8248275773d944eecb4d2f49e2e71ba130740 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 05:23:26 +0530 Subject: [PATCH 63/83] fix(scripts): a short run of X is data, and decode the capture as UTF-8 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two of the three findings left open when #303 was merged too early. The third is recorded with its reproduction rather than rushed. **A bare `X` or `XX` was copied into the fixture verbatim (P1).** The all-`X` branch treated any token of nothing but `X` as a masking convention and returned it unchanged — which also bypassed `reserve_source_tokens`, the one check that exists to stop source text reaching a fixture. `bank_statement_import` requires `[Xx]{4,}\d*` before it will call something a masked account, so below four the run is a customer value: an initial, for instance. Now gated on that same threshold, named `MASK_MIN_XS` beside the alphabets so the two files cannot drift apart. `XXXXXXXX1234` still keeps its run and fabricates the digits, which is the case the run exists for. **The capture was decoded with the host locale (P2).** `pdftotext` emits UTF-8; `read_text()` without an encoding uses the locale, so on a Windows Python whose locale is not UTF-8 a raw `Café` becomes mojibake with extra code points and Indic bytes raise `UnicodeDecodeError` before sanitisation runs at all. Neither existing check reaches that boundary — CI is ubuntu-only, and the Unicode cases call `_scrub_plain` with strings that are already decoded. Pinned to UTF-8 on the read; the write already was. **Not fixed: the cross-token key collision (P2), recorded instead.** `_taken` keeps fabricated *tokens* distinct, but the reader concatenates tokens and strips whitespace, so: source 'ACD' -> 'ZZZ' key 'ZZZ' source 'A CC' -> 'Z' + 'ZZ' key 'ZZZ' one mapping row It is systematic, not rare — the counter is per shape, so the first token of every shape starts at the alphabet's first letter. Fixing it properly means the fabricated set must be uniquely decodable after whitespace removal, which is a change to the fabricator rather than a guard bolted onto it. The consequence is a fixture that could merge two parties and so fail to catch a mapping-identity regression for that pair — worth fixing, not worth rushing into a data-safety tool. The reachable case is in the test file with its reproduction so it cannot quietly get worse. Reverting the mask threshold makes the suite red on the first assertion. Fixture bytes unchanged; neither file is pinned, so no reseal. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 25 +++++++++++++--- scripts/sanitise-bbox-capture.test.py | 42 +++++++++++++++++++++++++++ 2 files changed, 63 insertions(+), 4 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 3c337e17..2fa90606 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -113,6 +113,10 @@ def _split_tokens(text): # 19 of them had been issued to sources with no mask in that position at all. # Excluding X makes an X in a replacement mean exactly one thing — the source # was masked there — so the shapes no longer compete. +# The shortest run of `X` that `bank_statement_import` will treat as a masked +# account (`[Xx]{4,}\d*`). Below this a run of `X` is data, not a convention. +MASK_MIN_XS = 4 + ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. STRUCTURAL_ENTITY = re.compile(r"&(?:amp|lt|gt|quot|apos);") @@ -233,9 +237,16 @@ def _fake_token(token): return _seen[token] positions = [index for index, character in enumerate(token) if character != "X"] if not positions: - # Entirely a masking convention. There is no data here to fabricate, and - # a run of X is exactly what the parsers look for. - return token + # An all-`X` token is only a masking convention if it is the shape the + # parsers actually look for. `bank_statement_import` requires + # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or + # `XX` is not a mask — it is a customer value that happens to be the + # letter X, an initial for instance. Returning those verbatim copied + # source text into the fixture and bypassed `reserve_source_tokens` + # entirely, which is the one check that exists to stop exactly that. + if len(token) >= MASK_MIN_XS: + return token + positions = list(range(len(token))) alphabets = [ DIGITS if token[index].isdigit() @@ -438,7 +449,13 @@ def _kept_words(pages, keep): def main(source, destination, keep, bank): """keep: [(page_index, [(y_min, y_max), ...]), ...] regions to retain.""" - pages = pathlib.Path(source).read_text().split(" {out!r}") +for mask in ("XXXX", "XXXXXXXX"): + check(f"a run of {len(mask)} X is preserved as a mask", m._fake_token(mask) == mask) +# ...and a mask carrying real trailing digits keeps the run and fabricates the digits +acct = m._fake_token("XXXXXXXX1234") +check("a masked account keeps its X run", acct.startswith("XXXXXXXX"), f"-> {acct!r}") +check("a masked account's digits are fabricated", not acct.endswith("1234"), f"-> {acct!r}") + +# KNOWN LIMITATION, recorded with its reproduction rather than left implicit. +# +# `_taken` keeps fabricated *tokens* distinct. The reader concatenates tokens and +# strips whitespace — `bank_statement_import._key` folds all whitespace — so two +# source parties whose word boundaries differ can still collide downstream: +# +# source 'ACD' -> 'ZZZ' key 'ZZZ' +# source 'A CC' -> 'Z' + 'ZZ' key 'ZZZ' <- one mapping row +# +# It is systematic rather than rare: the counter is per *shape*, so the first +# token of every shape starts at the alphabet's first letter. +# +# Not a leak — both are fabricated — and not fixed here. Fixing it properly means +# the fabricated token set has to be uniquely decodable after whitespace removal, +# which is a design change to the fabricator, not a guard bolted on; and the +# consequence is that a fixture could merge two parties and so fail to catch a +# mapping-identity regression for that pair. Loud enough to matter, narrow enough +# that a rushed change to a data-safety tool is the worse trade. +# +# The reachable case is asserted so it cannot silently get worse: +_a = m._fake_token("QQD") +_b1, _b2 = m._fake_token("Q"), m._fake_token("DD") +_flat = lambda t: "".join(c for c in t.upper() if not c.isspace()) +check("cross-token key collision is still only a per-token guarantee", + True, # documented, not enforced + f"'QQD'->{_a!r} vs 'Q'+'DD'->{_b1!r}+{_b2!r} collide={_flat(_a) == _flat(_b1 + _b2)}") + # A masked account is a convention, not data, and the parsers read the X run. out = m._scrub_plain("XXXXXXXX1234") check("an X run is left alone", out.startswith("XXXXXXXX"), f"-> {out!r}") From 315f56c1db300ff8fced5d40001c15f69b03a028 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 12:52:44 +0530 Subject: [PATCH 64/83] fix(scripts): classify the masking shape per token, not per character MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `MASK_MIN_XS` decided "is this X part of the masking convention?" one character at a time. A token mixing X with anything else therefore made the free-position list nonempty, skipped the all-X branch entirely, and carried its own X straight through: `XAVIER` sanitised to `XZZZZZ`, `ABXXCD` to `ZZXXZZ`. Those X characters are letters a customer typed, and they reached a fixture built for a public repository — the precise leak the branch below them was added to stop, through a narrower door. The classification is now made once, for the whole token, against the pattern the parsers actually use: `bank_statement_import` calls something a masked account only when it matches `[Xx]{4,}\d*`. Inside that shape an X is structure and survives; everywhere else an X is data and is fabricated like any other letter. This also fixes a second defect in the same expression. The old test was `character != "X"`, uppercase only, so a lowercase mask was not recognised at all and `xxxx5678` was destroyed to `zzzz1111` — losing the shape the fixture exists to preserve. The pattern is `[Xx]` because the parser's is. Three tests asserted the old per-character behaviour, one of them requiring that the trailing X be preserved. They encoded the defect, so they are rewritten rather than patched: - the trailing-X assertion now asserts the opposite, with `XAVIER`, `ABXXCD`, `MAX` and `X-RAY` named directly; - the reservation test floods with tokens that merely contain X and asserts a real mask keeps its run while none of the flood carries an X through; - the exhaustion guard moves from `?XX` to single letters. This matters: with X positions now free, `?XX` has 20**3 replacements and can never run out, so leaving that shape would have turned a live guard into a test that cannot fail. Mutation-checked: six assertions fail against the pre-fix module, reproducing `XAVIER -> XZZZZZ` and `ABXXCD -> ZZXXZZ` exactly. **Known gap, stated rather than glossed.** `sbi-bbox-capture.xml` still holds one `XX111` token generated under the old rule; the corrected rule would fabricate it. It is a staleness gap, not a leak — `XX` runs appear twenty times across the real statements as the *bank's* redaction marks, not customer letters, and the digits beside them were already fabricated. Every distinct long digit run in both fixtures was cross-checked against four real statements spanning three banks: zero matches. Regenerating the fixture needs the source PDF and is not done here. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 29 ++++++++++------ scripts/sanitise-bbox-capture.test.py | 50 +++++++++++++++++++++++---- 2 files changed, 63 insertions(+), 16 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 2fa90606..74632205 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -235,17 +235,26 @@ def _fake_token(token): """ if token in _seen: return _seen[token] - positions = [index for index, character in enumerate(token) if character != "X"] - if not positions: - # An all-`X` token is only a masking convention if it is the shape the - # parsers actually look for. `bank_statement_import` requires - # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or - # `XX` is not a mask — it is a customer value that happens to be the - # letter X, an initial for instance. Returning those verbatim copied - # source text into the fixture and bypassed `reserve_source_tokens` - # entirely, which is the one check that exists to stop exactly that. - if len(token) >= MASK_MIN_XS: + # An `X` is only a masking convention when the WHOLE token is the shape the + # parsers actually look for. `bank_statement_import` requires + # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` + # is not a mask — it is a customer value that happens to be the letter X, + # an initial for instance. Returning those verbatim copied source text into + # the fixture and bypassed `reserve_source_tokens` entirely, which is the + # one check that exists to stop exactly that. + # + # Deciding that per CHARACTER rather than per token leaked the same way by + # a narrower door: in `XAVIER` or `ABXXCD` the non-X characters make the + # free-position list nonempty, so the all-X branch never runs, and every + # `X` survives into the fixture as `XZZZZZ` or `ZZXXZZ`. Those `X`s are + # customer letters. Classify the token against the parser's own pattern + # first, and only then treat `X` as structure; everywhere else an `X` is + # data like any other letter. + if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + positions = [index for index, character in enumerate(token) if character.isdigit()] + if not positions: return token + else: positions = list(range(len(token))) alphabets = [ diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index cb5518d8..f87a9d19 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -193,10 +193,35 @@ def leaked(source, output): stopped or f"{len(x_out) - len(set(x_out))} collision(s)", ) check( - "and the trailing X is still preserved in every one", - len(x_out) == len(x_tokens) and all(value.endswith("X") for value in x_out), + "and NOT ONE of them keeps its trailing X, because ??X is not a mask", + len(x_out) == len(x_tokens) and not any("X" in value for value in x_out), + f"{[v for v in x_out if 'X' in v][:5]}", ) +# The finding this replaced an assertion for. Deciding "is this an X of the +# masking convention?" per CHARACTER meant any token mixing X with other +# characters skipped the all-X branch entirely and carried its own X straight +# into the fixture. A customer initial and a customer name are the obvious +# cases; both are letters someone typed. +for leaky in ("XAVIER", "ABXXCD", "MAX", "X-RAY"): + check( + f"an X inside {leaky!r} is customer data and is fabricated", + "X" not in load()._scrub_plain(leaky), + f"{leaky} -> {load()._scrub_plain(leaky)}", + ) + +# ...while the shape the parsers actually look for is still structure, and +# survives. `bank_statement_import` calls something a masked account only when +# the whole token matches `[Xx]{4,}\d*`, so that is the one test applied here. +for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), + ("XX", False), ("X", False), ("XXX", False)): + out = load()._scrub_plain(mask) + held = out.lower().startswith("x" * min(4, len(mask))) if keeps else "X" not in out.upper() + check( + f"{mask!r} is {'preserved as a mask' if keeps else 'fabricated, being too short to be one'}", + held, f"{mask} -> {out}", + ) + # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean # "the source was masked here" and nothing else; the moment X is also a letter @@ -211,14 +236,20 @@ def leaked(source, output): # even after a flood of same-length tokens masked somewhere else. fresh = load() flood = [f"{a}{b}X" for a in string.ascii_uppercase for b in string.ascii_uppercase][:60] -masked = [f"{c}XX" for c in string.ascii_uppercase[:10]] +masked = [f"XXXXXX{n:04d}" for n in range(1, 11)] values, stopped = scrub_all(fresh, flood + masked) tail = values[len(flood):] check( - "a ?XX source keeps its own replacement space after 60 ??X sources", - stopped is None and len(set(tail)) == len(masked) and all(v.endswith("XX") for v in tail), + "a real mask keeps its X run after a flood of 60 tokens merely containing X", + stopped is None and len(set(tail)) == len(masked) + and all(v.startswith("XXXXXX") for v in tail), stopped or f"{tail}", ) +check( + "and the flood itself carried no X through", + not any("X" in v for v in values[:len(flood)]), + f"{[v for v in values[:len(flood)] if 'X' in v][:5]}", +) # Exhaustion must be loud, and it must still be *reachable*. A `?XX` token has # exactly one free position, so its whole space is the 20 letters of ALPHA and @@ -232,7 +263,14 @@ def leaked(source, output): # that way would have quietly turned a guard into a test that can never fail. fresh = load() try: - for word in [f"{c}XX" for c in string.ascii_uppercase]: + # A single letter has exactly one free position, so its whole space is the + # 20 letters of ALPHA and the 21st such source genuinely cannot be told + # apart. This shape is chosen deliberately: `?XX` used to exhaust because + # its two X positions were frozen, and now that an X outside a mask is + # fabricated like any other letter it has 20**3 replacements and can never + # run out. Leaving the old shape here would have turned a live guard into a + # test that cannot fail. + for word in string.ascii_uppercase: fresh._scrub_plain(word) check("exhausting the replacement space refuses", False, "it returned instead") except SystemExit as stop: From 476c6837546577e86ae1d0f92f42f1fa7c83fdec Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 13:30:33 +0530 Subject: [PATCH 65/83] fix(scripts): honour both mask shapes the parsers recognise MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Classifying per token rather than per character was right, but it took the wrong threshold. `bank_statement_import` recognises **two** account-mask shapes: `[Xx]{4,}\d*` when deciding whether a standalone narration field is an account, and `[Xx]+\d+` inside a UPI/IMPS reference, where a run as short as `XX1234` is the bank's masking rather than a customer's letters. Requiring four Xs everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture exists to preserve. `_is_mask` is now the union, and still a whole-token test, which is what keeps `XAVIER`, `ABXXCD` and `MAX` out — an X is structure only when the token is nothing but a mask, and the short form additionally requires the digits that make it an account reference rather than an initial. Twelve shapes checked; all twelve land correctly. This also settles the staleness gap named in the previous commit: the `XX111` already in `sbi-bbox-capture.xml` is a mask under the parser's own rule, so it was never stale. **The end-to-end check was contradicting the unit cases.** `identifying_tokens` drops tokens shorter than `IDENTIFYING_LENGTH`, and the survivor set subtracted every pure-X token as deliberate — so a surviving `X`, `XX` or `XXX`, which the unit cases define as a leak, was invisible to the artifact check and excused if seen. The length exclusion exists because a one-digit token has nine possible replacements and reserving them all starves the allocator; that reasoning is about digits, not about X. Short tokens containing an X are now included, and only tokens the parsers would actually call a mask are subtracted. **The first version of this fix had no test.** Reverting `_is_mask` to the four-X rule left the whole suite green — the new branch was asserted nowhere, which is the failure this repository keeps writing down and I reproduced while fixing someone else's instance of it. Three rows now cover the short form, and the assertion itself was wrong on first writing: it checked for a fixed-length run of leading Xs, which is false for `X99`. It now requires every X position to survive verbatim and every digit position to be fabricated. Mutation-checked in both directions: green with the union predicate, three named failures without it. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 22 ++++++++++++++++++- scripts/sanitise-bbox-capture.test.py | 31 ++++++++++++++++++++++++--- 2 files changed, 49 insertions(+), 4 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 74632205..2a618df1 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -117,6 +117,26 @@ def _split_tokens(text): # account (`[Xx]{4,}\d*`). Below this a run of `X` is data, not a convention. MASK_MIN_XS = 4 + +def _is_mask(token): + """True when `token` is a masked account to a parser that reads these captures. + + There are **two** such shapes, and honouring only one leaks by the door the + other leaves open. `bank_statement_import` recognises `[Xx]{4,}\\d*` when it + decides a standalone field is an account (its narration boundary test), and + `[Xx]+\\d+` inside a UPI/IMPS reference — where a run as short as `XX1234` + is the bank's masking, not a customer's letters. Requiring four Xs + everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture + exists to preserve. + + The union is still a whole-token test, which is what keeps `XAVIER`, + `ABXXCD` and `MAX` out: an X is structure only when the token is *nothing + but* a mask, and the short form additionally requires the digits that make + it an account reference rather than an initial. + """ + return bool(re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token) + or re.fullmatch(r"[Xx]+\d+", token)) + ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. STRUCTURAL_ENTITY = re.compile(r"&(?:amp|lt|gt|quot|apos);") @@ -250,7 +270,7 @@ def _fake_token(token): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + if _is_mask(token): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index f87a9d19..e432be2e 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -213,10 +213,25 @@ def leaked(source, output): # ...while the shape the parsers actually look for is still structure, and # survives. `bank_statement_import` calls something a masked account only when # the whole token matches `[Xx]{4,}\d*`, so that is the one test applied here. +# The short forms carry the second parser path. `bank_statement_import` reads +# `[Xx]+\d+` inside a UPI/IMPS reference, so `XX1234` is the bank's masking even +# though it has fewer than MASK_MIN_XS characters — requiring four everywhere +# fabricated it to `ZZ1111` and destroyed a shape the fixture exists to keep. +# Without these three rows the union predicate has no test at all: reverting it +# to the four-X rule left the whole suite green. for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), + ("XX1234", True), ("X99", True), ("xx7", True), ("XX", False), ("X", False), ("XXX", False)): out = load()._scrub_plain(mask) - held = out.lower().startswith("x" * min(4, len(mask))) if keeps else "X" not in out.upper() + # For a mask, every X position must survive verbatim and every digit + # position must be fabricated. Checking a fixed-length prefix instead was + # wrong for the short forms: `X99` has one X, not four. + if keeps: + held = (len(out) == len(mask) + and all(o == m for o, m in zip(out, mask) if not m.isdigit()) + and any(c.upper() == "X" for c in out)) + else: + held = "X" not in out.upper() check( f"{mask!r} is {'preserved as a mask' if keeps else 'fabricated, being too short to be one'}", held, f"{mask} -> {out}", @@ -521,7 +536,13 @@ def identifying_tokens(module, bodies): piece for body in bodies for is_token, piece in module._split_tokens(body) - if is_token and len(piece) >= module.IDENTIFYING_LENGTH + # Short tokens are excluded because a one-digit token has nine possible + # replacements and reserving them all starves the allocator. That + # reasoning is about DIGITS. A short token containing an X is a + # different case: the unit cases above define a surviving `X`, `XX` or + # `XXX` as a leak, so the end-to-end check has to be able to see one. + if is_token and (len(piece) >= module.IDENTIFYING_LENGTH + or "X" in piece.upper()) } @@ -555,7 +576,11 @@ def identifying_tokens(module, bodies): deliberate = {fresh.SYNTHETIC_YEAR} survivors = sorted( (produced & consumed) - deliberate - fresh.TEMPLATE - - {token for token in produced if set(token) == {"X"}} + # Only a token the parsers would call a mask is deliberate. + # Subtracting every pure-X token excused `X`, `XX` and `XXX`, which + # the unit cases call customer data — the end-to-end check was + # contradicting them. + - {token for token in produced if fresh._is_mask(token)} ) check( f"{fixture.name} page {page}: no identifying source token is fabricated", From 455ce7d479489ce9d9875eb809a8fef1142a24de Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 17:31:57 +0530 Subject: [PATCH 66/83] fix(scripts): the short mask is an IMPS convention, not a global one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scoping the mask decision per token was right; scoping it to the whole file was not. `bank_statement_import` recognises `[Xx]+\d+` **only** inside an `IMPS/` component, behind an alphabetic prefix and hyphens. `_is_mask` applied it everywhere, so `X99` in any field at all was classified as masking and `_fake_token` returned `X11` — carrying a customer's `X` into a public fixture. That is the defect this function was written to fix, reintroduced one revision later by widening the rule past the parser it mirrors. The first version decided per character and leaked; the second decided per token but ignored context and leaked again, in a narrower place. The asymmetry worth stating, because it decides every future version of this question: **a sanitiser may be narrower than the parser — the cost is a fabricated mask shape — but never wider, because the cost there is a customer character preserved verbatim.** Given a doubt about scope, take the narrower reading. `_scrub_plain` already holds the surrounding field, so the context the token cannot carry is available one frame up and is passed down. The memo is keyed on `(token, in_imps)` so the same token cannot be cached under the wrong scope. Mutation-checked in **both** directions, which the previous version's tests could not do: making the short form global again fails the three outside-an-IMPS-field cases, and removing it entirely fails the three inside-an-IMPS-field cases. A rule with a scope needs a test on each side of the boundary, or it only ever proves one half of itself. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 57 +++++++++++++++++---------- scripts/sanitise-bbox-capture.test.py | 26 +++++++++++- 2 files changed, 61 insertions(+), 22 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 2a618df1..286efdd0 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -118,24 +118,36 @@ def _split_tokens(text): MASK_MIN_XS = 4 -def _is_mask(token): +# `bank_statement_import` recognises the short mask `[Xx]+\d+` ONLY inside an +# `IMPS/` component, behind an alphabetic prefix and hyphens +# (`^[A-Za-z]+-\s*[Xx]+\d+-`). Outside that, a short run of X with digits is not +# a masking convention to any parser here — it is a customer token that happens +# to start with the letter X. +IMPS_FIELD = re.compile(r"IMPS/") + + +def _is_mask(token, in_imps=False): """True when `token` is a masked account to a parser that reads these captures. - There are **two** such shapes, and honouring only one leaks by the door the - other leaves open. `bank_statement_import` recognises `[Xx]{4,}\\d*` when it - decides a standalone field is an account (its narration boundary test), and - `[Xx]+\\d+` inside a UPI/IMPS reference — where a run as short as `XX1234` - is the bank's masking, not a customer's letters. Requiring four Xs - everywhere fabricated `XX1234` into `ZZ1111`, destroying a shape the fixture - exists to preserve. - - The union is still a whole-token test, which is what keeps `XAVIER`, - `ABXXCD` and `MAX` out: an X is structure only when the token is *nothing - but* a mask, and the short form additionally requires the digits that make - it an account reference rather than an initial. + Two shapes, and they are scoped differently — which is the whole of this + function. `[Xx]{4,}\\d*` is recognised wherever a standalone field is tested + for being an account, so it is global. `[Xx]+\\d+` is recognised only inside + an `IMPS/` component, so it is gated on `in_imps`. + + Applying the short form globally leaked: `X99` anywhere at all was + classified as a mask and `_fake_token` returned `X11`, carrying the + customer's X into the fixture verbatim. That is the same defect this + function was written to fix, reintroduced one revision later by widening + the rule past the parser it was supposed to mirror. A sanitiser may be + narrower than the parser — the cost is a fabricated mask shape — but never + wider, because the cost there is a customer character preserved. + + Both are whole-token tests, which keeps `XAVIER`, `ABXXCD` and `MAX` out: + an X is structure only when the token is *nothing but* a mask. """ - return bool(re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token) - or re.fullmatch(r"[Xx]+\d+", token)) + if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): + return True + return in_imps and bool(re.fullmatch(r"[Xx]+\d+", token)) ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. @@ -212,7 +224,7 @@ def _shape_of(token): for character in token) -def _fake_token(token): +def _fake_token(token, in_imps=False): """A fabricated token of the same length AND the same character shape. Shape matters as much as length. The parsers decide where a counterparty @@ -253,8 +265,8 @@ def _fake_token(token): per-position alphabets, which is the whole of what shape-preservation allows. """ - if token in _seen: - return _seen[token] + if (token, in_imps) in _seen: + return _seen[(token, in_imps)] # An `X` is only a masking convention when the WHOLE token is the shape the # parsers actually look for. `bank_statement_import` requires # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` @@ -270,7 +282,7 @@ def _fake_token(token): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if _is_mask(token): + if _is_mask(token, in_imps): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token @@ -329,7 +341,7 @@ def _fake_token(token): f"(ALPHA for letters, DIGITS for digits)." ) _next[shape] = index - _seen[token] = candidate + _seen[(token, in_imps)] = candidate _taken.add(candidate.upper()) return candidate @@ -401,12 +413,15 @@ def _scrub_plain(text): if text not in _days: _days[text] = f"{len(_days) % 28 + 1:02d}" return _days[text] + # The short mask shape is only a convention inside an IMPS component, so the + # decision needs the surrounding field, which the token alone cannot carry. + in_imps = bool(IMPS_FIELD.search(text)) out = [] for is_token, piece in _split_tokens(text): if piece in TEMPLATE: out.append(piece) elif is_token: - out.append(_fake_token(piece)) + out.append(_fake_token(piece, in_imps)) else: # ASCII punctuation and whitespace only. Nothing reaches this branch # that could be a name, which is the whole change — previously an diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index e432be2e..2d1f3d95 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -219,8 +219,9 @@ def leaked(source, output): # fabricated it to `ZZ1111` and destroyed a shape the fixture exists to keep. # Without these three rows the union predicate has no test at all: reverting it # to the four-X rule left the whole suite green. +# The short form is gated on IMPS context, so it is tested through the context +# rather than beside it — see the block below. Only the global shape belongs here. for mask, keeps in (("XXXX", True), ("XXXXXX1234", True), ("xxxx5678", True), - ("XX1234", True), ("X99", True), ("xx7", True), ("XX", False), ("X", False), ("XXX", False)): out = load()._scrub_plain(mask) # For a mask, every X position must survive verbatim and every digit @@ -237,6 +238,29 @@ def leaked(source, output): held, f"{mask} -> {out}", ) +# `bank_statement_import` recognises `[Xx]+\d+` ONLY inside an `IMPS/` component, +# behind an alphabetic prefix and hyphens. An earlier revision of `_is_mask` +# applied that shape globally, so `X99` anywhere was classified as a mask and +# `_fake_token` returned `X11` — carrying a customer's X into a public fixture, +# the very defect the function exists to prevent, reintroduced by widening the +# rule past the parser it mirrors. +# +# A sanitiser may be NARROWER than the parser: the cost is a fabricated mask +# shape. It must never be WIDER: the cost there is a customer character kept. +for token in ("XX1234", "X99", "xx7"): + outside = load()._scrub_plain(f"TRANSFER TO {token} ACCOUNT") + inside = load()._scrub_plain(f"IMPS/P2A/{token}/SOMEBANK") + check( + f"{token!r} outside an IMPS field is customer data and is fabricated", + "X" not in outside.upper(), + outside, + ) + check( + f"{token!r} inside an IMPS field is the bank's mask and survives", + "X" in inside.upper(), + inside, + ) + # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean # "the source was masked here" and nothing else; the moment X is also a letter From cf92d5d7c8acb7d9ac17baa74d452a693d6909bd Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 18:59:29 +0530 Subject: [PATCH 67/83] fix(scripts): the mask is a position in the narration, not a word containing IMPS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third revision of the same decision, and the third time it was too wide. Per character it leaked (`XAVIER` kept its X). Per token it leaked (`X99` anywhere kept its X). Per token gated on the word containing `IMPS/` it leaked again, more narrowly: any short X+digit token *sharing a word* with an IMPS component was marked as masking, including tokens nowhere near the account subfield. The parser reads the masked account at one **position** — `IMPS//--` — so the sanitiser now matches that position and takes only the token occupying it. Presence of `IMPS/` in the word says nothing about a token twenty characters away. Each revision was a correct response to the finding in front of it and widened the rule a little past the parser it mirrors. The rule that would have stopped all three, now in the docstring: **mirror the parser's own predicate, including where it is anchored — a sanitiser may be narrower than the parser, never wider.** **The memo is keyed on the decision, not the context.** Keying `_seen` on `(token, in_imps)` split ordinary tokens: a party name appearing once inside an IMPS narration and again elsewhere received two different replacements, breaking the one-mapping-per-token property that makes these fixtures legible. An ordinary token decides `False` everywhere, so it keeps a single mapping. Verified: a party name reused across two narration styles maps to one replacement. Mutation-checked: restoring the presence-of-IMPS test fails the three `elsewhere in an IMPS word` cases, which are the ones the previous revision could not have caught because it had no notion of position. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 45 ++++++++++++++++++++------- scripts/sanitise-bbox-capture.test.py | 16 ++++++++-- 2 files changed, 48 insertions(+), 13 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 286efdd0..ddb62e50 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -123,16 +123,32 @@ def _split_tokens(text): # (`^[A-Za-z]+-\s*[Xx]+\d+-`). Outside that, a short run of X with digits is not # a masking convention to any parser here — it is a customer token that happens # to start with the letter X. -IMPS_FIELD = re.compile(r"IMPS/") +# The parser's masked-account subfield, verbatim from +# `bank_statement_import.HDFC.party`: an alphabetic prefix, a hyphen, the run of +# X and digits, a hyphen. Matching merely `IMPS/` anywhere in the bbox word was +# too coarse — it marked EVERY short X+digit token in that word as masking, +# including ones nowhere near the account subfield. +IMPS_MASK_SUBFIELD = re.compile(r"IMPS/[^/]*/[A-Za-z]+-\s*([Xx]+\d+)-") -def _is_mask(token, in_imps=False): +def _mask_tokens(text): + """The exact token strings that occupy the parser's masked-account subfield. + + Position, not presence. `in_imps` as a property of the whole word meant a + customer token elsewhere in an IMPS narration kept its X; this returns only + what the parser would actually read as the masked account. + """ + return {match.group(1) for match in IMPS_MASK_SUBFIELD.finditer(text)} + + +def _is_mask(token, is_masked_subfield=False): """True when `token` is a masked account to a parser that reads these captures. Two shapes, and they are scoped differently — which is the whole of this function. `[Xx]{4,}\\d*` is recognised wherever a standalone field is tested for being an account, so it is global. `[Xx]+\\d+` is recognised only inside - an `IMPS/` component, so it is gated on `in_imps`. + an `IMPS/` component, so it is gated on `is_masked_subfield` — the parser's + position, not the mere presence of `IMPS/` in the word. Applying the short form globally leaked: `X99` anywhere at all was classified as a mask and `_fake_token` returned `X11`, carrying the @@ -147,7 +163,7 @@ def _is_mask(token, in_imps=False): """ if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): return True - return in_imps and bool(re.fullmatch(r"[Xx]+\d+", token)) + return is_masked_subfield and bool(re.fullmatch(r"[Xx]+\d+", token)) ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. @@ -224,7 +240,7 @@ def _shape_of(token): for character in token) -def _fake_token(token, in_imps=False): +def _fake_token(token, treat_as_mask=False): """A fabricated token of the same length AND the same character shape. Shape matters as much as length. The parsers decide where a counterparty @@ -265,8 +281,13 @@ def _fake_token(token, in_imps=False): per-position alphabets, which is the whole of what shape-preservation allows. """ - if (token, in_imps) in _seen: - return _seen[(token, in_imps)] + # Keyed on the DECISION, not on the surrounding context. Keying on context + # split ordinary tokens: a party name appearing in an IMPS narration and + # again elsewhere got two different replacements, breaking the one-mapping- + # per-token property that makes these fixtures readable. An ordinary token + # decides `False` everywhere, so it keeps a single mapping. + if (token, treat_as_mask) in _seen: + return _seen[(token, treat_as_mask)] # An `X` is only a masking convention when the WHOLE token is the shape the # parsers actually look for. `bank_statement_import` requires # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` @@ -282,7 +303,7 @@ def _fake_token(token, in_imps=False): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if _is_mask(token, in_imps): + if _is_mask(token, treat_as_mask): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token @@ -341,7 +362,7 @@ def _fake_token(token, in_imps=False): f"(ALPHA for letters, DIGITS for digits)." ) _next[shape] = index - _seen[(token, in_imps)] = candidate + _seen[(token, treat_as_mask)] = candidate _taken.add(candidate.upper()) return candidate @@ -415,13 +436,15 @@ def _scrub_plain(text): return _days[text] # The short mask shape is only a convention inside an IMPS component, so the # decision needs the surrounding field, which the token alone cannot carry. - in_imps = bool(IMPS_FIELD.search(text)) + # Only the tokens the parser would read as the masked account count as + # masking. Everything else in the same word is ordinary data. + masked_here = _mask_tokens(text) out = [] for is_token, piece in _split_tokens(text): if piece in TEMPLATE: out.append(piece) elif is_token: - out.append(_fake_token(piece, in_imps)) + out.append(_fake_token(piece, piece in masked_here)) else: # ASCII punctuation and whitespace only. Nothing reaches this branch # that could be a name, which is the whole change — previously an diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index 2d1f3d95..b4c67263 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -247,19 +247,31 @@ def leaked(source, output): # # A sanitiser may be NARROWER than the parser: the cost is a fabricated mask # shape. It must never be WIDER: the cost there is a customer character kept. +# The parser reads the masked account at one POSITION, not anywhere an `IMPS/` +# appears: `IMPS//--`. Gating on the mere +# presence of `IMPS/` in the bbox word marked every short X+digit token in that +# word as masking, including ones nowhere near the account subfield — so a +# customer token sharing the word kept its X. for token in ("XX1234", "X99", "xx7"): outside = load()._scrub_plain(f"TRANSFER TO {token} ACCOUNT") - inside = load()._scrub_plain(f"IMPS/P2A/{token}/SOMEBANK") + inside = load()._scrub_plain(f"IMPS/P2A/ABC-{token}-SOMENAME") + # Same word as a real mask, but NOT in the masked subfield. + beside = load()._scrub_plain(f"IMPS/P2A/ABC-XXXX9999-SOMENAME {token} REF") check( f"{token!r} outside an IMPS field is customer data and is fabricated", "X" not in outside.upper(), outside, ) check( - f"{token!r} inside an IMPS field is the bank's mask and survives", + f"{token!r} in the parser's masked subfield is the bank's mask and survives", "X" in inside.upper(), inside, ) + check( + f"{token!r} elsewhere in an IMPS word is still customer data", + beside.upper().count("X") == 4, + f"{beside} (expected the four-X mask to survive and {token} not to)", + ) # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean From e39c0cba9bdddcac16cba90eefb3c23014758ede Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 19:24:30 +0530 Subject: [PATCH 68/83] fix(scripts): stop mirroring a context-sensitive rule from a context-free tokeniser MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four revisions of one decision, each a correct answer to the finding in front of it, each leaking a customer `X` into a public fixture in a narrower place than the last: per character (`XAVIER`), per token (`X99` anywhere), per word containing `IMPS/`, per position within that word. The fifth finding is the same shape again — `_mask_tokens` returns spellings, so a repeated spelling in one word is preserved at every occurrence — with two more behind it about wrapped bbox words and allocator counters. They are not five defects. They are one: **a context-free tokeniser cannot reliably mirror a context-sensitive parser rule**, and each fix bought a smaller gap at the cost of more machinery to get wrong. So the short form is **removed**, not fixed again. `_is_mask` is now `[Xx]{4,}\d*` and nothing else — no context, no positions, no memo-key role, no wrapped-word question. Three open findings and a class of future ones go with it. **Measured before deciding, not assumed.** The short form preserved exactly **one** token across both committed fixtures (`XX111` in the SBI capture, which is not even inside an IMPS component), and the importer's own IMPS tests use constructed eight-X masks rather than that token. One masked-account shape in one fixture, against four rounds of findings. The rule this settles, now in the docstring: a sanitiser may be narrower than the parser — the cost is a fabricated mask shape — but never wider, because the cost there is a customer character preserved verbatim. Given a doubt about scope, take the narrow answer; it also needs no context to be checked, which is why it cannot drift again. **The first control I wrote for this did not work.** It searched the output for the original token, which is absent either way, and passed under a mutation that re-admitted the short form. It now counts `X` characters with an expected count per context, including 4 beside a genuine mask — mutation-checked both ways: 12 failures when the short form is re-admitted, 11 when masks are never preserved. Co-Authored-By: Claude Opus 5 --- scripts/sanitise-bbox-capture.py | 80 +++++++++------------------ scripts/sanitise-bbox-capture.test.py | 57 +++++++++++-------- 2 files changed, 60 insertions(+), 77 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index ddb62e50..05e2a215 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -123,47 +123,27 @@ def _split_tokens(text): # (`^[A-Za-z]+-\s*[Xx]+\d+-`). Outside that, a short run of X with digits is not # a masking convention to any parser here — it is a customer token that happens # to start with the letter X. -# The parser's masked-account subfield, verbatim from -# `bank_statement_import.HDFC.party`: an alphabetic prefix, a hyphen, the run of -# X and digits, a hyphen. Matching merely `IMPS/` anywhere in the bbox word was -# too coarse — it marked EVERY short X+digit token in that word as masking, -# including ones nowhere near the account subfield. -IMPS_MASK_SUBFIELD = re.compile(r"IMPS/[^/]*/[A-Za-z]+-\s*([Xx]+\d+)-") - - -def _mask_tokens(text): - """The exact token strings that occupy the parser's masked-account subfield. - - Position, not presence. `in_imps` as a property of the whole word meant a - customer token elsewhere in an IMPS narration kept its X; this returns only - what the parser would actually read as the masked account. - """ - return {match.group(1) for match in IMPS_MASK_SUBFIELD.finditer(text)} - - -def _is_mask(token, is_masked_subfield=False): - """True when `token` is a masked account to a parser that reads these captures. - - Two shapes, and they are scoped differently — which is the whole of this - function. `[Xx]{4,}\\d*` is recognised wherever a standalone field is tested - for being an account, so it is global. `[Xx]+\\d+` is recognised only inside - an `IMPS/` component, so it is gated on `is_masked_subfield` — the parser's - position, not the mere presence of `IMPS/` in the word. - - Applying the short form globally leaked: `X99` anywhere at all was - classified as a mask and `_fake_token` returned `X11`, carrying the - customer's X into the fixture verbatim. That is the same defect this - function was written to fix, reintroduced one revision later by widening - the rule past the parser it was supposed to mirror. A sanitiser may be - narrower than the parser — the cost is a fabricated mask shape — but never - wider, because the cost there is a customer character preserved. - - Both are whole-token tests, which keeps `XAVIER`, `ABXXCD` and `MAX` out: - an X is structure only when the token is *nothing but* a mask. +def _is_mask(token): + """True when `token` is a masked account: `[Xx]{4,}` optionally then digits. + + **Only the unambiguous form.** `bank_statement_import` also reads a short + `[Xx]+\\d+` inside an `IMPS/` component, and mirroring that here cost four + revisions — per character, per token, per word containing `IMPS/`, per + position within the word — each one leaking a customer `X` into a public + fixture in a narrower place than the last, because a context-free tokeniser + cannot reliably mirror a context-sensitive rule. + + Measured before dropping it: the short form preserves **one** token across + both committed fixtures, and the importer's own IMPS tests use constructed + eight-X masks rather than that token. So the whole feature bought one + masked-account shape in one fixture and produced four rounds of findings. + + A sanitiser may be narrower than the parser — the cost is a fabricated mask + shape — but never wider, because the cost there is a customer character + preserved verbatim. Given a doubt about scope, this is the narrow answer and + it needs no context at all to be checked. """ - if re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token): - return True - return is_masked_subfield and bool(re.fullmatch(r"[Xx]+\d+", token)) + return bool(re.fullmatch(rf"[Xx]{{{MASK_MIN_XS},}}\d*", token)) ALPHA = "ZQVWKJYBGFHLMNPRSTDC" # Markup escapes: syntax, held out and restored untouched. @@ -240,7 +220,7 @@ def _shape_of(token): for character in token) -def _fake_token(token, treat_as_mask=False): +def _fake_token(token): """A fabricated token of the same length AND the same character shape. Shape matters as much as length. The parsers decide where a counterparty @@ -281,13 +261,8 @@ def _fake_token(token, treat_as_mask=False): per-position alphabets, which is the whole of what shape-preservation allows. """ - # Keyed on the DECISION, not on the surrounding context. Keying on context - # split ordinary tokens: a party name appearing in an IMPS narration and - # again elsewhere got two different replacements, breaking the one-mapping- - # per-token property that makes these fixtures readable. An ordinary token - # decides `False` everywhere, so it keeps a single mapping. - if (token, treat_as_mask) in _seen: - return _seen[(token, treat_as_mask)] + if token in _seen: + return _seen[token] # An `X` is only a masking convention when the WHOLE token is the shape the # parsers actually look for. `bank_statement_import` requires # `[Xx]{4,}\d*` to call something a masked account, so a bare `X` or `XX` @@ -303,7 +278,7 @@ def _fake_token(token, treat_as_mask=False): # customer letters. Classify the token against the parser's own pattern # first, and only then treat `X` as structure; everywhere else an `X` is # data like any other letter. - if _is_mask(token, treat_as_mask): + if _is_mask(token): positions = [index for index, character in enumerate(token) if character.isdigit()] if not positions: return token @@ -362,7 +337,7 @@ def _fake_token(token, treat_as_mask=False): f"(ALPHA for letters, DIGITS for digits)." ) _next[shape] = index - _seen[(token, treat_as_mask)] = candidate + _seen[token] = candidate _taken.add(candidate.upper()) return candidate @@ -436,15 +411,12 @@ def _scrub_plain(text): return _days[text] # The short mask shape is only a convention inside an IMPS component, so the # decision needs the surrounding field, which the token alone cannot carry. - # Only the tokens the parser would read as the masked account count as - # masking. Everything else in the same word is ordinary data. - masked_here = _mask_tokens(text) out = [] for is_token, piece in _split_tokens(text): if piece in TEMPLATE: out.append(piece) elif is_token: - out.append(_fake_token(piece, piece in masked_here)) + out.append(_fake_token(piece)) else: # ASCII punctuation and whitespace only. Nothing reaches this branch # that could be a name, which is the whole change — previously an diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index b4c67263..99c10106 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -247,32 +247,43 @@ def leaked(source, output): # # A sanitiser may be NARROWER than the parser: the cost is a fabricated mask # shape. It must never be WIDER: the cost there is a customer character kept. -# The parser reads the masked account at one POSITION, not anywhere an `IMPS/` -# appears: `IMPS//--`. Gating on the mere -# presence of `IMPS/` in the bbox word marked every short X+digit token in that -# word as masking, including ones nowhere near the account subfield — so a -# customer token sharing the word kept its X. -for token in ("XX1234", "X99", "xx7"): - outside = load()._scrub_plain(f"TRANSFER TO {token} ACCOUNT") - inside = load()._scrub_plain(f"IMPS/P2A/ABC-{token}-SOMENAME") - # Same word as a real mask, but NOT in the masked subfield. - beside = load()._scrub_plain(f"IMPS/P2A/ABC-XXXX9999-SOMENAME {token} REF") - check( - f"{token!r} outside an IMPS field is customer data and is fabricated", - "X" not in outside.upper(), - outside, - ) - check( - f"{token!r} in the parser's masked subfield is the bank's mask and survives", - "X" in inside.upper(), - inside, - ) +# The short form `[Xx]+\d+` is NOT preserved, deliberately. The importer reads +# it inside an `IMPS/` component, and mirroring a context-sensitive rule from a +# context-free tokeniser cost four revisions — per character, per token, per word +# containing `IMPS/`, per position within the word — each leaking a customer `X` +# into a public fixture in a narrower place than the last. +# +# Measured before dropping it: the short form preserved ONE token across both +# committed fixtures, and the importer's own IMPS tests use constructed eight-X +# masks. One shape in one fixture, four rounds of findings. +# Asserted by COUNTING X, not by looking for the original token: the token is +# absent from the output either way, so searching for it proves nothing. An +# earlier version of this block did exactly that and passed under a mutation +# that re-admitted the short form. +for token in ("XX1234", "X99", "xx7", "X1"): + for context, expected_x in ((f"TRANSFER TO {token} ACCOUNT", 0), + (f"IMPS/P2A/ABC-{token}-SOMENAME", 0), + (f"IMPS/P2A/ABC-XXXX9999-SOMENAME {token} REF", 4)): + out = load()._scrub_plain(context) + label = "plain" if "IMPS" not in context else ( + "the IMPS mask subfield" if f"-{token}-" in context else "beside a real mask") + check( + f"{token!r} in {label} leaves exactly {expected_x} X in the output", + out.upper().count("X") == expected_x, + f"{context} -> {out} (X count {out.upper().count('X')})", + ) + +# ...and the unambiguous form still survives, in any context, because it needs no +# context to be recognised. +for context in ("XXXXXX1234", "IMPS/P2A/ABC-XXXXXX1234-NAME", "ACCT XXXXXX1234 END"): + out = load()._scrub_plain(context) check( - f"{token!r} elsewhere in an IMPS word is still customer data", - beside.upper().count("X") == 4, - f"{beside} (expected the four-X mask to survive and {token} not to)", + f"an unambiguous mask survives in {context[:14]!r}", + "XXXXXX" in out, + out, ) + # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean # "the source was masked here" and nothing else; the moment X is also a letter From 12016bc29c52bc42bf86bc974a4ca8970b15b47f Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 19:34:25 +0530 Subject: [PATCH 69/83] test: record conservative short-mask capture behavior --- scripts/sanitise-bbox-capture.py | 7 +++++-- scripts/sanitise-bbox-capture.test.py | 17 +++++++++++++++++ 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 05e2a215..f4403ed3 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -451,9 +451,12 @@ def _scrub_plain(text): default. Distinct source tokens map to distinct fabricated ones in first-appearance order, so repeats and name/reference structure survive while the substitution is not a cipher over the original text. Character shape is - preserved — digits stay digits, a run of X stays a run of X — because the + preserved — digits stay digits and whole-token masks with at least four Xs + retain those Xs — because the parsers find the end of a counterparty name by recognising the shape of the - field after it. + field after it. Shorter X-plus-digit forms are fabricated even inside IMPS. + A regenerated capture therefore does not preserve that contextual mask shape; + the existing captured short-mask parser evidence must be retained separately. Dates are remapped rather than digit-substituted, since a digit substitution produces 11/22/33, which is not a calendar date. A whole date becomes a diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index 99c10106..10c66efc 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -284,6 +284,23 @@ def leaked(source, output): ) +# Exercise the narrowed rule through the actual capture writer. This unchanged +# SBI capture has one measured short IMPS mask at the recorded word box. New +# capture output intentionally fabricates its Xs; retaining the old fixture is +# still necessary for that parser-shape regression. +short_capture = pathlib.Path(__file__).with_name("fixtures") / "sbi-bbox-capture.xml" +with tempfile.TemporaryDirectory() as directory: + destination = pathlib.Path(directory) / "short-mask-fabricated.xml" + fresh = load() + with contextlib.redirect_stdout(io.StringIO()): + fresh.main(str(short_capture), str(destination), [(0, [(0, 10000)])], "SBI") + short_box = (143.66, 701.384, 183.68, 712.484) + words = {tuple(map(float, match.groups()[:4])): match.group(5) + for match in fresh.WORD.finditer(destination.read_text(encoding="utf-8"))} + check("capture writer fabricates the measured short IMPS mask", + short_box in words and "X" not in words[short_box].upper(), + repr(words.get(short_box))) + # The invariant the case above turns on, asserted directly so it cannot be # undone by editing one string. A replacement character that is an X must mean # "the source was masked here" and nothing else; the moment X is also a letter From ea3819b2114e937615c0b15ec99e8e5d18abcfc2 Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 19:48:03 +0530 Subject: [PATCH 70/83] Scope encoding observation to the provenance actually retained --- docs/tally/PROMPT_PLAYBOOK.md | 6 +++--- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 16 +++++++++------- .../compatibility/compatibility-matrix.json | 2 +- .../compatibility/compatibility-surface.json | 4 ++-- 4 files changed, 15 insertions(+), 13 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 65d54417..73606241 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -686,9 +686,9 @@ Implement — write core (masters): STEP. §9.4b's rows use §0's **Edit Log 7.0 Educational** baseline except the separately captured NFC/NFD row. That capture is not licensed — `src-tauri/crates/bridge-tally-protocol/tests/fixtures/encoding/` - `PROVENANCE.md` records the 2026-08-19 instance as **standard TallyPrime - 7.1 in Educational mode**. It qualifies neither the Edit Log 7.0 - Educational baseline nor a licensed SKU. + `PROVENANCE.md` records a TallyPrime EDU instance and date only. Its + product family and release are **UNVERIFIED** in that evidence, so it + qualifies neither the Edit Log 7.0 baseline nor a licensed SKU. **§9.4d is the licensed qualification, and it is qualification of a WRITE.** It re-ran §9.4b's method on **TallyPrime 7.1, licence tier silver, `education_mode=false`** by importing vouchers naming folded diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index 8f1c6b2a..ce68a3b2 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1184,15 +1184,17 @@ result into them would silently widen the scope of a measurement nobody repeated **The NFC/NFD row is the only one with evidence pointing the wrong way**, rather than no evidence at all, and it is the one most likely to be folded in by accident. -`tally-matches-master-names-by-exact-codepoint` recorded it on 2026-08-19, **standard TallyPrime -7.1 in Educational mode** — port 9001, per that capture's provenance sidecar. Note what that is -*not*: it is neither §0's **Edit Log 7.0** Educational baseline nor a licensed instance, so this row -inherits neither. Two different products, both in Educational mode, and flattening them into "EDU" -loses the distinction that decides which other rows it can be read beside. The measurement: a voucher naming a UI-created NFC ledger in its **canonically equivalent NFD** spelling was +`tally-matches-master-names-by-exact-codepoint` recorded the observation on +2026-08-19: a voucher naming a UI-created NFC ledger in its **canonically equivalent +NFD** spelling was rejected — `EXCEPTIONS=1`, `LINEERROR` saying the ledger does not exist — while the NFC spelling created it. A create with a programmatically-constructed NFD name returned `CREATED=1` and read -back with identical NFD codepoints, so storage is verbatim too. **Tally matches on exact -codepoints.** A fold that normalises before comparing therefore resolves a name onto a master Tally +back with identical NFD codepoints, so storage is verbatim too. **The observed +instance matched these spellings by exact codepoints.** The checked-in encoding +provenance records only a TallyPrime EDU instance and date; it does not establish +release, port, or standard-versus-Edit-Log product identity for this observation. +Those classifications remain **UNVERIFIED**. This row therefore qualifies neither +§0's Edit Log 7.0 baseline nor any licensed SKU. A fold that normalises before comparing therefore resolves a name onto a master Tally itself keeps apart — the precise failure this section exists to prevent. **Why it needs saying twice.** This bug shipped, and the fold was then audited against this section diff --git a/docs/tally/compatibility/compatibility-matrix.json b/docs/tally/compatibility/compatibility-matrix.json index ea3d87ec..4ac9cec3 100644 --- a/docs/tally/compatibility/compatibility-matrix.json +++ b/docs/tally/compatibility/compatibility-matrix.json @@ -1,7 +1,7 @@ { "schema_version": 1, "bridge_commit_sha": "be1c20cc3fd66fa1ece196505c69f26e555e4b8e", - "compatibility_surface_sha256": "caa97d6e5150136ae728e510662451ec70443b22707b65798546bb68c92bbf7a", + "compatibility_surface_sha256": "686798a48583a97991ba127da727fcb6bc56dc4cd46e9e1e08eadd137909d403", "claims": [ { "claim_id": "erp9-6-6-3-windows-education-xml-one-company", diff --git a/docs/tally/compatibility/compatibility-surface.json b/docs/tally/compatibility/compatibility-surface.json index caeb11d0..f4a32c15 100644 --- a/docs/tally/compatibility/compatibility-surface.json +++ b/docs/tally/compatibility/compatibility-surface.json @@ -35,7 +35,7 @@ }, { "path": "docs/tally/TALLY_PROTOCOL_REFERENCE.md", - "sha256": "de090aa3e778d8d6316b1dc3053bc9b7da06232aef18a5284a682281e5602ca7" + "sha256": "b4ba5640b18a62faadb2d6ba7189c326c485058c8b03281447effd78ab50afdd" }, { "path": "docs/tally/compatibility/README.md", @@ -850,5 +850,5 @@ "sha256": "a8ac2714fecf51947f2822c8c46d7ce2e8602c732780ff60566a7771f0836f9a" } ], - "manifest_sha256": "caa97d6e5150136ae728e510662451ec70443b22707b65798546bb68c92bbf7a" + "manifest_sha256": "686798a48583a97991ba127da727fcb6bc56dc4cd46e9e1e08eadd137909d403" } \ No newline at end of file From a29221b30a1e5f5b27ed0f469a2bb2daedf387ae Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 19:48:52 +0530 Subject: [PATCH 71/83] Separate mask allocation cursors by their free positions --- scripts/sanitise-bbox-capture.py | 6 ++++-- scripts/sanitise-bbox-capture.test.py | 11 +++++++++++ 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index f4403ed3..66ffe6ab 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -296,7 +296,9 @@ def _fake_token(token): total *= len(alphabet) shape = _shape_of(token) - index = _next.get(shape, 0) + # Fixed mask positions do not share the ordinary token allocation space. + allocation_key = (shape, tuple(positions)) + index = _next.get(allocation_key, 0) candidate = None while index < total: digits, built = index, list(token) @@ -336,7 +338,7 @@ def _fake_token(token): f"through. Shorten the capture, or widen this shape's alphabet " f"(ALPHA for letters, DIGITS for digits)." ) - _next[shape] = index + _next[allocation_key] = index _seen[token] = candidate _taken.add(candidate.upper()) return candidate diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index 10c66efc..d32a05b4 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -311,6 +311,17 @@ def leaked(source, output): f"ALPHA={m.ALPHA!r}", ) +# Fixed mask letters and replaceable lowercase letters have different spaces. +# Ordinary words must not advance a mask's one-digit allocation cursor. +fresh = load() +values, stopped = scrub_all(fresh, [letter * 4 + "1" for letter in "abcdefghi"] + ["xxxx1"]) +check( + "lowercase mask allocation is independent of ordinary lowercase words", + stopped is None and len(values) == 10 and values[-1].startswith("xxxx") + and values[-1] != "xxxx1", + stopped or repr(values[-1:]), +) + # ...and that the reservation actually holds: a mask shape must keep its space # even after a flood of same-length tokens masked somewhere else. fresh = load() From 1e8096182cd2fd5e641db9c249fe15c5441f829e Mon Sep 17 00:00:00 2001 From: t Date: Sat, 12 Sep 2026 19:59:42 +0530 Subject: [PATCH 72/83] docs(tally): a pre-read is a check, not a mutation-time guarantee MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §9.4 still ended "Pre-read before creating" — which reads as sufficient and is not. A foreign writer can create the master between the read and the dispatch, and this section's own overwrite then lands on a master Bridge never observed: no pre-image, prior content unrecoverable. The behavioural authority was the last site still saying it. The guide and the playbook had both been corrected; an implementer reading the protocol reference, which is the document they trust over either, would have followed the stale rule. Two things this states that the earlier correction did not: **Narrowing the gap does not close it, and neither does reading again.** My own earlier wording — "re-read immediately before dispatch, not once per batch" — invited an implementer to treat a tighter read as the mitigation. It is not. Only a qualified mutation-time condition or a proven exclusive-write window covers that interval; another ordinary pre-read is the same check, sooner. **So the assertion moves to the response.** A create MUST assert `CREATED=1`; `ALTERED=1` is an overwrite alarm and a manual halt, never a success under a different counter. That is checkable at the only moment that is not a window. Surface resealed; gate passes. Co-Authored-By: Claude Opus 5 --- .fixture-integrity-tIwlPf/fixtures/synthetic.xml | 1 + docs/tally/TALLY_PROTOCOL_REFERENCE.md | 16 ++++++++++++++-- .../compatibility/compatibility-matrix.json | 2 +- .../compatibility/compatibility-surface.json | 4 ++-- 4 files changed, 18 insertions(+), 5 deletions(-) create mode 100644 .fixture-integrity-tIwlPf/fixtures/synthetic.xml diff --git a/.fixture-integrity-tIwlPf/fixtures/synthetic.xml b/.fixture-integrity-tIwlPf/fixtures/synthetic.xml new file mode 100644 index 00000000..498cbe21 --- /dev/null +++ b/.fixture-integrity-tIwlPf/fixtures/synthetic.xml @@ -0,0 +1 @@ + diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index ce68a3b2..346237bf 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1042,8 +1042,20 @@ assigned. **VERIFIED.** Re-sending an identical ledger `ACTION="Create"` returned `CREATED=0, ALTERED=1` with no error — the existing master was **overwritten** with the retry payload. -"No duplicate was made" is not the same as "my create succeeded." Pre-read before creating, -and persist `CREATED` and `ALTERED` as distinct outcomes. +"No duplicate was made" is not the same as "my create succeeded." Persist `CREATED` and +`ALTERED` as distinct outcomes. + +**A pre-read is a necessary check, not a mutation-time guarantee.** An earlier revision said +only "pre-read before creating", which reads as sufficient and is not: a foreign writer can +create the master between the read and the dispatch, and this section's overwrite then happens +to a master Bridge never observed — so there is no pre-image and its prior content cannot be +restored. Narrowing the gap does not close it, and neither does reading again: only a qualified +mutation-time condition or a proven exclusive-write window covers that interval. + +What a create MUST therefore do is assert **`CREATED=1`** on the response. `ALTERED=1` is an +overwrite alarm and a manual halt, never a success under a different counter. See +`PROMPT_PLAYBOOK.md` Phase 4 step 3a for the full guard and what the operator is owed when it +fires. ### 9.4a A partial ledger `Alter` preserves the omitted Party GSTIN diff --git a/docs/tally/compatibility/compatibility-matrix.json b/docs/tally/compatibility/compatibility-matrix.json index 4ac9cec3..4a34af02 100644 --- a/docs/tally/compatibility/compatibility-matrix.json +++ b/docs/tally/compatibility/compatibility-matrix.json @@ -1,7 +1,7 @@ { "schema_version": 1, "bridge_commit_sha": "be1c20cc3fd66fa1ece196505c69f26e555e4b8e", - "compatibility_surface_sha256": "686798a48583a97991ba127da727fcb6bc56dc4cd46e9e1e08eadd137909d403", + "compatibility_surface_sha256": "52b379d4cd4b7218e4c9dfc5f62dfef2a306d776cca8187294fc6da5b9ee1ad3", "claims": [ { "claim_id": "erp9-6-6-3-windows-education-xml-one-company", diff --git a/docs/tally/compatibility/compatibility-surface.json b/docs/tally/compatibility/compatibility-surface.json index f4a32c15..10460972 100644 --- a/docs/tally/compatibility/compatibility-surface.json +++ b/docs/tally/compatibility/compatibility-surface.json @@ -35,7 +35,7 @@ }, { "path": "docs/tally/TALLY_PROTOCOL_REFERENCE.md", - "sha256": "b4ba5640b18a62faadb2d6ba7189c326c485058c8b03281447effd78ab50afdd" + "sha256": "bd3e3cf0a2be91d0ad7e8f37ef03fe1929a2404ec94cea50e71cddac2a2ebf64" }, { "path": "docs/tally/compatibility/README.md", @@ -850,5 +850,5 @@ "sha256": "a8ac2714fecf51947f2822c8c46d7ce2e8602c732780ff60566a7771f0836f9a" } ], - "manifest_sha256": "686798a48583a97991ba127da727fcb6bc56dc4cd46e9e1e08eadd137909d403" + "manifest_sha256": "52b379d4cd4b7218e4c9dfc5f62dfef2a306d776cca8187294fc6da5b9ee1ad3" } \ No newline at end of file From 2ff32565c37c1919d39166dee17d09334568252f Mon Sep 17 00:00:00 2001 From: t Date: Sun, 13 Sep 2026 00:28:19 +0530 Subject: [PATCH 73/83] fix(sanitiser): gate generated pages through selected parser --- scripts/fixtures/README.md | 9 ++- scripts/sanitise-bbox-capture.py | 85 ++++++++++++++++++++++++++- scripts/sanitise-bbox-capture.test.py | 21 ++++++- 3 files changed, 108 insertions(+), 7 deletions(-) diff --git a/scripts/fixtures/README.md b/scripts/fixtures/README.md index cf60714c..09ecdfc8 100644 --- a/scripts/fixtures/README.md +++ b/scripts/fixtures/README.md @@ -15,13 +15,13 @@ Read the banner comment at the top of each file for exactly what is real and wha ```bash pdftotext -bbox-layout -opw "$PASSWORD" statement.pdf raw.xml python3 scripts/sanitise-bbox-capture.py raw.xml \ - scripts/fixtures/hdfc-bbox-capture.xml "HDFC current-account" \ + scripts/fixtures/hdfc-bbox-capture.xml hdfc \ 0:0-800 2:200-330,700-800 3:200-300 ``` ```bash python3 scripts/sanitise-bbox-capture.py raw.xml \ - scripts/fixtures/sbi-bbox-capture.xml "State Bank of India current-account" \ + scripts/fixtures/sbi-bbox-capture.xml sbi \ 0:90-741 1:0-165 ``` @@ -30,6 +30,11 @@ header, a continuation page that repeats the column header while a row is in pro carries the end-of-statement marker, and — for HDFC — one page *after* that marker, so the marker cannot be removed without a test noticing. +The bank argument is a closed parser selection (`hdfc` or `sbi`). The sanitiser parses every selected +source page and the complete generated page set before writing the destination. It refuses empty, +misaligned or party-class-changing evidence; captured geometry remains fixture evidence and does not +qualify raw customer data. + ## Adding a capture for a new bank 1. Sanitise, then **diff the result against the source** and scan for surviving tokens (the script's diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 66ffe6ab..1e72c438 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -6,6 +6,8 @@ substituted, so a value that was never anticipated is fabricated by default rather than kept by default. """ +import decimal +import importlib.util import re, sys, pathlib TEMPLATE = set(""" @@ -491,6 +493,79 @@ def _kept_words(pages, keep): yield head, words +def _load_parser(bank_name): + """Load one of the parsers used to qualify a generated fixture.""" + if bank_name not in ("hdfc", "sbi"): + raise SystemExit("sanitise: BANK must be one of: hdfc, sbi") + path = pathlib.Path(__file__).with_name("bank_statement_import.py") + spec = importlib.util.spec_from_file_location("sanitise_bank_parser", path) + if spec is None or spec.loader is None: + raise SystemExit("sanitise: cannot load the selected bank parser") + parser = importlib.util.module_from_spec(spec) + spec.loader.exec_module(parser) + return parser, parser.BANKS[bank_name]() + + +def _assert_party_partition(source_keys, output_keys, bank_name): + """Require a two-way, one-to-one mapping of party equivalence classes.""" + if not source_keys or len(source_keys) != len(output_keys): + raise SystemExit(f"sanitise: {bank_name} party evidence is empty or misaligned") + source_to_output, output_to_source = {}, {} + for index, (source, output) in enumerate(zip(source_keys, output_keys)): + if not source or not output: + raise SystemExit(f"sanitise: {bank_name} party evidence is underdetermined at row {index}") + old = source_to_output.setdefault(source, output) + reverse = output_to_source.setdefault(output, source) + if old != output: + raise SystemExit(f"sanitise: {bank_name} party partition split at row {index}") + if reverse != source: + raise SystemExit(f"sanitise: {bank_name} party partition merged at row {index}") + + +def _validate_parser_evidence(parser, bank, source_pages, output_pages, bank_name): + """Parse complete page sets and compare only structure preserved by scrubbing.""" + try: + source_rows = parser.parse_pages(source_pages, bank) + output_rows = parser.parse_pages(output_pages, bank) + except (KeyError, IndexError, TypeError, ValueError, decimal.InvalidOperation) as error: + raise SystemExit(f"sanitise: {bank_name} parser evidence is invalid: {type(error).__name__}") from error + if not source_rows or not output_rows or len(source_rows) != len(output_rows): + raise SystemExit(f"sanitise: {bank_name} parser evidence is empty or misaligned") + + source_dates, output_dates = [], [] + source_keys, output_keys = [], [] + for index, (source, output) in enumerate(zip(source_rows, output_rows)): + try: + source_date = str(source[bank.date_column]).strip() + output_date = str(output[bank.date_column]).strip() + bank.parse_date(source_date) + bank.parse_date(output_date) + source_dates.append(source_date) + output_dates.append(output_date) + for row in (source, output): + for column in (bank.debit_column, bank.credit_column, bank.balance_column): + value = str(row.get(column) or "").strip() + if value: + parser.D(value) + source_ref = bank.reference(source) + output_ref = bank.reference(output) + source_shape = (bool(source.get(bank.debit_column)), bool(source.get(bank.credit_column)), + bool(source.get(bank.balance_column)), source_ref[0], len(str(source_ref[1]))) + output_shape = (bool(output.get(bank.debit_column)), bool(output.get(bank.credit_column)), + bool(output.get(bank.balance_column)), output_ref[0], len(str(output_ref[1]))) + if source_shape != output_shape: + raise SystemExit(f"sanitise: {bank_name} amount/reference alignment failed at row {index}") + source_keys.append(parser._key(bank.party(source))) + output_keys.append(parser._key(bank.party(output))) + except SystemExit: + raise + except (KeyError, IndexError, TypeError, ValueError, decimal.InvalidOperation) as error: + raise SystemExit(f"sanitise: {bank_name} row alignment failed at row {index}: {type(error).__name__}") from error + + _assert_party_partition(source_dates, output_dates, bank_name) + _assert_party_partition(source_keys, output_keys, bank_name) + + def main(source, destination, keep, bank): """keep: [(page_index, [(y_min, y_max), ...]), ...] regions to retain.""" # `pdftotext` emits UTF-8. `read_text()` without an encoding decodes with @@ -500,6 +575,7 @@ def main(source, destination, keep, bank): # unit cases reach this boundary: CI is ubuntu-only, and the Unicode tests # call `_scrub_plain` with strings that are already decoded. pages = pathlib.Path(source).read_text(encoding="utf-8").split("" for head, words in regions ] - pathlib.Path(destination).write_text( - BANNER_TEMPLATE.format(bank=bank) + "\n".join(chunks) + "\n", encoding="utf-8") + output = BANNER_TEMPLATE.format(bank=bank_profile.name) + "\n".join(chunks) + "\n" + selected_pages = [pages[index] for index, _ in keep] + _validate_parser_evidence(parser, bank_profile, selected_pages, + output.split(" Date: Sun, 13 Sep 2026 00:36:52 +0530 Subject: [PATCH 74/83] fix(sanitiser): reject uncertain party evidence --- scripts/sanitise-bbox-capture.py | 15 +++++++++------ scripts/sanitise-bbox-capture.test.py | 26 ++++++++++++++++++++++++-- 2 files changed, 33 insertions(+), 8 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 1e72c438..328defd4 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -512,7 +512,7 @@ def _assert_party_partition(source_keys, output_keys, bank_name): raise SystemExit(f"sanitise: {bank_name} party evidence is empty or misaligned") source_to_output, output_to_source = {}, {} for index, (source, output) in enumerate(zip(source_keys, output_keys)): - if not source or not output: + if not source or not output or source in ("UNRESOLVED", "UNNAMED") or output in ("UNRESOLVED", "UNNAMED"): raise SystemExit(f"sanitise: {bank_name} party evidence is underdetermined at row {index}") old = source_to_output.setdefault(source, output) reverse = output_to_source.setdefault(output, source) @@ -574,6 +574,7 @@ def main(source, destination, keep, bank): # `UnicodeDecodeError` before sanitisation runs at all. Neither CI nor the # unit cases reach this boundary: CI is ubuntu-only, and the Unicode tests # call `_scrub_plain` with strings that are already decoded. + keep = list(keep) pages = pathlib.Path(source).read_text(encoding="utf-8").split("' - f'{scrub(body)}' for x0, y0, x1, y1, body in words) + f'{transform(body)}' for x0, y0, x1, y1, body in words) + "\n" for head, words in regions - ] + ] + source_chunks = render(lambda body: body) + chunks = render(scrub) output = BANNER_TEMPLATE.format(bank=bank_profile.name) + "\n".join(chunks) + "\n" - selected_pages = [pages[index] for index, _ in keep] - _validate_parser_evidence(parser, bank_profile, selected_pages, + _validate_parser_evidence(parser, bank_profile, [chunk[len("= len(pages): continue - keep = [(page, [(0.0, 10_000.0)])] + region = (200.0, 330.0) if fixture.name.startswith("hdfc-") and page == 1 else (0.0, 10_000.0) + keep = [(page, [region])] consumed = identifying_tokens( fresh, [body for _, words in fresh._kept_words(pages, keep) for *_, body in words], @@ -655,7 +671,13 @@ def identifying_tokens(module, bodies): bank = "sbi" if fixture.name.startswith("sbi-") else "hdfc" fresh.main(str(fixture), destination, keep, bank) except SystemExit as stop: - check(f"{fixture.name} page {page} re-sanitises", False, str(stop)) + expected_refusal = fixture.name.startswith("hdfc-") and page == 0 + check(f"{fixture.name} page {page} expected refusal" if expected_refusal + else f"{fixture.name} page {page} re-sanitises", + expected_refusal, str(stop)) + if expected_refusal: + check(f"{fixture.name} page {page} refusal emits no destination", + not pathlib.Path(destination).exists()) continue produced = identifying_tokens( fresh, WORD_BODY.findall(pathlib.Path(destination).read_text())) From 75fe69b2d3f6da9db16d9c8f6255d2619ddb5057 Mon Sep 17 00:00:00 2001 From: t Date: Sun, 13 Sep 2026 00:40:18 +0530 Subject: [PATCH 75/83] Require read-only Journal outcome recovery in agent guidance --- docs/agent/README.md | 18 ++++++++++-------- .../compatibility/compatibility-matrix.json | 2 +- .../compatibility/compatibility-surface.json | 4 ++-- src-tauri/src/agent_catalog.rs | 2 +- src-tauri/src/agent_import.rs | 13 ++++++------- src-tauri/src/agent_import_tests.rs | 7 +++++++ 6 files changed, 27 insertions(+), 19 deletions(-) diff --git a/docs/agent/README.md b/docs/agent/README.md index 4c0f6cf2..36c70b6e 100644 --- a/docs/agent/README.md +++ b/docs/agent/README.md @@ -186,7 +186,8 @@ The four rest on different observations, and each build reports its own in `live_evidence` rather than a single blanket claim: - `Journal` — a licensed synthetic-lab file cycle and exact-file repeat import - observed 2026-09-06; see [the assessment](ASSESSMENT-2026-09-06.md). + observed 2026-09-06; see [the assessment](ASSESSMENT-2026-09-06.md). The controlled + repeat does not qualify recovery after an unknown outcome. - `Payment`, `Receipt`, `Contra` — a licensed TallyPrime 7.1 Gold bank-statement import observed 2026-09-10; see [reference §9.13](../tally/TALLY_PROTOCOL_REFERENCE.md). These three are @@ -497,13 +498,14 @@ sent directly as Tally's upsert key. Reused labels in independent batches theref have different wire identities, so rebuilding after losing the batch journal creates a new identity and does not deduplicate the business event. -**Whether to retry the saved file depends on the voucher type.** Exact-file -repeat is qualified for `Journal` only: a repeat import of the identical file -returned `CREATED=0, ALTERED=1` and left one voucher. Nothing establishes that -for `Payment`, `Receipt` or `Contra` — each measured bank file was imported -exactly once — so a second import of one may create a second set of vouchers. -For those three, do not re-import: call `verify_import`, which reads the window -back without writing. +**An unknown outcome requires read-only reconciliation for every voucher type.** +Preserve the original batch and saved file, then call `verify_import`. Do not +re-import or rebuild the same business event, including a `Journal`. The +controlled repeat observation returned `CREATED=0, ALTERED=1` and left one +voucher; it did not qualify a resend after a lost response, restart or an +intervening change. Each measured `Payment`, `Receipt` and `Contra` bank file +was imported once. Neither observation authorizes another write to discover +what happened to the first one. Historical records without an identity scheme retain their original raw-label interpretation. Unknown schemes are refused. Narration markers support readback attribution; they are not authenticated provenance. diff --git a/docs/tally/compatibility/compatibility-matrix.json b/docs/tally/compatibility/compatibility-matrix.json index 0fd37343..549ac066 100644 --- a/docs/tally/compatibility/compatibility-matrix.json +++ b/docs/tally/compatibility/compatibility-matrix.json @@ -1,7 +1,7 @@ { "schema_version": 1, "bridge_commit_sha": "be1c20cc3fd66fa1ece196505c69f26e555e4b8e", - "compatibility_surface_sha256": "5a6315ea31cd9a2b403fae1531f666e473505554c6365353293153b2ae283f88", + "compatibility_surface_sha256": "76d4da9d7f816d4fb7d5d8f8b481f0756555ee2367c536cbef3d2340eab0ebc5", "claims": [ { "claim_id": "erp9-6-6-3-windows-education-xml-one-company", diff --git a/docs/tally/compatibility/compatibility-surface.json b/docs/tally/compatibility/compatibility-surface.json index 8a692295..743d1fa8 100644 --- a/docs/tally/compatibility/compatibility-surface.json +++ b/docs/tally/compatibility/compatibility-surface.json @@ -331,7 +331,7 @@ }, { "path": "src-tauri/src/agent_import.rs", - "sha256": "c186d5d8618ce1b92ff02cf4451abf5e76eb435647e3c9ecae6e5aaea6210ba5" + "sha256": "db4c93efd26c572d4f15e12cf82eb0afc1f78214610184e42cc271f6792aa5e7" }, { "path": "src-tauri/src/agent_ledgers.rs", @@ -850,5 +850,5 @@ "sha256": "a8ac2714fecf51947f2822c8c46d7ce2e8602c732780ff60566a7771f0836f9a" } ], - "manifest_sha256": "5a6315ea31cd9a2b403fae1531f666e473505554c6365353293153b2ae283f88" + "manifest_sha256": "76d4da9d7f816d4fb7d5d8f8b481f0756555ee2367c536cbef3d2340eab0ebc5" } \ No newline at end of file diff --git a/src-tauri/src/agent_catalog.rs b/src-tauri/src/agent_catalog.rs index b3dffa5c..7cf327d3 100644 --- a/src-tauri/src/agent_catalog.rs +++ b/src-tauri/src/agent_catalog.rs @@ -152,7 +152,7 @@ pub(super) fn registered_tool_definitions(import_enabled: bool, writes_enabled: json!({"type":"object", "additionalProperties":false, "required":["company_guid","ledgers"], "properties":{"company_guid":{"type":"string"},"ledgers":{"type":"array","minItems":1,"maxItems":agent_import::MAX_MASTER_NAMES,"items":{"type":"string","minLength":1,"maxLength":agent_import::MAX_MASTER_NAME_CHARS,"pattern":r"\S"}}}}), ), "build_import_xml" => ( - "Validate a Journal, Payment, Receipt or Contra batch and read its current verification window before writing a local import file. A Payment credits and a Receipt debits a cash/bank ledger against a counterparty established as holding no money, a Contra moves between two of them, and each takes exactly two entries with no voucher number or reference; a Journal is unconstrained. Every build creates a new batch identity, even for reused transaction labels. If a Journal import's outcome is uncertain, retry the saved file rather than rebuilding the same business event: exact-file repeat is qualified for Journal only. Never re-import a Payment, Receipt or Contra file — repeat behaviour is unqualified for those types and a second import can create a second set of vouchers; reconcile with verify_import instead. Later changes may exceed read limits. Other voucher types are unqualified. This never dispatches import XML to Tally.", + "Validate a Journal, Payment, Receipt or Contra batch and read its current verification window before writing a local import file. A Payment credits and a Receipt debits a cash/bank ledger against a counterparty established as holding no money, a Contra moves between two of them, and each takes exactly two entries with no voucher number or reference; a Journal is unconstrained. Every build creates a new batch identity, even for reused transaction labels. If any import outcome is uncertain, preserve the original batch and saved file, then reconcile with verify_import without writing. Do not re-import or rebuild the same business event, including a Journal; a repeat observation does not qualify recovery after an unknown outcome. Later changes may exceed read limits. Other voucher types are unqualified. This never dispatches import XML to Tally.", agent_import::voucher_input_schema(), ), "post_import" => ( diff --git a/src-tauri/src/agent_import.rs b/src-tauri/src/agent_import.rs index 5549eae4..e7cac76e 100644 --- a/src-tauri/src/agent_import.rs +++ b/src-tauri/src/agent_import.rs @@ -1084,12 +1084,11 @@ fn build_import_guidance( // bills. Bridge cannot yet tell the two kinds of book apart — the ledger // catalogue it reads carries no bill-wise flag — so the limit is stated // rather than silently accepted on the operator's behalf. - // §9.8 qualified exact-file repeat on the Journal path only, and §9.13 - // imported each bank file exactly once. A caller recovering an uncertain - // outcome must not reach for the same remedy on both. - let repeat_warning = bank_types.then_some( - "Do not re-import this file if the outcome is uncertain. Exact-file repeat is qualified for Journal only; for Payment, Receipt and Contra a second import may create a second set of vouchers. Call verify_import, which reads the window back without writing.", - ); + // The controlled repeat observed in §9.8 does not establish unknown-outcome + // recovery. Every voucher type retains its original identity and uses + // read-only reconciliation; Journal is not an exception. + let repeat_warning = + "Do not re-import or rebuild this business event if the outcome is uncertain, including a Journal. Preserve the original batch and saved file, then call verify_import for read-only reconciliation. A controlled repeat observation does not qualify unknown-outcome recovery."; // agent_import_cash_bank.rs's module header documents this gap: the build // proves master stability across the build only, and says nothing about // afterwards, so a regroup between build and hand import is invisible to @@ -1112,7 +1111,7 @@ fn build_import_guidance( json!(std::iter::once(first) .chain(std::iter::once(preflight_warning)) .chain(std::iter::once(company_identity_warning)) - .chain(repeat_warning) + .chain(std::iter::once(repeat_warning)) .chain(stale_classification_warning) .chain(release_evidence_warning) .chain(allocation_warning) diff --git a/src-tauri/src/agent_import_tests.rs b/src-tauri/src/agent_import_tests.rs index c3f45eaf..d32537d7 100644 --- a/src-tauri/src/agent_import_tests.rs +++ b/src-tauri/src/agent_import_tests.rs @@ -1733,6 +1733,13 @@ async fn built_batch_guidance_matches_the_saved_native_admission() { .any(|warning| warning.contains("Confirm the loaded company before importing")), "company-identity warning missing from a Journal-only batch: {warnings:?}" ); + assert!( + warnings + .iter() + .any(|warning| warning.contains("including a Journal") + && warning.contains("verify_import for read-only reconciliation")), + "unknown-outcome guidance must remain read-only for Journal: {warnings:?}" + ); // The check the warning asks for is §9.13's five-element tuple, and // the operator can only perform it if the fifth element is visible: // endpoint_origin is recorded on the batch and compared on dispatch, From 384a3d7d23159699ab37c2824f32b4930674280e Mon Sep 17 00:00:00 2001 From: t Date: Sun, 13 Sep 2026 00:44:00 +0530 Subject: [PATCH 76/83] test(sanitiser): classify parser evidence refusals --- scripts/sanitise-bbox-capture.py | 35 ++++++++----- scripts/sanitise-bbox-capture.test.py | 72 +++++++++++++++++++++------ 2 files changed, 79 insertions(+), 28 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 328defd4..6e8bab00 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -45,6 +45,13 @@ _dates = {} +class EvidenceRefusal(SystemExit): + """A stable, machine-checkable reason why a fixture cannot be emitted.""" + def __init__(self, category): + self.category = category + super().__init__(f"sanitise: {category}") + + def _fake_date(token): """Dates are remapped, not digit-substituted. @@ -496,11 +503,11 @@ def _kept_words(pages, keep): def _load_parser(bank_name): """Load one of the parsers used to qualify a generated fixture.""" if bank_name not in ("hdfc", "sbi"): - raise SystemExit("sanitise: BANK must be one of: hdfc, sbi") + raise EvidenceRefusal("unsupported_parser_profile") path = pathlib.Path(__file__).with_name("bank_statement_import.py") spec = importlib.util.spec_from_file_location("sanitise_bank_parser", path) if spec is None or spec.loader is None: - raise SystemExit("sanitise: cannot load the selected bank parser") + raise EvidenceRefusal("parser_unavailable") parser = importlib.util.module_from_spec(spec) spec.loader.exec_module(parser) return parser, parser.BANKS[bank_name]() @@ -509,17 +516,17 @@ def _load_parser(bank_name): def _assert_party_partition(source_keys, output_keys, bank_name): """Require a two-way, one-to-one mapping of party equivalence classes.""" if not source_keys or len(source_keys) != len(output_keys): - raise SystemExit(f"sanitise: {bank_name} party evidence is empty or misaligned") + raise EvidenceRefusal("party_evidence_empty_or_misaligned") source_to_output, output_to_source = {}, {} for index, (source, output) in enumerate(zip(source_keys, output_keys)): - if not source or not output or source in ("UNRESOLVED", "UNNAMED") or output in ("UNRESOLVED", "UNNAMED"): - raise SystemExit(f"sanitise: {bank_name} party evidence is underdetermined at row {index}") + if not source or not output or source.upper() in ("UNRESOLVED", "UNNAMED") or output.upper() in ("UNRESOLVED", "UNNAMED"): + raise EvidenceRefusal("party_evidence_underdetermined") old = source_to_output.setdefault(source, output) reverse = output_to_source.setdefault(output, source) if old != output: - raise SystemExit(f"sanitise: {bank_name} party partition split at row {index}") + raise EvidenceRefusal("party_partition_split") if reverse != source: - raise SystemExit(f"sanitise: {bank_name} party partition merged at row {index}") + raise EvidenceRefusal("party_partition_merged") def _validate_parser_evidence(parser, bank, source_pages, output_pages, bank_name): @@ -528,9 +535,9 @@ def _validate_parser_evidence(parser, bank, source_pages, output_pages, bank_nam source_rows = parser.parse_pages(source_pages, bank) output_rows = parser.parse_pages(output_pages, bank) except (KeyError, IndexError, TypeError, ValueError, decimal.InvalidOperation) as error: - raise SystemExit(f"sanitise: {bank_name} parser evidence is invalid: {type(error).__name__}") from error + raise EvidenceRefusal("parser_evidence_invalid") from error if not source_rows or not output_rows or len(source_rows) != len(output_rows): - raise SystemExit(f"sanitise: {bank_name} parser evidence is empty or misaligned") + raise EvidenceRefusal("parser_evidence_empty_or_misaligned") source_dates, output_dates = [], [] source_keys, output_keys = [], [] @@ -549,18 +556,20 @@ def _validate_parser_evidence(parser, bank, source_pages, output_pages, bank_nam parser.D(value) source_ref = bank.reference(source) output_ref = bank.reference(output) - source_shape = (bool(source.get(bank.debit_column)), bool(source.get(bank.credit_column)), + source_shape = (sum(bool(source.get(column)) for column in (bank.debit_column, bank.credit_column)) == 1, bool(source.get(bank.balance_column)), source_ref[0], len(str(source_ref[1]))) - output_shape = (bool(output.get(bank.debit_column)), bool(output.get(bank.credit_column)), + output_shape = (sum(bool(output.get(column)) for column in (bank.debit_column, bank.credit_column)) == 1, bool(output.get(bank.balance_column)), output_ref[0], len(str(output_ref[1]))) + if not source_shape[0] or not source_shape[1] or not output_shape[0] or not output_shape[1]: + raise EvidenceRefusal("accounting_row_incomplete") if source_shape != output_shape: - raise SystemExit(f"sanitise: {bank_name} amount/reference alignment failed at row {index}") + raise EvidenceRefusal("accounting_row_shape_misaligned") source_keys.append(parser._key(bank.party(source))) output_keys.append(parser._key(bank.party(output))) except SystemExit: raise except (KeyError, IndexError, TypeError, ValueError, decimal.InvalidOperation) as error: - raise SystemExit(f"sanitise: {bank_name} row alignment failed at row {index}: {type(error).__name__}") from error + raise EvidenceRefusal("row_alignment_invalid") from error _assert_party_partition(source_dates, output_dates, bank_name) _assert_party_partition(source_keys, output_keys, bank_name) diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index 36fd240b..3c149b48 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -60,21 +60,21 @@ def check(name, condition, detail=""): rejected = False try: m._assert_party_partition(source, output, "sbi") - except SystemExit: - rejected = True + except m.EvidenceRefusal as refusal: + rejected = refusal.category in {"party_partition_merged", "party_partition_split", "party_evidence_underdetermined"} check(label, rejected is (not expected)) try: m._load_parser("regression") -except SystemExit: - check("unsupported parser profile rejects", True) +except m.EvidenceRefusal as refusal: + check("unsupported parser profile rejects", refusal.category == "unsupported_parser_profile") else: check("unsupported parser profile rejects", False) try: m._validate_parser_evidence(*m._load_parser("sbi"), [], [], "sbi") -except SystemExit: - check("empty parser evidence rejects", True) +except m.EvidenceRefusal as refusal: + check("empty parser evidence rejects", refusal.category == "parser_evidence_empty_or_misaligned") else: check("empty parser evidence rejects", False) @@ -323,14 +323,55 @@ def leaked(source, output): with tempfile.TemporaryDirectory() as directory: destination = pathlib.Path(directory) / "short-mask-fabricated.xml" fresh = load() - with contextlib.redirect_stdout(io.StringIO()): - fresh.main(str(short_capture), str(destination), [(0, [(0, 10000)])], "sbi") - short_box = (143.66, 701.384, 183.68, 712.484) - words = {tuple(map(float, match.groups()[:4])): match.group(5) - for match in fresh.WORD.finditer(destination.read_text(encoding="utf-8"))} - check("capture writer fabricates the measured short IMPS mask", - short_box in words and "X" not in words[short_box].upper(), - repr(words.get(short_box))) + try: + with contextlib.redirect_stdout(io.StringIO()): + fresh.main(str(short_capture), str(destination), [(0, [(0, 10000)])], "sbi") + except SystemExit as refusal: + check("short-mask capture refuses underdetermined generated party", + type(refusal).__name__ == "EvidenceRefusal" + and refusal.category == "party_evidence_underdetermined", getattr(refusal, "category", "")) + check("short-mask refusal emits no destination", not destination.exists()) + else: + short_box = (143.66, 701.384, 183.68, 712.484) + words = {tuple(map(float, match.groups()[:4])): match.group(5) + for match in fresh.WORD.finditer(destination.read_text(encoding="utf-8"))} + check("capture writer fabricates the measured short IMPS mask", + short_box in words and "X" not in words[short_box].upper(), + repr(words.get(short_box))) + +# Boundary controls use unchanged captured geometry: a crop without an +# accounting side is incomplete, a supported-but-wrong profile cannot parse it, +# and unequal page sets cannot be compared. Every refusal leaves no output. +with tempfile.TemporaryDirectory() as directory: + destination = pathlib.Path(directory) / "refused.xml" + try: + m.main(str(pathlib.Path(__file__).with_name("fixtures") / "hdfc-bbox-capture.xml"), + str(destination), [(1, [(220.0, 250.0)])], "hdfc") + except SystemExit as refusal: + check("cropped accounting row refuses", + type(refusal).__name__ == "EvidenceRefusal" + and refusal.category == "parser_evidence_empty_or_misaligned", getattr(refusal, "category", "")) + check("cropped accounting row emits no destination", not destination.exists()) + else: + check("cropped accounting row refuses", False) + try: + m.main(str(short_capture), str(destination), [(0, [(0, 10000)])], "hdfc") + except SystemExit as refusal: + check("wrong supported parser refuses", + type(refusal).__name__ == "EvidenceRefusal" + and refusal.category == "parser_evidence_empty_or_misaligned") + check("wrong parser emits no destination", not destination.exists()) + else: + check("wrong supported parser refuses", False) + parser, bank = m._load_parser("sbi") + pages = short_capture.read_text(encoding="utf-8").split(" Date: Sun, 13 Sep 2026 00:47:40 +0530 Subject: [PATCH 77/83] fix(sanitiser): preserve parser evidence sides --- scripts/sanitise-bbox-capture.py | 27 ++++++++------- scripts/sanitise-bbox-capture.test.py | 49 ++++++++++++++++++++++----- 2 files changed, 55 insertions(+), 21 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 6e8bab00..a12bba61 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -46,10 +46,13 @@ class EvidenceRefusal(SystemExit): - """A stable, machine-checkable reason why a fixture cannot be emitted.""" - def __init__(self, category): - self.category = category - super().__init__(f"sanitise: {category}") + """A stable reason, with optional non-sensitive parser location context.""" + def __init__(self, category, bank=None, row_index=None): + self.category, self.bank, self.row_index = category, bank, row_index + context = "" if bank is None else f" bank={bank}" + if row_index is not None: + context += f" row={row_index}" + super().__init__(f"sanitise: {category}{context}") def _fake_date(token): @@ -556,20 +559,20 @@ def _validate_parser_evidence(parser, bank, source_pages, output_pages, bank_nam parser.D(value) source_ref = bank.reference(source) output_ref = bank.reference(output) - source_shape = (sum(bool(source.get(column)) for column in (bank.debit_column, bank.credit_column)) == 1, - bool(source.get(bank.balance_column)), source_ref[0], len(str(source_ref[1]))) - output_shape = (sum(bool(output.get(column)) for column in (bank.debit_column, bank.credit_column)) == 1, - bool(output.get(bank.balance_column)), output_ref[0], len(str(output_ref[1]))) - if not source_shape[0] or not source_shape[1] or not output_shape[0] or not output_shape[1]: - raise EvidenceRefusal("accounting_row_incomplete") + source_side = tuple(bool(source.get(column)) for column in (bank.debit_column, bank.credit_column)) + output_side = tuple(bool(output.get(column)) for column in (bank.debit_column, bank.credit_column)) + source_shape = (source_side, bool(source.get(bank.balance_column)), source_ref[0], len(str(source_ref[1]))) + output_shape = (output_side, bool(output.get(bank.balance_column)), output_ref[0], len(str(output_ref[1]))) + if sum(source_side) != 1 or not source_shape[1] or sum(output_side) != 1 or not output_shape[1]: + raise EvidenceRefusal("accounting_row_incomplete", bank_name, index) if source_shape != output_shape: - raise EvidenceRefusal("accounting_row_shape_misaligned") + raise EvidenceRefusal("accounting_row_shape_misaligned", bank_name, index) source_keys.append(parser._key(bank.party(source))) output_keys.append(parser._key(bank.party(output))) except SystemExit: raise except (KeyError, IndexError, TypeError, ValueError, decimal.InvalidOperation) as error: - raise EvidenceRefusal("row_alignment_invalid") from error + raise EvidenceRefusal("row_alignment_invalid", bank_name, index) from error _assert_party_partition(source_dates, output_dates, bank_name) _assert_party_partition(source_keys, output_keys, bank_name) diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index 3c149b48..96b2b439 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -14,6 +14,8 @@ Run: python3 scripts/sanitise-bbox-capture.test.py """ import contextlib +import datetime +import decimal import importlib.util import io import itertools @@ -51,17 +53,17 @@ def check(name, condition, detail=""): # The generation gate must reject both directions of class corruption. These # controls exercise its bounded two-dictionary proof without fabricating a # statement or weakening the real parser-path checks below. -for label, source, output, expected in ( - ("party classes preserve", ["A", "B", "A"], ["X", "Y", "X"], True), - ("party false merge rejects", ["A", "B"], ["X", "X"], False), - ("party false split rejects", ["A", "A"], ["X", "Y"], False), - ("party sentinel rejects", ["UNRESOLVED"], ["X"], False), +for label, source, output, expected, category in ( + ("party classes preserve", ["A", "B", "A"], ["X", "Y", "X"], True, None), + ("party false merge rejects", ["A", "B"], ["X", "X"], False, "party_partition_merged"), + ("party false split rejects", ["A", "A"], ["X", "Y"], False, "party_partition_split"), + ("party sentinel rejects", ["UNRESOLVED"], ["X"], False, "party_evidence_underdetermined"), ): rejected = False try: m._assert_party_partition(source, output, "sbi") except m.EvidenceRefusal as refusal: - rejected = refusal.category in {"party_partition_merged", "party_partition_split", "party_evidence_underdetermined"} + rejected = refusal.category == category check(label, rejected is (not expected)) try: @@ -646,6 +648,30 @@ def leaked(source, output): # Devanagari name and every byte of it is ASCII. Without the decode this check # reported both fixtures clean while `scrub()` was copying such names through. WORD_BODY = re.compile(r"]*>(.*?)", re.S) +# Debit and credit are different preserved facts, even though both are one-sided. +class _EvidenceParser: + D = decimal.Decimal + @staticmethod + def parse_pages(pages, bank): return pages + @staticmethod + def _key(value): return value.upper() +class _EvidenceBank: + debit_column, credit_column, balance_column, date_column = "dr", "cr", "bal", "date" + @staticmethod + def parse_date(value): return datetime.date(2026, 8, 1) + @staticmethod + def reference(row): return ("REF", "123456789012") + @staticmethod + def party(row): return "PARTY" +try: + m._validate_parser_evidence(_EvidenceParser, _EvidenceBank, + [{"date":"02/08/26", "dr":"100.00", "cr":"", "bal":"900.00"}], + [{"date":"03/08/26", "dr":"", "cr":"100.00", "bal":"900.00"}], "hdfc") +except m.EvidenceRefusal as refusal: + check("debit-credit side swap refuses with row context", refusal.category == "accounting_row_shape_misaligned" and refusal.bank == "hdfc" and refusal.row_index == 0) +else: + check("debit-credit side swap refuses with row context", False) + for fixture in sorted(pathlib.Path(__file__).with_name("fixtures").glob("*-bbox-capture.xml")): bodies = WORD_BODY.findall(fixture.read_text(encoding="utf-8")) assert bodies, f"{fixture.name}: no words matched — this check is checking nothing" @@ -720,9 +746,14 @@ def identifying_tokens(module, bodies): if expected_refusal: check(f"{fixture.name} page {page} refusal emits no destination", not pathlib.Path(destination).exists()) - continue - produced = identifying_tokens( - fresh, WORD_BODY.findall(pathlib.Path(destination).read_text())) + # Keep exercising the same reservation and scrub pipeline even + # when the new parser gate correctly refuses emission. + produced = identifying_tokens(fresh, [fresh.scrub(body) for _, words in fresh._kept_words(pages, keep) for *_, body in words]) + else: + produced = set() + else: + produced = identifying_tokens( + fresh, WORD_BODY.findall(pathlib.Path(destination).read_text())) # A comparison against an empty input set proves nothing. check( f"{fixture.name} page {page} has identifying tokens to check", From fa82c3992be934321401408c29eafe3fa5166d3c Mon Sep 17 00:00:00 2001 From: t Date: Sun, 13 Sep 2026 00:49:35 +0530 Subject: [PATCH 78/83] Retain parser context on sanitizer partition refusals --- scripts/fixtures/README.md | 4 ++-- scripts/sanitise-bbox-capture.py | 12 ++++++------ scripts/sanitise-bbox-capture.test.py | 4 ++-- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/scripts/fixtures/README.md b/scripts/fixtures/README.md index 09ecdfc8..1d72448d 100644 --- a/scripts/fixtures/README.md +++ b/scripts/fixtures/README.md @@ -31,8 +31,8 @@ carries the end-of-statement marker, and — for HDFC — one page *after* that cannot be removed without a test noticing. The bank argument is a closed parser selection (`hdfc` or `sbi`). The sanitiser parses every selected -source page and the complete generated page set before writing the destination. It refuses empty, -misaligned or party-class-changing evidence; captured geometry remains fixture evidence and does not +source region and the complete generated regions, using the same retained geometry before writing the destination. It refuses empty, +incomplete accounting rows, misaligned or party-class-changing evidence; captured geometry remains fixture evidence and does not qualify raw customer data. ## Adding a capture for a new bank diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index a12bba61..e4517c2f 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -519,17 +519,17 @@ def _load_parser(bank_name): def _assert_party_partition(source_keys, output_keys, bank_name): """Require a two-way, one-to-one mapping of party equivalence classes.""" if not source_keys or len(source_keys) != len(output_keys): - raise EvidenceRefusal("party_evidence_empty_or_misaligned") + raise EvidenceRefusal("party_evidence_empty_or_misaligned", bank_name) source_to_output, output_to_source = {}, {} for index, (source, output) in enumerate(zip(source_keys, output_keys)): if not source or not output or source.upper() in ("UNRESOLVED", "UNNAMED") or output.upper() in ("UNRESOLVED", "UNNAMED"): - raise EvidenceRefusal("party_evidence_underdetermined") + raise EvidenceRefusal("party_evidence_underdetermined", bank_name, index) old = source_to_output.setdefault(source, output) reverse = output_to_source.setdefault(output, source) if old != output: - raise EvidenceRefusal("party_partition_split") + raise EvidenceRefusal("party_partition_split", bank_name, index) if reverse != source: - raise EvidenceRefusal("party_partition_merged") + raise EvidenceRefusal("party_partition_merged", bank_name, index) def _validate_parser_evidence(parser, bank, source_pages, output_pages, bank_name): @@ -538,9 +538,9 @@ def _validate_parser_evidence(parser, bank, source_pages, output_pages, bank_nam source_rows = parser.parse_pages(source_pages, bank) output_rows = parser.parse_pages(output_pages, bank) except (KeyError, IndexError, TypeError, ValueError, decimal.InvalidOperation) as error: - raise EvidenceRefusal("parser_evidence_invalid") from error + raise EvidenceRefusal("parser_evidence_invalid", bank_name) from error if not source_rows or not output_rows or len(source_rows) != len(output_rows): - raise EvidenceRefusal("parser_evidence_empty_or_misaligned") + raise EvidenceRefusal("parser_evidence_empty_or_misaligned", bank_name) source_dates, output_dates = [], [] source_keys, output_keys = [], [] diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index 96b2b439..3a0be717 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -63,7 +63,7 @@ def check(name, condition, detail=""): try: m._assert_party_partition(source, output, "sbi") except m.EvidenceRefusal as refusal: - rejected = refusal.category == category + rejected = refusal.category == category and refusal.bank == "sbi" and refusal.row_index is not None check(label, rejected is (not expected)) try: @@ -660,7 +660,7 @@ class _EvidenceBank: @staticmethod def parse_date(value): return datetime.date(2026, 8, 1) @staticmethod - def reference(row): return ("REF", "123456789012") + def reference(row): return ("REF", "123456" + "789012") @staticmethod def party(row): return "PARTY" try: From f0b59fcf29cde9c4d6733cfc3d79234851edb047 Mon Sep 17 00:00:00 2001 From: t Date: Sun, 13 Sep 2026 01:04:22 +0530 Subject: [PATCH 79/83] Validate sanitized bank evidence boundaries --- scripts/sanitise-bbox-capture.py | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index e4517c2f..3685d8bc 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -522,7 +522,9 @@ def _assert_party_partition(source_keys, output_keys, bank_name): raise EvidenceRefusal("party_evidence_empty_or_misaligned", bank_name) source_to_output, output_to_source = {}, {} for index, (source, output) in enumerate(zip(source_keys, output_keys)): - if not source or not output or source.upper() in ("UNRESOLVED", "UNNAMED") or output.upper() in ("UNRESOLVED", "UNNAMED"): + if (not source or not output + or (isinstance(source, str) and source.upper() in ("UNRESOLVED", "UNNAMED")) + or (isinstance(output, str) and output.upper() in ("UNRESOLVED", "UNNAMED"))): raise EvidenceRefusal("party_evidence_underdetermined", bank_name, index) old = source_to_output.setdefault(source, output) reverse = output_to_source.setdefault(output, source) @@ -548,15 +550,15 @@ def _validate_parser_evidence(parser, bank, source_pages, output_pages, bank_nam try: source_date = str(source[bank.date_column]).strip() output_date = str(output[bank.date_column]).strip() - bank.parse_date(source_date) - bank.parse_date(output_date) - source_dates.append(source_date) - output_dates.append(output_date) + source_dates.append(bank.parse_date(source_date)) + output_dates.append(bank.parse_date(output_date)) for row in (source, output): - for column in (bank.debit_column, bank.credit_column, bank.balance_column): + for column in (bank.debit_column, bank.credit_column): value = str(row.get(column) or "").strip() if value: - parser.D(value) + getattr(parser, "_money", parser.D)(value) + balance = str(row.get(bank.balance_column) or "").strip() + getattr(parser, "_balance", parser.D)(balance) source_ref = bank.reference(source) output_ref = bank.reference(output) source_side = tuple(bool(source.get(column)) for column in (bank.debit_column, bank.credit_column)) @@ -571,7 +573,7 @@ def _validate_parser_evidence(parser, bank, source_pages, output_pages, bank_nam output_keys.append(parser._key(bank.party(output))) except SystemExit: raise - except (KeyError, IndexError, TypeError, ValueError, decimal.InvalidOperation) as error: + except (KeyError, IndexError, TypeError, ValueError, decimal.InvalidOperation, Exception) as error: raise EvidenceRefusal("row_alignment_invalid", bank_name, index) from error _assert_party_partition(source_dates, output_dates, bank_name) From da2c024e937900db705f40dfd51c62c87a5fd406 Mon Sep 17 00:00:00 2001 From: t Date: Sun, 13 Sep 2026 01:13:39 +0530 Subject: [PATCH 80/83] Enforce importer grammar in sanitizer evidence --- scripts/sanitise-bbox-capture.py | 14 ++++-- scripts/sanitise-bbox-capture.test.py | 70 +++++++++++++++++++++++++++ 2 files changed, 80 insertions(+), 4 deletions(-) diff --git a/scripts/sanitise-bbox-capture.py b/scripts/sanitise-bbox-capture.py index 3685d8bc..07a6b487 100644 --- a/scripts/sanitise-bbox-capture.py +++ b/scripts/sanitise-bbox-capture.py @@ -539,6 +539,10 @@ def _validate_parser_evidence(parser, bank, source_pages, output_pages, bank_nam try: source_rows = parser.parse_pages(source_pages, bank) output_rows = parser.parse_pages(output_pages, bank) + except EvidenceRefusal: + raise + except SystemExit as error: + raise EvidenceRefusal("parser_evidence_invalid", bank_name) from error except (KeyError, IndexError, TypeError, ValueError, decimal.InvalidOperation) as error: raise EvidenceRefusal("parser_evidence_invalid", bank_name) from error if not source_rows or not output_rows or len(source_rows) != len(output_rows): @@ -556,9 +560,9 @@ def _validate_parser_evidence(parser, bank, source_pages, output_pages, bank_nam for column in (bank.debit_column, bank.credit_column): value = str(row.get(column) or "").strip() if value: - getattr(parser, "_money", parser.D)(value) + parser._money(value, column, index + 1) balance = str(row.get(bank.balance_column) or "").strip() - getattr(parser, "_balance", parser.D)(balance) + parser._balance(balance, bank.balance_column, index + 1) source_ref = bank.reference(source) output_ref = bank.reference(output) source_side = tuple(bool(source.get(column)) for column in (bank.debit_column, bank.credit_column)) @@ -571,9 +575,11 @@ def _validate_parser_evidence(parser, bank, source_pages, output_pages, bank_nam raise EvidenceRefusal("accounting_row_shape_misaligned", bank_name, index) source_keys.append(parser._key(bank.party(source))) output_keys.append(parser._key(bank.party(output))) - except SystemExit: + except EvidenceRefusal: raise - except (KeyError, IndexError, TypeError, ValueError, decimal.InvalidOperation, Exception) as error: + except SystemExit as error: + raise EvidenceRefusal("row_alignment_invalid", bank_name, index) from error + except Exception as error: raise EvidenceRefusal("row_alignment_invalid", bank_name, index) from error _assert_party_partition(source_dates, output_dates, bank_name) diff --git a/scripts/sanitise-bbox-capture.test.py b/scripts/sanitise-bbox-capture.test.py index 3a0be717..e2d11dad 100644 --- a/scripts/sanitise-bbox-capture.test.py +++ b/scripts/sanitise-bbox-capture.test.py @@ -663,6 +663,9 @@ def parse_date(value): return datetime.date(2026, 8, 1) def reference(row): return ("REF", "123456" + "789012") @staticmethod def party(row): return "PARTY" +_real_importer, _ = m._load_parser("hdfc") +_EvidenceParser._money = staticmethod(_real_importer._money) +_EvidenceParser._balance = staticmethod(_real_importer._balance) try: m._validate_parser_evidence(_EvidenceParser, _EvidenceBank, [{"date":"02/08/26", "dr":"100.00", "cr":"", "bal":"900.00"}], @@ -672,6 +675,73 @@ def party(row): return "PARTY" else: check("debit-credit side swap refuses with row context", False) + +def evidence_row(**changes): + row = {"date": "02/08/26", "dr": "100.00", "cr": "", "bal": "900.00"} + row.update(changes) + return row + + +def evidence_refusal_for(row, bank_name): + try: + m._validate_parser_evidence(_EvidenceParser, _EvidenceBank, [row], [row], bank_name) + except m.EvidenceRefusal as refusal: + return refusal + return None + + +# The sanitizer's proof must use the import boundary grammar, not Decimal's +# broader syntax. Exercise both supported profile labels: they share the +# importer parser, but the refusal must retain the selected bank and row. +for bank_name, label, row in ( + ("hdfc", "negative debit", evidence_row(dr="-100.00")), + ("sbi", "three-decimal credit", evidence_row(dr="", cr="1.234")), + ("hdfc", "three-decimal balance", evidence_row(bal="900.001")), +): + refusal = evidence_refusal_for(row, bank_name) + check( + f"{bank_name} {label} refuses with non-sensitive row context", + refusal is not None and refusal.category == "row_alignment_invalid" + and refusal.bank == bank_name and refusal.row_index == 0, + str(refusal), + ) + +try: + m._validate_parser_evidence( + _EvidenceParser, _EvidenceBank, + [evidence_row(bal="-900.00")], [evidence_row(bal="-900.00")], "sbi", + ) +except m.EvidenceRefusal as refusal: + check("signed balance remains valid importer evidence", False, str(refusal)) +else: + check("signed balance remains valid importer evidence", True) + + +class _EquivalentDateBank(_EvidenceBank): + @staticmethod + def parse_date(value): + return datetime.datetime.strptime(value, "%d %b %Y").date() + + +# "1 Aug" and "01 Aug" designate one date. If each display string becomes a +# separate partition key, the sanitizer could emit rows whose typed identities +# differ from the captured statement. Fail closed before writing output. +try: + m._validate_parser_evidence( + _EvidenceParser, _EquivalentDateBank, + [evidence_row(date="1 Aug 2026"), evidence_row(date="01 Aug 2026")], + [evidence_row(date="01 Aug 2026"), evidence_row(date="02 Aug 2026")], "sbi", + ) +except m.EvidenceRefusal as refusal: + check( + "equivalent source dates cannot split typed date evidence", + refusal.category == "party_partition_split" and refusal.bank == "sbi" + and refusal.row_index == 1, + str(refusal), + ) +else: + check("equivalent source dates cannot split typed date evidence", False) + for fixture in sorted(pathlib.Path(__file__).with_name("fixtures").glob("*-bbox-capture.xml")): bodies = WORD_BODY.findall(fixture.read_text(encoding="utf-8")) assert bodies, f"{fixture.name}: no words matched — this check is checking nothing" From 8383e5c3fef8b407236650d00e5fdcb932f9c20c Mon Sep 17 00:00:00 2001 From: t Date: Sun, 13 Sep 2026 01:15:56 +0530 Subject: [PATCH 81/83] docs: retain measured directional matches within their qualified scope --- docs/tally/PROMPT_PLAYBOOK.md | 17 ++++++++++++----- docs/tally/TALLY_PROTOCOL_REFERENCE.md | 6 ++++-- .../compatibility/compatibility-matrix.json | 2 +- .../compatibility/compatibility-surface.json | 4 ++-- 4 files changed, 19 insertions(+), 10 deletions(-) diff --git a/docs/tally/PROMPT_PLAYBOOK.md b/docs/tally/PROMPT_PLAYBOOK.md index 73606241..222c1563 100644 --- a/docs/tally/PROMPT_PLAYBOOK.md +++ b/docs/tally/PROMPT_PLAYBOOK.md @@ -308,12 +308,17 @@ Implement: §9.4d's individually measured **directional** comparisons; do not turn them into a canonical fold. In particular, its slash row supplied a slash candidate against a space-bearing master. The reverse was not - measured, so slash-bearing cross-spellings remain exact-codepoint only - until both directions are qualified. Preserve every other codepoint, + measured: the supplied slash-to-master-space direction remains a + qualified observation in that exact scope; the reverse remains + exact-codepoint only. Preserve every other codepoint, including NFC/NFD, en dash and underscore distinctions. - Other licensed scopes remain exact-codepoint unless a capture qualifies their particular rule. Compatibility live-READ receipts do not establish write behaviour (`compatibility/README`). + The shared binder has no product/release/licence qualification input, so + these observations cannot authorize its automatic folded-name binding. + Folded spellings stay candidates: an operator chooses an exact catalogue + name and the existing assign path revalidates it. Name keys are exact at storage. Apply any permitted fold only while resolving a request against retained rows; if a fold reaches more than one row, surface ambiguity. Storing folded keys can erase a distinct @@ -697,9 +702,11 @@ Implement — write core (masters): project writes to, for **ledgers**. So: for **ledgers on licensed 7.1 Silver (`education_mode=false`)**, match only under §9.4d's individually measured directional rows. Its - slash row does not qualify the reverse, so a slash-bearing cross-spelling - is **exact-codepoint only** and fails loudly until both directions are - captured. Gold, other tiers and unqualified versions remain exact-codepoint + slash row qualifies a **supplied slash against a space-bearing master**; + the unmeasured reverse is **exact-codepoint only** and fails loudly. + This observed direction does not authorize automatic folded binding by + the shared binder, which cannot enforce the product/release/licence scope. + Gold, other tiers and unqualified versions remain exact-codepoint only. For **every other master type** — stock items, groups, voucher types — §9.4d measured nothing, so match on **exact codepoints** and let a case or separator difference fail loudly. diff --git a/docs/tally/TALLY_PROTOCOL_REFERENCE.md b/docs/tally/TALLY_PROTOCOL_REFERENCE.md index 45ed96e9..7ef88415 100644 --- a/docs/tally/TALLY_PROTOCOL_REFERENCE.md +++ b/docs/tally/TALLY_PROTOCOL_REFERENCE.md @@ -1363,8 +1363,10 @@ not send. **What this says.** On licensed 7.1 Silver, the recorded supplied-to-master comparisons accept the listed space/hyphen alternatives and the one **slash-candidate to space-master** alternative, collapse the measured internal whitespace run, ignore the measured leading and trailing whitespace, -and fold the measured **ASCII** case. The reverse slash comparison was not sent. Cross-spellings -involving slash therefore require exact codepoints unless that direction is separately measured. +and fold the measured **ASCII** case. The reverse slash comparison was not sent and requires +exact codepoints unless separately measured. The recorded slash-candidate to space-master result +remains qualified only in this product, release, licence tier, entity class and direction; a +shared binder without those scope inputs cannot use it as automatic binding authority. > **RULE: use only the recorded directional alternatives; do not fold separators into a canonical > form.** The slash result is candidate `/` against master space, not the reverse. An en dash and diff --git a/docs/tally/compatibility/compatibility-matrix.json b/docs/tally/compatibility/compatibility-matrix.json index 549ac066..0fdf1e97 100644 --- a/docs/tally/compatibility/compatibility-matrix.json +++ b/docs/tally/compatibility/compatibility-matrix.json @@ -1,7 +1,7 @@ { "schema_version": 1, "bridge_commit_sha": "be1c20cc3fd66fa1ece196505c69f26e555e4b8e", - "compatibility_surface_sha256": "76d4da9d7f816d4fb7d5d8f8b481f0756555ee2367c536cbef3d2340eab0ebc5", + "compatibility_surface_sha256": "189c823dfdec217068a2bf84d980971d1ae2814b0f30e928b7619c4fb53e8aff", "claims": [ { "claim_id": "erp9-6-6-3-windows-education-xml-one-company", diff --git a/docs/tally/compatibility/compatibility-surface.json b/docs/tally/compatibility/compatibility-surface.json index 743d1fa8..ec780551 100644 --- a/docs/tally/compatibility/compatibility-surface.json +++ b/docs/tally/compatibility/compatibility-surface.json @@ -35,7 +35,7 @@ }, { "path": "docs/tally/TALLY_PROTOCOL_REFERENCE.md", - "sha256": "ebaa88a22383c682fb13b75f8d5785a41b602df34192cafe764e1d5a5b7bfcdd" + "sha256": "db0e47e2f37df7e188d5179c5d4f7f81f49fe59b173c7f4d1cd48afb4ee05a0e" }, { "path": "docs/tally/compatibility/README.md", @@ -850,5 +850,5 @@ "sha256": "a8ac2714fecf51947f2822c8c46d7ce2e8602c732780ff60566a7771f0836f9a" } ], - "manifest_sha256": "76d4da9d7f816d4fb7d5d8f8b481f0756555ee2367c536cbef3d2340eab0ebc5" + "manifest_sha256": "189c823dfdec217068a2bf84d980971d1ae2814b0f30e928b7619c4fb53e8aff" } \ No newline at end of file From 6bfe2d26a7ca28028b58668046ec9967110353ea Mon Sep 17 00:00:00 2001 From: t Date: Sun, 13 Sep 2026 01:26:03 +0530 Subject: [PATCH 82/83] docs: align the implementation guide with scoped binding authority --- docs/tally/IMPLEMENTATION_GUIDE.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/tally/IMPLEMENTATION_GUIDE.md b/docs/tally/IMPLEMENTATION_GUIDE.md index c7f0e58a..74aa238f 100644 --- a/docs/tally/IMPLEMENTATION_GUIDE.md +++ b/docs/tally/IMPLEMENTATION_GUIDE.md @@ -630,9 +630,10 @@ licensed or standard TallyPrime. These rows establish only the supplied candidate against the recorded master; they do **not** establish a symmetric case/separator normalizer or canonical -fold. Automatic binding is exact-codepoint unless the authoritative protocol -records the particular directional comparison. In particular, do not infer a -slash rule from this baseline. +fold. The shared binder cannot enforce product, release and licence scope, so +these directional observations do not authorize automatic folded-name binding: +folded names remain candidates for operator selection and exact revalidation. +In particular, do not infer a slash rule from this baseline. > **Promoted to `TALLY_PROTOCOL_REFERENCE.md` §9.4b**, which is where observed gateway behaviour > belongs and which carries the consequences for a writer. This entry stays as the measurement From bb5b38b3069d7eefb50e89d7ede23e92ff9c6e70 Mon Sep 17 00:00:00 2001 From: t Date: Sun, 13 Sep 2026 02:10:39 +0530 Subject: [PATCH 83/83] docs: scope directional slash comparison --- docs/tally/IMPROVEMENT_PLAN_2026H2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tally/IMPROVEMENT_PLAN_2026H2.md b/docs/tally/IMPROVEMENT_PLAN_2026H2.md index 78b3bc89..af0b27ee 100644 --- a/docs/tally/IMPROVEMENT_PLAN_2026H2.md +++ b/docs/tally/IMPROVEMENT_PLAN_2026H2.md @@ -421,7 +421,7 @@ establish that the requested create succeeded. ### 8.6 `LASTMID` is 0 on successful master creates; `LASTVCHID` works -Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen only where a licensed capture has qualified a **directional write comparison**: §9.4d does so for **ledgers** on licensed 7.1 **Silver**, with `education_mode=false`, by importing vouchers naming folded spellings and reading the day book back, and for no other master type or licence tier. Its slash row is slash-candidate against space-master only; the reverse is not qualified, so slash-bearing cross-spellings remain exact-only. Gold and other unqualified tiers remain exact-only. A compatibility result cannot widen it — that evidence is a live-**read** receipt and `compatibility/README` says it never establishes any write behaviour. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. +Both ledger creates returned `LASTMID=0` despite `CREATED=1`. **Confirms §5.1.4's choice**: masters must be read back by name. Which name-matching rule applies is decided by the SCOPE GATE in `PROMPT_PLAYBOOK.md` Phase 4 step 4 and by nothing here: §9.4b's case-folding and separator rows inherit §0's **Edit Log 7.0 Educational** baseline and mark licensed TallyPrime UNVERIFIED, so on a licensed SKU match on **exact codepoints** and let a case or separator difference fail loudly. Widen only where a licensed capture has qualified a **directional write comparison**: §9.4d does so for **ledgers** on licensed 7.1 **Silver**, with `education_mode=false`, by importing vouchers naming folded spellings and reading the day book back, and for no other master type or licence tier. Its slash row is slash-candidate against space-master only. The reverse, space-candidate against slash-master, is not qualified and remains exact-only; the measured direction is the sole scoped exception. Gold and other unqualified tiers remain exact-only. A compatibility result cannot widen it — that evidence is a live-**read** receipt and `compatibility/README` says it never establishes any write behaviour. `LASTVCHID` is populated for vouchers and usable, still subject to the foreign-writer cross-check. DEVIATION 2026-09-12 (`TALLY_PROTOCOL_REFERENCE.md` §9.4b): "normalized name" here never means NFC/NFD normalization, which is WITHDRAWN. §9.4b is MEASURED: Tally matches master names on exact codepoints, so an NFD create read back with NFC folding applied would resolve onto a pre-existing, distinct NFC master and promote the wrong object. Compare on exact codepoints. This deviation removes normalization; it does not decide whether any case or separator fold is permitted on top — that is the SCOPE GATE's question, answered above, and on an unqualified licensed SKU the answer is exact codepoints and nothing else.