Capture the 14 CTA ground-truth values from in-database twins (#2152)

[jakebromberg]

Fills pending_cta_repair with the 14 real rows, turning scripts/audit/bs_replacement_char_cta.sql from the verified no-op it shipped as into a live repair. Already applied to prod — see the run record on #2152.

Where the values came from

Not from any of the three channels the script's capture procedure lists. All three read tubafrenzy, and tubafrenzy had already retired when the capture was attempted; channel (b)'s escape hatch never materialised either, since no library.db snapshot ever gained per-track data. The procedure is preserved as the record of what was intended, annotated as unrunnable.

The values were still in the table. The script's own "Sequencing vs #1996" note records that post-#454 the ETL derives the correct string, and importCompilationTracks inserts via a bare, untargeted onConflictDoNothing() — so a correct row does not conflict with a corrupt one, because their (library_id, artist_name, track_title) tuples differ, and it lands beside it rather than being suppressed. That is the same 98.5% double-ingest shape #1996 measured, and the same fact this script's twin detection already depends on. What had not been noticed is that it makes the twin a source of ground truth, not merely a unique-index hazard to route around.

Release 8844 carries three generations of one credit:

110864   Wanda Sá    Só Danço Samba = Jazz 'N' Samba     <- clean original
1228848  Wanda Sá   Só Danço Samba = Jazz 'N' Samba    <- #1996 double-encoding
3185320  Wanda Sá   Só Dan��o Samba = Jazz 'N' Samba   <- this ticket's row

Method

For each corrupt row the undamaged column is the join key against clean rows in the same release; the damaged column is then matched with its U+FFFD runs as bounded wildcards, against each candidate in both stored and re-encoded form. 13 of the 14 resolved to a value present byte-exact as a live row in the same release — located, not reconstructed.

Two decoding subtleties mattered:

• CP1252's undefined bytes. The corrupting client passed through the five bytes CP1252 leaves undefined (0x81, 0x8D, 0x8F, 0x90, 0x9D). Python's codec refuses them, which left six values looking irreversibly double-encoded until the codec was corrected.
• Truncation. Some twins are cut mid-character at varchar(255), so the reversal must decode tolerantly at the tail only — never mid-string.

One row needs the closest look at review: legacy_release_id 12988. Its only pattern-matching twin is itself double-encoded and truncated, so reversing it drops the final 章. The captured value instead comes from id 36019, which holds the untruncated string. This is the single row whose value comes from a sibling rather than a direct correspondent.

Test changes

The spec's placeholder gate named its own succession plan — it asserted the block carried no string literals and said it "goes red the moment real data lands, forcing whoever adds it to add the codepoint assertion in the SAME change." That assertion is here rather than the test being deleted: byte pins for all 14 true values. This matters more than it sounds — release 59194 contains both · (U+00B7 MIDDLE DOT) and ・ (U+30FB KATAKANA MIDDLE DOT) in different titles, and an editor normalising one into the other would be invisible in review and would break the downstream byte-exact parity comparison.

The no-op test becomes its inverse: the block declares 14 rows, splits 11 track_title / 3 artist_name as the header promises, survives the placeholder scrub, and clears every operator-error guard.

The DB-backed tests were not run locally — the integration harness wants a test-user role absent from every running container, and the only bootstrap is npm run db:stop/db:start, where db:stop is down -v against volumes this change did not create. Left to CI. The static assertions were verified directly against the file: 14/14 byte pins, exactly 14 VALUES rows, no dollar-quoting.

Prod run

Dry-run first, via the script's documented sed 's/^COMMIT;$/ROLLBACK;/', which executes the real guards and the real DELETE/UPDATE inside a real transaction so a would-be cta_unique_idx violation surfaces exactly as it would for real. It reported pending_declared 14 | pending_matched 14 | pending_unmatched 0 — every captured current_* matched a live row byte-exactly — then UPDATE 10 / DELETE 10 / UPDATE 4 and residual 0, and discarded everything.

The real run produced identical counts and committed. Independent verification after the fact: compilation_track_artist U+FFFD residual is 0 / 0; ten of the fourteen original ids are gone, four repaired in place.

10 rows took the DELETE-twin branch and 4 took UPDATE-no-twin, because the twin join requires both columns to match post-fix — repairing only the damaged column leaves the other in whatever form it had, so a fully-clean row is not a twin. Those 4 now carry a clean value in the repaired column and a still-double-encoded value in the other. That is #1996's scope, not this ticket's, and it is why the header says to land this first so #1996's twin analysis re-derives after these writes.

Closes #2152

[jakebromberg]

Note for review: this branch carries a third commit, 70ce0b10, that reformats plans/mu-ziq-replacement-char-remediation.md — 90 lines, entirely unrelated to the capture.

It is pre-existing breakage on main, not something this change introduced. That file reached main unformatted because lint-and-typecheck skipped on its own PR (#2385): the change was docs-only, and the workflow's paths-filter routes src — not markdown — to the lint job. So npm run format:check never saw it, and it now fails on every branch that touches a prettier-checked path.

It is included here only because it blocks CI. Worth its own follow-up: the plan in that very file lists the paths-filter gap as a finding to fix, and this is the gap demonstrating itself on the plan document that documents it.

CI is green with it in: lint-and-typecheck, unit-tests, and Integration-Tests all pass. The three new spec assertions ran against a real Postgres — byte pins for all 14 values, the 14-row / 11-3 split, and the operator-error guards.