fix(#43): reuse live document on same content hash

[nexus69420]

Summary

• Same tenant + same file bytes now reuse the existing live document (200 + duplicate: true) instead of starting a second OCR run. Path-identical uploads still dedup the way they do on main (fix(api): shared ingest _dedup_or_none with duplicate:true contract #104).
• Soft-deleted matches are not duplicates. The same hash in another tenant is allowed. If several live rows share a hash, the completed one is preferred.
• Ingest UI says Already ingested and opens that card. Marqo purge is unchanged.

Closes #43.

Test plan

• pytest tests/test_ingest_dedup.py (12 passed)
• Related API/DB/tenancy/route-contract tests (183 passed)
• npm run build in ui/
• Manual: upload the same PDF under a different filename when a completed row already exists → existing workflow_id, no new pipeline
• Manual: exact same filename still duplicate: true
• Manual: other tenant can still ingest the same bytes

[KDwevedi]

Two blockers:

1. The documented “soft-deleted match starts a fresh ingest” contract is not enforced for path-identical uploads. dedup_or_none() checks db.get_document(workflow_id) before the live-fingerprint query and never excludes is_disabled; if that Temporal workflow still answers, it returns the disabled row as duplicate: true. If Temporal no longer answers, the caller reuses/upserts the same disabled row rather than allocating a fresh run, so it can remain hidden. Apply the disabled-row rule to the path-identity branch and add an exact same filename/path regression for both queryable and closed Temporal cases.

2. The full synthetic merge suite now trips the shared /upload limiter in the two final new dedup tests with 429, while the other branches do not. tests/test_ingest_dedup.py passes alone, which confirms suite-order/global limiter leakage. Reset or disable the limiter in these route tests so the repository suite stays green.

Separately, the existing taxonomy-console assertion still fails on main and is unrelated.

[KDwevedi]

No new fix commit is present yet. After #136/#141/#144 merged, this branch also now conflicts with current main in pipeline/routers/documents.py and ui/src/lib/pipelineUi.js. Please rebase while addressing the existing same-path soft-delete and limiter-test findings.

[nexus69420]

Addressed the review and rebased/merged current main (#136/#141/#144).

1. Soft-deleted path identity: dedup_or_none() now allocates a *-rerun-* id when the same workflow_id row is disabled, whether Temporal still answers or not. Same-filename/path regressions cover both cases.
2. tests/test_ingest_dedup.py disables the shared /upload limiter for that module so suite-order 429s cannot trip the new route tests.

tests/test_ingest_dedup.py: 16 passed locally.

[KDwevedi]

The previous soft-delete and limiter findings are fixed, but one dedup blocker remains: fingerprint lookup and document creation are still a read-then-write sequence with no uniqueness guard.

Two simultaneous uploads of the same bytes under different filenames derive different workflow IDs. Both requests can complete find_live_document_by_fingerprint() before either reaches upsert_document(), then both create rows and start OCR. Since documents has no unique constraint on tenant + live fingerprint, this recreates the same-document_id alias condition that #43 is intended to prevent.

Please make reservation/creation atomic (for example, a database uniqueness invariant for a normalized tenant + active fingerprint with conflict handling that returns the winning row) and add a concurrent-upload regression. Soft-deleted rows must remain eligible for a fresh run.

[nexus69420]

Addressed the remaining atomic-reservation blocker.

Live tenant + source_file_fingerprint is now a unique index on non-disabled rows. A lost INSERT race raises LiveFingerprintConflict and both ingest doors return the winning row as duplicate: true (insert_or_duplicate). POST /upload claims that row before MinIO so the loser does not store a second object. Soft-deleted rows are outside the index, so a fresh *-rerun-* ingest still inserts.

Regressions: concurrent two-thread INSERT (exactly one live row), insert_or_duplicate loser returns the winner, and a disabled row still allows a new live insert. tests/test_ingest_dedup.py: 20 passed.

[KDwevedi]

The clean-database race is fixed, but the migration path still leaves the blocker active on legacy databases.

init_db() catches IntegrityError while creating ux_documents_live_tenant_fingerprint and only logs/skips the index when duplicate live tenant+fingerprint rows already exist. That is exactly the legacy state discussed in #43. Once the index is skipped, there is no uniqueness protection for any future fingerprint, so concurrent first-time uploads can still create duplicates.

I reproduced this by starting with two legacy live aliases: the unique index remained absent after init_db(), and two subsequent inserts sharing a new tenant+fingerprint both succeeded.

Please make the migration preserve the invariant even when legacy duplicates exist—for example, reconcile/fail explicitly before enabling writes, or use a separate unique fingerprint-reservation table that can select a winner without requiring the historical document rows themselves to be unique. Add a regression that initializes from a legacy duplicate database and proves concurrent inserts for a new fingerprint cannot both succeed.

[nexus69420]

Addressed the leftover uniqueness hole on legacy duplicate DBs.

init_db still skips ux_documents_live_tenant_fingerprint when live tenant+hash rows already duplicate, but uniqueness for new hashes no longer depends on that index. document_live_fingerprints is a separate PK on (tenant_instance, fingerprint):

• Backfilled from live rows (completed, then non-failed, then oldest created_at) with INSERT OR IGNORE, so historical extra aliases stay in documents.
• Claimed in the same transaction as the documents INSERT; conflict still surfaces as LiveFingerprintConflict / duplicate: true.
• Released on soft-delete so a fresh ingest can claim the hash again.

Regression: test_legacy_duplicate_db_blocks_concurrent_new_fingerprint seeds two live aliases, runs init_db (unique index absent), then concurrent inserts of a new fingerprint — only one succeeds. Existing soft-delete fresh-run coverage is unchanged.

Also merged upstream/main (includes #140) so this is not behind.