fix(core): harden scan durability and idempotency (#303)

[SHAURYAKSHARMA24]

Summary

This draft PR implements the complete #303 hardening contract: transaction recovery, fenced scan leases, idempotent result persistence, durable scan admission, durable CVE enrichment, and bounded operational signals.

Problems fixed

• Aborted PostgreSQL transactions no longer poison worker progress; broken connections are discarded and reacquired.
• Expired or stale scan workers cannot write authoritative completion, failure, evaluations, or findings.
• Repeated result delivery no longer creates duplicate evaluations or findings.
• Concurrent/replayed API triggers cannot create uncontrolled duplicate active scans.
• CVE enrichment is no longer owned by a Gunicorn daemon thread and no longer stops after a single NVD page.
• Operators now have worker, queue, lease, retry, and last-success visibility.

Architecture

• Scan leases and fencing: PostgreSQL claims have owner, expiry, monotonic fencing token, renewal, stale recovery, and fenced final writes.
• Idempotent persistence: findings are uniquely identified by scan, rule, canonical resource scope, and an optional rule-specific discriminator. Mutable fields are updated with ON CONFLICT; rule evaluations are uniquely keyed by scan, rule, and resource and are upserted.
• Admission: a transaction-scoped PostgreSQL advisory lock plus partial unique indexes enforce one pending/running scan per subscription and a unique subscription/idempotency-key pair. Same semantics replay the logical scan; changed semantics conflict. OPENSHIELD_MAX_SCANS_PER_SUBSCRIPTION_PER_HOUR provides an explicit optional time-window policy; one active scan remains the enforced concurrency quota.
• Durable enrichment: a completed fenced scan atomically creates one PostgreSQL enrichment job. The existing durable worker claims, renews, checkpoints, retries with bounded exponential backoff, recovers expired leases, and fences stale writers. NVD retrieval follows totalResults through every page.
• Metrics: /metrics reads bounded PostgreSQL aggregates; labels are only queue (scan/enrichment) and worker_type.

Database migrations

1. e4f7a9b2c6d8 — renewable scan leases and fencing tokens.
2. f2b6d8e1a4c9 — finding identities and rule evaluations. Existing findings receive distinct legacy:<id> keys; no legacy rows are silently collapsed.
3. a7c5e9d2f1b4 — durable scan admission/idempotency indexes.
4. c9e1a5b7d3f2 — durable fenced enrichment jobs.
5. d4a8c1e6b2f9 — worker heartbeat storage for operational metrics.

There is one Alembic head. Clean base-to-head, #325's original e4f7a9b2c6d8-to-head, and downgrade/upgrade paths were validated on PostgreSQL.

Concurrency guarantees

All authoritative scan-result writes re-check lease owner, fencing token, running state, and unexpired lease under FOR UPDATE in the same transaction as persistence. Once worker A loses its lease and worker B reclaims with a newer token, A cannot update scan state, findings, evaluations, or enrichment progress. PostgreSQL unique constraints and upserts make duplicate API/result/job delivery converge on one logical record.

Deployment

1. Stop or drain old scan workers; mixed old/new workers are unsafe because old workers cannot satisfy the fencing contract.
2. Apply Alembic migrations through d4a8c1e6b2f9.
3. Deploy this API and the existing scanner/worker.py process. The worker now processes both scan and enrichment jobs.
4. Monitor /metrics for worker liveness, queue age, lease age, retries, and last successful scan.

Tests

• core: harden scan transactions, leases, idempotency, and durable background work #303-focused tests against repository Compose postgres:16-alpine: 150 passed, 0 skipped.
• Full backend suite: 1 failed, 885 passed, 2 skipped. The sole failure is existing local test_vector_store_purity, because this checkout has an ai/vectorstore directory without its BM25 index. This PR does not alter AI/RAG code.
• ruff check .: passed.
• ruff format --check .: passed.
• Alembic single-head, base-to-head, original-fix(core): harden scan durability and idempotency (#303) #325-migration-to-head, and downgrade/upgrade checks: passed.

Acceptance criteria

• All transactions rollback on failure and discard/reacquire broken connections.
• Claims use renewable leases with owner, expiry and fencing token.
• Heartbeat and completion updates require the current fencing token.
• Evaluation/finding persistence is idempotent using stable unique keys/upserts.
• Scan admission has per-subscription quotas, one-active-scan deduplication and idempotency keys.
• Enrichment is a durable claimed job with retries, stale recovery and complete pagination.
• PostgreSQL-backed fault-injection tests cover abort, restart, duplicate delivery, lease expiry and two-worker races.
• Metrics include worker heartbeat, oldest queue age, lease age, retry count and last successful complete scan.

Related

Closes #303

[m-khan-97]

@SHAURYAKSHARMA24, this is the canonical track for #303’s durability layer: transaction recovery, leases/fencing, idempotent admission and writes, durable enrichment, and worker telemetry. One integration boundary must be resolved before lead review: migration f2b6d8e1a4c9 creates rule evaluations and associated persistence semantics that overlap #321, which is the already-agreed #263 evaluation-contract implementation. Please coordinate with Dipesh and either stack/rebase #325 on the accepted #321 contract or remove the duplicate evaluation-schema ownership from this PR. We must not merge two competing rule_evaluations definitions or aggregation contracts. Keep the fencing/idempotency guarantees around whichever canonical evaluation model is selected.