chore: local-dev + backfill tooling (read-only against prod by default)

[swaroopvarma1]

What

Dev/ops tooling only — zero runtime code touched. These scripts backed the 2026-07-11→16 "one template per merchant" backfill and the local dev environment:

• db_env_switch.py — flips .env between local/prod DB profiles; forces the dispatcher/background-task kill-switches OFF whenever the prod profile is selected. The prod profile ships with a blank password on purpose — typed per session, never stored.
• seed_local_from_prod.py — one-pass, read-only seed of a local clairvoyance_local DB.
• fleet_census.sql / orphan_merchants.sql / db_readonly_report.sh — read-only census + orphan detectors.
• backfill_merchant_templates.py / link_config_template_ids.py / revert_backfill_run.py — the journaling backfill executor (dry-run by default), the config→template id linker, and snapshot-based revert.
• docs/FLEET_CENSUS_2026-07-11.md — the aggregate census that sized the backfill (merchant/template counts only, no customer data).
• .gitignore — ignores scripts/backfill_runs/ (run journals can contain lead payload snapshots and must never be committed), plus the db-switch env profiles and the pre-commit error dump.

Safety

• Audited for secrets/PII before push: connection params are env-vars/prompts only; the census doc is aggregate counts.
• Lowest priority of the slice PRs — merge whenever convenient; nothing else depends on it except the .gitignore hygiene.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

@swaroopvarma1, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 49 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d9763b03-eac5-4a76-bdaf-a869ecba269d

📥 Commits

Reviewing files that changed from the base of the PR and between d6c6e47 and affe8ce.

📒 Files selected for processing (10)

• .gitignore
• docs/FLEET_CENSUS_2026-07-11.md
• scripts/backfill_merchant_templates.py
• scripts/db_env_switch.py
• scripts/db_readonly_report.sh
• scripts/fleet_census.sql
• scripts/link_config_template_ids.py
• scripts/orphan_merchants.sql
• scripts/revert_backfill_run.py
• scripts/seed_local_from_prod.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch chore/local-dev-tooling

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[murdore]

This docstring's safety claim — "the DB FK (lead_call_tracker.template_id → template.id) enforces this even if the check races" — doesn't hold: migration 008 declares that FK ON DELETE SET NULL, not RESTRICT (only call_execution_config/chat_session/widget_config are RESTRICT). So the one table named as the backstop is exactly the one that won't block the delete — it silently nulls the referencing lead's template_id instead. Combined with the two-pass structure (count leads in pass 1, DELETE in pass 2 with no re-check in the same transaction), a lead that lands between the check and the delete gets its analytics linkage silently severed. Recommend a WHERE NOT EXISTS (SELECT 1 FROM lead_call_tracker WHERE template_id = template.id) guard on the delete itself (or a SELECT … FOR UPDATE re-check in the delete txn) so a race fails loudly instead of orphaning a live lead.

[murdore]

new_tpl = await conn.fetchval(TEMPLATE_COPY_SQL, …) is an INSERT … SELECT … WHERE id=$1 AND merchant_id IS NULL RETURNING id — if that WHERE ever matches 0 rows (shared row deleted/reassigned mid-run), fetchval returns None without raising, but created_t += 1 fires unconditionally and the journal records "template_id": "None" tagged created. That later crashes revert_backfill_run.py when it casts "None" to UUID in its unguarded scan loop, and (if a shared config exists) CONFIG_COPY_SQL inserts an orphaned call_execution_config with template_id=NULL. Suggest if new_tpl is None: raise RuntimeError(...) right after the fetchval, matching how link_config_template_ids.py already guards its own write.

[murdore]

scripts/seed_local_from_prod.py — prod credentials + customer PII land on developer laptops unredacted

This script copies real production data into the local dev DB with no anonymization step anywhere in the file:

FULL_TABLES = [
    "merchants",
    "template",
    "call_execution_config",
    "outbound_number",
    "widget_config",
    "credentials",
    "knowledge_base",
    "kb_document",
]

credentials is copied in full — copy_rows just does SELECT {cols} FROM "{table}" with every column that exists on both sides, no column exclusion or redaction. If that table holds per-merchant calling-provider secrets (which this very PR's db_env_switch.py implies, noting ".env carries real Plivo/Exotel/Twilio credentials"), then every merchant's live telephony credentials end up in cleartext in whatever local Postgres instance runs this script.

The "bounded slices" pull real customer traffic verbatim too:

total += await copy_rows(
    prod, local, "lead_call_tracker",
    "SELECT {cols} FROM lead_call_tracker "
    f"WHERE created_at >= now() - interval '{LEAD_DAYS} days' "
    "ORDER BY created_at DESC LIMIT $1",
    [LEAD_CAP],
)
...
total += await copy_rows(
    prod, local, "chat_message",
    "SELECT {cols} FROM chat_message WHERE session_id = ANY($1)",
    [session_ids],
)

Up to 25k leads and 5k chat sessions (with their full chat_message/chat_turn_metrics content) get copied byte-for-byte — customer phone numbers and chat transcripts, no scrubbing/anonymization pass anywhere in the script.

The PR description's safety framing ("Audited for secrets/PII before push... connection params are env-vars/prompts only") covers the scripts' own source, but this is the one script whose actual job is to move real secrets and real customer PII off prod onto a laptop, and that risk isn't mentioned or mitigated. Worth at minimum excluding credentials from FULL_TABLES (or redacting its secret columns) and anonymizing the customer-identifying columns (phone numbers, chat text) in the bounded slices before they land locally.

[murdore]

scripts/link_config_template_ids.py — the "in-flight leads" safety check is a no-op for reseller-level (shared) configs, due to a plain = on a nullable column

CANDIDATES_SQL's deferral guard:

  AND NOT EXISTS (
        SELECT 1 FROM lead_call_tracker l
        WHERE l.merchant_id = c.merchant_id
          AND l.template = c.template
          AND l.status IN ('BACKLOG', 'RETRY', 'PROCESSING')
          AND l.template_id IS DISTINCT FROM t.id)

uses plain l.merchant_id = c.merchant_id, while every other scope comparison in this same query (and in SKIPPED_SQL) correctly uses IS NOT DISTINCT FROM to handle merchant_id IS NULL:

 AND t.merchant_id IS NOT DISTINCT FROM c.merchant_id

For reseller-level configs (c.merchant_id IS NULL — the unlinked_shared bucket the script itself reports in POPULATION_SQL/the printed summary), l.merchant_id = NULL is UNKNOWN for every lead row, so the NOT EXISTS is vacuously true regardless of any actual in-flight leads. The result: shared-scope configs always land in CANDIDATES_SQL (never in SKIPPED_SQL's "in-flight leads pinned to a different template" bucket) and get linked immediately under --apply, even when leads with status IN ('BACKLOG','RETRY','PROCESSING') are pinned to a different template_id for that same reseller/name — exactly the race this check exists to defer. Given the inconsistency is only on this one comparison while the identical NULL-scope problem is handled correctly two lines above, this looks like an unintentional gap rather than deliberate scoping. Suggest l.merchant_id IS NOT DISTINCT FROM c.merchant_id to match the rest of the query.