Tathya

Tathya (तथ्य, “fact”) is an autonomous, non-partisan record of India's Union Government. It does not manually select topics or issue AI verdicts: configured public sources are watched continuously and clustered into sourced case files showing what government, media, and citizens said. You decide.

Live

• Site: tathya-1.vercel.app
• API docs (Swagger UI): tathya-zi9p.onrender.com/docs

The API runs on Render's free tier and sleeps after 15 minutes of inactivity (cold start ~30–60s on the next request); an external uptime pinger keeps it warm during normal use. case_file_persist.py now calls Gemini-grounded generation by default (25 Jul 2026) — this needs db/migrations/008_topic_status_promotion.sql applied to Supabase and GEMINI_API_KEY set as a repo secret before topics actually show as Live with real narrative titles; see docs/audit_and_next_steps.md Section 10 for the full status.

Current foundation

The pipeline (ingestion → snapshotting → clustering → extractive/Gemini-grounded generation → persistence) and API v1 (api/main.py, typed against apps/web/lib/types.ts) are both built and deployed. The Next.js frontend (apps/web/) is fully wired to the real API — no mock data. What's still incomplete: pipeline/case_file_persist.py only calls the deterministic extractive builder, not the Gemini-grounded path, so live topics currently show raw cluster titles/templated summaries rather than final generated case files; the manual Phase 3 audit gate hasn't run yet either. See docs/audit_and_next_steps.md for the up-to-date gap list — it is more current than this file for "what's actually left."

Frontend (apps/web)

cd apps/web
npm install
npm run dev

Runs at http://localhost:3000. Set NEXT_PUBLIC_TATHYA_API_URL in apps/web/.env.local to point at a running API (defaults to http://localhost:8000 if unset — see apps/web/lib/api.ts). In production this points at the deployed Render API instead.

Deployment

• Frontend → Vercel: connect the repo, set Root Directory to apps/web, set NEXT_PUBLIC_TATHYA_API_URL to the deployed API URL. Currently live at tathya-1.vercel.app.
• API → Render (free tier): Root Directory blank (repo root), Build Command pip install ".[api]", Start Command uvicorn api.main:app --host 0.0.0.0 --port $PORT, Health Check Path /health, env vars SUPABASE_URL / SUPABASE_SERVICE_ROLE_KEY / TATHYA_CORS_ORIGINS. Currently live at tathya-zi9p.onrender.com/docs — sleeps after 15 minutes of inactivity on the free tier, kept warm with an external ping (e.g. cron-job.org) hitting /health every ~10 minutes rather than Render's own (paid) Cron Jobs product.
• Full reasoning, RAM caveats for /signals/search on the free tier, and the Oracle Cloud always-on alternative are documented in docs/audit_and_next_steps.md Section 8.

Run locally (pipeline/API)

Use Python 3.11+ (repo pins 3.12.7 via runtime.txt for hosted deploys).

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
python -m pytest
python -m pipeline.scheduler

Database setup

1. Create a Supabase project.
2. In its SQL Editor, run db/schema.sql, db/seed_sources.sql, then db/seed_entities_core.sql.
3. Copy .env.example to .env and fill in the project URL and service-role key. Keep .env private.

The schema makes snapshots append-only at the database level; source pages may change or disappear, but a captured source record cannot silently change.

After the database and source seed are in place, persist the enabled watcher output with:

python -m pipeline.scheduler --persist

This calls the database's atomic record_signal_snapshot function. Re-running it is safe: the same source URL does not create a second signal, and exact content duplicates are linked to their first canonical signal.

Each persisted run also records its source count. After three successful historical runs, a run below 20% of its recent median is printed as an alert; Telegram delivery is the remaining deployment integration.

PIB, Lok Sabha, Rajya Sabha, and YouTube now have source-specific adapters. They remain disabled in shared/config.py until their live endpoints/channels are validated; enabling a configured source automatically selects its correct adapter.

Scheduled ingestion

Three GitHub Actions workflows run on a schedule once SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY are added as repository secrets (offset within the same 2-hour window so each stage runs after the previous one has fresh data):

• ingest.yml — fetches enabled sources (:17).
• embed.yml — embeds newly persisted signals (:35), installs the embeddings extra.
• case-file.yml — persists promotable case-file drafts (:50). Calls Gemini-grounded generation by default, falling back to the deterministic extractive builder per topic on any failure (see pipeline/generation/grounded_case_file_draft.py). Needs GEMINI_API_KEY as a repo secret and db/migrations/008_topic_status_promotion.sql applied to Supabase to actually produce Live topics with narrative titles — without either, it still runs safely, just via the extractive fallback.
• lifecycle.yml — archives Live topics with no new signal in ARCHIVE_AFTER_DAYS (shared/config.py, currently 60). Runs once daily, not on the 2-hour cycle. Reopening a dormant topic needs no separate workflow — it happens automatically inside case_file_persist.py's persist step whenever a promotable cluster matches an existing (possibly archived) topic's title.

All four can also be run manually from the Actions tab (workflow_dispatch).

Optional source-failure and low-volume alerts need TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID added as GitHub repository secrets (and, for local runs, to .env).

Verify that Telegram independently of a source failure with:

python -m pipeline.scheduler --test-alert

Source activation

List all configured sources:

python -m pipeline.source_audit --list

Inspect a disabled candidate source without writing to Supabase:

python -m pipeline.source_audit --source hindustan-times-india --limit 10

Persist one disabled candidate once for manual Supabase inspection without enabling GitHub Actions polling:

python -m pipeline.source_activation --source hindustan-times-india --limit 100

If your Supabase project predates the activation command, first run db/migrations/001_source_activation.sql in the SQL Editor.

If you want to test pmindia-news-updates, first run db/migrations/002_official_website_sources.sql, then rerun db/seed_sources.sql. Existing projects need the migration because official_website is a new source_type enum value.

The scheduler currently polls only enabled sources. Keep candidates disabled until their adapter, timestamp quality, canonical URLs, raw text, snapshots, duplicate behavior, and terms posture are checked. Add sources to shared/config.py, never individual stories.

See docs/source_research.md for the current multi-source activation register and the reason every non-enabled source is held back.

Phase gates

See docs/mission_ethics.md and docs/phase0_phase1_checklist.md. Do not move to Phase 2 AI/clustering until Phase 1 has several reliable official and independent enabled sources with clean snapshots and source-run metrics.

Private Phase 2 report

After enabled sources have been persisted, build the first private candidate-topic report:

python -m pipeline.topic_report --signals 300 --topics 10

This is deterministic entity matching and grouping only. It does not call Gemini and does not publish case files.

Review likely wire-copy or near-duplicate records:

python -m pipeline.duplicate_scan --signals 300 --threshold 0.82

After running db/migrations/003_mark_signal_duplicate.sql, clearly reviewed candidates can be marked with:

python -m pipeline.duplicate_scan --signals 300 --threshold 0.82 --apply

Build private Phase 3 extractive case-file drafts:

python -m pipeline.case_file_report --signals 300 --topics 5
python -m pipeline.case_file_report --signals 300 --topics 5 --json

These drafts are audit material only: no Gemini, no public publishing, and every claim/event/fact is copied from a source row with a URL.

After running db/migrations/004_case_file_persistence.sql, persist private extractive drafts:

python -m pipeline.case_file_persist --signals 300 --topics 5

Use --promotable-only when you only want clusters that pass the official-plus-nonofficial significance gate.

Semantic retrieval

Tathya uses Supabase Postgres with pgvector; embeddings are stored in signals.embedding. The first model target is intfloat/multilingual-e5-base, a 768-dimensional multilingual sentence-transformers model.

Install the optional local embedding dependencies:

python -m pip install -e ".[embeddings]"

After running db/migrations/005_signal_embeddings.sql, embed recent signals:

python -m pipeline.embed_signals --limit 300 --batch-size 16

Search embedded signals:

python -m pipeline.semantic_search "Sonam Wangchuk Parliament march" --limit 10

Grounded generation and audit

Install optional Gemini generation dependencies:

python -m pip install -e ".[generation]"

Generate private grounded JSON drafts:

python -m pipeline.gemini_case_file_report --signals 300 --topics 3

Export rows for the mandatory Phase 3 manual audit:

python -m pipeline.audit_export --target claims --limit 50 > audit_claims.csv
python -m pipeline.audit_export --target events --limit 50 > audit_events.csv
python -m pipeline.audit_export --target facts --limit 50 > audit_facts.csv