RAG Quality Workbench

A production-shaped Retrieval-Augmented Generation assistant — Next.js chat UI, streaming FastAPI backend, three swappable LLM providers, hybrid retrieval with reranking, source citations with abstention, and a real evaluation harness that shows why one chunking or retrieval configuration outperforms another. Independent portfolio reference implementation by Raghu Sharma; not represented as client work.

The differentiator

Most public RAG demos are "upload a PDF, chat with it." This one adds the piece that's actually hard and actually matters in production: an evaluation harness that re-ingests the same source documents under different chunking/retrieval configurations and runs a labeled query set through the real pipeline — not a parallel reimplementation of retrieval, the literal same rag.pipeline.run_pipeline() the chat API uses — then reports recall@k, mean reciprocal rank, keyword coverage, and abstention rate per configuration. See it in the Evaluation tab, or read backend/src/rag_workbench/evaluation/runner.py.

Architecture

flowchart LR
    subgraph Frontend["Next.js"]
        Chat["Chat"]
        Docs["Documents"]
        Eval["Evaluation"]
    end

    subgraph Backend["FastAPI"]
        Ingest["Ingestion\n(extract → chunk → embed)"]
        Pipeline["RAG pipeline\n(hybrid search → rerank → cite/abstain → generate)"]
        Runner["Evaluation runner"]
    end

    subgraph Providers["Provider adapters"]
        OpenAI["OpenAI"]
        Anthropic["Anthropic"]
        Gemini["Gemini"]
    end

    PG[("Postgres + pgvector")]

    Docs -->|upload| Ingest --> PG
    Chat -->|SSE stream| Pipeline
    Pipeline --> PG
    Pipeline --> Providers
    Eval -->|run| Runner --> Pipeline
    Ingest --> Providers

Loading

What it demonstrates

• Multi-provider LLM adapters (OpenAI, Anthropic Claude, Google Gemini) behind one ProviderAdapter interface — swap chat provider per request; embedding provider is fixed per index since mixing embedding spaces produces incomparable vectors (documented in config.py). Anthropic has no native embeddings API, handled explicitly rather than faked (see anthropic_provider.py).
• PDF/DOCX/text ingestion with page-aware chunking, so every chunk still knows which page it came from for citations.
• Postgres + pgvector storage, with an HNSW index for vector search and a GIN full-text index for keyword search.
• Hybrid retrieval: vector + keyword search fused via normalized weighted scoring (hybrid_vector_weight), so a strong keyword-only match isn't lost to a purely vector-ranked list. Storage-agnostic — unit-tested with an in-memory store, no database required.
• Reranking: a dependency-free lexical reranker by default (query-term coverage + density), behind a Reranker protocol so swapping in a hosted cross-encoder is a one-line change, not a rewrite.
• Source citations and abstention: every answer cites [1], [2], … back to a document/page; when the best reranked score falls below a threshold, the pipeline says so instead of guessing.
• A real evaluation harness (see above) — the actual differentiator.
• Tracing: per-stage latency, token usage, and estimated cost surfaced in the chat UI, not just server logs.
• Docker Compose for local dev (db + backend + frontend), with unit/integration tests that need none of it — pytest runs the whole backend suite against in-memory fakes.

Quickstart

cp .env.example .env
# fill in at least one provider key
docker compose up --build

• Frontend: http://localhost:3000
• Backend: http://localhost:8000 (/health, /docs for the OpenAPI schema)

Running locally without Docker

# Backend — needs a local Postgres with the pgvector extension available
cd backend
python -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'
cp ../.env.example .env   # then edit DATABASE_URL to point at your local Postgres
uvicorn rag_workbench.app:app --reload

# Frontend
cd frontend
npm install
npm run dev

Tests

cd backend && pip install -e '.[dev]' && pytest -q   # 77 tests, no live DB or API keys needed
cd frontend && npm install && npm run lint && npm run build

The backend test suite runs entirely against InMemoryVectorStore and a fake provider (tests/fakes.py) — including full pipeline and evaluation-runner integration tests — so CI doesn't need Postgres or real API credentials. Provider adapters are tested against a mocked HTTP transport (httpx.MockTransport), verifying real request/response shapes without live network calls.

Running an evaluation

# via the UI: Evaluation tab → upload eval/sample_dataset.jsonl + eval/sample_docs/*.txt
# or via the API directly:
curl -X POST http://localhost:8000/evaluation/run \
  -F "dataset=@eval/sample_dataset.jsonl" \
  -F "documents=@eval/sample_docs/refund-policy.txt" \
  -F "documents=@eval/sample_docs/warranty-terms.txt" \
  -F "documents=@eval/sample_docs/subscription-terms.txt" \
  -F 'configs=[{"label":"small","chunk_size_tokens":150,"chunk_overlap_tokens":20,"hybrid_vector_weight":0.6},{"label":"large","chunk_size_tokens":500,"chunk_overlap_tokens":80,"hybrid_vector_weight":0.6}]'

Known limitations, stated plainly

• The default reranker is a lexical heuristic, not a cross-encoder — good enough to demonstrably beat unreranked hybrid search (see the evaluation harness), but a real deployment handling ambiguous queries would likely swap in a hosted rerank API.
• Cost estimates in the tracing panel use a hardcoded, approximate pricing table per provider — directionally useful, not a billing source of truth.
• The in-memory rate/rerank paths are single-process; nothing here assumes horizontal scaling.
• PII masking, if you're looking for it: this project doesn't do PII redaction (that's the companion secure-mcp-gateway project's job) — this one assumes indexed documents are already appropriate to surface to the chat UI.

License

MIT