PaperTalk

Ask questions about your PDFs. Get answers with sources.

Demo

papertalk.mp4

What it is

PaperTalk lets you upload PDFs/TXT into personal Spaces and chat with them. Under the hood it uses retrieval-augmented generation (semantic search + an LLM) so answers stay grounded in your documents, with citations.

What you can do

• Upload PDFs or text to a Space
• Ask natural-language questions and get cited answers
• Compare info across multiple docs in one go
• Sign in with Google; your data stays in your account

How it works (quickly)

1. We split documents into chunks and index them with pgvector.
2. Your question retrieves the most relevant chunks.
3. Gemini generates a concise answer, citing the sources it used.

Quick start

Prereqs: Python 3.11+, Bun, PostgreSQL (with pgvector), Google OAuth creds, Gemini API key.

• Create env files: backend .env, frontend .env.local (DB URL, OAuth, JWT, Gemini key, etc.).

From the repo root:

bun setup
bun dev

bun setup installs the root/frontend Bun packages, creates backend/.venv, and installs the backend Python dependencies.

Open http://localhost:3000.

Retrieval evaluation

The retrieval stack is measured with a committed eval harness (backend/evals) on a fixed public corpus (3 arXiv papers: Attention, BERT, Adam — 107 chunks) and 45 questions whose ground truth is validated verbatim against the extracted corpus before every run. Three retrieval configs are compared: pure vector search (dense), vector + keyword boost (hybrid — what the app uses), and the multi-query expansion pipeline (multi-query).

config	precision@5	recall@5	p50 latency	p95 latency
dense	0.30	0.86	3,149 ms	3,508 ms
hybrid (in app)	0.30	0.87	3,177 ms	3,580 ms
multi-query	0.28	0.84	7,158 ms	11,495 ms

Findings from this benchmark:

• The keyword boost contributes marginally on this corpus (recall +0.01 over dense).
• The multi-query expansion path did not improve recall here (0.84 vs 0.87) while costing ~2.3× latency, because each user query fans out into 2–4 embedding calls. Latency includes query embedding against the Gemini API; the relative comparison between configs is the meaningful part.
• precision@5 is bounded low by single-gold questions (one relevant chunk out of 5 → max 0.20 for those), so treat it as a consistency signal rather than an absolute quality score.

Reproduce:

python -m backend.evals.run            # retrieval table
python -m backend.evals.run --judge    # + LLM-as-judge groundedness (slower)

The harness ingests the corpus into an isolated throwaway space (papertalk_eval_*), runs against it, and deletes it afterwards; your real spaces are never touched.

Tech

FastAPI + PostgreSQL/pgvector, Next.js + React + Tailwind, Gemini embeddings for retrieval, Gemini for generation.

Notes

• Currently supports PDF/TXT (up to ~5MB, ~25 pages). Text-only; scanned PDFs need OCR first.
• Best results in English.