A practical, fully-local evaluation study on a real agentic-RAG memory system (an LLM coding agent's own session-store vector database). We ran the four classic RAGAS metrics with a local LLM judge, then compared them against cheap, deterministic shortcuts built from embeddings, cross-encoders, NLI, and ROUGE-L — and measured how well each shortcut predicts what the judge would say.
Headline: for two of four metrics the shortcut is as good as the judge for ranking; for the other two the judge itself is the unreliable part.
| Metric | Shortcut | Spearman ρ | Verdict |
|---|---|---|---|
| AnswerCorrectness | 0.75·ROUGE-L + 0.25·nomic cosine | 0.894 | STRONG |
| AnswerRelevancy | nomic query/doc cosine | 0.709 | STRONG |
| AnswerRelevancy | TAS-B dot(q, a) | 0.576 | MODERATE |
| Faithfulness | sentence NLI (DeBERTa-v3) | −0.185 | WEAK (judge saturates) |
| ContextPrecision | graded NDCG (bge / MiniLM) | ≈ 0 | WEAK (no signal) |
LLM-as-a-judge evaluation is the industry default for RAG, but it is slow (hours per dataset with local models), non-deterministic, and expensive. Deterministic proxies that agree with the judge can replace it for regression gates and system tuning; disagreements pinpoint where the judge is measuring something different (or something meaningless). This study quantifies both effects on a real, messy, production-shaped corpus — agent conversations, not synthetic QA pairs.
opencode is an open-source agentic coding CLI whose entire session history is
stored in a SQLite database and mirrored into a local Chroma vector store
(opencode_sessions, cosine, nomic-embed-text embeddings on a local GPU via
Ollama). Retrieval answers are generated by gemma4:12b on the same GPU.
The dataset of 100 query/response/reference triples was sampled from that store
with a fixed seed.
This repo also documents the memory architecture that motivated the study:
docs/opencode_memory_architecture.md.
ragas-opencode-vector-eval/
├── README.md ← you are here
├── research_report.md ← full formal report (aggregates only)
├── blog_post.md ← long-form narrative
├── linkedin_post.md ← short-form announcement
├── lit_digest.md ← per-paper digest of foundational works
├── METHODOLOGY.md ← reproduction guide
├── arxiv_paper.tex ← LaTeX paper source
├── CHANGELOG.md / CONTRIBUTING.md / AUTHORS / LICENSE / CITATION.cff
├── requirements.txt
├── results/ ← SUMMARY.md, aggregates.csv, correlations.csv
├── figures/ ← comparison_metrics.png, correlations.png
├── src/ ← shortcut_metrics.py, compare_eval.py, build_aggregates.py, ...
├── tests/ ← unit tests for the pure functions
├── data/README.md ← provenance + no-examples policy (no content shipped)
├── docs/opencode_memory_architecture.md
└── research/ ← compiled PDFs (paper, report, blog, LinkedIn)
- AnswerCorrectness is replaceable. A 0.75·ROUGE-L-F1 + 0.25·semantic cosine blend reproduces the judge's ranking with ρ = 0.89 (ROUGE alone 0.87, semantic alone 0.87). Use the shortcut for ranking; keep the judge only when absolute calibration matters.
- AnswerRelevancy: nomic query/document cosine agrees STRONGLY (ρ = 0.71); TAS-B is MODERATE (ρ = 0.58). The judge shows a blind spot on refusals ("I don't know" scores 0.5–0.8), so gate with the embedding cosine.
- Faithfulness cannot be judged this way. The RAGAS judge sits at median 1.000 while the NLI shortcut reads median 0.000 → ρ = −0.18. One of them is lenient, one is strict; an oracle audit is required before trusting either.
- ContextPrecision has no signal on this corpus. All variants are near-constant (bge NDCG ≈ 0.99, RAGAS median 0.70). Every retrieved chunk is relevant — so the metric cannot rank systems here.
python -m pip install -r requirements.txt
# 1. point config at your own private eval workspace (see data/README.md)
export EVAL_BASE=/path/to/your/eval-workspace
# 2. rebuild the dataset from a vector store (needs Ollama + your corpus)
python -m src.build_eval_dataset
# 3. RAGAS with a local judge (hours) — see src/run_ragas.py
python -m src.reproduce_eval --stage ragas
# 4. cheap shortcuts (minutes)
python -m src.reproduce_eval --stage shortcuts
# 5. correlate + write aggregates/figures
python -m src.reproduce_eval --stage compare
python -m src.reproduce_eval --stage aggregatesTests (offline, no models/data needed):
python -m unittest discover tests/ -vNo conversation content, no per-sample rows, and no examples are shipped in
this repository — only aggregate statistics (results/), figures, and code.
The raw dataset and RAGAS outputs remain in the private eval workspace. See
data/README.md.
Dual: MIT for code (src/, tests/, data/download.sh), CC-BY-4.0
for prose and figures. See LICENSE.
See CITATION.cff for the canonical citation.