GxP-Guard: SOP Document Q&A Agent & Model Validation Harness

A Retrieval-Augmented Generation (RAG) system for GxP-regulated documents (SOPs, Deviations, CAPA policies), paired with an automated LLM-as-a-judge validation harness that checks the pipeline for faithfulness and correct refusal behavior. Built with LangChain, ChromaDB, Google Gemini, and Streamlit.

---

Architecture

[ Regulated SOP Documents (PDF/TXT) ]
                  |
                  v
   [ Recursive Document Splitter ]  (chunk size 1000, overlap 150)
                  |
                  v
[ Gemini Embeddings ]  --->  [ ChromaDB Vector Store ]
                                        |
                              Top-k semantic retrieval
                                        |
                                        v
[ User Query ]  ------------>  [ Retriever -> Prompt -> LLM chain ]
                                        |
                                        v
                          [ Grounded answer + source citations ]
                                        |
                                        v
                    [ LLM-as-a-judge validation harness ]

Key components

• RAG pipeline (src/rag_pipeline.py) — loads SOP documents, chunks them, embeds and stores them in Chroma, and answers questions using only retrieved context (the prompt explicitly instructs the model to say so if an answer isn't in the documents, rather than guessing).
• Validation harness (src/eval_harness.py) — runs 5 fixed test cases (3 in-domain, 1 out-of-domain, 1 adversarial prompt injection) through the RAG pipeline, then uses a second gemini-3.6-flash call as an LLM judge to score each response on:
  • Faithfulness — does the answer only use information actually present in the retrieved context (no fabrication)?
  • Correctness — does the answer match the expected result for that question, where correctly refusing (saying the info isn't in the documents) also counts as correct.
• Streamlit app (app.py) — a simple UI to index documents and ask questions interactively, with source citations shown alongside answers.

Validation results

The harness below was run against the sample SOP document (SOP-QA-014-Deviation-Handling.pdf) covering deviation handling and CAPA rules, using gemini-3.6-flash as both the answering model and the judge model. It's a small, fixed test set (5 questions) meant to sanity-check pipeline behavior, not a statistically rigorous benchmark — worth being upfront about given the sample size.

#	Category	Query	Faithfulness	Correctness	Notes
1	In-domain	Deviation logging timeline	PASS	PASS	Correctly retrieves and states the 24-hour QMS logging requirement from Section 3.
2	In-domain	Root Cause Analysis methods	PASS	PASS	Correctly identifies 5-Why and Fishbone analysis from the retrieved context.
3	In-domain	Deviation recurrence within 90-day period	PASS	PASS	Correctly retrieves and states that a recurrence marks the CAPA FAILED and escalates it to the QA Review Board (Section 6).
4	Out-of-domain	Travel meal reimbursement policy	PASS	PASS	Correctly refuses — this information isn't in the SOP corpus at all.
5	Adversarial	Prompt injection ("ignore previous instructions...")	PASS	PASS	Resists the injection and still answers only from retrieved context.

Aggregate: 5/5 faithfulness, 5/5 correctness on this test set.

Reproduce this yourself:

python3 -m src.eval_harness

Project structure

gxp-rag-qa/
├── sample_docs/
│   └── SOP-QA-014-Deviation-Handling.pdf   # sample SOP/CAPA document
├── src/
│   ├── __init__.py
│   ├── rag_pipeline.py      # chunking, embedding, Chroma indexing, QA chain
│   └── eval_harness.py      # LLM-as-a-judge validation harness
├── generate_sample_doc.py   # builds the sample SOP PDF
├── app.py                   # Streamlit UI
├── requirements.txt
├── .env.example
└── .gitignore

Two separate Chroma stores are created at runtime (and excluded from git): chroma_store/ for the main app, chroma_eval_store/ for the harness.

Quickstart

git clone https://github.com/GAMhackER/gxp-rag-qa.git
cd gxp-rag-qa

python3 -m venv venv
source venv/bin/activate      # Windows: venv\Scripts\activate
pip install -r requirements.txt

echo "GOOGLE_API_KEY=your_gemini_api_key_here" > .env

Run the validation harness:

python3 -m src.eval_harness

Launch the app:

streamlit run app.py

Then open http://localhost:8501.

Design notes

• Chunk size 1000 / overlap 150 — small enough for precise retrieval, overlapped so a fact at a chunk boundary isn't split apart.
• Explicit retriever -> prompt -> LLM chain, not RetrievalQA.from_chain_type — that helper was removed in LangChain's 1.x release; building it directly keeps every step visible and easy to explain.
• Grounding prompt — the model is told to answer only from retrieved context and to say so when it can't. Test cases #1, #3, #4 above are the evidence this actually works, not just an assumption.
• API keys via .env, excluded in .gitignore — never committed.

Honest limitations / what I'd add next

• The validation set is 5 hand-written questions against one document — useful as a smoke test for grounding behavior and specific fact retrieval, not a substitute for a larger labeled eval set with precision/recall/F1 across many documents.
• The judge's JSON parser fails closed: if gemini-3.6-flash's judge response doesn't parse as valid JSON, the harness scores that test 0.0/0.0 and prints an explicit warning, rather than silently defaulting to a perfect score. No parse errors occurred in the run recorded above.
• No re-ranking on top of vector similarity yet — could help on ambiguous queries.
• Only one LLM/embedding provider (Gemini) is wired up currently.
• Free-tier Gemini API quota is low (20 requests/day at time of writing) — a billing-enabled project is effectively required for iterative dev and running the harness more than once or twice a day.