AI Security RAG with LangSmith Observability

A production-style Retrieval-Augmented Generation (RAG) system built around AI security documentation using FastAPI, Qdrant, Redis Queue (RQ), and LangSmith for tracing and evaluation.

The assistant answers questions grounded strictly in trusted AI security documents, supports asynchronous request processing, and includes an automated evaluation pipeline with multiple RAG-specific metrics.

---

Features

• Retrieval-Augmented Generation (RAG)
• AI Security knowledge base
• PDF ingestion and indexing
• Semantic search using Qdrant
• FastAPI REST API
• Asynchronous processing with Redis Queue (RQ)
• LangSmith tracing
• LangSmith benchmark evaluation
• Grounded responses with page citations
• Modular project architecture

---

Knowledge Base

The assistant currently indexes the following documents located in:

app/data/

• NIST AI Risk Management Framework (AI RMF)
• SAFE-AI Full Report

During ingestion, these PDFs are:

1. Loaded
2. Split into semantic chunks
3. Embedded
4. Indexed into Qdrant

Every answer generated by the assistant is based solely on the retrieved document context.

---

Project Modules

Path	Purpose
app/data/	Stores the PDF knowledge base used by the RAG pipeline.
app/rag/	Core RAG application containing the API server, worker, clients, and pipeline logic.
app/rag/ingestion/	Loads PDFs, performs chunking, generates embeddings, and indexes them into Qdrant. Run this whenever new documents are added.
app/rag/queues/	Contains the RQ worker responsible for retrieval, prompt construction, LLM invocation, and answer generation.
app/rag/clients/	Client configurations such as the Redis Queue connection.
app/rag/server.py	Defines the FastAPI application and REST endpoints.
app/rag/main.py	Entry point for starting the FastAPI server.
app/evaluation/	Contains the benchmark dataset, evaluation script, and custom LangSmith evaluators.
app/reports/	Stores evaluation reports and other generated artifacts.
docker-compose.yml	Starts the required infrastructure services (Qdrant and Redis).
qdrant_storage/	Persistent storage for the local Qdrant database.

---

Tech Stack

• Python
• FastAPI
• LangChain
• LangSmith
• Qdrant
• Redis
• Redis Queue (RQ)
• Docker
• Pandas
• OpenAI Compatible APIs

---

Getting Started

1. Clone the Repository

git clone <repository-url>
cd rag_observalibility

---

2. Create a Virtual Environment

python3 -m venv venv

Activate it.

Linux/macOS

source venv/bin/activate

Windows

venv\Scripts\activate

---

3. Install Dependencies

cd app
pip install -r requirements.txt

---

4. Configure Environment Variables

Create a .env file inside the app/ directory.

Example:

LANGSMITH_API_KEY=
GROQ_API_KEY=

---

Running the Project

Step 1 — Start Required Services

From the project root:

docker compose up -d

This starts:

• Qdrant (Vector Database)
• Redis (Message Queue)

---

Step 2 — Index the Documents

Open a terminal.

Navigate to:

rag_observalibility/app

Run:

python -m rag.ingestion.index

This step:

• Loads every PDF inside app/data
• Chunks the documents
• Generates embeddings
• Stores them inside Qdrant

This only needs to be executed again when the document collection changes.

---

Step 3 — Start the Worker

Open a new terminal.

Navigate to:

rag_observalibility/app

Run:

rq worker

The worker:

• Retrieves relevant document chunks
• Constructs prompts
• Calls the LLM
• Generates the final grounded response

You may start multiple workers depending on workload.

---

Step 4 — Start the FastAPI Server

Open another terminal.

Navigate to:

rag_observalibility/app

Run:

python -m rag.main

The API will start on:

http://localhost:8000

Interactive API Documentation:

http://localhost:8000/docs

---

Step 5 — Query the Assistant

Using the Swagger UI available at:

http://localhost:8000/docs

Submit a request to:

POST /chat

The endpoint returns a Job ID.

Use:

GET /job-status

with the Job ID to retrieve the generated answer once processing completes.

---

Running the Evaluation Pipeline

The evaluation module benchmarks the RAG system using a custom dataset and multiple LLM-as-a-Judge metrics.

Supported Metrics

• Correctness
• Answer Relevance
• Retrieval Relevance
• Faithfulness (Groundedness)

---

Before Running

Ensure your .env file contains valid API keys, especially:

LANGSMITH_API_KEY=
GROQ_API_KEY=

---

Run the Evaluation

From the project root:

python -m app.evaluation.evaluate

The evaluation script will:

• Load the benchmark dataset
• Execute the RAG pipeline
• Run all evaluation metrics
• Upload the experiment to LangSmith

---

LangSmith Report

The latest evaluation results are available here:

https://smith.langchain.com/o/6bb11db4-bbaa-43db-a5f0-9d6164f5c573/datasets/92849185-c26e-4cb7-9975-1d207824945f

---

RAG Pipeline Overview

User Question
      │
      ▼
FastAPI API
      │
      ▼
Redis Queue (RQ)
      │
      ▼
RQ Worker
      │
      ▼
Semantic Retrieval (Qdrant)
      │
      ▼
Prompt Construction
      │
      ▼
LLM
      │
      ▼
Grounded Response
      │
      ▼
User

---

Future Improvements

• Hybrid Retrieval (Dense + BM25)
• Cross-Encoder Reranking
• Metadata Filtering
• Streaming Responses
• Multi-turn Conversation Support
• Conversation Memory
• Hybrid Search
• Larger Benchmark Dataset
• CI/CD Integration
• Containerized Deployment

---

License

This project is intended for educational, research, and demonstration purposes.