RAG Pipeline for DOU Articles

https://www.kaggle.com/competitions/ua-agent-builder-lab-dev-track/overview

End-to-end Retrieval Augmented Generation pipeline for answering questions from a corpus of 3,851 DOU (dou.ua) articles.

Overview

This system builds an AI agent capable of:

• Navigating a large, noisy IT corpus
• Finding relevant sources among 3,851 documents
• Extracting precise, atomic answers (one word, number, year, or surname)
• Answering three sub-questions per query

Architecture

Four-stage pipeline:

1. Data Ingestion: Load articles, chunk text, generate embeddings, store in ChromaDB
2. Retrieval: Query vector store for initial top-K=20 chunks
3. Reranking: Re-rank chunks using BGE-reranker (Top-K=5)
4. Generation: Constrained LLM to produce atomic answers

Installation

1. Install dependencies:

pip install -e .

Or create a virtual environment:

python -m venv .venv
.venv\Scripts\activate  # Windows
# or
source .venv/bin/activate  # Linux/Mac
pip install -e .

2. Set up API key:

For OpenAI:

export OPENAI_API_KEY="your-api-key"
# or
set OPENAI_API_KEY=your-api-key  # Windows

For Gemini (free tier):

export GEMINI_API_KEY="your-api-key"
# or
set GEMINI_API_KEY=your-api-key  # Windows

Tip: Use Gemini for free-tier optimization (gemini-2.0-flash-exp is free)

Usage

Build Index Only

python run_rag_pipeline.py --build-only

Calibration (Validate on Example Questions)

python run_rag_pipeline.py --calibrate --validate

Run on Test Questions

python run_rag_pipeline.py --test --validate

Custom Options

python run_rag_pipeline.py \
  --test \
  --output results.csv \
  --questions ua-agent-builder-lab-dev-track/test_questions.jsonl \
  --no-rerank \
  --initial-top-k 20 \
  --final-top-k 5

Project Structure

.
├── data/
│   └── embeddings/              # ChromaDB storage (auto-created)
├── ua-agent-builder-lab-dev-track/
│   ├── dou_articles.jsonl       # 3,851 articles
│   ├── example_questions.jsonl  # Calibration data
│   ├── test_questions.jsonl     # Leaderboard test pool
│   └── sample_submission.csv    # Expected output format
├── src/
│   ├── __init__.py
│   ├── data_loader.py           # JSONL parsing
│   ├── chunking.py              # Text splitting
│   ├── embedding_service.py     # ChromaDB vector database
│   ├── retriever.py             # Vector search
│   ├── reranker.py              # Cross-encoder reranking
│   ├── generator.py             # Constrained LLM generation
│   └── orchestrator.py          # Main workflow
├── run_rag_pipeline.py         # Entry point script
├── evaluate.py                 # Validation script
└── pyproject.toml               # Dependencies

Components

Data Loader

• Parses JSONL files with robust error handling
• Extracts article metadata (url, text, title, author)

Chunker

• Uses LangChain RecursiveCharacterTextSplitter
• 512 tokens/chunk, 10% overlap
• Ukrainian preprocessing: Normalizes text (collapses whitespace, lowercase, cleans non-essential characters)
• Preserves metadata with each chunk

Embedding Service

• Sentence Transformers (all-MiniLM-L6-v2)
• 384-dimensional vectors
• ChromaDB for persistent storage

Retriever

• Cosine similarity search
• Retrieves Top-K=20 initial chunks

Reranker

• BGE-reranker-v2-m3 cross-encoder
• Re-ranks chunks for improved precision

Generator

• OpenAI: GPT-4o or GPT-4o-mini (paid)
• Gemini: gemini-2.0-flash-exp (free tier, optimized for high-volume)
• Pydantic schema for structured output
• Strict format: one word, number, year, or surname
• Language-aware prompt engineering for Ukrainian/English queries

Orchestrator

• Coordinates all pipeline stages
• Batch processing support
• Progress tracking

Output Format

Results are saved in CSV format:

question_id,source_urls,c1,c2,c3
q_001,https://dou.ua/lenta/articles/...|https://dou.ua/forums/...,SwiftData,2023,15
q_002,https://dou.ua/lenta/articles/...,Kyiv,2024,7

Validation

Use the evaluation script to check accuracy:

python evaluate.py --results sample_submission.csv --ground-truth ua-agent-builder-lab-dev-track/example_questions.jsonl

Notes

• Embeddings are cached locally for reuse
• Reranking is optional (can be disabled for speed)
• Answers are constrained to one word/number/year/surname
• Case-insensitive matching for years and surnames
• Supports both English and Ukrainian queries

Development

The code is designed to be run in Jupyter Notebooks for experimentation. See sample.ipynb for examples.

License

See project license file.