ReportIQ Intelligence Engine

The Intelligence Engine is the AI backend for the ReportIQ platform. It is a Python-based microservice responsible for processing jobs, running local/cloud AI models, and managing vector embeddings.

Architecture

• Language: Python 3.11
• Framework: FastAPI (for API) + SQLAlchemy (for DB Access)
• AI Framework: LangChain (for RAG & Embeddings)
• Concurrency: Event Loop with SKIP LOCKED polling (No RabbitMQ required)

Project Structure

src/
├── domain/                     # Pydantic contracts (data models)
│   └── models.py               # IngestedContent, SearchResult, etc.
├── core/                       # Shared, reusable modules
│   ├── ingestion/              # ONLY parsing logic
│   │   ├── parsers.py          # File parsing (PDF, Excel, etc.)
│   │   └── utils.py            # Helper utilities
│   ├── intelligence/           # ONLY AI/LLM logic
│   │   ├── embeddings.py       # Ollama vector embeddings
│   │   └── llm_sql.py          # Gemini SQL generation
│   ├── search/                 # ONLY search logic
│   │   └── service.py          # Hybrid semantic + keyword search
│   └── storage/                # ONLY MinIO logic
│       └── minio_service.py    # File storage operations
├── features/                   # Feature-specific modules (isolated)
│   ├── document_chat/          # PDF RAG feature
│   │   ├── service.py          # Business logic for document chat
│   │   └── router.py           # API endpoints (/documents/*)
│   └── data_analyst/           # Excel/SQL feature
│       ├── service.py          # Business logic for SQL generation
│       └── router.py           # API endpoints (/data/*)
├── db/                         # Database layer
│   ├── models.py               # SQLAlchemy ORM models
│   └── session.py              # Database session management
├── config.py                   # Configuration (pydantic-settings)
├── main.py                     # FastAPI app entry point
└── worker.py                   # Background job processor

tests/
└── safety_net.py               # Integration tests

Job Loop Logic

This service does not simply wait for HTTP requests. Instead, it actively polls the shared PostgreSQL database for work.

1. Poll: Every 1 second, it checks job_queue for status='PENDING'.
2. Lock: It uses SELECT ... FOR UPDATE SKIP LOCKED to atomically claim a job.
3. Process: It routes the job to a specific handler based on job.type.
   • INGEST: Convert Text -> Vector (via Ollama/Nomic) -> Save to documents.
   • REPORT: Retrieve Vectors -> Summarize (via LLM) -> Save Report.
4. Complete: It marks the job as DONE or FAILED.

Technical Stack

Component	Tool	Purpose
Database	PostgreSQL + pgvector	Stores Jobs and Vector Embeddings
Embeddings	Ollama (nomic-embed-text)	Converts text to 768-dim vectors locally
LLM	Ollama (mistral) / Gemini	Generates summaries and SQL queries
Config	pydantic-settings	Loads .env variables securely
Storage	MinIO	Object storage for uploaded files

API Endpoints

Legacy Endpoints (Backward Compatible)

Method	Endpoint	Description
GET	/health	Health check
POST	/embed	Generate vector embedding
POST	/parse_files	Parse uploaded file
POST	/search	Hybrid document search
POST	/chat/sql	Natural language to SQL

Feature Endpoints (New)

Method	Endpoint	Description
POST	/documents/parse	Parse uploaded document
POST	/documents/embed	Generate embedding for text
POST	/documents/search	Search documents
POST	/data/chat/sql	Natural language to SQL
GET	/data/schema	Get available data schema keys

Quick Start

1. Environment Setup

Create a .env file in the root directory:

ENV_NAME=local
DATABASE_URL=postgresql://admin:adminpassword@localhost:5432/reportiq_db
OLLAMA_URL=http://localhost:11434
MINIIO_URL=localhost:9000
MINIIO_USERNAME=minioadmin
MINIIO_PASSWORD=minioadminpassword
MINIIO_SECURE=false
GEMINI_APIKEY=your_gemini_api_key

2. Install Dependencies

pip install -r requirements.txt

3. Run the Worker

This starts the background polling loop.

python -m src.worker

4. Run the API

Start the FastAPI server.

python -m src.main

Or with auto-reload for development:

uvicorn src.main:app --reload --host 0.0.0.0 --port 8000

5. Run Safety Net Tests

Before starting any new feature, run the integration tests:

python tests/safety_net.py

Key Files

• src/main.py: FastAPI application entry point (imports feature routers)
• src/worker.py: Background job processor with polling loop
• src/domain/models.py: Pydantic models for data contracts
• src/features/*/service.py: Business logic for each feature
• src/features/*/router.py: API endpoints for each feature
• src/core/*/: Shared modules (ingestion, intelligence, search, storage)

Common Issues

• 404 Model Not Found: The OLLAMA_URL is reachable, but the specific model (e.g., nomic-embed-text) hasn't been pulled on that server.
  • Fix: Run ollama pull nomic-embed-text on the server hosting Ollama.
• Connection Refused: The Python script cannot see the Database.
  • Fix: If running locally, ensure DATABASE_URL uses localhost. If running in Docker, use db.

About ReportIQ

ReportIQ is an enterprise AI platform designed to transform raw company data into actionable intelligence. Our mission is to enable organizations to plug in diverse data sources—including databases, emails, spreadsheets, and documents—and instantly generate reports, summaries, and answers without manual effort.

Core Capabilities

• Unified Data Ingestion: Seamlessly ingest and normalize data from disparate sources such as SQL databases, PDF documents, Excel spreadsheets, and email servers.
• Hybrid AI Intelligence: Intelligently route queries between efficient local models for privacy and powerful cloud models for complex reasoning.
• Automated Reporting: Generate scheduled executive summaries, operational dashboards, and anomaly reports automatically.
• Enterprise Security: Built with strict multi-tenancy, row-level security, and identity propagation to ensure data isolation and compliance.

Technology Stack Overview

• Core Platform: Java 21 / Spring Boot (Orchestration, Security, API Gateway)
• Intelligence Engine: Python / FastAPI / LangChain (RAG, Vector Processing, LLM Inference)
• Frontend Console: TypeScript / SvelteKit (Reactive User Interface)
• Data Layer: PostgreSQL 17 (Relational Data) + pgvector (Semantic Search) + pgai (In-Database AI)
• Infrastructure: Docker Compose (Local Development) / Docker Swarm (Deployment)

Key Repositories

Core Services

• reportiq-core: The central backend service handling authentication, API gateway functions, and business logic.
• reportiq-intelligence: The AI worker service responsible for document parsing, vector embedding, and LLM interactions.
• reportiq-console: The web-based administration and user interface.

Shared & Infrastructure

• reportiq-infra: Infrastructure as Code (IaC), Docker configurations, and deployment scripts.

Contact & Collaboration

This organization is currently in active development. For inquiries regarding access, security, or contribution guidelines, please contact the engineering team directly.