test: Add 46 tests for RAG coverage accuracy improvements

.claude/prompts/behavioral-contracts.md

@@ -0,0 +1,104 @@
+# Behavioral Contracts Full Audit
+
+Run a comprehensive audit of all behavioral promises the system makes to users. This is the manual skill invoked with `/behavioral-contracts`.
+
+## Procedure
+
+### Step 1: Load Contract Definitions
+
+Read `.claude/references/behavioral-contracts.md` for the full list of contracts with their IDs, promises, implementation locations, and validation criteria.
+
+### Step 2: Verify Each Contract
+
+For each contract, read the implementing code and run a pass/fail assessment:
+
+**Memory Contracts:**
+
+| ID | What to Check |
+|----|---------------|
+| MEM-001 | Read `conversations.py` — find `.limit(N)` for history and `message_count >= M` for facts. Is M > N? If so, dead zone exists. |
+| MEM-002 | Read `memory_consolidation.py` `_apply_decay()` and `_prune_facts()` — do they skip identity facts? Read `memory_scoring.py` — is identity priority 1.0? |
+| MEM-003 | Search for consolidation calls outside of `tasks/` and `celery_app.py`. If none, consolidation only runs in beat tasks (not during active conversations). |
+| MEM-004 | Read `fact_extraction.py` dedup logic. When a fact key matches an existing fact, is the value compared or just the key? |
+
+**Citation Contracts:**
+
+| ID | What to Check |
+|----|---------------|
+| CIT-001 | Search entire backend for `was_used_in_response = False`. If not found, the field is always True (broken). |
+| CIT-002 | Read system prompt in `conversations.py` for how chunks are numbered. Check if marker bounds are validated after LLM response. |
+| CIT-003 | Read jump URL builder. Does it handle None timestamps? Does it correctly convert seconds to `t=` parameter? |
+
+**Accuracy Contracts:**
+
+| ID | What to Check |
+|----|---------------|
+| ACC-001 | Read `storage_calculator.py` BYTES_PER_VECTOR constant. Read `config.py` for embedding_model. Do the dimensions match? |
+| ACC-002 | Read token estimation in `conversations.py`. Is it `word_count * 1.3` or a proper tokenizer? How far off could it be? |
+| ACC-003 | Read `bm25_search.py` `_should_skip_bm25()`. Does it skip queries with proper nouns that have <3 tokens? |
+
+**Content Parity Contracts:**
+
+| ID | What to Check |
+|----|---------------|
+| PAR-001 | Compare enrichment calls in `document_tasks.py` vs `video_tasks.py`. Are parameters equivalent? |
+| PAR-002 | Read `enrichment.py` truncation logic. Is there a `logger.warning()` when full_text > 48K? |
+
+**Retrieval Contracts:**
+
+| ID | What to Check |
+|----|---------------|
+| RET-001 | Read `two_level_retriever.py` around lines 785-828. When `enable_relevance_grading=True`, do REFORMULATE/EXPAND_SCOPE actually trigger re-retrieval? Or are they just logged? |
+
+### Step 3: Produce Report
+
+```
+## Behavioral Contracts Audit Report
+
+**Date:** [current date]
+**Audited by:** Claude Code
+
+### Summary
+- Total contracts: X
+- Passing: Y
+- Broken: Z
+- Degraded: W
+
+### Detailed Results
+
+| ID | Promise | Status | Evidence |
+|----|---------|--------|----------|
+| MEM-001 | No memory dead zone | PASS/BROKEN/DEGRADED | [file:line citation + specific finding] |
+| MEM-002 | Identity facts survive | PASS/BROKEN/DEGRADED | [file:line citation] |
+| ... | ... | ... | ... |
+
+### Critical Issues (Must Fix)
+[List broken contracts that cause user-visible problems, ordered by severity]
+
+### Degraded Contracts (Should Fix)
+[List degraded contracts with impact assessment]
+
+### Passing Contracts
+[Brief confirmation of passing contracts]
+
+### Recommendations
+[Prioritized list of fixes with specific file:line targets and estimated effort]
+```
+
+### Step 4: Prioritize Recommendations
+
+Order recommendations by:
+1. **User impact** — Does the broken contract cause visible problems?
+2. **Fix complexity** — How many files need to change?
+3. **Risk** — Could the fix introduce regressions?
+
+### Step 5: Offer to Fix
+
+For each broken contract, offer a specific code change. Focus on the highest-impact fixes first.
+
+## Contract Status Legend
+
+- **PASS** — Implementation matches promise, validation confirms
+- **BROKEN** — Implementation contradicts promise (e.g., `was_used_in_response` always True)
+- **DEGRADED** — Implementation partially fulfills promise (e.g., dead zone exists but is small)
+- **UNTESTED** — Cannot verify without live infrastructure (e.g., needs Docker)

.claude/prompts/citation-accuracy.md

@@ -0,0 +1,70 @@
+# Citation Accuracy Analysis
+
+Analyze citation behavioral contracts after the shell script has run. The shell script checks structural patterns; your job is to trace the full citation flow from system prompt through storage to frontend display.
+
+## Context
+
+Read `.claude/references/behavioral-contracts.md` for contracts CIT-001 through CIT-003.
+
+## Your Tasks
+
+### 1. Evaluate Shell Script Output
+
+Review the output from `citation-accuracy.sh`. Note which contracts passed and which were flagged.
+
+### 2. Trace the Full Citation Flow
+
+Read the following files and trace how citations work end-to-end:
+
+1. **System prompt** (`backend/app/api/routes/conversations.py`): How does the prompt instruct the LLM to cite sources? What format (e.g., `[1]`, `[Source 1]`)?
+2. **Chunk assembly**: How are chunks numbered when building the context? Is numbering 0-indexed or 1-indexed?
+3. **LLM response**: After streaming completes, is the response parsed for citation markers?
+4. **Storage** (`backend/app/models/message.py`): How are `MessageChunkReference` records created? Is `was_used_in_response` ever updated?
+5. **Frontend** (`frontend/src/components/shared/CitationBadge.tsx`): How does the UI render citations? Does it rely on `was_used_in_response`?
+
+### 3. Deep Analysis of Flagged Contracts
+
+**CIT-001 (was_used_in_response tracking):**
+- Search the entire codebase for any code that sets `was_used_in_response = False`
+- Read the message creation flow — does it parse `[N]` markers from the completed LLM response?
+- If broken: every citation appears "used" even if the LLM ignored the chunk — this undermines citation quality metrics
+
+**CIT-002 (Marker bounds):**
+- Count how many chunks are provided to the LLM in the system prompt
+- Check if there's validation that prevents `[5]` when only 4 chunks exist
+- Check for off-by-one errors (0-indexed chunks but 1-indexed markers, or vice versa)
+
+**CIT-003 (Jump URL timestamps):**
+- Read the URL builder function
+- Does it handle `None` timestamps (e.g., for document chunks that don't have timestamps)?
+- Does it correctly convert chunk `start_timestamp` (seconds) to YouTube `t=` parameter?
+- Are there edge cases (timestamp=0, negative timestamps, very large timestamps)?
+
+### 4. Report
+
+```
+## Citation Accuracy Report
+
+### Citation Flow Trace
+[Step-by-step description of how citations flow through the system]
+
+### Contract Status
+| Contract | Status | Evidence |
+|----------|--------|----------|
+| CIT-001  | PASS/BROKEN | [specific finding with file:line] |
+| CIT-002  | PASS/BROKEN | [specific finding] |
+| CIT-003  | PASS/BROKEN | [specific finding] |
+
+### Impact Assessment
+[What user-visible problems do broken contracts cause?]
+
+### Recommended Fixes
+[Prioritized list with specific code changes]
+```
+
+### 5. Offer to Fix
+
+Common citation fixes:
+- CIT-001: After LLM streaming completes, parse response text for `[N]` markers using regex, set `was_used_in_response=False` for chunk_refs not referenced
+- CIT-002: Add bounds validation before storing chunk references
+- CIT-003: Add null-timestamp guard in URL builder, handle document chunks separately

.claude/prompts/content-parity.md

@@ -0,0 +1,79 @@
+# Content Parity Analysis
+
+Analyze document vs video processing parity after the shell script has run. The shell script checks structural patterns; your job is to read both pipelines side-by-side and identify features present in one but missing from the other.
+
+## Context
+
+Read `.claude/references/behavioral-contracts.md` for contracts PAR-001 and PAR-002.
+
+## Your Tasks
+
+### 1. Evaluate Shell Script Output
+
+Review the output from `content-parity.sh`. Note which contracts passed and which were flagged.
+
+### 2. Side-by-Side Pipeline Comparison
+
+Read both processing pipelines in full:
+- `backend/app/tasks/video_tasks.py` — the video processing pipeline
+- `backend/app/tasks/document_tasks.py` — the document processing pipeline
+
+For each pipeline stage, compare:
+
+| Stage | Video Pipeline | Document Pipeline | Parity? |
+|-------|---------------|-------------------|---------|
+| Download/Extract | Audio download + Whisper | Text extraction (PDF/DOCX) | N/A (different sources) |
+| Chunking | `chunking.py` (semantic, timestamp-aware) | `document_chunker.py` (section/page-aware) | Check |
+| Enrichment | ContextualEnricher with full transcript | ContextualEnricher with full text | Check |
+| Embedding | Same service for both? | Same service for both? | Check |
+| Indexing | Qdrant with video_id | Qdrant with document_id | Check |
+| Summary | Video summary generation | Document summary generation | Check |
+
+### 3. Deep Analysis of Flagged Contracts
+
+**PAR-001 (Enrichment Parity):**
+- Read the enrichment calls in both task files
+- Are the parameters equivalent? (full_text, content_type, usage_collector)
+- Does the document pipeline pass full_text for contextual enrichment?
+- Does the document pipeline get the same cache benefits as video pipeline?
+
+**PAR-002 (Truncation Warning):**
+- Read `enrichment.py` line 94: `self.full_text = full_text[:48000]...`
+- Is there a `logger.warning()` before or after truncation?
+- If not: large documents are silently losing content context without any log trace
+
+### 4. Feature Parity Checklist
+
+Check each feature in the video pipeline and verify the document pipeline has it too:
+- [ ] Status tracking (pending → processing → completed → failed)
+- [ ] Cancellation support (can cancel mid-processing)
+- [ ] Reprocessing support (can reprocess failed/canceled)
+- [ ] Storage quota tracking (track storage usage)
+- [ ] Error handling with status rollback
+- [ ] Idempotency guards (skip if already completed)
+
+### 5. Report
+
+```
+## Content Parity Report
+
+### Pipeline Comparison
+| Feature | Video | Document | Status |
+|---------|-------|----------|--------|
+| Contextual enrichment | Yes (full transcript) | ? | Check |
+| Truncation logging | ? | ? | Check |
+| Status tracking | Yes | ? | Check |
+| ... | ... | ... | ... |
+
+### Contract Status
+| Contract | Status | Evidence |
+|----------|--------|----------|
+| PAR-001  | PASS/BROKEN | [specific finding] |
+| PAR-002  | PASS/BROKEN | [specific finding] |
+
+### Features Missing from Document Pipeline
+[List any features present in video pipeline but absent from document pipeline]
+
+### Recommended Fixes
+[Prioritized list of parity gaps to close]
+```

.claude/prompts/conversation-quality.md

@@ -0,0 +1,70 @@
+# Conversation Quality Analysis
+
+Analyze conversation behavioral contracts after the shell script has run. The shell script checks structural patterns; your job is to perform semantic analysis that requires reading and understanding code.
+
+## Context
+
+Read `.claude/references/behavioral-contracts.md` for the full contract definitions (MEM-001 through MEM-004, CIT-001).
+
+## Your Tasks
+
+### 1. Evaluate Shell Script Output
+
+Review the output from `conversation-quality.sh`. Note which contracts passed and which were flagged.
+
+### 2. Deep Analysis of Flagged Contracts
+
+For each flagged contract, read the implementing code and determine:
+
+**MEM-001 (Memory Dead Zone):**
+- Read `backend/app/api/routes/conversations.py` around line 1242 (`.limit()`) and line 1445 (`message_count >= 15`)
+- Read `backend/app/api/utils.py` for `truncate_history_messages()`
+- Map the exact lifecycle: when are messages loaded? When are they truncated? When does fact extraction run?
+- Is there a bridging mechanism (e.g., facts extracted incrementally before messages leave the window)?
+- Calculate the exact dead zone: messages N+1 through M-1 where N=history_limit and M=fact_threshold
+
+**CIT-001 (Citation Tracking):**
+- Read `backend/app/models/message.py` line 114 for the default
+- Search the codebase for any code that sets `was_used_in_response = False`
+- Read the message creation flow in `conversations.py` — does it parse `[N]` markers from LLM output?
+- If broken: the field is a lie — every citation is marked "used" regardless of whether the LLM referenced it
+
+**MEM-003 (Active Consolidation):**
+- Read `backend/app/services/memory_consolidation.py` for consolidation logic
+- Read `backend/app/core/celery_app.py` beat_schedule for when consolidation runs
+- Is consolidation ever triggered during the message send flow?
+- If only in beat tasks: conversations can accumulate unlimited facts until the next beat run
+
+**MEM-004 (Fact Value Merge):**
+- Read `backend/app/services/fact_extraction.py` dedup logic
+- When a fact key matches an existing fact, is the value compared?
+- If "speaker=Alice" is updated to "speaker=Alice and Bob", does the new value replace the old?
+
+### 3. Report
+
+```
+## Conversation Quality Report
+
+### Contract Status
+| Contract | Status | Evidence |
+|----------|--------|----------|
+| MEM-001  | PASS/BROKEN/DEGRADED | [specific finding with file:line] |
+| MEM-002  | PASS/BROKEN/DEGRADED | [specific finding] |
+| MEM-003  | PASS/BROKEN/DEGRADED | [specific finding] |
+| MEM-004  | PASS/BROKEN/DEGRADED | [specific finding] |
+| CIT-001  | PASS/BROKEN/DEGRADED | [specific finding] |
+
+### Impact Assessment
+[What user-visible problems do broken contracts cause?]
+
+### Recommended Fixes
+[Prioritized list of fixes, with specific file:line targets]
+```
+
+### 4. Offer to Fix
+
+If contracts are broken, offer specific code changes. Common fixes:
+- MEM-001: Lower fact threshold or extract facts incrementally
+- CIT-001: Add post-generation marker parsing
+- MEM-003: Call consolidation inline when fact_count > MAX_FACTS
+- MEM-004: Compare fact values in dedup, update if changed