[LightRAG][Step 5] Add shadow retrieval, unified tracing, and evaluation metrics

[rickisba]

Parent Epic: #162
Depends on: #165, #166
Related observability contract: #141

Goal

Run LightRAG retrieval alongside the active embedding backend without changing production knowledge selection or routing. Record enough bounded evidence to compare retrieval quality, latency, consistency, and downstream system cost.

Shadow Semantics

• The active backend determines Knowledge_List, KDN selection, Proxy selection, and the data-plane request.
• The shadow backend runs under a strict independent timeout.
• Shadow success, failure, cancellation, or disagreement must not alter the active request result.
• Shadow work must be bounded and cancellable under overload.

Scope

• Implement shadow retrieval orchestration using the embedding backend as active and LightRAG as shadow by default.
• Add sampling controls so shadow execution can be enabled for a bounded percentage or explicit request cohort.
• Record active and shadow candidate KIDs, scores, latency, backend/mode, errors, timeout, and fallback state using bounded trace fields.
• Compute deterministic comparison metrics such as top-k overlap, exact-set match, rank correlation where valid, duplicate rate, unresolved/stale KID rate, and total candidate token length.
• Add hooks or offline utilities for Recall@K, MRR, and NDCG when labeled relevance data is available; do not invent relevance labels online.
• Correlate retrieval observations with existing request/system metrics, including TTFT, E2E latency, selected KDN/Proxy, transfer bytes, injected tokens, KV hit/miss, and fallback behavior where those measurements exist.
• Add counters/histograms for shadow timeout, error, overload skip, stale candidate, and comparison latency.
• Ensure prompts, full knowledge text, credentials, embeddings, and unbounded entity/relationship payloads are not emitted into normal logs or metrics.
• Add a reproducible evaluation script/configuration for a fixed query set without turning it into a pytest-collected live-service script.

Overload and Privacy Rules

• Shadow work may be skipped when concurrency or queue limits are reached; record the skip reason.
• Cancelling an active request must cancel unnecessary shadow work.
• Trace payload sizes must be bounded.
• Full user prompts and retrieved documents must not be logged by default.
• Secrets and auth headers must never enter traces.

Non-Goals

• Do not make LightRAG the active default backend.
• Do not modify Scheduler pool routing based on shadow results.
• Do not modify Proxy Instance selection.
• Do not claim retrieval-quality improvement without labeled evaluation.
• Do not implement Knowledge Demand Profile yet.

Acceptance Criteria

• Shadow mode cannot change the active Knowledge_List or routing decision.
• Shadow failures and overload skips are isolated and observable.
• Trace and metrics are bounded, provenance-tagged, and compatible with the v1 observability vocabulary.
• CPU-only tests verify success, timeout, error, cancellation, sampling, overload skip, and metric calculations with mocked backends.
• An offline evaluation workflow produces a machine-readable comparison report.
• A demo or smoke test shows active embedding retrieval and shadow LightRAG retrieval on the same requests.

Validation

Run focused shadow/trace/metric tests plus Step 4 Scheduler retrieval tests and relevant observability tests. Include actual results in the PR and use Closes #167.