Implement Langfuse tracing integration for invoke-rag calls and consolidate usage functions

[NathalieFroissart]

Summary

This pull request consolidates duplicated logic for extracting per-LLM-call token usage and stage timings from agent results into a new shared namespace, and integrates Langfuse tracing for RAG runs. The main benefit is to eliminate code duplication and prevent future bugs caused by diverging logic, while also enabling optional tracing of RAG runs through Langfuse. There are no behavior changes to existing CSV output or API responses except for the new tracing feature (which is off by default).

Refactoring: Usage and Stage Timing Extraction

• Introduced new namespace digdir.skills.usage containing shared functions for extracting stage timings and token usage from agent results. This replaces duplicated logic in both digdir.sweep.runner and digdir.api.routes.endpoints.openai_compat.
• Updated all call sites in digdir.sweep.runner and digdir.api.routes.endpoints.openai_compat to delegate to the new shared functions instead of their local copies, removing the old definitions. [1] [2] [3] [4] [5] [6] [7] [8] [9]

Feature: Optional Langfuse Tracing

• Added configuration options for Langfuse tracing to .env.example, with detailed documentation and warnings about data leaving the deployment.
• Integrated Langfuse tracing into the main RAG invocation path (digdir.skills.invoke/invoke-rag), wrapping the entire skill graph execution so that traces are emitted for both successful and failed runs if enabled. [1] [2] [3]

Codebase Maintenance

• Updated namespace requires to use the new digdir.skills.usage and digdir.telemetry.langfuse modules where needed. [1] [2] [3]

No behavior is changed for existing users unless Langfuse tracing is explicitly enabled and configured. The refactoring ensures future maintainability and reduces risk of bugs from duplicated logic.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 5a2c7ddd-bc8a-40cd-8521-521275bb70ac

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Copilot review overview

🟡 Changes recommended

Address the tracing correctness, resource-management, and automated-coverage issues before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 2

Open (3)

• Bound fire-and-forget telemetry tasks · New
• Use plain string encoding for Langfuse observation level · New
• Record per-call generation start times · New

What changed in this PR

Adds optional Langfuse tracing for RAG invocations and centralizes usage and stage-timing extraction.

Changes:

• Added Langfuse tracing and configuration.
• Added shared usage and timing utilities.
• Updated invocation, API, and sweep paths to use shared logic.

File	Summary
server/​src/​digdir/​telemetry/​langfuse.clj	Implements Langfuse tracing; issues remain with error attributes, generation timing, aggregated usage, and unbounded background tasks.
server/​src/​digdir/​skills/​usage.clj	Adds shared usage and timing extraction utilities.
server/​src/​digdir/​skills/​invoke.clj	Wraps RAG execution with tracing; focused automated coverage is needed.
server/​src/​digdir/​api/​routes/​endpoints/​openai_compat.clj	Uses shared usage extraction.
server/​src-dev/​digdir/​sweep/​runner.clj	Uses shared stage-timing extraction.
.env.example	Documents Langfuse configuration.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Unresolved moderate tracing reliability, testing, and model-attribute issues remain.

Review effort: Lite
Findings: None

Resolved since last review (3)

• Bound fire-and-forget telemetry tasks
• Record per-call generation start times
• Use plain string encoding for Langfuse observation level

Previously missed (1)

In code that hasn't changed since last review

Add tests for Langfuse payloads, gates, and failures

server/​src/​digdir/​telemetry/​langfuse.clj:342

This introduces the production Langfuse integration and its payload construction, environment gate, async dispatch, and failure-swallowing behavior without tests. The repository already tests analogous HTTP integrations by redefining clj-http.client/post (for example server/test/digdir/llm/marker_test.clj:74-85); add deterministic tests for the OTLP payload, disabled/misconfigured gates, error traces, and rejected/failed posts so regressions do not silently turn tracing off.