diff --git a/CHANGELOG.md b/CHANGELOG.md index 59e633c..1e6232f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,18 +8,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added +- `Client#list_observations`, `Client#query_metrics`, and `Client#list_scores` expose the current v2 observations and metrics APIs and the v3 scores API. +- Text and correction score types are available through the score creation and read APIs. - `ScoreClient#create!`, `Client#create_score!`, and `Langfuse.create_score!` create scores through the synchronous Scores API, return the created score ID, and raise API errors. The existing `create` methods retain fire-and-forget ingestion batching. - `Langfuse.configured?` checks locally whether the global client can be constructed without accessing the network. - `Config#metrics_reporter` forwards OpenTelemetry batch span processor metrics to an application-owned reporter without allowing reporter failures to interrupt tracing. - `Config#span_exporter` lets applications inject an OpenTelemetry span exporter into Langfuse's existing tracing pipeline. +- `TextPromptClient#variables` and `ChatPromptClient#variables` expose referenced Mustache variables in source order. +- `Config#tracing_enabled` and `LANGFUSE_TRACING_ENABLED` provide an application-wide tracing and scoring kill switch. ### Changed +- Trace export uses direct Langfuse v4 OTLP ingestion so current observation and metric reads can see new spans without the legacy ingestion delay. +- `LANGFUSE_TIMEOUT`, `LANGFUSE_FLUSH_AT`, `LANGFUSE_FLUSH_INTERVAL`, and `LANGFUSE_DEBUG` now configure their corresponding defaults. +- The asynchronous score queue is bounded. Score flushes split batches before a multi-score JSON payload exceeds 2.5 MB. - Client construction now rejects invalid `batch_size` and `flush_interval` values before score batching can fail later. - Configuration validation now requires `public_key` and `secret_key` to be non-empty Strings and `base_url` to be an absolute HTTP or HTTPS URL. - Assigning `nil` to `Config#logger` now selects a null logger so SDK logger calls remain safe. - Implicit observations warn once and use a no-op tracer when tracing configuration is invalid. Explicit `Langfuse.tracer_provider` access still raises `ConfigurationError`. ### Fixed +- Export-stage masking can transform third-party OpenTelemetry spans while preserving the original span for other exporters. +- SDK-owned queues and workers reset after Ruby `fork` so parent work is not duplicated in child processes. +- Pending spans and scores flush once during normal process exit. - Score creation now uses the configured client environment when a score does not provide an environment override. - Configuration validation now reports invalid numeric types, stale cache settings, and tracing callables as `ConfigurationError`. - Tracing validates batching and sampling settings before it creates an OpenTelemetry span processor. diff --git a/docs/API_REFERENCE.md b/docs/API_REFERENCE.md index c1647f1..e9db91c 100644 --- a/docs/API_REFERENCE.md +++ b/docs/API_REFERENCE.md @@ -9,7 +9,7 @@ Complete method reference for the Langfuse Ruby SDK. - [Prompt Management](#prompt-management) - [Trace ID Generation](#trace-id-generation) - [Tracing & Observability](#tracing--observability) -- [Traces](#traces) +- [Data Access](#data-access) - [Scoring](#scoring) - [Datasets](#datasets) - [Experiments](#experiments) @@ -50,11 +50,13 @@ Block receives a `Langfuse::Config` object with these properties: | `prompt_cache_observer` | Callable | No | `nil` | Prompt cache event hook | | `batch_size` | Integer | No | `50` | Score + trace export batch size | | `flush_interval` | Integer | No | `10` | Score + trace export interval (s) | +| `score_queue_capacity` | Integer | No | `100000` | Maximum pending asynchronous scores | | `sample_rate` | Float | No | `1.0` | Trace + trace-linked score sampling rate (`0.0..1.0`) | | `logger` | Logger | No | Auto-detected | Logger instance | -| `tracing_async` | Boolean | No | `true` | ⚠️ Experimental (OTel batch scheduling) | +| `tracing_async` | Boolean | No | `true` | Experimental OTel batch scheduling | +| `tracing_enabled` | Boolean | No | `true` | Langfuse tracing and scoring kill switch | | `job_queue` | Symbol | No | `:default` | Reserved/no-op for future job integration | -| `environment` | String | No | `nil` (or `ENV["LANGFUSE_TRACING_ENVIRONMENT"]`) | Default trace environment | +| `environment` | String | No | `nil` (or `ENV["LANGFUSE_TRACING_ENVIRONMENT"]`) | Default trace, observation, and score environment | | `release` | String | No | `nil` (or `ENV["LANGFUSE_RELEASE"]` / common CI commit SHA env) | Default release identifier | | `should_export_span` | `#call` | No | `nil` | Span export filter callback | | `mask` | `#call` | No | `nil` | Mask callable for input/output/metadata (receives `data:` keyword) | @@ -126,6 +128,23 @@ This method does not access the network. A `true` result does not prove that cre Tracing does not require this guard. `Langfuse.observe` warns once and uses a no-op tracer when tracing configuration is invalid, so application code can use the same observation wrapper in every environment. +### `Config#valid?` + +Check whether a configuration object can construct a client: + +```ruby +config = Langfuse::Config.new do |candidate| + candidate.public_key = "pk-lf-..." + candidate.secret_key = "sk-lf-..." +end + +config.valid? # => true or false +``` + +The check is local and does not raise an error. +It does not validate credentials, network access, or backend ingestion. +Use `Langfuse.configured?` for the global configuration. + ### `Langfuse.tracer_provider` Return Langfuse's internal tracer provider so you can explicitly install it as the global OpenTelemetry provider. @@ -249,7 +268,7 @@ get_prompt(name, version: nil, label: nil, fallback: nil, type: nil, cache_ttl: **Raises:** -- `NotFoundError` if prompt doesn't exist (unless `fallback` provided) +- `NotFoundError` if the prompt does not exist and no `fallback` is present - `UnauthorizedError` if credentials invalid - `ApiError` on network/server errors @@ -498,10 +517,11 @@ Returned by `get_prompt` for text prompts. | `commit_message` | String, nil | Commit message for the prompt version | | `resolution_graph` | Hash, nil | Dependency resolution graph for composed prompts when returned by Langfuse | | `is_fallback` | Boolean | Whether the client uses caller-provided fallback content | +| `variables` | Array | Referenced Mustache variables in source order | **Methods:** -#### `compile` +#### `TextPromptClient#compile` ```ruby compile(**variables) # => String @@ -516,6 +536,16 @@ prompt = client.get_prompt("greeting") message = prompt.compile(name: "Alice", time: "morning") ``` +#### `TextPromptClient#variables` + +```ruby +variables # => Array +``` + +Returns unique Mustache variable and section names in source order. +Dotted paths do not change. +Invalid Mustache syntax raises `Mustache::Parser::SyntaxError`. + ### `ChatPromptClient` Returned by `get_prompt` for chat prompts. @@ -524,7 +554,7 @@ Returned by `get_prompt` for chat prompts. **Methods:** -#### `compile` +#### `ChatPromptClient#compile` ```ruby compile(**variables) # => Array @@ -556,6 +586,15 @@ messages = prompt.compile( # ] ``` +#### `ChatPromptClient#variables` + +```ruby +variables # => Array +``` + +Returns unique Mustache variables from message content in message order. +The result does not include Langfuse message placeholders. + ## Trace ID Generation ### `Langfuse.create_trace_id` @@ -838,7 +877,10 @@ obs.update( ) ``` -## Traces +## Data Access + +Use [DATA_ACCESS.md](DATA_ACCESS.md) to select a read API. +The guide also explains cursor pagination and SDK and CLI verification. ### `Client#list_traces` @@ -913,7 +955,7 @@ get_trace(id) # => Hash **Raises:** -- `NotFoundError` if the trace doesn't exist +- `NotFoundError` if the trace does not exist - `UnauthorizedError` if authentication fails - `ApiError` for other API errors @@ -1159,10 +1201,12 @@ flush_scores **Example:** ```ruby -# Before shutdown +# Before an immediate verification read Langfuse.client.flush_scores ``` +Normal process exit flushes pending scores automatically. + ### Module-Level Scoring Convenience methods delegating to `Langfuse.client`: @@ -1283,7 +1327,7 @@ get_dataset(name) # => DatasetClient **Returns:** `DatasetClient` -**Raises:** `NotFoundError` if the dataset doesn't exist +**Raises:** `NotFoundError` if the dataset does not exist ### `Client#list_datasets` @@ -1351,7 +1395,7 @@ Fetch a dataset item by ID. get_dataset_item(id) # => DatasetItemClient ``` -**Raises:** `NotFoundError` if the item doesn't exist +**Raises:** `NotFoundError` if the item does not exist ### `Client#list_dataset_items` @@ -1679,7 +1723,7 @@ Extends `ApiError`. Resource not found (404). **Raised when:** -- Prompt doesn't exist +- Prompt does not exist - Invalid version/label ### `Langfuse::CacheWarmingError` @@ -1769,10 +1813,12 @@ Langfuse.shutdown(timeout: 30) **Example:** ```ruby -# Before process exit +# At an earlier application-owned shutdown boundary Langfuse.shutdown ``` +The SDK invokes shutdown automatically during normal process exit. + ### `Langfuse.force_flush` Force flush all pending data. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 58aedaa..ad04d7f 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,6 +1,6 @@ # Architecture Overview -High-level architecture and key design decisions for the Langfuse Ruby SDK. +This document describes the Langfuse Ruby SDK architecture and its main design decisions. ## Table of Contents @@ -16,23 +16,25 @@ The Langfuse Ruby SDK follows these core principles: ### 1. LaunchDarkly-Inspired API -**Flat API surface** - All methods on `Client`, not nested managers: +Put all public methods on `Client`. +Do not use nested managers: ```ruby -# ✅ Good - Flat API +# Correct: flat API client.get_prompt("name") client.compile_prompt("name", variables: {}) -# ❌ Avoid - Nested managers +# Incorrect: nested managers client.prompts.get("name") client.prompts.compile("name") ``` -**Why?** Simpler, more discoverable API with better IDE autocomplete support. +This API reduces the number of classes that a user must know. +It also gives direct IDE autocomplete results. ### 2. Rails-Friendly -Global configuration pattern that feels natural in Rails: +Use one global configuration block in Rails: ```ruby # config/initializers/langfuse.rb @@ -41,7 +43,7 @@ Langfuse.configure do |config| config.secret_key = ENV['LANGFUSE_SECRET_KEY'] end -# Use anywhere +# Use the configured client client = Langfuse.client ``` @@ -54,21 +56,22 @@ client = Langfuse.client ### 4. Minimal Dependencies -Only add dependencies when absolutely necessary: +Add a dependency only when the SDK requires it: -- **Faraday** - HTTP client (industry standard) -- **Mustache** - Variable templating (logic-less, secure) -- **OpenTelemetry** - Distributed tracing (CNCF standard) +- **Faraday** - HTTP client +- **Mustache** - Logic-free variable templates +- **OpenTelemetry** - Distributed tracing -No Rails dependency - works in any Ruby project. +The SDK has no Rails dependency. +It operates in any supported Ruby project. ### 5. Thread-Safe by Default -All components use proper synchronization: +SDK components use these synchronization mechanisms: - `PromptCache` uses Monitor - `RailsCacheAdapter` uses Redis atomic operations -- `ScoreClient` uses Queue and Mutex for thread-safe batching +- `ScoreClient` uses a bounded `PendingScoreQueue` and mutexes for ordered batching - OpenTelemetry handles context propagation ## Core Components @@ -88,7 +91,8 @@ end **Responsibilities:** - Store SDK configuration - Validate required settings -- Provide defaults +- Read supported environment defaults +- Control tracing and scoring independently from the OpenTelemetry trace switch ### 2. HTTP Client (`Langfuse::ApiClient`) @@ -116,7 +120,7 @@ prompt_data = api_client.get_prompt("name") #### TextPromptClient -For simple string templates: +Use this client for string templates: ```ruby prompt = TextPromptClient.new(api_response) @@ -183,6 +187,7 @@ text = client.compile_prompt("name", variables: { name: "Alice" }) - Cache backend selection - High-level API methods - Score creation and management (delegates to ScoreClient) +- Current observation, metric, and score reads (delegates through ReadApi) ### 6. Tracing Layer (OpenTelemetry-based) @@ -230,6 +235,10 @@ span.end - **OtelSetup** - Initializes OpenTelemetry SDK with OTLP exporter - **SpanProcessor** - Propagates trace-level attributes to child spans - **OtelAttributes** - Converts Langfuse attributes to OpenTelemetry format +- **MaskingExporter** - Applies export-stage patches to Langfuse's span copy +- **AppRootTracking** - Preserves logical application roots across filtered parents +- **ForkSafety** - Rebuilds SDK-owned background state in child processes +- **ExitHook** - Flushes pending tracing and scores during normal process exit **Responsibilities:** - Wrap OpenTelemetry spans with Langfuse-specific functionality @@ -247,13 +256,15 @@ score_client.create(name: "quality", value: 0.85, trace_id: "abc123...") ``` **Features:** -- Thread-safe queuing with `Queue` -- Automatic batching (configurable batch_size and flush_interval) +- Bounded, ordered asynchronous queue +- Synchronous score creation through the Scores API +- Automatic batching by count, payload size, and flush interval - Background flush timer thread - Integration with OpenTelemetry spans (extracts trace_id/observation_id) **Responsibilities:** - Queue score events for batching +- Reject or drop work at documented boundaries instead of allowing unbounded memory growth - Extract trace/observation IDs from active OTel spans - Batch and send scores to ingestion API - Handle graceful shutdown and flush @@ -284,79 +295,83 @@ end **Decision:** Build tracing on OpenTelemetry instead of custom implementation -**Why?** -- Industry standard (CNCF) -- W3C Trace Context compatibility when the host app configures propagation -- Works with existing APM tools (Datadog, New Relic, etc.) -- Battle-tested context and propagation model +**Reasons:** +- OpenTelemetry is a Cloud Native Computing Foundation standard. +- It supports W3C Trace Context when the application configures propagation. +- It can integrate with application performance monitoring tools. +- It provides context and propagation components. -**Trade-offs:** -- ✅ More robust, future-proof -- ✅ Cross-service tracing when the app installs an OpenTelemetry propagator -- ❌ ~10 additional gem dependencies -- ❌ Slightly more complex setup +**Benefits:** +- The application can use one trace context across services. +- The application can install an OpenTelemetry propagator. + +**Limits:** +- OpenTelemetry adds approximately 10 gem dependencies. +- OpenTelemetry configuration adds setup steps. ### 2. Dual Cache Backend **Decision:** Support both in-memory and Rails.cache backends -**Why?** -- In-memory: Perfect for single-process apps, scripts, small deployments -- Rails.cache: Essential for large multi-process deployments (100+ processes) +**Reasons:** +- The in-memory cache applies to scripts and single-process applications. +- `Rails.cache` can share prompt data across processes. + +**Benefits:** +- Applications can select a cache for their process model. +- The default cache does not require an external service. -**Trade-offs:** -- ✅ Flexibility for different use cases -- ✅ Zero external dependencies by default -- ❌ More code to maintain -- ❌ Two code paths to test +**Limits:** +- The SDK must maintain two cache implementations. +- Tests must cover both cache implementations. ### 3. Stampede Protection via Distributed Locks **Decision:** Use Redis atomic operations for stampede protection -**Why?** -- Prevents thundering herd (1,200 simultaneous API calls → 1 call) -- Critical for large-scale deployments -- Works automatically with Rails.cache backend +**Reasons:** +- A distributed lock prevents concurrent cache misses from producing duplicate API calls. +- The `Rails.cache` backend can coordinate many application processes. + +**Benefits:** +- One process refreshes a stale cache entry. +- Other processes wait for the shared result. +- No additional SDK option is necessary. -**Trade-offs:** -- ✅ Massive performance improvement at scale -- ✅ Automatic - no user configuration needed -- ❌ Only works with Rails.cache backend -- ❌ Slight latency increase for waiting processes +**Limits:** +- Distributed locking applies only to the `Rails.cache` backend. +- Waiting processes have additional latency. ### 4. Mustache for Variable Substitution **Decision:** Use Mustache templating instead of ERB or custom solution -**Why?** -- Logic-less (no code execution = secure) -- Same syntax as Langfuse JavaScript SDK (consistency) -- Well-tested, mature library -- Supports nested objects, arrays, conditionals +**Reasons:** +- Mustache templates do not execute Ruby code. +- The syntax matches the Langfuse JavaScript SDK. +- Mustache supports nested objects, arrays, and sections. -**Alternatives considered:** -- ERB: Too powerful, security concerns -- String interpolation: Not flexible enough -- Custom: Reinventing the wheel +**Alternatives:** +- ERB can execute Ruby code. +- String interpolation does not support the required template structures. +- A custom parser would duplicate Mustache behavior. ### 5. Flat API Surface **Decision:** All methods on `Client`, not nested managers -**Why?** -- Inspired by LaunchDarkly Ruby SDK -- Simpler mental model -- Better IDE autocomplete -- Fewer classes to remember +**Reasons:** +- The LaunchDarkly Ruby SDK uses this API shape. +- Direct methods reduce the number of public classes. +- Direct methods give direct IDE autocomplete results. **Example:** ```ruby -# ✅ Flat API +# Correct: flat API client.get_prompt("name") client.compile_prompt("name", variables: {}) -# ❌ Nested (rejected) +# Incorrect: nested API client.prompts.get("name") client.prompts.compile("name", variables: {}) ``` @@ -365,49 +380,54 @@ client.prompts.compile("name", variables: {}) **Decision:** `Langfuse.configure` block pattern with global client -**Why?** -- Rails-friendly (feels natural in initializers) -- Reduces boilerplate (don't pass client everywhere) -- Thread-safe singleton pattern -- Easy to reset for testing +**Reasons:** +- Rails initializers commonly use global configuration blocks. +- Application code does not have to pass a client to each object. +- The singleton uses synchronization. +- Tests can call `Langfuse.reset!`. -**Trade-offs:** -- ✅ Convenient for most use cases -- ✅ Follows Rails conventions -- ❌ Global state (can be problematic in tests) -- ✅ Mitigated with `Langfuse.reset!` method +**Benefits:** +- Applications use one configuration entry point. +- The configuration shape follows Rails conventions. + +**Limit:** +- The singleton is global state. ### 7. Observation-Based Tracing Model **Decision:** Use unified observation model instead of separate trace/span/generation classes -**Why?** -- Aligns with Langfuse JavaScript SDK architecture -- Single `start_observation()` method with `as_type` parameter -- Flexible - supports 10+ observation types (span, generation, event, embedding, agent, tool, chain, retriever, evaluator, guardrail) -- Consistent API for all observation types +**Reasons:** +- The model matches the Langfuse JavaScript SDK architecture. +- One `start_observation()` method accepts an `as_type` parameter. +- The API supports all Langfuse observation types. + +**Benefits:** +- All observation types use the same API. +- The model can include new observation types. +- The model matches the Langfuse platform. + +**Limit:** +- Callers must select the correct `as_type` value. -**Trade-offs:** -- ✅ Consistent API across all observation types -- ✅ Easy to add new observation types -- ✅ Aligns with Langfuse platform model -- ❌ Slightly more complex than separate classes +### 8. OTLP as the Default Export Protocol -### 8. OTLP Export Instead of Custom Exporter +**Decision:** Use the OpenTelemetry OTLP exporter by default. +Require explicit exporter injection. -**Decision:** Use OpenTelemetry OTLP exporter instead of custom Langfuse exporter +**Reasons:** +- OTLP is an OpenTelemetry protocol. +- The Langfuse server converts OTLP data to the Langfuse format. +- `BatchSpanProcessor` supplies batch export. +- `Config#span_exporter` supplies an explicit test and integration interface. -**Why?** -- Standard OpenTelemetry protocol (OTLP) -- Langfuse server handles OTLP → Langfuse format conversion -- Future-proof (OTLP is industry standard) -- Automatic batching via BatchSpanProcessor +**Benefits:** +- The SDK uses one export protocol. +- The server owns format conversion. +- An injected exporter uses the normal sampler, filter, enrichment, masking, and batch pipeline. -**Trade-offs:** -- ✅ Standard protocol (OTLP) -- ✅ Server-side conversion (simpler SDK) -- ✅ Works with any OTLP-compatible backend -- ❌ Requires Langfuse server to support OTLP (which it does) +**Limit:** +- The Langfuse server must support OTLP. ## Data Flow @@ -456,8 +476,10 @@ User Code └─> gen.usage_details = {...} → Sets token attributes via OTel span.set_attribute() ├─> OTel BatchSpanProcessor collects spans ├─> SpanProcessor propagates trace-level attributes to new spans + ├─> MaskingExporter transforms the Langfuse copy when configured └─> OTLP Exporter sends spans to Langfuse ├─> POST /api/public/otel/v1/traces (OTLP format) + ├─> x-langfuse-ingestion-version: 4 ├─> Batch export (50 spans per batch, configurable) └─> Langfuse server converts OTLP → Langfuse ingestion format ``` @@ -466,15 +488,26 @@ User Code ``` User Code - └─> Langfuse.create_score(name: "quality", value: 0.85, trace_id: "abc123") - ├─> ScoreClient.create() validates and normalizes score - ├─> Build score event hash - ├─> Queue event (thread-safe Queue) - ├─> Check if batch_size reached → trigger flush - └─> Background flush timer (every flush_interval seconds) - ├─> Collect queued events - ├─> ApiClient.send_batch() → POST /api/public/ingestion - └─> Retry on transient errors (429, 503, 504) +User Code + ├─> Langfuse.create_score(...) + │ ├─> Validate, normalize, and snapshot the score payload + │ ├─> Add to bounded PendingScoreQueue + │ └─> Flush by count, payload size, timer, or lifecycle boundary + │ └─> ApiClient.send_batch() → POST /api/public/ingestion + └─> Langfuse.create_score!(...) + ├─> Validate, normalize, and snapshot the same score payload + └─> ApiClient.create_score() → POST /api/public/scores +``` + +### Process Lifecycle + +``` +Application boot + └─> Langfuse.configure stores settings + └─> First trace or client call starts required resources lazily + ├─> fork child rebuilds SDK-owned queues and workers + ├─> explicit force_flush sends pending spans without shutdown + └─> normal process exit shuts down tracing and scores once ``` ## Technology Choices diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 0f4676d..432b6b6 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -18,7 +18,7 @@ Call this once at application startup (Rails initializer, boot script, etc.). ## Tracing Ownership -This is the part people get wrong. +Select the tracing owner before you configure OpenTelemetry. - `Langfuse.configure` stores configuration only. - Module-level tracing initializes lazily on first use. @@ -26,7 +26,7 @@ This is the part people get wrong. - `Langfuse.tracer_provider` is the explicit seam for installing Langfuse as the global OpenTelemetry provider. - `should_export_span` only runs on spans handled by Langfuse's provider. - Filtering is not the fix for ambient-span overcapture. Isolation is. -- Langfuse does not auto-configure a second OpenTelemetry backend or any multi-export pipeline for you. +- Langfuse does not configure a second OpenTelemetry backend or an export pipeline with multiple destinations. Default isolated setup: @@ -48,7 +48,22 @@ end OpenTelemetry.tracer_provider = Langfuse.tracer_provider ``` -If you also want propagation or another OpenTelemetry backend, configure those in your application. Langfuse does not infer or install them. +If the application needs propagation or another OpenTelemetry backend, configure them in the application. +Langfuse does not install them. + +### Runtime Modes + +Do not combine the Langfuse telemetry switch with the OpenTelemetry switch: + +| Configuration | Traces | Asynchronous scores | Synchronous scores | Prompt and read APIs | +| --- | --- | --- | --- | --- | +| Default | Enabled | Enabled | Enabled | Enabled | +| `tracing_enabled = false` or `LANGFUSE_TRACING_ENABLED=false` | Disabled | Disabled | Disabled and returns `nil` | Available with valid credentials | +| `OTEL_SDK_DISABLED=true` | Disabled | Enabled | Enabled | Enabled | + +`tracing_enabled` is the Langfuse telemetry switch. +It disables tracing and scoring. +`OTEL_SDK_DISABLED` disables only OpenTelemetry trace export. ## All Configuration Options @@ -177,8 +192,8 @@ This flag does not independently turn SWR on or off. SWR activates when `cache_s **Compatibility:** -- ✅ Works with `:memory` backend -- ✅ Works with `:rails` backend +- Available with the `:memory` backend +- Available with the `:rails` backend See [CACHING.md](CACHING.md#stale-while-revalidate-swr) for detailed usage. @@ -276,7 +291,13 @@ config.flush_interval = 5 # Flush more frequently config.score_queue_capacity = 20_000 ``` -A full queue does not wait for capacity. The SDK logs an error and drops only the new asynchronous score. Synchronous `create_score!` calls do not use this queue. Flush requests remain below 2.5 MB when a batch contains more than one score. Failed batches stay queued in their original order for a later flush. +A full queue does not wait for capacity. +The SDK logs an error and drops only the new asynchronous score. +Synchronous `create_score!` calls do not use this queue. +A multi-score flush request stays below 2.5 MB. +Retryable failures keep the batch at the front of the queue. +The SDK logs and discards permanent batch failures. +Thus, the SDK can send later valid scores. #### `sample_rate` @@ -328,7 +349,7 @@ config.logger = nil The SDK normalizes `nil` to a null logger. Logger calls remain safe when output is disabled. -#### `tracing_async` ⚠️ Experimental +#### `tracing_async` (Experimental) - **Type:** Boolean - **Default:** `true` @@ -422,7 +443,7 @@ Use one shared DogStatsD client. At application shutdown, call `Langfuse.shutdow before `statsd.close` so the final batch processor metrics can leave the DogStatsD client's buffer. -#### `job_queue` ⚠️ Experimental +#### `job_queue` (Experimental) - **Type:** Symbol - **Default:** `:default` @@ -439,7 +460,7 @@ config.job_queue = :langfuse # Reserved/no-op today - **Type:** String - **Default:** `nil` (or `ENV["LANGFUSE_TRACING_ENVIRONMENT"]` if set) -- **Description:** Default tracing environment applied to new traces/observations +- **Description:** Default environment applied to new traces, observations, and scores ```ruby config.environment = "production" @@ -472,7 +493,7 @@ This callback only runs for spans processed by Langfuse's tracer provider. Under The SDK calls the callback once after each span finishes. The callback can use final attributes, duration, and status. -If you want shared OpenTelemetry spans to be eligible for this filter, install Langfuse explicitly: +To make shared OpenTelemetry spans eligible for this filter, install the Langfuse provider explicitly: ```ruby OpenTelemetry.tracer_provider = Langfuse.tracer_provider @@ -484,7 +505,8 @@ When Langfuse processes a span and no custom filter is configured, default behav - Spans with `gen_ai.*` attributes - Spans from conservative LLM-related instrumentation scopes such as `langsmith.*`, `openinference.*`, and `opentelemetry.instrumentation.anthropic.*` -Composing with `Langfuse.default_export_span?` keeps that allowlist and lets you add tighter exclusions. +Combine the callback with `Langfuse.default_export_span?` to keep that allowlist. +The callback can add more exclusions. Use this callback to narrow a provider path Langfuse already owns. Do not treat it as the fix for default ambient-span overcapture. The isolated default already prevents that problem. @@ -573,7 +595,7 @@ There are three states worth documenting. ### Explicit Global Install with `Langfuse.tracer_provider` -If you want Langfuse to own the global OpenTelemetry provider, install it explicitly: +To make Langfuse the global OpenTelemetry provider, install it explicitly: ```ruby require "opentelemetry/trace/propagation/trace_context" @@ -587,14 +609,18 @@ OpenTelemetry.tracer_provider = Langfuse.tracer_provider OpenTelemetry.propagation = OpenTelemetry::Trace::Propagation::TraceContext::TextMapPropagator.new ``` -That global install is a lifecycle commitment. `Langfuse.shutdown` and `Langfuse.reset!` stop the internal provider. If you reset or reconfigure Langfuse, reinstall the tracer provider and any propagators you want afterward. +The application controls the lifecycle of a global provider installation. +`Langfuse.shutdown` and `Langfuse.reset!` stop the internal provider. +After a reset or new configuration, install the required tracer provider and propagators again. ### Additional OTel Backends Are Application-Owned -If you want spans in another OpenTelemetry backend as well, configure that pipeline in your application. Langfuse does not auto-install multi-export. That can mean: +To send spans to another OpenTelemetry backend, configure the export pipeline in the application. +Langfuse does not install export to multiple destinations. +Use one of these configurations: -- adding processors/exporters to the provider you own -- or managing your own provider pipeline explicitly +- Add processors or exporters to the application-owned provider. +- Manage an application-owned provider pipeline. After the first successful tracing initialization, these settings require `Langfuse.reset!` before changes take effect: @@ -631,9 +657,11 @@ The SDK automatically reads these environment variables as defaults when no expl - `LANGFUSE_FLUSH_AT` — score flush threshold and maximum trace batch size (defaults to `50`) - `LANGFUSE_FLUSH_INTERVAL` — maximum batch wait in seconds (defaults to `10`) - `LANGFUSE_DEBUG` — set to `true` to write SDK logs to stdout at the `DEBUG` level +- `LANGFUSE_TRACING_ENABLED` — set to `false` to disable Langfuse tracing and scoring (defaults to `true`) - `LANGFUSE_TRACING_ENVIRONMENT` — default trace environment - `LANGFUSE_RELEASE` — default release identifier (falls back to common CI commit envs if present) - `LANGFUSE_SAMPLE_RATE` — trace sampling rate (`0.0..1.0`, defaults to `1.0`) +- `OTEL_SDK_DISABLED` — set to `true` to disable OpenTelemetry trace export without disabling score, prompt, or read APIs Explicit configuration always takes precedence: @@ -655,7 +683,11 @@ LANGFUSE_TIMEOUT=10 # Optional LANGFUSE_FLUSH_AT=100 # Optional LANGFUSE_FLUSH_INTERVAL=5 # Optional LANGFUSE_DEBUG=true # Optional +LANGFUSE_TRACING_ENABLED=true # Optional +LANGFUSE_TRACING_ENVIRONMENT=production # Recommended +LANGFUSE_RELEASE=release-2026-08-17 # Optional LANGFUSE_SAMPLE_RATE=0.25 # Optional +# OTEL_SDK_DISABLED=true # Optional trace-only kill switch ``` ## Rails-Specific Configuration @@ -749,6 +781,10 @@ Call `Langfuse.shutdown` when the application must flush before process exit. An The exit callback runs once and does not allow shutdown errors to escape the process-exit path. Abrupt termination, such as `SIGKILL`, cannot run process-exit callbacks. +Call `Langfuse.force_flush` or `Langfuse.flush_scores` before an immediate verification read. +Do not flush on every request. +Batch export is the normal runtime operation. + ## Configuration by Environment ### Development @@ -841,7 +877,11 @@ Tracing reuses the credential, batching, sampling, and logger rules. It also val ### Rails cache validation and boot order -Setting `cache_backend = :rails` while caching is enabled requires `Rails.cache` to be populated at validation time. Rails assigns `Rails.cache` during its own `initialize_cache` step, so a Langfuse initializer that builds a client earlier than that now raises `ConfigurationError` instead of failing later on the first cache read. Build the client lazily, or use `:auto`, which falls back to the in-memory cache when `Rails.cache` is unavailable. +When caching is enabled, `cache_backend = :rails` requires an available `Rails.cache` object during validation. +Rails assigns `Rails.cache` during its `initialize_cache` step. +An earlier Langfuse client build raises `ConfigurationError`. +Build the client after `initialize_cache`. +Alternatively, use `:auto` to select the in-memory cache when `Rails.cache` is unavailable. ## Accessing Current Configuration diff --git a/docs/DATA_ACCESS.md b/docs/DATA_ACCESS.md new file mode 100644 index 0000000..19386c5 --- /dev/null +++ b/docs/DATA_ACCESS.md @@ -0,0 +1,220 @@ +# Data Access and Verification + +Use the read APIs to examine stored Langfuse data. +You can also use them for bounded exports and ingestion verification. +These methods return a Langfuse response envelope, not model objects. + +## Choose the Right Read + +| Need | SDK method | Langfuse endpoint | +| --- | --- | --- | +| Individual spans, generations, or events | `client.list_observations` | `GET /api/public/v2/observations` | +| Aggregated volume, latency, token, cost, or score data | `client.query_metrics` | `GET /api/public/v2/metrics` | +| Individual typed scores | `client.list_scores` | `GET /api/public/v3/scores` | +| Legacy trace objects | `client.list_traces` / `client.get_trace` | Legacy trace API | + +Use observations to traverse current traces. +Use metrics to calculate an aggregate on the server. +Do not download raw rows to calculate an aggregate. + +The v2 observations and metrics APIs require Langfuse Cloud or self-hosted Langfuse v4. +The SDK sends the v4 ingestion header. +Thus, new SDK traces do not have the legacy ingestion delay. + +## Read Observations + +Broad observation reads must include both time bounds. A trace-specific read is already bounded: + +```ruby +page = Langfuse.client.list_observations( + trace_id: trace_id, + fields: "core,basic,io,model,usage,prompt" +) + +page.fetch("data").each do |observation| + puts "#{observation['type']} #{observation['name']}" +end +``` + +For project-level queries, pass an inclusive start and exclusive end: + +```ruby +page = Langfuse.client.list_observations( + from_start_time: Time.now.utc - 3600, + to_start_time: Time.now.utc, + type: "GENERATION", + environment: ["production"], + fields: "core,basic,model,usage", + limit: 100 +) +``` + +Without `fields`, Langfuse returns only `core` and `basic`. +Request other field groups only when they are necessary. + +### Cursor Pagination + +Pass the cursor from `meta.cursor` into the same query: + +```ruby +rows = [] +cursor = nil + +loop do + page = Langfuse.client.list_observations( + from_start_time: window_start, + to_start_time: window_end, + fields: "core,basic", + limit: 1_000, + cursor: cursor + ) + + rows.concat(page.fetch("data")) + cursor = page.dig("meta", "cursor") + break unless cursor +end +``` + +Keep the filters and time bounds unchanged between pages. + +### Logical Roots + +Use `is_root_observation: true` to select application entry points. +A logical root can have a physical parent. +This condition occurs when an upstream or filtered OpenTelemetry span exists. +Use `parent_observation_id` only when you need the physical tree relationship. + +## Query Metrics + +Metrics queries aggregate server-side. This example counts observations for one trace: + +```ruby +result = Langfuse.client.query_metrics( + query: { + view: "observations", + metrics: [{ measure: "count", aggregation: "count" }], + filters: [ + { column: "traceId", operator: "=", value: trace_id, type: "string" } + ], + fromTimestamp: (Time.now.utc - 3600).iso8601, + toTimestamp: (Time.now.utc + 60).iso8601 + } +) + +count = result.dig("data", 0, "count_count") +``` + +The supported views are `observations`, `scores-numeric`, `scores-boolean`, and `scores-categorical`. +Use high-cardinality values such as trace IDs as filters. +Do not use them as grouping dimensions. + +## Read Scores + +The scores v3 API returns one polymorphic `value` field: + +| `dataType` | Ruby value | +| --- | --- | +| `NUMERIC` | Numeric | +| `BOOLEAN` | `true` or `false` | +| `CATEGORICAL` | String | +| `TEXT` | String | +| `CORRECTION` | String | + +```ruby +page = Langfuse.client.list_scores( + trace_id: trace_id, + data_type: "BOOLEAN,CORRECTION", + fields: "details,subject", + limit: 100 +) + +page.fetch("data").each do |score| + case score.fetch("dataType") + when "BOOLEAN" + puts score.fetch("value") ? "passed" : "failed" + when "CORRECTION" + puts score.fetch("value") + end +end +``` + +Score pages also use `meta.cursor`. Keep the original filters when requesting the next page. + +## Verify an SDK Write End to End + +A reliable verification separates three claims: + +1. The SDK accepted the operation. +2. The SDK exporter or HTTP client completed delivery. +3. An independent API read found the persisted backend record. + +Use unique names or IDs so a previous run cannot satisfy the assertion: + +```ruby +require "securerandom" + +run_id = SecureRandom.hex(6) +trace_id = nil + +Langfuse.observe("sdk-e2e-#{run_id}", input: { run_id: run_id }) do |root| + trace_id = root.trace_id + root.update(output: { status: "passed" }) +end + +Langfuse.force_flush + +deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 30 + +loop do + rows = Langfuse.client.list_observations( + trace_id: trace_id, + fields: "core,basic,io" + ).fetch("data") + + break if rows.any? { |row| row["name"] == "sdk-e2e-#{run_id}" } + raise "Langfuse ingestion timed out" if Process.clock_gettime(Process::CLOCK_MONOTONIC) >= deadline + + sleep 1 +end +``` + +For asynchronous scores, call `Langfuse.flush_scores` before readback. +`create_score!` is synchronous. +An independent read verifies that the backend stored the expected typed value. + +## Verify Through the Langfuse CLI + +The CLI provides an independent client path through the same public APIs: + +```bash +export LANGFUSE_HOST="${LANGFUSE_BASE_URL:-https://cloud.langfuse.com}" + +npx --yes langfuse-cli@latest api observations list \ + --trace-id "$TRACE_ID" \ + --fields core,basic,io \ + --json + +npx --yes langfuse-cli@latest api scores list \ + --trace-id "$TRACE_ID" \ + --fields details,subject \ + --json +``` + +Use `npx --yes langfuse-cli@latest api __schema` to list resources. +Use ` --help` to examine the current contract. + +With `--json`, the CLI returns an envelope with `status`, `headers`, and `body`. +Read observations and scores from `body.data`. +For HTTP `429`, wait for the returned retry interval. + +Do not print API keys or include them in command arguments. Supply them through the environment. + +## See Also + +- [TRACING.md](TRACING.md) — create well-structured observations +- [SCORING.md](SCORING.md) — create scores and choose delivery semantics +- [API_REFERENCE.md](API_REFERENCE.md#data-access) — complete read signatures and filters +- [Langfuse Observations API](https://langfuse.com/docs/api-and-data-platform/features/observations-api) +- [Langfuse Metrics API](https://langfuse.com/docs/metrics/features/metrics-api) +- [Langfuse Scores API](https://langfuse.com/docs/api-and-data-platform/features/scores-api) +- [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli) diff --git a/docs/ERROR_HANDLING.md b/docs/ERROR_HANDLING.md index 890b22c..3b07fe3 100644 --- a/docs/ERROR_HANDLING.md +++ b/docs/ERROR_HANDLING.md @@ -79,25 +79,20 @@ client.get_prompt("greeting") **Solution:** 1. Verify keys in Langfuse UI (Project Settings → API Keys) -2. Check you're using keys from correct project +2. Check that you use keys from the correct project 3. Regenerate keys if compromised -```ruby -# Debug: Print first few chars of keys -config = Langfuse.configuration -puts "Public key: #{config.public_key[0..10]}..." -puts "Secret key: #{config.secret_key[0..10]}..." -``` +Do not print, inspect, or log any part of an API key. Compare the configured project and environment-variable names instead. Use a bounded authenticated read to validate the complete credential pair. ### `Langfuse::NotFoundError` -**Cause:** Requested resource doesn't exist (404 response) +**Cause:** The requested resource does not exist (404 response) **Common scenarios:** - Prompt name misspelled - Prompt not deployed in Langfuse UI -- Requesting specific version that doesn't exist -- Label doesn't exist +- The requested version does not exist +- The label does not exist **Example:** @@ -122,7 +117,7 @@ prompt = client.get_prompt( fallback: "Hello {{name}}!", type: :text ) -# If prompt doesn't exist, uses fallback without error +# If the prompt does not exist, use the fallback without an error ``` **Option 3:** Graceful degradation @@ -257,7 +252,7 @@ The SDK automatically retries certain operations: - Max 2 retries (3 total attempts) - Same error conditions and backoff -You don't need to implement retries for these operations. +You do not need to implement retries for these operations. ### Application-Level Retries @@ -296,8 +291,8 @@ end ``` **Don't retry:** -- `UnauthorizedError` (credentials won't fix themselves) -- `NotFoundError` (resource doesn't exist) +- `UnauthorizedError` (a retry does not correct invalid credentials) +- `NotFoundError` (the resource does not exist) - `ConfigurationError` (code issue, not transient) ## Fallback Patterns @@ -428,9 +423,13 @@ This logs: ```ruby config = Langfuse.configuration -puts config.inspect +puts "Langfuse host: #{config.base_url}" +puts "Langfuse environment: #{config.environment || 'default'}" +puts "Langfuse locally configured: #{Langfuse.configured?}" ``` +Do not call `config.inspect` in logs. The configuration object contains the secret key. + ### Check Cache State ```ruby @@ -444,11 +443,11 @@ puts "Cache enabled: #{stats[:enabled]}" ```ruby begin prompts = Langfuse.client.list_prompts(limit: 1) - puts "✓ Credentials valid, found #{prompts.size} prompt(s)" + puts "OK: Credentials are valid. Found #{prompts.size} prompt(s)." rescue Langfuse::UnauthorizedError - puts "✗ Invalid credentials" + puts "ERROR: Credentials are invalid." rescue Langfuse::ApiError => e - puts "✗ API error: #{e.message}" + puts "ERROR: API request failed: #{e.message}" end ``` diff --git a/docs/GETTING_STARTED.md b/docs/GETTING_STARTED.md index e9f9ed8..66b9f8b 100644 --- a/docs/GETTING_STARTED.md +++ b/docs/GETTING_STARTED.md @@ -1,14 +1,28 @@ -# Getting Started with Langfuse Ruby SDK +# Getting Started -This is the happy path for a new consumer. The goal is simple: configure the SDK once, fetch a real prompt, send a real trace, and know where to go next without guessing how tracing works. +This guide shows how to install the SDK and create a trace. +It also shows how to verify the trace in Langfuse. +You can add prompts and scores after this verification. -## Before You Start +## 1. Prepare the Environment -- Ruby `>= 3.2.0` -- A Langfuse project with API keys -- At least one prompt created in the Langfuse UI +You need Ruby `>= 3.2.0`, a Langfuse project, and project API keys. -## 1. Install the Gem +Set these variables in the process that runs the application: + +```bash +LANGFUSE_PUBLIC_KEY=pk-lf-... +LANGFUSE_SECRET_KEY=sk-lf-... +LANGFUSE_BASE_URL=https://cloud.langfuse.com +``` + +`LANGFUSE_BASE_URL` is optional for Langfuse Cloud. +The SDK does not load `.env` files. +Load a `.env` file before SDK initialization. + +## 2. Install the Gem + +Add the SDK to your `Gemfile`: ```ruby gem "langfuse-rb" @@ -20,76 +34,44 @@ Then install dependencies: bundle install ``` -## 2. Configure Langfuse Once at Startup +## 3. Configure the Singleton Client -Rails first, because that is the common consumer path. - -```ruby -# config/initializers/langfuse.rb -Langfuse.configure do |config| - config.public_key = Rails.application.credentials.dig(:langfuse, :public_key) - config.secret_key = Rails.application.credentials.dig(:langfuse, :secret_key) - config.base_url = ENV.fetch("LANGFUSE_BASE_URL", "https://cloud.langfuse.com") - - config.cache_backend = :rails - config.cache_ttl = 300 - config.cache_stale_ttl = 300 -end -``` - -Plain Ruby uses the same API: +The SDK reads credentials from the environment. +Use `Langfuse.configure` for application settings: ```ruby require "langfuse" Langfuse.configure do |config| - config.public_key = ENV["LANGFUSE_PUBLIC_KEY"] - config.secret_key = ENV["LANGFUSE_SECRET_KEY"] - config.base_url = ENV.fetch("LANGFUSE_BASE_URL", "https://cloud.langfuse.com") + config.environment = ENV.fetch("APP_ENV", "development") + config.release = ENV["APP_RELEASE"] end ``` -`Langfuse.configure` stores configuration only. It does not replace `OpenTelemetry.tracer_provider`. The default onboarding path is isolated tracing through the Langfuse helpers. If you want Langfuse to become the global OpenTelemetry provider, that is an explicit later step in [TRACING.md](TRACING.md#opentelemetry-integration). - -For the full config surface, see [CONFIGURATION.md](CONFIGURATION.md). - -## 3. Fetch and Compile a Prompt +In a Rails application, put this block in `config/initializers/langfuse.rb`. +See [RAILS.md](RAILS.md) for cache and job patterns. -Create a prompt in the Langfuse UI first. For example: +`Langfuse.configure` stores configuration. +It does not replace the process-wide `OpenTelemetry.tracer_provider`. +The helper API uses an isolated Langfuse provider by default. +Read [TRACING.md](TRACING.md#opentelemetry-integration) before you install the provider globally. -- Name: `support-answer` -- Type: chat -- Label: `production` - -Then fetch and compile it in your app: +You can check local client readiness without a network request: ```ruby -prompt = Langfuse.client.get_prompt("support-answer", label: "production") - -messages = prompt.compile( - customer_name: "Alice", - question: "How do I reset my password?" -) +abort "Langfuse configuration is incomplete" unless Langfuse.configured? ``` -If you prefer the one-call version: +This check does not validate credentials. +It also does not prove ingestion. +The tracing helpers use a no-op tracer when the configuration is invalid. +Thus, most request paths do not need this guard. -```ruby -messages = Langfuse.client.compile_prompt( - "support-answer", - label: "production", - variables: { - customer_name: "Alice", - question: "How do I reset my password?" - } -) -``` - -More prompt patterns live in [PROMPTS.md](PROMPTS.md). +## 4. Create a Useful Trace -## 4. Send Your First Real Trace - -Use a root observation for the workflow, then nest the model call as a generation. This is the pattern most consumers actually want. +Use one trace for each self-contained unit of work. +Use stable action names. +Put meaningful input and output on the root observation. ```ruby class SupportAnswerService @@ -100,43 +82,25 @@ class SupportAnswerService def call(user:, question:) Langfuse.propagate_attributes( user_id: user.id.to_s, - session_id: "support-#{user.id}" + session_id: "support-#{user.id}", + tags: ["support"] ) do - Langfuse.observe("support-answer", input: { question: question }) do |root| - prompt = Langfuse.client.get_prompt("support-answer", label: "production") - messages = prompt.compile( - customer_name: user.name, - question: question - ) - - answer = root.start_observation("llm-response", as_type: :generation) do |gen| - gen.model = "gpt-4.1-mini" - gen.input = messages - - response = @llm_client.chat( - parameters: { - model: "gpt-4.1-mini", - messages: messages - } - ) + Langfuse.observe("answer-support-question", input: { question: question }) do |root| + answer = root.start_observation("generate-answer", as_type: :generation) do |generation| + generation.model = "gpt-4.1-mini" + generation.input = { question: question } - answer = response.dig("choices", 0, "message", "content") + response = @llm_client.chat(question) - gen.update( - output: answer, - usage_details: { - prompt_tokens: response.dig("usage", "prompt_tokens"), - completion_tokens: response.dig("usage", "completion_tokens"), - total_tokens: response.dig("usage", "total_tokens") - } + generation.update( + output: response.fetch(:content), + usage_details: response.fetch(:usage) ) - answer + response.fetch(:content) end - root.event(name: "reply-generated", input: { channel: "support" }) root.update(output: { answer: answer }) - answer end end @@ -144,60 +108,122 @@ class SupportAnswerService end ``` -Why this shape matters: +This creates: + +- One root observation for the support workflow. +- One nested generation for the model call. +- Root input and output for trace-level review. +- Model and token information on the generation. +- User, session, environment, and tag values for filters. + +See [TRACING.md](TRACING.md) for events, background jobs, custom trace IDs, masking, and OpenTelemetry ownership. + +## 5. Flush Before Immediate Readback + +The SDK exports data in batches. +A normal process exit flushes pending data automatically. +Long-running applications do not need to flush each request. + +Call `Langfuse.force_flush` at an explicit durability boundary. +Also call it before an immediate verification read: + +```ruby +Langfuse.force_flush +``` + +The normal exit hook cannot run after abrupt termination such as `SIGKILL`. -- The root observation gives you a real trace entrypoint -- The nested generation carries model-specific fields like `model` and `usage_details` -- `root.event(...)` persists a point-in-time annotation on the active observation -- `root.update(...)` persists the final workflow output instead of leaving the trace half-empty +## 6. Verify Backend Ingestion -Plain Ruby is the same pattern without Rails wrappers: +Keep the trace ID returned by the root observation when you need deterministic readback: ```ruby -Langfuse.observe("support-answer", input: { question: question }) do |root| - answer = root.start_observation("llm-response", as_type: :generation) do |gen| - gen.model = "gpt-4.1-mini" - # ... - end +trace_id = nil - root.update(output: { answer: answer }) +Langfuse.observe("verify-sdk", input: { source: "getting-started" }) do |root| + trace_id = root.trace_id + root.update(output: { status: "ok" }) end + +Langfuse.force_flush + +rows = Langfuse.client.list_observations( + trace_id: trace_id, + fields: "core,basic,io" +).fetch("data") + +raise "trace was not ingested" unless rows.any? { |row| row["name"] == "verify-sdk" } +``` + +Ingestion can have a short delay. +A deployment probe must retry the bounded read. +Do not use an unbounded project scan. + +The Langfuse CLI provides an independent read path: + +```bash +export LANGFUSE_HOST="${LANGFUSE_BASE_URL:-https://cloud.langfuse.com}" +npx --yes langfuse-cli@latest api observations list \ + --trace-id "$TRACE_ID" \ + --fields core,basic,io \ + --json ``` -For deeper tracing patterns, see [TRACING.md](TRACING.md). +CLI JSON output contains `status`, `headers`, and `body`. +Observation rows are in `body.data`. +See [DATA_ACCESS.md](DATA_ACCESS.md) for pagination, metrics, score reads, and a complete verification procedure. + +## 7. Add a Managed Prompt -## 5. Verify It Worked +Create a prompt in Langfuse. +Then, fetch the prompt by a stable label: -After running the code: +```ruby +prompt = Langfuse.client.get_prompt("support-answer", label: "production") -1. Open the Langfuse UI. -2. Find the `support-answer` trace. -3. Confirm you can see: - - the root observation input and output - - the nested `llm-response` generation - - usage details on the generation - - the `reply-generated` event +expected_variables = ["customer.name", "question"] +raise "prompt contract changed" unless prompt.variables == expected_variables -If you do not see traces, start with [ERROR_HANDLING.md](ERROR_HANDLING.md) and the Rails operational checks in [RAILS.md](RAILS.md#troubleshooting). +messages = prompt.compile( + customer: { name: "Alice" }, + question: "How do I reset my password?" +) +``` -## 6. Add Scores Once Traces Exist +See [PROMPTS.md](PROMPTS.md) for text and chat prompts, message placeholders, fallbacks, versioning, and caching. -Do not invent a scoring workflow before the trace is working. First make the trace visible, then attach evaluation or feedback signals. +## 8. Add Scores Deliberately -Example: +Use asynchronous score creation for inline telemetry: ```ruby -Langfuse.observe("support-answer") do |root| - # ... do work ... - root.score_trace(name: "customer-satisfaction", value: 5) -end +Langfuse.create_score( + name: "helpful", + value: true, + trace_id: trace_id, + data_type: :boolean +) +``` + +Use synchronous score creation when the caller needs delivery confirmation: + +```ruby +score_id = Langfuse.create_score!( + id: "feedback-#{feedback.id}", + name: "helpful", + value: true, + trace_id: trace_id, + data_type: :boolean +) ``` -Scoring details live in [SCORING.md](SCORING.md). +See [SCORING.md](SCORING.md) for delivery semantics, score types, environment inheritance, batching, and idempotency. -## What to Read Next +## Next Steps -- [PROMPTS.md](PROMPTS.md) if prompt versioning and fallbacks are your next problem -- [TRACING.md](TRACING.md) if you need nested workflows, events, jobs, or OpenTelemetry integration -- [SCORING.md](SCORING.md) if you want feedback or eval signals on traces -- [RAILS.md](RAILS.md) if you are wiring this into controllers, services, or background jobs +- [Prompt Management](PROMPTS.md) for managed prompt workflows +- [Tracing](TRACING.md) for trace design and OpenTelemetry integration +- [Scoring](SCORING.md) for evaluation and feedback +- [Data Access](DATA_ACCESS.md) for SDK and CLI queries +- [Configuration](CONFIGURATION.md) for production controls +- [Testing Tracing](TESTING.md) for network-free span assertions diff --git a/docs/PROMPTS.md b/docs/PROMPTS.md index 7e42f6e..bbf4ff7 100644 --- a/docs/PROMPTS.md +++ b/docs/PROMPTS.md @@ -4,7 +4,8 @@ Complete guide to managing LLM prompts with Langfuse. ## Overview -Langfuse centralizes prompt management, allowing you to: +Langfuse provides central prompt management. +You can: - Version and iterate prompts without code changes - A/B test prompt variations - Roll back to previous versions @@ -20,6 +21,7 @@ The SDK supports two prompt types: | Method | Description | |--------|-------------| | `get_prompt(name)` | Fetch a prompt by name | +| `prompt.variables` | List the template variables a caller must provide | | `compile_prompt(name, variables:)` | Fetch and compile in one call | | `create_prompt(name:, prompt:, type:)` | Create a new prompt or version | | `update_prompt(name:, version:, labels:)` | Update labels on a version | @@ -45,6 +47,7 @@ prompt.tags # => ["marketing", "seo"] prompt.config # => { "temperature" => 0.7, "model" => "gpt-4" } prompt.prompt # => "Write a {{tone}} product description for {{product_name}}..." prompt.type # => "text" +prompt.variables # => ["tone", "product_name"] prompt.commit_message # => "Improve product tone" prompt.resolution_graph # => nil prompt.is_fallback # => false @@ -67,6 +70,40 @@ puts description # => "Write a professional product description for Wireless Headphones..." ``` +### Inspecting Required Variables + +Call `variables` before compilation to validate a prompt contract. +You can also use it to generate a form or find an incompatible prompt change: + +```ruby +prompt = client.get_prompt("support-answer", label: "production") + +prompt.variables +# => ["customer.name", "question", "context"] +``` + +The method parses the Mustache template. +It does not use a text search: + +- Names stay in the order of their first occurrence. +- Each duplicate name occurs one time. +- Dotted names such as `customer.name` do not change. +- The result includes section and inverted-section names. +- Variables in a section include the full section scope. +- The result does not include Mustache comments. +- The result does not include chat message placeholders. +- Invalid Mustache syntax raises `Mustache::Parser::SyntaxError`. + +For chat prompts, the SDK examines the content in message order. +It returns one list without duplicate names: + +```ruby +prompt = client.get_prompt("support-chat", label: "production") +prompt.variables # => ["agent.role", "customer.name", "question"] +``` + +The method reports referenced variables. It does not determine whether a Mustache section is optional for a specific call. + ### Using Metadata The `config` hash stores prompt-specific settings: @@ -475,7 +512,7 @@ message = client.compile_prompt( ### `list_prompts` - Browse Available Prompts -List all prompts in your project: +List all prompts in the project: ```ruby prompts = client.list_prompts @@ -492,7 +529,7 @@ prompts = client.list_prompts(page: 2, limit: 50) ## Fallback Handling -Provide a fallback template for development or when prompts don't exist: +Provide a fallback template for development or when a prompt does not exist: ```ruby prompt = client.get_prompt( @@ -502,7 +539,7 @@ prompt = client.get_prompt( ) message = prompt.compile(name: "Bob") -# If prompt doesn't exist in Langfuse, uses fallback +# If the prompt does not exist in Langfuse, use the fallback ``` **Important:** You must specify `type:` when using `fallback`. @@ -598,7 +635,7 @@ Langfuse.observe("generate-response", as_type: :generation) do |gen| } ) - # Record prompt metadata in trace + # Record generation data and link the managed prompt version gen.model = prompt.config["model"] gen.input = messages gen.output = response.dig("choices", 0, "message", "content") @@ -607,17 +644,14 @@ Langfuse.observe("generate-response", as_type: :generation) do |gen| completion_tokens: response.dig("usage", "completion_tokens"), total_tokens: response.dig("usage", "total_tokens") } - gen.metadata = { - prompt_name: prompt.name, - prompt_version: prompt.version, - prompt_labels: prompt.labels - } + gen.update(prompt: prompt) response end ``` -This creates a trace with full prompt provenance, making it easy to correlate outputs with specific prompt versions. +The trace contains the prompt version and other provenance data. +Use this data to correlate outputs with prompt versions. See [TRACING.md](TRACING.md) for more tracing patterns. @@ -628,3 +662,4 @@ See [TRACING.md](TRACING.md) for more tracing patterns. - [CACHING.md](CACHING.md) - Optimizing prompt fetch performance - [ERROR_HANDLING.md](ERROR_HANDLING.md) - Handling prompt errors - [API_REFERENCE.md](API_REFERENCE.md) - Complete method signatures for all prompt methods +- [Langfuse prompt variables](https://langfuse.com/docs/prompt-management/features/variables) - Platform variable concepts diff --git a/docs/RAILS.md b/docs/RAILS.md index 498c134..44b87e9 100644 --- a/docs/RAILS.md +++ b/docs/RAILS.md @@ -1,6 +1,7 @@ # Rails Integration Guide -This guide assumes you already read [GETTING_STARTED.md](GETTING_STARTED.md). It is for applied Rails patterns, not basic setup repetition. +Read [GETTING_STARTED.md](GETTING_STARTED.md) before you use this guide. +This guide contains Rails integration patterns. ## Initializer Pattern @@ -18,17 +19,15 @@ Langfuse.configure do |config| config.cache_stale_ttl = Rails.env.production? ? 300 : 0 config.logger = Rails.logger end - -at_exit do - Langfuse.shutdown(timeout: 10) -end ``` Notes: - `Langfuse.configure` stores config only - module-level tracing works without replacing the global `OpenTelemetry.tracer_provider` -- if you do choose a global install with `Langfuse.tracer_provider`, that is a separate explicit step and you own its lifecycle +- If you install `Langfuse.tracer_provider` globally, the application controls its lifecycle. +- the SDK flushes pending traces and scores during normal process exit; do not register another `at_exit` hook +- the SDK resets its queues and background workers in forked Puma, Unicorn, or Resque children ## Controller Pattern @@ -116,7 +115,7 @@ This keeps the trace shape honest: ## Background Jobs -Jobs are where people usually lie to themselves about propagation. Rails does not continue Langfuse trace context across processes for you. +Rails does not continue Langfuse trace context across job processes automatically. Pass the required context explicitly. ### Enqueue with Explicit Trace Context @@ -165,7 +164,9 @@ class ProcessDocumentJob < ApplicationJob end ``` -Passing `trace_id` is the pragmatic default. It rejoins the same trace, but it does not restore an exact parent span relationship across process boundaries. If you need that, carry and restore OpenTelemetry context yourself. +Pass `trace_id` to continue the same trace. +This value does not restore an exact parent span relationship across process boundaries. +For that relationship, pass and restore OpenTelemetry context in the application. ## Testing @@ -210,7 +211,7 @@ For SDK integration coverage, prefer the repo's existing WebMock/VCR patterns in ### Cache Settings -For multi-process Rails deployments, `cache_backend = :rails` is usually the right default if `Rails.cache` is already backed by Redis. +For a multi-process Rails deployment, use `cache_backend = :rails` when Redis supplies `Rails.cache`. ```ruby Langfuse.configure do |config| @@ -238,15 +239,15 @@ prompt = Langfuse.client.get_prompt( ### Global OTel Install -If you install `Langfuse.tracer_provider` as the global provider, remember the lifecycle contract: +If you install `Langfuse.tracer_provider` as the global provider, apply this lifecycle contract: - `Langfuse.reset!` tears down the internal provider - `Langfuse.shutdown` shuts it down -- after either one, you must reinstall the provider yourself if the app still expects it globally +- After either call, install the provider again if the application still requires it globally. ## Operational Debugging -Turn logging up when you need to see configuration and export behavior: +Set the log level to `DEBUG` to examine configuration and export behavior: ```ruby Langfuse.configure do |config| @@ -269,9 +270,9 @@ Langfuse.client.prompt_cache_stats The usual problem is stale cache, not a broken prompt API. 1. Wait for `cache_ttl` to expire. -2. Refresh the specific prompt if you need fresh state now. +2. Refresh the specific prompt when the application needs the current state. 3. Invalidate the prompt name or clear the prompt cache namespace. -4. Lower `cache_ttl` in development if you are iterating quickly. +4. Lower `cache_ttl` in development when you test prompt changes. ```ruby Langfuse.client.refresh_prompt("greeting", label: "production") @@ -292,9 +293,12 @@ Langfuse.configuration.base_url.present? Then check the ownership assumption: - `Langfuse.observe(...)` should work once Langfuse is configured -- third-party ambient OpenTelemetry spans do not go to Langfuse unless you explicitly install `Langfuse.tracer_provider` +- The SDK does not export third-party ambient OpenTelemetry spans unless you install `Langfuse.tracer_provider`. - `should_export_span` only runs for spans handled by Langfuse's provider ### Unexpected Spans After Global Install -That means you chose the global-provider path and now Langfuse is seeing application-wide spans. That is expected. Narrow the export path with `should_export_span` if needed, but do not confuse that with the isolated default behavior. +This result occurs when the application installs the Langfuse provider globally. +In this mode, Langfuse receives application-wide spans. +Use `should_export_span` to reduce this export set. +The isolated default mode does not export these spans. diff --git a/docs/README.md b/docs/README.md index 1fed643..30cc723 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,44 +1,52 @@ # Langfuse Ruby SDK Documentation -This is the consumer hub. Start here unless you are already looking for a specific reference page. +Use this page to find the correct guide for your task. +The guides explain workflows and limits. +[API_REFERENCE.md](API_REFERENCE.md) contains the exact public method signatures. ## Start Here -1. **[Getting Started](GETTING_STARTED.md)** — Rails-first first run: install, configure, fetch a prompt, send a real trace -2. **[Prompts](PROMPTS.md)** — Fetch, compile, version, and fall back safely -3. **[Tracing](TRACING.md)** — Root observations, nested generations, events, propagation, and OpenTelemetry ownership -4. **[Scoring](SCORING.md)** — Add evaluation and feedback signals to traces and observations -5. **[Rails](RAILS.md)** — Applied controller, service, job, testing, and operational patterns -6. **[Testing tracing](TESTING.md)** — In-memory exporter recipes and lifecycle constraints - -## By Intent - -### First Run - -- **[Getting Started](GETTING_STARTED.md)** — The shortest path from zero to a visible prompt + trace -- **[Prompts](PROMPTS.md)** — The next thing most consumers need after installation -- **[Tracing](TRACING.md)** — The actual tracing lifecycle, without the hand-wavy OpenTelemetry claims - -### Instrument an App - -- **[Tracing](TRACING.md)** — Observation hierarchy, propagation, background jobs, explicit global install -- **[Rails](RAILS.md)** — Rails-specific patterns for controllers, services, jobs, and tests -- **[Scoring](SCORING.md)** — Capture quality signals after a trace exists - -### Production Hardening - -- **[Configuration](CONFIGURATION.md)** — Config surface, tracing ownership, export filtering, environment defaults -- **[Caching](CACHING.md)** — Prompt cache backends, stale-while-revalidate, cache warming -- **[Error Handling](ERROR_HANDLING.md)** — Failure modes, retry boundaries, debugging -- **[Migration Guide](MIGRATION.md)** — Move hardcoded prompts into Langfuse-managed prompts without breaking runtime behavior - -### Evaluation - -- **[Datasets](DATASETS.md)** — Dataset primitives and management -- **[Experiments](EXPERIMENTS.md)** — Experiment runner workflows - -### Reference - -- **[API Reference](API_REFERENCE.md)** — Exact public signatures and types -- **[Configuration](CONFIGURATION.md)** — Option-by-option config reference -- **[Architecture](ARCHITECTURE.md)** — Implementation and internal design reference, not required for the first run +1. [Getting Started](GETTING_STARTED.md) — install, configure, create a useful trace, and verify backend ingestion +2. [Prompt Management](PROMPTS.md) — fetch, inspect, compile, version, and cache prompts +3. [Tracing](TRACING.md) — model observation trees, propagate context, mask data, and integrate OpenTelemetry +4. [Scoring](SCORING.md) — attach synchronous or asynchronous evaluation and feedback signals +5. [Data Access](DATA_ACCESS.md) — query current observations, metrics, and scores through the SDK or CLI + +## Guides by Task + +| Task | Canonical guide | +| --- | --- | +| Configure keys, batching, sampling, masking, exporters, or telemetry controls | [Configuration](CONFIGURATION.md) | +| Add tracing to a workflow | [Tracing](TRACING.md) | +| Verify newly exported records | [Data Access](DATA_ACCESS.md) | +| Fetch or compile managed prompts | [Prompt Management](PROMPTS.md) | +| Tune prompt caching or stale-while-revalidate | [Caching](CACHING.md) | +| Record user feedback or evaluation results | [Scoring](SCORING.md) | +| Build dataset-backed evaluations | [Datasets](DATASETS.md) and [Experiments](EXPERIMENTS.md) | +| Integrate controllers, services, and jobs | [Rails](RAILS.md) | +| Test trace output without network access | [Testing Tracing](TESTING.md) | +| Diagnose configuration, API, or cache failures | [Error Handling](ERROR_HANDLING.md) | +| Move hardcoded prompts into Langfuse | [Migration](MIGRATION.md) | + +## Reference + +- [API Reference](API_REFERENCE.md) — exact methods, parameters, return values, and exceptions +- [Configuration](CONFIGURATION.md) — option and environment-variable reference +- [Architecture](ARCHITECTURE.md) — contributor-facing components, ownership, and data flow +- [Changelog](../CHANGELOG.md) — release behavior changes + +## Production Checklist + +- Use one trace for each self-contained unit of work. +- Use stable, action-oriented observation names. +- Put meaningful input and output on the root observation. +- Record model, usage, and prompt details on generation observations. +- Set `environment` to keep development and staging records out of production analysis. +- Configure masking before tracing starts when payloads can contain sensitive data. +- Select asynchronous `create_score` or synchronous `create_score!` for the required delivery contract. +- Use bounded observation reads and cursor pagination for data extraction. +- Verify one real trace and score in the target Langfuse project before deployment. + +A normal process exit flushes pending data. +The SDK resets background workers after Ruby `fork`. +An abrupt termination such as `SIGKILL` cannot run flush callbacks. diff --git a/docs/SCORING.md b/docs/SCORING.md index 6ce08bf..088cee0 100644 --- a/docs/SCORING.md +++ b/docs/SCORING.md @@ -4,15 +4,41 @@ Add quality scores to your traces and observations for evaluation and analytics. ## Overview -Scores let you evaluate LLM outputs: +Use scores to evaluate LLM output: + - **Human feedback:** User thumbs up/down, star ratings - **Automated metrics:** Accuracy, relevance, safety checks - **A/B testing:** Compare prompt/model performance Scores can be attached to: + - Entire traces (end-to-end quality) - Individual observations (LLM call quality) +## Choose a Delivery Mode + +The SDK provides two score creation contracts: + +| Method | Delivery | Return value | Best for | +| --- | --- | --- | --- | +| `create_score` | Asynchronous ingestion queue | `nil` | Inline telemetry and high-volume evaluation | +| `create_score!` | Synchronous Scores API request | Created score ID | User feedback, durable verdicts, and workflows that must observe API failure | + +`create_score` confirms that the SDK accepted the score into its bounded queue. +It does not confirm backend storage. +`create_score!` confirms that Langfuse accepted the HTTP request. +For retryable synchronous work, use a stable `id`. +After an uncertain network failure, use the same complete payload for the retry. + +Both methods use the same score validation and environment order. +An explicit score `environment` has first priority. +`config.environment` has second priority. +Otherwise, Langfuse uses its `default` environment. + +When `tracing_enabled` is false, both methods are no-ops. +In this mode, `create_score!` returns `nil`. +`OTEL_SDK_DISABLED=true` does not disable scores. + ## Score Data Types ### Numeric @@ -184,6 +210,18 @@ Langfuse.create_score( ) ``` +Use the synchronous module-level method when delivery is part of the caller's contract: + +```ruby +score_id = Langfuse.create_score!( + id: "feedback-#{feedback.id}", + name: "user_feedback", + value: true, + trace_id: trace_id, + data_type: :boolean +) +``` + ### Scoring Active Observations Score the currently active observation (from OpenTelemetry context): @@ -203,7 +241,7 @@ Langfuse.observe("generate-summary", as_type: :generation) do |gen| end ``` -This is useful when you don't have the observation ID but want to score from within the traced block. +Use this method when the traced block does not have an observation ID. ### Scoring Active Traces @@ -403,7 +441,14 @@ Langfuse.configure do |config| end ``` -The asynchronous queue is bounded. If the queue is full, the SDK logs an error, drops the new score, and returns without waiting for capacity. The SDK splits flush requests before a multi-score JSON payload exceeds 2.5 MB. A failed batch stays at the front of the queue for a later flush. Use `create_score!` when the caller needs synchronous delivery and an API error result. +The asynchronous queue has a fixed capacity. +If the queue is full, the SDK logs an error and drops the new score. +The call does not wait for capacity. +The SDK keeps each multi-score JSON payload below 2.5 MB. +Retryable failures keep the batch at the front of the queue. +The SDK logs and discards permanent batch failures. +Thus, the SDK can send later valid scores. +Use `create_score!` when the caller needs synchronous delivery and an API error result. **Manual flush:** @@ -414,13 +459,35 @@ Langfuse.create_score(name: "critical", value: 1, trace_id: "abc", data_type: :n Langfuse.flush_scores # Send immediately ``` -Use before shutdown: +Use a manual flush before an immediate readback. +Also use it at an explicit durability boundary: ```ruby -# Before process exit Langfuse.flush_scores ``` +A normal process exit flushes pending scores automatically. +Do not flush on every request. +An abrupt termination such as `SIGKILL` cannot run the exit hook. + +## Reading Scores + +Use `client.list_scores` for typed score records and cursor pagination: + +```ruby +page = Langfuse.client.list_scores( + trace_id: trace_id, + data_type: "BOOLEAN,CORRECTION", + fields: "details,subject" +) + +page.fetch("data").each do |score| + puts "#{score['dataType']}: #{score['value'].inspect}" +end +``` + +See [DATA_ACCESS.md](DATA_ACCESS.md) for the v3 response contract, independent CLI readback, and end-to-end verification. + ## Sampling Behavior Trace-linked scores follow the same deterministic `sample_rate` decision as traces. @@ -485,7 +552,7 @@ Langfuse.observe("process-order", trace_id: trace_id) do |obs| obs.update(output: result) end -# Later (background job, different service) — recompute, don't look up +# Later, in a background job or different service, calculate the ID again trace_id = Langfuse.create_trace_id(seed: "order-#{order.id}") Langfuse.create_score( @@ -498,7 +565,7 @@ Langfuse.create_score( #### Alternative: Store the Trace ID -If you don't have a stable external identifier, capture and persist the trace ID: +If you do not have a stable external identifier, capture and store the trace ID: ```ruby # During request @@ -559,3 +626,4 @@ Langfuse.create_score( - [TRACING.md](TRACING.md) - Creating observations to score - [API_REFERENCE.md](API_REFERENCE.md) - Complete scoring API reference - [CONFIGURATION.md](CONFIGURATION.md) - Batch configuration +- [DATA_ACCESS.md](DATA_ACCESS.md) - Read and verify persisted scores diff --git a/docs/TESTING.md b/docs/TESTING.md index 52c207a..a87938a 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -9,6 +9,8 @@ send trace data over HTTP. Create one exporter for the test process. Configure it before tracing starts. Dummy API keys are still required because tracing validates all connection settings. +Ensure the test process does not inherit `LANGFUSE_TRACING_ENABLED=false` or +`OTEL_SDK_DISABLED=true`; either setting prevents the exporter from receiving spans. ```ruby # spec/support/langfuse.rb diff --git a/docs/TRACING.md b/docs/TRACING.md index b8107c0..0f70dbc 100644 --- a/docs/TRACING.md +++ b/docs/TRACING.md @@ -1,14 +1,41 @@ # LLM Tracing Guide -This guide is about the tracing behavior the SDK actually implements today. If you just need install and first-run setup, start with [GETTING_STARTED.md](GETTING_STARTED.md). +This guide explains the current SDK tracing behavior. +For installation and initial configuration, start with [GETTING_STARTED.md](GETTING_STARTED.md). ## Mental Model -- A root observation becomes the root of a trace. You do not create traces separately. +- A root observation becomes the root of a trace. +- Do not create a trace separately. - Child observations create nested spans inside that trace. -- `:generation` is the right type for model calls because it carries model-specific fields like `model`, `usage_details`, and `cost_details`. +- Use `:generation` for model calls. +- A generation can contain `model`, `usage_details`, and `cost_details`. - `:event` is a point-in-time observation with no duration. -- `Langfuse.configure` stores configuration only. Module-level tracing uses Langfuse's internal tracer provider when tracing is ready. +- `Langfuse.configure` stores configuration only. +- Module-level tracing uses the internal Langfuse tracer provider when tracing is ready. + +## Design the Trace Before Instrumenting It + +Use a stable trace data contract for diagnostics, dashboards, evaluations, and experiments: + +- Use one trace for each self-contained unit of work. +- Use a shared `session_id` for a sequence of related traces. +- Use stable, action-oriented names such as `retrieve-context` and `generate-answer`. +- Do not put IDs, retry numbers, or model names in observation names. +- Put the workflow input and final output on the root observation. +- Use the most specific child type. +- Use `:generation` for model calls and `:retriever` for retrievals. +- Use `:tool` for tools and `:agent` for agent executions. +- Nest tool and generation work under the workflow or agent that owns it. +- Record model and token usage on generations. +- Link the managed prompt when you use one. +- Put request IDs, feature flags, and diagnostic context in metadata. +- Use tags for stable business dimensions. +- Set `environment`. +- Mask sensitive data before export. + +Treat names as durable API identifiers. +If you change a name, saved views, metric queries, and evaluators can stop matching the observation. ## Start with a Root Observation @@ -27,22 +54,23 @@ result = Langfuse.observe("draft-summary", input: { document_id: document.id }) end ``` -This pattern does three things: +The root observation: -- creates the trace entrypoint -- gives you a place to persist workflow-level output -- gives child work somewhere correct to hang +- Creates the trace entry point. +- Stores workflow-level output. +- Provides a parent for child work. ## Nest Generations Inside the Workflow -Use a child generation for the actual model call instead of stuffing everything into one giant root span. +Use a child generation for the model call. +Do not put all model data in the root span. ```ruby Langfuse.observe("support-answer", input: { question: question }) do |root| prompt = Langfuse.client.get_prompt("support-answer", label: "production") messages = prompt.compile(customer_name: user.name, question: question) - answer = root.start_observation("openai-chat", as_type: :generation) do |gen| + answer = root.start_observation("openai-chat", { prompt: prompt }, as_type: :generation) do |gen| gen.model = "gpt-4.1-mini" gen.input = messages gen.model_parameters = { temperature: 0.2 } @@ -73,7 +101,8 @@ Langfuse.observe("support-answer", input: { question: question }) do |root| end ``` -That shape is better than a flat one-span trace because it keeps workflow state on the root and model-specific state on the generation where Langfuse expects it. +This shape keeps workflow state on the root observation. +It keeps model-specific state on the generation observation. ## Record Events That Actually Persist Payloads @@ -133,11 +162,13 @@ Important boundaries: - it also applies to spans created after the propagation block starts - it does not retroactively rewrite spans that already ended -If you need cross-service propagation via OpenTelemetry baggage, use `as_baggage: true` and make sure the host app has the baggage gem and its own header propagation pipeline configured. +For cross-service propagation through OpenTelemetry baggage, use `as_baggage: true`. +The application must also configure the baggage gem and its header propagation pipeline. ## Background Jobs and Async Work -ActiveJob or Sidekiq does not magically continue Langfuse trace context across processes. You need to pass something explicit. +ActiveJob and Sidekiq do not continue Langfuse trace context across processes. +Pass the required context explicitly. ### Good Default: Pass the `trace_id` @@ -231,18 +262,20 @@ Use `Langfuse.propagate_attributes` for trace fields that must be available on c The exporter sends each completed span once. Do not reuse an observation ID to send an update after export. -There are three states. Keep them separate in your head or you will misconfigure this. +The SDK has three OpenTelemetry ownership states. +Configure each state explicitly. ### 1. Default Isolated Langfuse Tracing -This is the default behavior: +The SDK has this default behavior: -- `Langfuse.configure` does not mutate `OpenTelemetry.tracer_provider` -- `Langfuse.configure` does not mutate `OpenTelemetry.propagation` -- `Langfuse.observe(...)` uses Langfuse's internal tracer provider once tracing is configured -- if tracing config is incomplete, module-level tracing falls back to a no-op tracer and logs one warning +- `Langfuse.configure` does not change `OpenTelemetry.tracer_provider`. +- `Langfuse.configure` does not change `OpenTelemetry.propagation`. +- `Langfuse.observe(...)` uses the internal Langfuse tracer provider after tracing configuration. +- If tracing configuration is incomplete, module-level tracing uses a no-op tracer. +- The SDK logs one warning for the incomplete tracing configuration. -This is why ambient spans from some unrelated global OpenTelemetry provider are not exported to Langfuse by default. +Thus, the SDK does not export ambient spans from an unrelated global OpenTelemetry provider by default. ### 2. Explicit Global Install with `Langfuse.tracer_provider` @@ -257,32 +290,56 @@ end OpenTelemetry.tracer_provider = Langfuse.tracer_provider ``` -That is an ownership decision, not a free convenience: +This configuration makes Langfuse the owner of the global provider: -- spans created through the global provider now run through Langfuse's provider -- `should_export_span` now applies to those spans because Langfuse is actually processing them -- if you call `Langfuse.shutdown` or `Langfuse.reset!`, you own reinstalling the provider afterward +- Spans from the global provider now use the Langfuse provider. +- `should_export_span` applies to the spans that Langfuse processes. +- After `Langfuse.shutdown` or `Langfuse.reset!`, the application must install the provider again. -If your app also needs W3C propagation or baggage propagation, configure `OpenTelemetry.propagation` yourself. Langfuse does not install that for you. +If the application needs W3C or baggage propagation, configure `OpenTelemetry.propagation` in the application. +Langfuse does not install propagation. ### 3. Additional OpenTelemetry Backends Are Application-Owned Langfuse does not automatically configure multi-destination OpenTelemetry export. -If you want Langfuse plus another OTel backend, wire that in explicitly in your application: +To use Langfuse with another OpenTelemetry backend, configure both backends in the application: -- add more processors/exporters to the provider you own -- or build and manage your own provider pipeline +- Add more processors or exporters to the application-owned provider. +- Or, build and manage an application-owned provider pipeline. -What Langfuse will not do for you: +Langfuse does not: -- replace your app's exporter topology automatically -- install a second backend by default -- infer whether you want multi-export just because you called `Langfuse.configure` +- Replace the application exporter topology automatically. +- Install a second backend by default. +- Enable export to multiple backends after a `Langfuse.configure` call. + +## Operational Lifecycle + +The SDK controls its internal tracing and scoring lifecycle: + +- Invalid tracing configuration logs one warning and uses a no-op tracer. +- `Langfuse.configured?` checks local client readiness without network access. +- `LANGFUSE_TRACING_ENABLED=false` disables Langfuse tracing and scoring. +- `OTEL_SDK_DISABLED=true` disables trace export. +- `OTEL_SDK_DISABLED=true` does not disable score, prompt, or read APIs. +- A normal process exit flushes pending traces and scores. +- Queues and background workers reset in child processes from the Ruby `fork` path. + +Do not add a second `at_exit` callback. +Use `Langfuse.shutdown` for an earlier shutdown. +Use `Langfuse.force_flush` before immediate readback. +An abrupt termination such as `SIGKILL` cannot flush buffered data. + +For tests, inject `config.span_exporter` before tracing starts. +For batch processor telemetry, provide a fast and thread-safe `config.metrics_reporter`. +The application controls the reporter lifecycle. +Langfuse controls the configured exporter lifecycle. +See [CONFIGURATION.md](CONFIGURATION.md) and [TESTING.md](TESTING.md). ## Export Filtering -`config.should_export_span` is a filter on spans handled by Langfuse's provider. That is it. +`config.should_export_span` filters spans that the Langfuse provider processes. The SDK calls the filter once after each span finishes. The callback can use the span's final attributes. @@ -299,9 +356,11 @@ Langfuse.configure do |config| end ``` -Use it when you want to narrow what Langfuse exports after Langfuse owns the provider path. +Use the filter to reduce the spans that the Langfuse provider exports. -Do not pretend filtering is the fix for ambient-span overcapture. Isolation is the fix. The default isolated setup already prevents random global spans from leaking into Langfuse. +Export filtering does not prevent ambient-span overcapture. +Use provider isolation for this purpose. +The default isolated configuration does not export unrelated global spans to Langfuse. Public helper predicates: @@ -316,15 +375,21 @@ The exact signatures live in [API_REFERENCE.md](API_REFERENCE.md). ## Best Practices - Put workflow-level output on the root observation and model-level output on the generation. -- Capture `usage_details` on every generation you care about. +- Capture `usage_details` on each applicable generation. - Use descriptive observation names tied to real workflow steps. - Use `Langfuse.propagate_attributes` early, before you start child observations. - Keep `should_export_span` allocation-light and side-effect free. - Add scores only after you have a stable trace flow. See [SCORING.md](SCORING.md). +After you instrument a path, execute the path. +Flush data at the verification boundary. +Then, fetch the stored observations. +See [DATA_ACCESS.md](DATA_ACCESS.md) for SDK reads and independent Langfuse CLI verification. + ## Masking -If inputs or outputs contain sensitive data, configure `mask` and let the SDK redact values before serialization: +If input or output contains sensitive data, configure `mask`. +The SDK changes the selected values before serialization: ```ruby Langfuse.configure do |config| @@ -376,7 +441,16 @@ batch unchanged. The hook receives `Langfuse::MaskOtelSpansParams` and returns `Langfuse::MaskOtelSpansResult` with typed sparse patches. Errors fail closed: the batch, span, or invalid attribute is omitted rather than exported unmasked. -Only the Langfuse export copy is transformed. If you also send telemetry to -another OpenTelemetry backend, that backend receives the original spans and needs -its own masking. The two hooks are independent; applications may configure either -or both. The full contract is in [CONFIGURATION.md](CONFIGURATION.md#mask_otel_spans). +The SDK changes only the Langfuse export copy. +Another OpenTelemetry backend receives the original spans. +Configure separate masking for that backend. +The two hooks are independent. +An application can configure one hook or both hooks. +The full contract is in [CONFIGURATION.md](CONFIGURATION.md#mask_otel_spans). + +## See Also + +- [DATA_ACCESS.md](DATA_ACCESS.md) — verify traces and query observations or metrics +- [CONFIGURATION.md](CONFIGURATION.md) — tracing controls, batching, exporters, and masking contracts +- [SCORING.md](SCORING.md) — attach evaluation and feedback signals +- [Langfuse trace best practices](https://langfuse.com/docs/observability/best-practices) — platform guidance for trace structure diff --git a/lib/langfuse.rb b/lib/langfuse.rb index a586ab0..b6cb6b3 100644 --- a/lib/langfuse.rb +++ b/lib/langfuse.rb @@ -179,8 +179,8 @@ def tracer_provider # Shutdown Langfuse and flush any pending traces and scores # - # Call this when shutting down your application to ensure - # all traces and scores are sent to Langfuse. + # Normal process exit calls this automatically. Use it directly when the + # application needs an earlier, explicit shutdown boundary. # # @param timeout [Integer] Timeout in seconds # @return [void] diff --git a/lib/langfuse/chat_prompt_client.rb b/lib/langfuse/chat_prompt_client.rb index 23dffe8..7743c73 100644 --- a/lib/langfuse/chat_prompt_client.rb +++ b/lib/langfuse/chat_prompt_client.rb @@ -12,7 +12,7 @@ module Langfuse # @example Basic usage # prompt_data = api_client.get_prompt("support_chat") # chat_prompt = Langfuse::ChatPromptClient.new(prompt_data) - # chat_prompt.compile(variables: { user_name: "Alice", issue: "login" }) + # chat_prompt.compile(user_name: "Alice", issue: "login") # # => [{ role: "system", content: "You are a support agent..." }, ...] # # @example Accessing metadata diff --git a/lib/langfuse/config.rb b/lib/langfuse/config.rb index cd4a337..4919b74 100644 --- a/lib/langfuse/config.rb +++ b/lib/langfuse/config.rb @@ -79,7 +79,7 @@ class Config # @return [Symbol] Reserved no-op queue name for future async job integration attr_accessor :job_queue - # @return [String, nil] Default tracing environment applied to new traces/observations + # @return [String, nil] Default environment applied to traces, observations, and scores attr_accessor :environment # @return [String, nil] Default release identifier applied to new traces/observations diff --git a/lib/langfuse/text_prompt_client.rb b/lib/langfuse/text_prompt_client.rb index e9d1348..2254a37 100644 --- a/lib/langfuse/text_prompt_client.rb +++ b/lib/langfuse/text_prompt_client.rb @@ -12,7 +12,7 @@ module Langfuse # @example Basic usage # prompt_data = api_client.get_prompt("greeting") # text_prompt = Langfuse::TextPromptClient.new(prompt_data) - # text_prompt.compile(variables: { name: "Alice" }) + # text_prompt.compile(name: "Alice") # # => "Hello Alice!" # # @example Accessing metadata