From dd5e57de0a1d01674e45d5094256d5520114efb6 Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Sat, 1 Aug 2026 16:49:21 -0500 Subject: [PATCH 01/17] docs(otel): add design spec and implementation plan for the SDK metrics exporter --- docs/design/otel-metrics-exporter.md | 202 ++++ .../plans/2026-08-01-otel-metrics-exporter.md | 869 ++++++++++++++++++ 2 files changed, 1071 insertions(+) create mode 100644 docs/design/otel-metrics-exporter.md create mode 100644 docs/superpowers/plans/2026-08-01-otel-metrics-exporter.md diff --git a/docs/design/otel-metrics-exporter.md b/docs/design/otel-metrics-exporter.md new file mode 100644 index 0000000000..4e24df4ef5 --- /dev/null +++ b/docs/design/otel-metrics-exporter.md @@ -0,0 +1,202 @@ +# Design: OpenTelemetry metrics exporter (isolated from existing telemetry) + +Status: proposed. Implementation plan: +[`docs/superpowers/plans/2026-08-01-otel-metrics-exporter.md`](../superpowers/plans/2026-08-01-otel-metrics-exporter.md). + +## Problem + +`TelemetryReporter` (`crates/core/src/tracing/telemetry.rs`) hand-builds OTLP-JSON +log records and POSTs them to `{telemetry-endpoint}/v1/logs` with `reqwest` +(`telemetry.rs:1415-1500`). It is the feed for the project's central dashboard and +it works. It is not an OpenTelemetry SDK pipeline: no meter provider, no +instruments, no standard `OTEL_*` env-var handling, no metrics or traces — so +there is nowhere to hang actual metrics. + +## Goal + +Add a real OpenTelemetry SDK metrics pipeline that can be pointed at any OTLP +collector and configured the standard way. The existing telemetry path is +untouched. + +## Non-goals + +- Any change to `TelemetryReporter`, `to_otlp_logs`, the `/v1/logs` wire format, + `telemetry-enabled`, or `telemetry-endpoint`. +- Migrating existing events onto the new pipeline. +- Logs and traces exporters. The endpoint resolution is designed to serve all + three signals; only metrics ships now. +- Any instrumentation beyond the single proof-of-life gauge. + +## Isolation requirement (hard) + +`otel-telemetry-enabled` and `telemetry-enabled` are strictly independent +features. The two pipelines are not expected to share a backend. Concretely: + +- Separate config structs. `OtelConfig` is a sibling of `TelemetryConfig`, never a + field on it. +- `otel::init` takes `&OtelConfig` only and must never read `TelemetryConfig`. +- No endpoint fallback between them. `otel-endpoint` never defaults to + `DEFAULT_TELEMETRY_ENDPOINT` (nova). +- Enabling or disabling one has no effect on the other. +- The only shared code is the test-harness detection helper, a free function. + +## Configuration + +New `OtelArgs` (clap + serde) and `OtelConfig` (resolved), beside +`TelemetryArgs`/`TelemetryConfig` in `crates/core/src/config.rs`: + +| Key | Env | Default | Meaning | +|---|---|---|---| +| `otel-telemetry-enabled` | `FREENET_OTEL_TELEMETRY_ENABLED` | `false` | Enable the SDK metrics exporter | +| `otel-endpoint` | (see below) | none | OTLP/HTTP collector base URL | + +Default is `false`: nothing is exported yet, so off is the no-behavior-change +default. Operators opt in. + +### Endpoint precedence + +`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` > `OTEL_EXPORTER_OTLP_ENDPOINT` > +`otel-endpoint` in config.toml > `http://localhost:4318` (the SDK's own default). + +This is deliberately *not* what the SDK does by itself. In +`opentelemetry-otlp` 0.32, `resolve_http_endpoint` +(`src/exporter/http/mod.rs:719-749`) gives a programmatic `with_endpoint` value +**priority over both env vars**, and uses it **verbatim** — `build_endpoint_uri` +appends `/v1/metrics` only on the env-var path. So to get env-wins precedence the +code must: + +- call `with_endpoint` only when neither env var is set, and +- append `/v1/metrics` itself when passing the config-file value. + +`otel-endpoint` therefore has no clap `env =` binding — binding it would merge the +standard variable into the config layer and invert the precedence. + +Every other standard variable — `OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`, +`OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_TIMEOUT`, +`OTEL_EXPORTER_OTLP_COMPRESSION`, `OTEL_METRIC_EXPORT_INTERVAL` (default 60s, +`opentelemetry_sdk/src/metrics/periodic_reader.rs:24-43`) — is read by the SDK. +No code for them. + +## Dependencies + +`opentelemetry` is already non-optional in `crates/core/Cargo.toml`. +`opentelemetry_sdk` and `opentelemetry-otlp` are optional and reachable only +through the `trace-ot` feature; both are already in `Cargo.lock`. Making them +non-optional is the whole dependency change — their default features already +cover what is needed: + +- `opentelemetry-otlp` defaults: `http-proto`, `reqwest-blocking-client`, + `metrics`, `trace`, `logs`, `internal-logs`. +- `opentelemetry_sdk` defaults: `metrics`, `trace`, `logs` (workspace decl adds + `rt-tokio`). + +`reqwest-blocking-client` is load-bearing, not incidental: `PeriodicReader` runs +exports on a dedicated thread through `futures_executor::block_on` +(`periodic_reader.rs:419`). An async reqwest client on that thread has no tokio +reactor and would fail at export time. + +Because a feature array may not name a non-optional dependency, `trace-ot` drops +`"opentelemetry-otlp"` from its list. + +## Suppression + +`otel::init` returns `None` — no provider, no exporter, no global registration — +when any of these hold, mirroring `telemetry_suppression_reason` +(`telemetry.rs:660-679`): + +1. `!cfg.enabled` +2. `cfg.is_test_environment` (the `--id` flag) +3. `cfg!(test)` +4. `running_under_cargo_test()` (executable's parent dir is `deps/`) + +Same rationale as #4366: keyed only on signals a real release binary never trips, +and deliberately not on `cfg!(feature = "testing")`, which leaks onto the shipped +binary via Cargo feature unification with `fdev`. + +The decision lives in a pure function so both directions are unit-testable from +inside a test process, which by construction trips signals 3 and 4. + +## Pipeline + +`crates/core/src/tracing/otel.rs`: + +``` +MetricExporter::builder().with_http()[.with_endpoint(resolved)].build() + → SdkMeterProvider::builder() + .with_periodic_exporter(exporter) + .with_resource(Resource::builder() + .with_service_name("freenet-peer") + .with_attribute(KeyValue::new("peer.id", local_peer_id)) + .build()) + .build() + → opentelemetry::global::set_meter_provider(provider.clone()) +``` + +Exporter build failure logs a WARN and returns `None`. Metrics export must never +fail node startup. + +No wrapper type, no registry, no facade. Future instrumentation is +`opentelemetry::global::meter("freenet").u64_counter(…)` at the call site. + +### Proof-of-life metric + +One observable gauge, `freenet.process.memory.rss`, over +`crate::node::resource_metrics::rss_bytes()` (already exists, +`node/resource_metrics.rs:79`). A real datapoint end to end, and the thing to look +at in a collector to confirm the pipeline works. + +### Shutdown + +Not wired. `global::set_meter_provider` holds a reference for the process +lifetime, and `PeriodicReader` exports every 60s, so at most one partial interval +is lost at exit. Flushing on the signal path is plumbing this does not need yet; +the code carries a `ponytail:` comment naming the ceiling and the upgrade path. + +## Wire-up + +`crates/core/src/node.rs`, in `build_with_flush_handle` beside the +`TelemetryReporter::new` call (`node.rs:754`): + +```rust +otel::init(&self.config.otel, self.local_peer_id_string()); +``` + +The provider is registered globally; nothing is stored on `Node`. + +## Testing + +- Endpoint precedence: metrics env > generic env > config > default. Pure + function, no network. +- Suppression: the pure decision function returns "export" only for a + production-shaped input, and a reason for each of disabled / `--id` / + `cfg(test)` / cargo-`deps` harness. Both directions. +- Isolation: enforced structurally — the otel decision function takes + `&OtelConfig` and cannot reach `TelemetryConfig` — plus a review check that + the diff touches `telemetry.rs` in exactly one line (the `pub(crate)` + widening of the harness detector). +- CLI/env parsing: `--otel-telemetry-enabled=false` parses as false. A bare + clap flag with an `env` binding treats any value of the variable as true, + which would turn the exporter on for an operator trying to turn it off. +- Provider construction succeeds inside a tokio runtime against an unreachable + endpoint (guards the reqwest-blocking-in-async-context concern; export failure + is asynchronous and must not surface at build time). +- Config round-trip through `ConfigArgs::build()` — mandatory per + `.claude/rules/code-style.md`. + +## Risks + +- Making the two crates non-optional grows the default build. Both already + compile in any `trace-ot` build and share reqwest with the existing HTTP + client, so the marginal cost is small — confirm cross-compile targets still + build (`.github/workflows/cross-compile.yml`, + `crates/core/tests/cross_compile_feature_split.rs`). +- `global::set_meter_provider` is process-global. A `trace-ot` build also sets + OTel globals (tracer provider, not meter provider) — confirm no conflict. +- `reqwest/blocking` gets enabled workspace-wide by feature unification, which + pulls in a background runtime thread for blocking clients. + +## Process note + +This is a feature, not a bug fix. Per +[CONTRIBUTING.md](../../CONTRIBUTING.md) it needs a maintainer-approved issue +before implementation starts. diff --git a/docs/superpowers/plans/2026-08-01-otel-metrics-exporter.md b/docs/superpowers/plans/2026-08-01-otel-metrics-exporter.md new file mode 100644 index 0000000000..a559eb7eee --- /dev/null +++ b/docs/superpowers/plans/2026-08-01-otel-metrics-exporter.md @@ -0,0 +1,869 @@ +# OpenTelemetry Metrics Exporter Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a standards-configured OpenTelemetry SDK metrics pipeline to `freenet`, strictly isolated from the existing `telemetry-enabled` reporter, so real metrics can be exported to any OTLP collector. + +**Architecture:** A new `OtelArgs`/`OtelConfig` config pair (sibling of, never nested in, `TelemetryArgs`/`TelemetryConfig`) gates a new `crates/core/src/tracing/otel.rs` module. That module builds an OTLP/HTTP `MetricExporter` and an `SdkMeterProvider`, registers it as the process-global meter provider, and registers one observable RSS gauge as proof of life. The endpoint is resolved env-first, which requires working *around* `opentelemetry-otlp`'s own precedence. Everything in `telemetry.rs` is untouched. + +**Tech Stack:** Rust, clap + serde config, `opentelemetry` / `opentelemetry_sdk` / `opentelemetry-otlp` 0.32, OTLP over HTTP/protobuf with the blocking reqwest client. + +**Design spec:** [`docs/design/otel-metrics-exporter.md`](../../design/otel-metrics-exporter.md) + +## Global Constraints + +- **This is a feature, not a bug fix.** Per `CONTRIBUTING.md` a maintainer-approved issue MUST exist before implementation starts. Do not open a PR without it. +- Branch name: `feat/otel-metrics-exporter`. +- Conventional-commit subjects, under 72 chars, body explains WHY (`.claude/rules/git-workflow.md`). +- Before every commit: `cargo fmt` and `cargo clippy -p freenet -- -D warnings`. CI treats any warning as failure. +- No behavior change to `TelemetryReporter`, `to_otlp_logs`, the `/v1/logs` path, `telemetry-enabled`, or `telemetry-endpoint`. If a diff touches those, it is wrong. +- `otel::init` and everything it calls must never read `TelemetryConfig`. +- `otel-endpoint` must NEVER fall back to `DEFAULT_TELEMETRY_ENDPOINT` (`http://nova.locut.us:4318`). Its default is `http://localhost:4318`. +- Default of `otel-telemetry-enabled` is `false`. +- Crate under test is `freenet` (`crates/core`). Unit tests run with `cargo test -p freenet --lib `. +- Production code in `crates/core/` uses `TimeSource` for time and `GlobalRng` for randomness (`.claude/rules/code-style.md`). Neither is needed here — do not introduce `Instant::now()` or `rand::random()`. +- Any deliberate simplification gets a `ponytail:` comment naming the ceiling and the upgrade path. + +--- + +### Task 1: Config — `OtelArgs` / `OtelConfig` + +**Files:** +- Modify: `crates/core/src/config.rs` (add structs after `TelemetryConfig`'s helpers ~line 2237; add `ConfigArgs` field ~line 236; add `ConfigArgs::default()` entry ~line 294; add merge block in `build()` ~line 629; add `Config` field ~line 1312; add `Config` construction in `build()` ~line 1109) +- Test: `crates/core/src/config.rs` (the `mod tests` block — new tests plus the existing guard at line 5241) + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: + - `pub const DEFAULT_OTEL_ENDPOINT: &str = "http://localhost:4318";` + - `pub struct OtelArgs { pub enabled: bool, pub endpoint: Option }` + - `pub struct OtelConfig { pub enabled: bool, pub endpoint: Option, pub is_test_environment: bool }` + - `Config::otel: OtelConfig`, `ConfigArgs::otel: OtelArgs` + +- [ ] **Step 1: Write the failing tests** + +Add to the `mod tests` block in `crates/core/src/config.rs`: + +```rust +#[test] +fn otel_args_default_is_off_and_endpointless() { + // The new pipeline exports nothing yet, so shipping it on would be a + // behavior change. Operators opt in explicitly. + let args = OtelArgs::default(); + assert!(!args.enabled, "otel-telemetry-enabled must default to false"); + assert_eq!(args.endpoint, None, "no implicit collector"); +} + +#[test] +fn otel_flag_parses_from_cli() { + use clap::Parser; + let none = ConfigArgs::try_parse_from(["freenet"]).expect("bare parse"); + assert!(!none.otel.enabled, "no flag -> off"); + let set = ConfigArgs::try_parse_from(["freenet", "--otel-telemetry-enabled"]) + .expect("flag parse"); + assert!(set.otel.enabled, "--otel-telemetry-enabled -> on"); + // Explicit `=false` must parse and mean false. Without this form the flag + // would be a bare ArgAction::SetTrue, and clap turns ANY value of the bound + // env var — including "false" — into true. + let off = ConfigArgs::try_parse_from(["freenet", "--otel-telemetry-enabled=false"]) + .expect("explicit false parse"); + assert!(!off.otel.enabled, "--otel-telemetry-enabled=false -> off"); + let with_ep = ConfigArgs::try_parse_from([ + "freenet", + "--otel-endpoint", + "http://collector.example:4318", + ]) + .expect("endpoint parse"); + assert_eq!( + with_ep.otel.endpoint.as_deref(), + Some("http://collector.example:4318") + ); +} + +#[test] +fn otel_endpoint_never_defaults_to_the_dashboard_collector() { + // Hard isolation requirement: the two pipelines share no backend. + assert_ne!( + DEFAULT_OTEL_ENDPOINT, DEFAULT_TELEMETRY_ENDPOINT, + "otel must not default to the central dashboard collector" + ); + assert_eq!(DEFAULT_OTEL_ENDPOINT, "http://localhost:4318"); +} +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cargo test -p freenet --lib config::tests::otel_ -- --nocapture` +Expected: FAIL — `cannot find type OtelArgs in this scope`, `cannot find value DEFAULT_OTEL_ENDPOINT`. + +- [ ] **Step 3: Add the config types** + +Insert after `fn default_iface_tx_enabled()` (~line 2237) in `crates/core/src/config.rs`: + +```rust +/// Default OTLP/HTTP endpoint for the SDK metrics pipeline, used when neither +/// the standard `OTEL_EXPORTER_OTLP_*` env vars nor `otel-endpoint` are set. +/// +/// Deliberately NOT `DEFAULT_TELEMETRY_ENDPOINT`: `otel-telemetry-enabled` and +/// `telemetry-enabled` are strictly isolated features that are not expected to +/// share a backend. Pointing this pipeline at the central dashboard collector +/// must always be an explicit operator choice. +pub const DEFAULT_OTEL_ENDPOINT: &str = "http://localhost:4318"; + +/// CLI/file args for the OpenTelemetry SDK metrics exporter. +/// +/// Strictly independent of [`TelemetryArgs`]: no shared field, no shared +/// default, no fallback in either direction. +#[derive(clap::Parser, Debug, Clone, Default, Serialize, Deserialize)] +pub struct OtelArgs { + /// Enable the OpenTelemetry SDK metrics exporter. Independent of + /// `telemetry-enabled`; enabling or disabling one has no effect on the + /// other. + /// + /// `num_args`/`default_missing_value` rather than a bare flag: with an + /// `env` binding, clap's `SetTrue` action treats ANY value of the variable + /// as true, so `FREENET_OTEL_TELEMETRY_ENABLED=false` would silently turn + /// the exporter ON. This form accepts `--otel-telemetry-enabled`, + /// `--otel-telemetry-enabled=false`, and a properly parsed env value. + #[arg( + long = "otel-telemetry-enabled", + env = "FREENET_OTEL_TELEMETRY_ENABLED", + num_args = 0..=1, + default_value = "false", + default_missing_value = "true", + action = clap::ArgAction::Set + )] + #[serde(rename = "otel-telemetry-enabled", default)] + pub enabled: bool, + + /// OTLP/HTTP collector base URL (e.g. `http://collector:4318`). + /// + /// No clap `env =` binding on purpose. The standard + /// `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` + /// variables must take priority over this file-level value, and binding + /// them here would merge them into the config layer and invert that + /// precedence. They are resolved in `tracing::otel` instead. + #[arg(long = "otel-endpoint")] + #[serde(rename = "otel-endpoint", skip_serializing_if = "Option::is_none")] + pub endpoint: Option, +} + +/// Resolved configuration for the OpenTelemetry SDK metrics exporter. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct OtelConfig { + /// Whether the SDK metrics exporter is enabled. + #[serde(default, rename = "otel-telemetry-enabled")] + pub enabled: bool, + + /// Operator-configured OTLP/HTTP collector base URL, if any. `None` means + /// "let the SDK resolve it" — see `tracing::otel::resolve_metrics_endpoint`. + #[serde(default, rename = "otel-endpoint", skip_serializing_if = "Option::is_none")] + pub endpoint: Option, + + /// Whether this is a test environment (detected via `--id`). Mirrors + /// [`TelemetryConfig::is_test_environment`]; suppresses export so test + /// networks can't ship data to a collector. + #[serde(skip)] + pub is_test_environment: bool, +} +``` + +- [ ] **Step 4: Hang the args off `ConfigArgs`** + +In `crates/core/src/config.rs`, after the `pub telemetry: TelemetryArgs,` field (~line 236): + +```rust + #[command(flatten)] + pub otel: OtelArgs, +``` + +And in `impl Default for ConfigArgs`, after `telemetry: Default::default(),` (~line 294): + +```rust + otel: Default::default(), +``` + +- [ ] **Step 5: Run the tests to verify they pass** + +Run: `cargo test -p freenet --lib config::tests::otel_ -- --nocapture` +Expected: PASS (3 tests). + +- [ ] **Step 6: Write the failing round-trip test** + +In `crates/core/src/config.rs`, extend the existing guard test +`all_persisted_config_fields_round_trip_through_build` (line 5241). + +In the `seed` literal, after the `telemetry: TelemetryConfig { … },` block (~line 5323): + +```rust + otel: OtelConfig { + enabled: true, + endpoint: Some("http://example.invalid:4319".to_string()), + is_test_environment: false, // #[serde(skip)] — derived from --id + }, +``` + +In the exhaustive `let Config { … } = rebuilt;` destructure, after `telemetry,` (~line 5357): + +```rust + otel, +``` + +And after the `shutdown_drain_secs` assertion (~line 5403): + +```rust + assert_eq!(otel.enabled, seed.otel.enabled, "otel.enabled"); + assert_eq!( + otel.endpoint, seed.otel.endpoint, + "otel.endpoint — an operator's collector URL must survive the \ + config.toml merge" + ); +``` + +- [ ] **Step 7: Run it to verify it fails** + +Run: `cargo test -p freenet --lib config::tests::all_persisted_config_fields_round_trip_through_build` +Expected: FAIL to COMPILE — `struct Config has no field named otel`. + +- [ ] **Step 8: Add the `Config` field, the merge, and the construction** + +Three edits in `crates/core/src/config.rs`. + +(a) On `pub struct Config`, after `pub telemetry: TelemetryConfig,` (~line 1312): + +```rust + /// OpenTelemetry SDK metrics exporter settings. Strictly isolated from + /// `telemetry` above — see `docs/design/otel-metrics-exporter.md`. + #[serde(default)] + pub otel: OtelConfig, +``` + +(b) In `ConfigArgs::build()`, inside the `if let Some(cfg) = …` merge block, after the `iface_tx_enabled` merge (~line 629): + +```rust + // otel-telemetry-enabled defaults to false via clap, so only the + // file-says-true direction needs handling — same one-directional + // override as reference-ping/iface-tx above. Kept separate from the + // telemetry merge on purpose: the two features are independent. + if cfg.otel.enabled { + self.otel.enabled = true; + } + if let Some(endpoint) = cfg.otel.endpoint { + self.otel.endpoint.get_or_insert(endpoint); + } +``` + +(c) In the `Config { … }` literal in `build()`, after the `telemetry: TelemetryConfig { … },` block (~line 1109): + +```rust + otel: OtelConfig { + enabled: self.otel.enabled, + endpoint: self.otel.endpoint, + // Same --id rule as telemetry: simulated networks and + // integration tests must not ship data to a collector. + is_test_environment: self.id.is_some(), + }, +``` + +- [ ] **Step 9: Run the round-trip test to verify it passes** + +Run: `cargo test -p freenet --lib config::tests::all_persisted_config_fields_round_trip_through_build` +Expected: PASS. + +- [ ] **Step 10: Run the whole config module and lint** + +Run: `cargo test -p freenet --lib config::` +Expected: PASS, no regressions. + +Run: `cargo fmt && cargo clippy -p freenet -- -D warnings` +Expected: clean. + +- [ ] **Step 11: Commit** + +```bash +git add crates/core/src/config.rs +git commit -m "feat(otel): add isolated otel-telemetry config + +New OtelArgs/OtelConfig sit beside TelemetryArgs/TelemetryConfig rather +than inside them: the SDK metrics pipeline and the dashboard reporter are +independent features that are not expected to share a backend, so neither +enable-flag nor endpoint may fall back to the other." +``` + +--- + +### Task 2: Pure decision functions in `tracing::otel` + +**Files:** +- Create: `crates/core/src/tracing/otel.rs` +- Modify: `crates/core/src/tracing.rs` (add module declaration next to `pub mod telemetry;`, ~line 42) +- Modify: `crates/core/src/tracing/telemetry.rs:615` (widen `running_under_cargo_test` to `pub(crate)`) +- Test: `crates/core/src/tracing/otel.rs` (inline `mod tests`) + +**Interfaces:** +- Consumes: `crate::config::OtelConfig` from Task 1. +- Produces: + - `pub(crate) enum OtelSuppression { Disabled, TestEnvironmentFlag, TestHarness }` + - `pub(crate) fn otel_suppression_reason(cfg: &OtelConfig, is_test_build: bool, running_under_cargo_test: bool) -> Option` + - `pub(crate) fn resolve_metrics_endpoint(cfg_endpoint: Option<&str>, metrics_env: Option<&str>, generic_env: Option<&str>) -> Option` — `None` means "let the SDK resolve it". + +- [ ] **Step 1: Write the failing tests** + +Create `crates/core/src/tracing/otel.rs` containing ONLY this test module for now: + +```rust +#[cfg(test)] +mod tests { + use super::*; + use crate::config::OtelConfig; + + fn enabled_config() -> OtelConfig { + OtelConfig { + enabled: true, + endpoint: None, + is_test_environment: false, + } + } + + #[test] + fn production_shaped_input_is_not_suppressed() { + assert_eq!( + otel_suppression_reason(&enabled_config(), false, false), + None, + "a real release binary with the flag on must export" + ); + } + + #[test] + fn every_test_signal_suppresses() { + let disabled = OtelConfig { + enabled: false, + ..enabled_config() + }; + assert_eq!( + otel_suppression_reason(&disabled, false, false), + Some(OtelSuppression::Disabled) + ); + + let test_env = OtelConfig { + is_test_environment: true, + ..enabled_config() + }; + assert_eq!( + otel_suppression_reason(&test_env, false, false), + Some(OtelSuppression::TestEnvironmentFlag) + ); + + assert_eq!( + otel_suppression_reason(&enabled_config(), true, false), + Some(OtelSuppression::TestHarness), + "cfg(test) build" + ); + assert_eq!( + otel_suppression_reason(&enabled_config(), false, true), + Some(OtelSuppression::TestHarness), + "running from a cargo deps/ harness" + ); + } + + #[test] + fn metrics_env_wins_over_generic_env_and_config() { + // Both env forms mean "let the SDK resolve it", because + // opentelemetry-otlp gives a programmatic endpoint priority over the + // env vars — passing one would invert the required precedence. + assert_eq!( + resolve_metrics_endpoint( + Some("http://from-config:4318"), + Some("http://from-metrics-env:4318/v1/metrics"), + Some("http://from-generic-env:4318"), + ), + None + ); + assert_eq!( + resolve_metrics_endpoint( + Some("http://from-config:4318"), + None, + Some("http://from-generic-env:4318"), + ), + None + ); + } + + #[test] + fn config_endpoint_gets_the_signal_path_appended() { + // The SDK appends /v1/metrics only on the env-var path; a programmatic + // endpoint is used verbatim, so we append it ourselves. + assert_eq!( + resolve_metrics_endpoint(Some("http://collector:4318"), None, None), + Some("http://collector:4318/v1/metrics".to_string()) + ); + assert_eq!( + resolve_metrics_endpoint(Some("http://collector:4318/"), None, None), + Some("http://collector:4318/v1/metrics".to_string()), + "trailing slash must not double up" + ); + } + + #[test] + fn nothing_configured_defers_to_the_sdk_default() { + assert_eq!(resolve_metrics_endpoint(None, None, None), None); + assert_eq!( + resolve_metrics_endpoint(Some(" "), None, None), + None, + "a blank endpoint is not a configuration" + ); + } +} +``` + +- [ ] **Step 2: Declare the module and run the tests to verify they fail** + +In `crates/core/src/tracing.rs`, after `pub use telemetry::TelemetryReporter;` (~line 43): + +```rust +/// Standards-configured OpenTelemetry SDK metrics pipeline. Strictly isolated +/// from `telemetry` above — see `docs/design/otel-metrics-exporter.md`. +pub mod otel; +``` + +Run: `cargo test -p freenet --lib tracing::otel` +Expected: FAIL — `cannot find function otel_suppression_reason in this scope`. + +- [ ] **Step 3: Write the implementation** + +Prepend to `crates/core/src/tracing/otel.rs`, above the test module: + +```rust +//! Standards-configured OpenTelemetry SDK metrics pipeline. +//! +//! Strictly isolated from [`super::telemetry`]: nothing here reads +//! `TelemetryConfig`, and the endpoint never falls back to +//! `DEFAULT_TELEMETRY_ENDPOINT`. The two features are independent by design — +//! see `docs/design/otel-metrics-exporter.md`. + +use crate::config::OtelConfig; + +/// Why the OTel metrics exporter was not started. +/// +/// Mirrors `telemetry::TelemetrySuppression` so both pipelines refuse to ship +/// data from a test process, but the decision is computed from `OtelConfig` +/// alone — the two flags never consult each other. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum OtelSuppression { + /// Operator left `otel-telemetry-enabled` off (the default). + Disabled, + /// `--id` test environment (integration/CLI harness sets `is_test_environment`). + TestEnvironmentFlag, + /// A `cfg(test)` build or a binary running under a cargo test/bench harness. + TestHarness, +} + +/// Decide whether the metrics exporter should be suppressed. +/// +/// Pure and side-effect free: callers pass `cfg!(test)` and the result of +/// `telemetry::running_under_cargo_test()` so this is testable for a +/// production release binary (must NOT suppress) from inside a test process, +/// which by construction trips both test signals. +/// +/// Suppression is keyed only on signals a real release binary never matches, +/// and deliberately NOT on `cfg!(feature = "testing")` — that flag leaks onto +/// the shipped binary through Cargo feature unification with `fdev` and +/// silently disabled telemetry across the fleet once already (#4366, the +/// 0.2.81 blackout). See `telemetry::telemetry_suppression_reason`. +pub(crate) fn otel_suppression_reason( + config: &OtelConfig, + is_test_build: bool, + running_under_cargo_test: bool, +) -> Option { + if !config.enabled { + return Some(OtelSuppression::Disabled); + } + if config.is_test_environment { + return Some(OtelSuppression::TestEnvironmentFlag); + } + if is_test_build || running_under_cargo_test { + return Some(OtelSuppression::TestHarness); + } + None +} + +/// Endpoint to hand to `MetricExporter`'s builder, or `None` to let the SDK +/// resolve it. +/// +/// Required precedence is env > config file > SDK default, but +/// `opentelemetry-otlp` 0.32 inverts the first two: `resolve_http_endpoint` +/// (`src/exporter/http/mod.rs`) checks the programmatic value FIRST and only +/// then `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` / `OTEL_EXPORTER_OTLP_ENDPOINT`. +/// So whenever either variable is set we return `None` and stay out of the +/// way. It also appends the `/v1/metrics` signal path only on the env-var +/// path, so a config-file value gets the path appended here. +pub(crate) fn resolve_metrics_endpoint( + cfg_endpoint: Option<&str>, + metrics_env: Option<&str>, + generic_env: Option<&str>, +) -> Option { + let is_set = |v: Option<&str>| v.is_some_and(|s| !s.trim().is_empty()); + if is_set(metrics_env) || is_set(generic_env) { + return None; + } + let base = cfg_endpoint.map(str::trim).filter(|s| !s.is_empty())?; + Some(format!("{}/v1/metrics", base.trim_end_matches('/'))) +} +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `cargo test -p freenet --lib tracing::otel` +Expected: PASS (5 tests). + +- [ ] **Step 5: Widen the shared harness detector** + +In `crates/core/src/tracing/telemetry.rs`, change line 615 from: + +```rust +fn running_under_cargo_test() -> bool { +``` + +to: + +```rust +pub(crate) fn running_under_cargo_test() -> bool { +``` + +Leave its doc comment unchanged. This is the only code shared between the two +pipelines — a free function with no config in it. + +- [ ] **Step 6: Verify the crate still builds and lints** + +Run: `cargo test -p freenet --lib tracing:: && cargo fmt && cargo clippy -p freenet -- -D warnings` +Expected: PASS, clean. + +- [ ] **Step 7: Commit** + +```bash +git add crates/core/src/tracing.rs crates/core/src/tracing/otel.rs crates/core/src/tracing/telemetry.rs +git commit -m "feat(otel): add suppression and endpoint-precedence logic + +Both decisions are pure functions so the production direction is testable +from inside a test process. Endpoint resolution deliberately returns None +when a standard OTEL_* var is set: opentelemetry-otlp gives a programmatic +endpoint priority over the env vars, which is the opposite of the +precedence operators expect." +``` + +--- + +### Task 3: Exporter, meter provider, RSS gauge, and node wire-up + +**Files:** +- Modify: `crates/core/Cargo.toml` (deps ~lines 129-130, `trace-ot` feature ~line 233) +- Modify: `crates/core/src/tracing/otel.rs` (add `init`, `build_provider`, `register_process_metrics`) +- Modify: `crates/core/src/node.rs` (~line 754, beside the `TelemetryReporter::new` call) +- Test: `crates/core/src/tracing/otel.rs` (inline `mod tests`) + +**Interfaces:** +- Consumes: `otel_suppression_reason`, `resolve_metrics_endpoint` (Task 2); `OtelConfig` (Task 1); `crate::node::resource_metrics::rss_bytes() -> Option` (existing, `node/resource_metrics.rs:79`); `NodeConfig::local_peer_id_string() -> String` (existing, `node.rs:441`). +- Produces: + - `pub fn init(config: &OtelConfig, local_peer_id: String)` + - `pub(crate) fn build_provider(endpoint: Option<&str>, local_peer_id: String) -> Result` + +- [ ] **Step 1: Make the OTel crates non-optional** + +In `crates/core/Cargo.toml`, change lines 129-130 from: + +```toml +opentelemetry-otlp = { workspace = true, optional = true } +opentelemetry_sdk = { workspace = true, optional = true } +``` + +to: + +```toml +# Non-optional: the SDK metrics pipeline (tracing::otel) ships in every build. +# Default features already give us metrics + http-proto + reqwest-blocking-client. +# The blocking client is load-bearing, not incidental: PeriodicReader exports +# from a dedicated thread via futures_executor::block_on, where an async reqwest +# client would have no tokio reactor. +opentelemetry-otlp = { workspace = true } +opentelemetry_sdk = { workspace = true } +``` + +And on line 233, drop `"opentelemetry-otlp"` from the feature list (a feature +array may not name a non-optional dependency): + +```toml +trace-ot = ["opentelemetry-jaeger", "trace", "tracing-opentelemetry"] +``` + +- [ ] **Step 2: Verify the crate still builds** + +Run: `cargo build -p freenet` +Expected: SUCCESS. + +Run: `cargo test -p freenet --test cross_compile_feature_split` +Expected: PASS. + +- [ ] **Step 3: Write the failing test** + +Add to the `mod tests` block in `crates/core/src/tracing/otel.rs`: + +```rust + #[tokio::test] + async fn provider_builds_inside_a_tokio_runtime() { + // Two things under test. First, exporter construction must not panic + // when it happens inside an async context — the OTLP HTTP exporter uses + // reqwest's BLOCKING client because PeriodicReader exports from its own + // thread. Second, an unreachable collector must not surface as a build + // error: export failures are asynchronous and must never fail node + // startup. Port 1 is chosen because nothing can be listening there. + let provider = build_provider( + Some("http://127.0.0.1:1/v1/metrics"), + "peer-under-test".to_string(), + ) + .expect("exporter build must succeed against an unreachable collector"); + provider.shutdown().expect("clean shutdown"); + } +``` + +- [ ] **Step 4: Run it to verify it fails** + +Run: `cargo test -p freenet --lib tracing::otel::tests::provider_builds_inside_a_tokio_runtime` +Expected: FAIL — `cannot find function build_provider in this scope`. + +- [ ] **Step 5: Write the implementation** + +Add to `crates/core/src/tracing/otel.rs`, after `resolve_metrics_endpoint` and +before the test module: + +```rust +use opentelemetry::{KeyValue, global}; +use opentelemetry_otlp::{ExporterBuildError, MetricExporter, WithExportConfig}; +use opentelemetry_sdk::{Resource, metrics::SdkMeterProvider}; + +/// Instrumentation scope name for every instrument this crate registers. +const METER_NAME: &str = "freenet"; + +/// Start the OpenTelemetry SDK metrics pipeline and install it as the +/// process-global meter provider. +/// +/// No-op when suppressed (see [`otel_suppression_reason`]) and best-effort +/// otherwise: an exporter that cannot be built logs a warning and the node +/// starts anyway. Metrics export must never be a startup dependency. +/// +/// After this returns, instrumentation anywhere in the crate is just +/// `opentelemetry::global::meter("freenet")` — there is deliberately no wrapper +/// type or registry to keep in sync. +pub fn init(config: &OtelConfig, local_peer_id: String) { + if let Some(reason) = otel_suppression_reason( + config, + cfg!(test), + super::telemetry::running_under_cargo_test(), + ) { + tracing::debug!(?reason, "OTel metrics exporter not started"); + return; + } + + let endpoint = resolve_metrics_endpoint( + config.endpoint.as_deref(), + std::env::var(opentelemetry_otlp::OTEL_EXPORTER_OTLP_METRICS_ENDPOINT) + .ok() + .as_deref(), + std::env::var(opentelemetry_otlp::OTEL_EXPORTER_OTLP_ENDPOINT) + .ok() + .as_deref(), + ); + + match build_provider(endpoint.as_deref(), local_peer_id) { + Ok(provider) => { + // ponytail: no shutdown hook. `set_meter_provider` holds a + // reference for the process lifetime and PeriodicReader exports + // every 60s (OTEL_METRIC_EXPORT_INTERVAL), so at most one partial + // interval is lost at exit. If that tail ever matters, keep the + // provider in a OnceLock and call `shutdown()` from the graceful + // shutdown path in `bin/freenet.rs`. + global::set_meter_provider(provider); + register_process_metrics(); + tracing::info!( + endpoint = endpoint.as_deref().unwrap_or(""), + "OTel metrics exporter started" + ); + } + Err(error) => { + tracing::warn!( + %error, + "OTel metrics exporter failed to start; node continues without metrics" + ); + } + } +} + +/// Build the OTLP/HTTP exporter and meter provider. +/// +/// `endpoint` is `None` when the standard env vars should win — see +/// [`resolve_metrics_endpoint`] for why calling `with_endpoint` at all would +/// override them. +pub(crate) fn build_provider( + endpoint: Option<&str>, + local_peer_id: String, +) -> Result { + let mut builder = MetricExporter::builder().with_http(); + if let Some(endpoint) = endpoint { + builder = builder.with_endpoint(endpoint); + } + let exporter = builder.build()?; + + // `service.name` is overridden by OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES + // when the operator sets them; the SDK reads those itself. + let resource = Resource::builder() + .with_service_name("freenet-peer") + .with_attribute(KeyValue::new("peer.id", local_peer_id)) + .build(); + + Ok(SdkMeterProvider::builder() + .with_periodic_exporter(exporter) + .with_resource(resource) + .build()) +} + +/// Register the instruments this crate owns. +/// +/// Must run AFTER `global::set_meter_provider`: `global::meter` binds to +/// whatever provider is installed at call time. +fn register_process_metrics() { + let meter = global::meter(METER_NAME); + // The handle is dropped on purpose — the callback is registered into the + // pipeline at `build()` and observed on every collection cycle regardless. + let _rss = meter + .u64_observable_gauge("freenet.process.memory.rss") + .with_unit("By") + .with_description("Resident set size of the freenet process") + .with_callback(|observer| { + if let Some(rss) = crate::node::resource_metrics::rss_bytes() { + observer.observe(rss, &[]); + } + }) + .build(); +} +``` + +- [ ] **Step 6: Run the test to verify it passes** + +Run: `cargo test -p freenet --lib tracing::otel` +Expected: PASS (6 tests). + +- [ ] **Step 7: Wire it into node startup** + +In `crates/core/src/node.rs`, in `build_with_flush_handle`, immediately after the +`if let Some(telemetry) = TelemetryReporter::new(…) { … }` block (~line 758) and +before `(DynamicRegister::new(registers), flush_handle)`: + +```rust + // Independent of the TelemetryReporter above: a separate opt-in + // (`otel-telemetry-enabled`), a separate endpoint, and a separate + // collector. It is not a NetEventRegister — it installs a global + // meter provider that instrumentation reaches via + // `opentelemetry::global::meter`. + crate::tracing::otel::init(&self.config.otel, self.local_peer_id_string()); +``` + +- [ ] **Step 8: Verify the node still builds and its tests pass** + +Run: `cargo build -p freenet && cargo test -p freenet --lib node::` +Expected: SUCCESS, PASS. + +- [ ] **Step 9: Lint** + +Run: `cargo fmt && cargo clippy -p freenet -- -D warnings` +Expected: clean. + +- [ ] **Step 10: Commit** + +```bash +git add crates/core/Cargo.toml crates/core/src/tracing/otel.rs crates/core/src/node.rs +git commit -m "feat(otel): export metrics through the OpenTelemetry SDK + +Installs a global meter provider backed by an OTLP/HTTP exporter, plus one +RSS gauge so the pipeline carries a real datapoint end to end. Future +instrumentation is a global::meter call at the site, with no registry to +keep in sync. The OTel crates become non-optional because the pipeline +ships in every build, not just trace-ot ones." +``` + +--- + +### Task 4: Operator documentation + +**Files:** +- Modify: `AGENTS.md` (new section after "Delegate secrets-at-rest") +- Reference: `docs/design/otel-metrics-exporter.md` (already written; do not restate it) + +**Interfaces:** +- Consumes: the config keys from Task 1 and the env-var precedence from Task 2. +- Produces: nothing code depends on. + +- [ ] **Step 1: Add the section** + +In `AGENTS.md`, after the `## Delegate secrets-at-rest` section, insert: + +```markdown +## Two independent telemetry pipelines + +`telemetry-enabled` / `telemetry-endpoint` feed the project's central +dashboard through a hand-rolled OTLP-JSON log POST (`tracing/telemetry.rs`). + +`otel-telemetry-enabled` / `otel-endpoint` are a **separate, unrelated** +OpenTelemetry SDK metrics pipeline (`tracing/otel.rs`). The two share no +config, no endpoint, and no fallback in either direction — enabling or +disabling one has no effect on the other, and `otel-endpoint` must never +default to the dashboard collector. + +The otel pipeline honors the standard variables, which take priority over +`otel-endpoint` in `config.toml`: +`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, `OTEL_EXPORTER_OTLP_ENDPOINT`, +`OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_SERVICE_NAME`, +`OTEL_RESOURCE_ATTRIBUTES`, `OTEL_METRIC_EXPORT_INTERVAL`. Without any of +them it exports to `http://localhost:4318`. + +Adding an instrument is one call at the site — no registry, no wrapper: + + opentelemetry::global::meter("freenet").u64_counter("freenet.some.thing").build() + +Design: [`docs/design/otel-metrics-exporter.md`](docs/design/otel-metrics-exporter.md). +``` + +- [ ] **Step 2: Verify the design doc links resolve** + +Run: `ls docs/design/otel-metrics-exporter.md docs/superpowers/plans/2026-08-01-otel-metrics-exporter.md` +Expected: both listed. + +- [ ] **Step 3: Commit** + +```bash +git add AGENTS.md +git commit -m "docs(otel): document the two independent telemetry pipelines + +The isolation between telemetry-enabled and otel-telemetry-enabled is a +design constraint, not an accident, so it belongs where the next person +reads before touching either." +``` + +--- + +## Verification + +After Task 4, before opening the PR: + +- [ ] `cargo fmt --check` — clean +- [ ] `cargo clippy -p freenet -- -D warnings` — clean +- [ ] `cargo test -p freenet --lib config:: tracing::` — pass +- [ ] `cargo test -p freenet --test cross_compile_feature_split` — pass +- [ ] `cargo build -p freenet --features trace-ot` — the feature-list edit in Task 3 did not break the jaeger path +- [ ] `git diff main --stat` — `tracing/telemetry.rs` shows exactly one changed line (the `pub(crate)` widening). Anything more means the isolation constraint was violated. + +Manual smoke check (optional, needs a collector): + +```bash +docker run --rm -p 4318:4318 otel/opentelemetry-collector:latest +FREENET_OTEL_TELEMETRY_ENABLED=true cargo run -p freenet --bin freenet -- network +# collector log should show freenet.process.memory.rss within ~60s +``` From 94ba5126f82ec21d35108523fc59ae19c0b639e0 Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Sat, 1 Aug 2026 17:05:06 -0500 Subject: [PATCH 02/17] feat(otel): add isolated otel-telemetry config New OtelArgs/OtelConfig sit beside TelemetryArgs/TelemetryConfig rather than inside them: the SDK metrics pipeline and the dashboard reporter are independent features that are not expected to share a backend, so neither enable-flag nor endpoint may fall back to the other. --- crates/core/src/config.rs | 159 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 159 insertions(+) diff --git a/crates/core/src/config.rs b/crates/core/src/config.rs index f1501185b6..2c1d45bf46 100644 --- a/crates/core/src/config.rs +++ b/crates/core/src/config.rs @@ -234,6 +234,9 @@ pub struct ConfigArgs { #[command(flatten)] pub telemetry: TelemetryArgs, + + #[command(flatten)] + pub otel: OtelArgs, } impl Default for ConfigArgs { @@ -302,6 +305,7 @@ impl Default for ConfigArgs { shutdown_drain_secs: None, disable_auto_update: false, telemetry: Default::default(), + otel: Default::default(), } } } @@ -967,6 +971,16 @@ impl ConfigArgs { if cfg.telemetry.iface_tx_enabled { self.telemetry.iface_tx_enabled = true; } + // otel-telemetry-enabled defaults to false via clap, so only the + // file-says-true direction needs handling — same one-directional + // override as reference-ping/iface-tx above. Kept separate from the + // telemetry merge on purpose: the two features are independent. + if cfg.otel.enabled { + self.otel.enabled = true; + } + if let Some(endpoint) = cfg.otel.endpoint { + self.otel.endpoint.get_or_insert(endpoint); + } } // Validate the effective config (CLI + values merged from config.toml). @@ -1473,6 +1487,13 @@ impl ConfigArgs { reference_ping_enabled: self.telemetry.reference_ping_enabled, iface_tx_enabled: self.telemetry.iface_tx_enabled, }, + otel: OtelConfig { + enabled: self.otel.enabled, + endpoint: self.otel.endpoint, + // Same --id rule as telemetry: simulated networks and + // integration tests must not ship data to a collector. + is_test_environment: self.id.is_some(), + }, }; fs::create_dir_all(this.config_dir())?; @@ -1707,6 +1728,12 @@ pub struct Config { /// Telemetry configuration #[serde(flatten)] pub telemetry: TelemetryConfig, + + /// OpenTelemetry SDK metrics exporter settings. Strictly isolated from + /// `telemetry` above — see `docs/design/otel-metrics-exporter.md`. + #[serde(default)] + pub otel: OtelConfig, + /// Maximum seconds to wait on graceful shutdown for in-flight /// client-originated operations (PUT/UPDATE/GET/SUBSCRIBE) to /// finish before tearing down peer connections. @@ -2809,6 +2836,77 @@ fn default_iface_tx_enabled() -> bool { false } +/// Default OTLP/HTTP endpoint for the SDK metrics pipeline, used when neither +/// the standard `OTEL_EXPORTER_OTLP_*` env vars nor `otel-endpoint` are set. +/// +/// Deliberately NOT `DEFAULT_TELEMETRY_ENDPOINT`: `otel-telemetry-enabled` and +/// `telemetry-enabled` are strictly isolated features that are not expected to +/// share a backend. Pointing this pipeline at the central dashboard collector +/// must always be an explicit operator choice. +pub const DEFAULT_OTEL_ENDPOINT: &str = "http://localhost:4318"; + +/// CLI/file args for the OpenTelemetry SDK metrics exporter. +/// +/// Strictly independent of [`TelemetryArgs`]: no shared field, no shared +/// default, no fallback in either direction. +#[derive(clap::Parser, Debug, Clone, Default, Serialize, Deserialize)] +pub struct OtelArgs { + /// Enable the OpenTelemetry SDK metrics exporter. Independent of + /// `telemetry-enabled`; enabling or disabling one has no effect on the + /// other. + /// + /// `num_args`/`default_missing_value` rather than a bare flag: with an + /// `env` binding, clap's `SetTrue` action treats ANY value of the variable + /// as true, so `FREENET_OTEL_TELEMETRY_ENABLED=false` would silently turn + /// the exporter ON. This form accepts `--otel-telemetry-enabled`, + /// `--otel-telemetry-enabled=false`, and a properly parsed env value. + #[arg( + id = "otel_telemetry_enabled", + long = "otel-telemetry-enabled", + env = "FREENET_OTEL_TELEMETRY_ENABLED", + num_args = 0..=1, + default_value = "false", + default_missing_value = "true", + action = clap::ArgAction::Set + )] + #[serde(rename = "otel-telemetry-enabled", default)] + pub enabled: bool, + + /// OTLP/HTTP collector base URL (e.g. `http://collector:4318`). + /// + /// No clap `env =` binding on purpose. The standard + /// `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` + /// variables must take priority over this file-level value, and binding + /// them here would merge them into the config layer and invert that + /// precedence. They are resolved in `tracing::otel` instead. + #[arg(id = "otel_endpoint", long = "otel-endpoint")] + #[serde(rename = "otel-endpoint", skip_serializing_if = "Option::is_none")] + pub endpoint: Option, +} + +/// Resolved configuration for the OpenTelemetry SDK metrics exporter. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct OtelConfig { + /// Whether the SDK metrics exporter is enabled. + #[serde(default, rename = "otel-telemetry-enabled")] + pub enabled: bool, + + /// Operator-configured OTLP/HTTP collector base URL, if any. `None` means + /// "let the SDK resolve it" — see `tracing::otel::resolve_metrics_endpoint`. + #[serde( + default, + rename = "otel-endpoint", + skip_serializing_if = "Option::is_none" + )] + pub endpoint: Option, + + /// Whether this is a test environment (detected via `--id`). Mirrors + /// [`TelemetryConfig::is_test_environment`]; suppresses export so test + /// networks can't ship data to a collector. + #[serde(skip)] + pub is_test_environment: bool, +} + impl Default for TelemetryConfig { fn default() -> Self { Self { @@ -6420,6 +6518,54 @@ shutdown-drain-secs = 42 } } + #[test] + fn otel_args_default_is_off_and_endpointless() { + // The new pipeline exports nothing yet, so shipping it on would be a + // behavior change. Operators opt in explicitly. + let args = OtelArgs::default(); + assert!( + !args.enabled, + "otel-telemetry-enabled must default to false" + ); + assert_eq!(args.endpoint, None, "no implicit collector"); + } + + #[test] + fn otel_flag_parses_from_cli() { + use clap::Parser; + let none = ConfigArgs::try_parse_from(["freenet"]).expect("bare parse"); + assert!(!none.otel.enabled, "no flag -> off"); + let set = ConfigArgs::try_parse_from(["freenet", "--otel-telemetry-enabled"]) + .expect("flag parse"); + assert!(set.otel.enabled, "--otel-telemetry-enabled -> on"); + // Explicit `=false` must parse and mean false. Without this form the flag + // would be a bare ArgAction::SetTrue, and clap turns ANY value of the bound + // env var — including "false" — into true. + let off = ConfigArgs::try_parse_from(["freenet", "--otel-telemetry-enabled=false"]) + .expect("explicit false parse"); + assert!(!off.otel.enabled, "--otel-telemetry-enabled=false -> off"); + let with_ep = ConfigArgs::try_parse_from([ + "freenet", + "--otel-endpoint", + "http://collector.example:4318", + ]) + .expect("endpoint parse"); + assert_eq!( + with_ep.otel.endpoint.as_deref(), + Some("http://collector.example:4318") + ); + } + + #[test] + fn otel_endpoint_never_defaults_to_the_dashboard_collector() { + // Hard isolation requirement: the two pipelines share no backend. + assert_ne!( + DEFAULT_OTEL_ENDPOINT, DEFAULT_TELEMETRY_ENDPOINT, + "otel must not default to the central dashboard collector" + ); + assert_eq!(DEFAULT_OTEL_ENDPOINT, "http://localhost:4318"); + } + #[tokio::test] async fn test_serde_config_args() { // Use tempfile for a guaranteed-writable directory (avoids CI permission issues on /tmp) @@ -7421,6 +7567,7 @@ shutdown-drain-secs = 42 shutdown_drain_secs: None, disable_auto_update: false, telemetry: Default::default(), + otel: Default::default(), } } @@ -7578,6 +7725,11 @@ shutdown-drain-secs = 42 reference_ping_enabled: true, iface_tx_enabled: true, }, + otel: OtelConfig { + enabled: true, + endpoint: Some("http://example.invalid:4319".to_string()), + is_test_environment: false, // #[serde(skip)] — derived from --id + }, shutdown_drain_secs: 77, disable_auto_update: true, // #[serde(skip)] — see destructure below } @@ -7631,6 +7783,7 @@ shutdown-drain-secs = 42 module_cache_budget_bytes, enable_event_log, telemetry, + otel, shutdown_drain_secs, // #[serde(skip)] runtime CLI/env flag — set from --disable-auto-update // at build() time, intentionally not persisted, so it does not @@ -7677,6 +7830,12 @@ shutdown-drain-secs = 42 shutdown_drain_secs, seed.shutdown_drain_secs, "shutdown_drain_secs" ); + assert_eq!(otel.enabled, seed.otel.enabled, "otel.enabled"); + assert_eq!( + otel.endpoint, seed.otel.endpoint, + "otel.endpoint — an operator's collector URL must survive the \ + config.toml merge" + ); let NetworkApiConfig { address, From e2c2f7640716b064a075f96d27b669fbd276c874 Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Sat, 1 Aug 2026 17:22:07 -0500 Subject: [PATCH 03/17] feat(otel): add suppression and endpoint-precedence logic Both decisions are pure functions so the production direction is testable from inside a test process. Endpoint resolution deliberately returns None when a standard OTEL_* var is set: opentelemetry-otlp gives a programmatic endpoint priority over the env vars, which is the opposite of the precedence operators expect. --- crates/core/src/tracing.rs | 4 + crates/core/src/tracing/otel.rs | 181 +++++++++++++++++++++++++++ crates/core/src/tracing/telemetry.rs | 2 +- 3 files changed, 186 insertions(+), 1 deletion(-) create mode 100644 crates/core/src/tracing/otel.rs diff --git a/crates/core/src/tracing.rs b/crates/core/src/tracing.rs index 3c964931d0..bf2fd42f40 100644 --- a/crates/core/src/tracing.rs +++ b/crates/core/src/tracing.rs @@ -43,6 +43,10 @@ pub mod event_aggregator; pub mod telemetry; pub use telemetry::TelemetryReporter; +/// Standards-configured OpenTelemetry SDK metrics pipeline. Strictly isolated +/// from `telemetry` above — see `docs/design/otel-metrics-exporter.md`. +pub mod otel; + /// Automatic state verification through telemetry linearization. pub mod state_verifier; diff --git a/crates/core/src/tracing/otel.rs b/crates/core/src/tracing/otel.rs new file mode 100644 index 0000000000..9f28ca8d9e --- /dev/null +++ b/crates/core/src/tracing/otel.rs @@ -0,0 +1,181 @@ +//! Standards-configured OpenTelemetry SDK metrics pipeline. +//! +//! Strictly isolated from [`super::telemetry`]: nothing here reads +//! `TelemetryConfig`, and the endpoint never falls back to +//! `DEFAULT_TELEMETRY_ENDPOINT`. The two features are independent by design — +//! see `docs/design/otel-metrics-exporter.md`. + +use crate::config::OtelConfig; + +/// Why the OTel metrics exporter was not started. +/// +/// Mirrors `telemetry::TelemetrySuppression` so both pipelines refuse to ship +/// data from a test process, but the decision is computed from `OtelConfig` +/// alone — the two flags never consult each other. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[allow(dead_code)] +pub(crate) enum OtelSuppression { + /// Operator left `otel-telemetry-enabled` off (the default). + Disabled, + /// `--id` test environment (integration/CLI harness sets `is_test_environment`). + TestEnvironmentFlag, + /// A `cfg(test)` build or a binary running under a cargo test/bench harness. + TestHarness, +} + +/// Decide whether the metrics exporter should be suppressed. +/// +/// Pure and side-effect free: callers pass `cfg!(test)` and the result of +/// `telemetry::running_under_cargo_test()` so this is testable for a +/// production release binary (must NOT suppress) from inside a test process, +/// which by construction trips both test signals. +/// +/// Suppression is keyed only on signals a real release binary never matches, +/// and deliberately NOT on `cfg!(feature = "testing")` — that flag leaks onto +/// the shipped binary through Cargo feature unification with `fdev` and +/// silently disabled telemetry across the fleet once already (#4366, the +/// 0.2.81 blackout). See `telemetry::telemetry_suppression_reason`. +#[allow(dead_code)] +pub(crate) fn otel_suppression_reason( + config: &OtelConfig, + is_test_build: bool, + running_under_cargo_test: bool, +) -> Option { + if !config.enabled { + return Some(OtelSuppression::Disabled); + } + if config.is_test_environment { + return Some(OtelSuppression::TestEnvironmentFlag); + } + if is_test_build || running_under_cargo_test { + return Some(OtelSuppression::TestHarness); + } + None +} + +/// Endpoint to hand to `MetricExporter`'s builder, or `None` to let the SDK +/// resolve it. +/// +/// Required precedence is env > config file > SDK default, but +/// `opentelemetry-otlp` 0.32 inverts the first two: `resolve_http_endpoint` +/// (`src/exporter/http/mod.rs`) checks the programmatic value FIRST and only +/// then `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` / `OTEL_EXPORTER_OTLP_ENDPOINT`. +/// So whenever either variable is set we return `None` and stay out of the +/// way. It also appends the `/v1/metrics` signal path only on the env-var +/// path, so a config-file value gets the path appended here. +#[allow(dead_code)] +pub(crate) fn resolve_metrics_endpoint( + cfg_endpoint: Option<&str>, + metrics_env: Option<&str>, + generic_env: Option<&str>, +) -> Option { + let is_set = |v: Option<&str>| v.is_some_and(|s| !s.trim().is_empty()); + if is_set(metrics_env) || is_set(generic_env) { + return None; + } + let base = cfg_endpoint.map(str::trim).filter(|s| !s.is_empty())?; + Some(format!("{}/v1/metrics", base.trim_end_matches('/'))) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::config::OtelConfig; + + fn enabled_config() -> OtelConfig { + OtelConfig { + enabled: true, + endpoint: None, + is_test_environment: false, + } + } + + #[test] + fn production_shaped_input_is_not_suppressed() { + assert_eq!( + otel_suppression_reason(&enabled_config(), false, false), + None, + "a real release binary with the flag on must export" + ); + } + + #[test] + fn every_test_signal_suppresses() { + let disabled = OtelConfig { + enabled: false, + ..enabled_config() + }; + assert_eq!( + otel_suppression_reason(&disabled, false, false), + Some(OtelSuppression::Disabled) + ); + + let test_env = OtelConfig { + is_test_environment: true, + ..enabled_config() + }; + assert_eq!( + otel_suppression_reason(&test_env, false, false), + Some(OtelSuppression::TestEnvironmentFlag) + ); + + assert_eq!( + otel_suppression_reason(&enabled_config(), true, false), + Some(OtelSuppression::TestHarness), + "cfg(test) build" + ); + assert_eq!( + otel_suppression_reason(&enabled_config(), false, true), + Some(OtelSuppression::TestHarness), + "running from a cargo deps/ harness" + ); + } + + #[test] + fn metrics_env_wins_over_generic_env_and_config() { + // Both env forms mean "let the SDK resolve it", because + // opentelemetry-otlp gives a programmatic endpoint priority over the + // env vars — passing one would invert the required precedence. + assert_eq!( + resolve_metrics_endpoint( + Some("http://from-config:4318"), + Some("http://from-metrics-env:4318/v1/metrics"), + Some("http://from-generic-env:4318"), + ), + None + ); + assert_eq!( + resolve_metrics_endpoint( + Some("http://from-config:4318"), + None, + Some("http://from-generic-env:4318"), + ), + None + ); + } + + #[test] + fn config_endpoint_gets_the_signal_path_appended() { + // The SDK appends /v1/metrics only on the env-var path; a programmatic + // endpoint is used verbatim, so we append it ourselves. + assert_eq!( + resolve_metrics_endpoint(Some("http://collector:4318"), None, None), + Some("http://collector:4318/v1/metrics".to_string()) + ); + assert_eq!( + resolve_metrics_endpoint(Some("http://collector:4318/"), None, None), + Some("http://collector:4318/v1/metrics".to_string()), + "trailing slash must not double up" + ); + } + + #[test] + fn nothing_configured_defers_to_the_sdk_default() { + assert_eq!(resolve_metrics_endpoint(None, None, None), None); + assert_eq!( + resolve_metrics_endpoint(Some(" "), None, None), + None, + "a blank endpoint is not a configuration" + ); + } +} diff --git a/crates/core/src/tracing/telemetry.rs b/crates/core/src/tracing/telemetry.rs index c286f32e28..7850c4b546 100644 --- a/crates/core/src/tracing/telemetry.rs +++ b/crates/core/src/tracing/telemetry.rs @@ -612,7 +612,7 @@ fn try_enqueue_event( /// /// Best-effort: if `current_exe()` fails we return `false` and fall back to the /// compile-time guards, which already cover unit tests and CI. -fn running_under_cargo_test() -> bool { +pub(crate) fn running_under_cargo_test() -> bool { std::env::current_exe() .ok() .and_then(|exe| { From d43f0a992dcf320a8248a55460e1e17e0636b1a7 Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Sat, 1 Aug 2026 17:43:23 -0500 Subject: [PATCH 04/17] feat(otel): export metrics through the OpenTelemetry SDK Installs a global meter provider backed by an OTLP/HTTP exporter, plus one RSS gauge so the pipeline carries a real datapoint end to end. Future instrumentation is a global::meter call at the site, with no registry to keep in sync. The OTel crates become non-optional because the pipeline ships in every build, not just trace-ot ones. --- crates/core/Cargo.toml | 11 ++- crates/core/src/node.rs | 7 ++ crates/core/src/tracing/otel.rs | 130 +++++++++++++++++++++++++++++++- 3 files changed, 142 insertions(+), 6 deletions(-) diff --git a/crates/core/Cargo.toml b/crates/core/Cargo.toml index a97212b004..909a81bd21 100644 --- a/crates/core/Cargo.toml +++ b/crates/core/Cargo.toml @@ -126,8 +126,13 @@ tracing-core = { workspace = true } tracing-opentelemetry = { workspace = true, optional = true } tracing-subscriber = { workspace = true, features = ["json"], optional = true } tracing-appender = { workspace = true, optional = true } -opentelemetry-otlp = { workspace = true, optional = true } -opentelemetry_sdk = { workspace = true, optional = true } +# Non-optional: the SDK metrics pipeline (tracing::otel) ships in every build. +# Default features already give us metrics + http-proto + reqwest-blocking-client. +# The blocking client is load-bearing, not incidental: PeriodicReader exports +# from a dedicated thread via futures_executor::block_on, where an async reqwest +# client would have no tokio reactor. +opentelemetry-otlp = { workspace = true } +opentelemetry_sdk = { workspace = true } hkdf = { workspace = true } keyring = { workspace = true } @@ -230,7 +235,7 @@ winres = "0.1" default = ["redb", "trace", "websocket", "wasmtime-backend"] sqlite = ["sqlx"] trace = ["tracing-subscriber", "tracing-appender"] -trace-ot = ["opentelemetry-jaeger", "trace", "tracing-opentelemetry", "opentelemetry-otlp"] +trace-ot = ["opentelemetry-jaeger", "trace", "tracing-opentelemetry"] websocket = ["axum/ws"] testing = ["freenet-stdlib/testing", "parking_lot/deadlock_detection"] console-subscriber = ["dep:console-subscriber"] diff --git a/crates/core/src/node.rs b/crates/core/src/node.rs index d899fb75aa..17c363698b 100644 --- a/crates/core/src/node.rs +++ b/crates/core/src/node.rs @@ -818,6 +818,13 @@ impl NodeConfig { registers.push(Box::new(telemetry)); } + // Independent of the TelemetryReporter above: a separate opt-in + // (`otel-telemetry-enabled`), a separate endpoint, and a separate + // collector. It is not a NetEventRegister — it installs a global + // meter provider that instrumentation reaches via + // `opentelemetry::global::meter`. + crate::tracing::otel::init(&self.config.otel, self.local_peer_id_string()); + (DynamicRegister::new(registers), flush_handle) }; let cfg = self.config.clone(); diff --git a/crates/core/src/tracing/otel.rs b/crates/core/src/tracing/otel.rs index 9f28ca8d9e..9a4d46242c 100644 --- a/crates/core/src/tracing/otel.rs +++ b/crates/core/src/tracing/otel.rs @@ -13,7 +13,6 @@ use crate::config::OtelConfig; /// data from a test process, but the decision is computed from `OtelConfig` /// alone — the two flags never consult each other. #[derive(Debug, Clone, Copy, PartialEq, Eq)] -#[allow(dead_code)] pub(crate) enum OtelSuppression { /// Operator left `otel-telemetry-enabled` off (the default). Disabled, @@ -35,7 +34,6 @@ pub(crate) enum OtelSuppression { /// the shipped binary through Cargo feature unification with `fdev` and /// silently disabled telemetry across the fleet once already (#4366, the /// 0.2.81 blackout). See `telemetry::telemetry_suppression_reason`. -#[allow(dead_code)] pub(crate) fn otel_suppression_reason( config: &OtelConfig, is_test_build: bool, @@ -63,7 +61,6 @@ pub(crate) fn otel_suppression_reason( /// So whenever either variable is set we return `None` and stay out of the /// way. It also appends the `/v1/metrics` signal path only on the env-var /// path, so a config-file value gets the path appended here. -#[allow(dead_code)] pub(crate) fn resolve_metrics_endpoint( cfg_endpoint: Option<&str>, metrics_env: Option<&str>, @@ -77,6 +74,117 @@ pub(crate) fn resolve_metrics_endpoint( Some(format!("{}/v1/metrics", base.trim_end_matches('/'))) } +use opentelemetry::{KeyValue, global}; +use opentelemetry_otlp::{ExporterBuildError, MetricExporter, WithExportConfig}; +use opentelemetry_sdk::{Resource, metrics::SdkMeterProvider}; + +/// Instrumentation scope name for every instrument this crate registers. +const METER_NAME: &str = "freenet"; + +/// Start the OpenTelemetry SDK metrics pipeline and install it as the +/// process-global meter provider. +/// +/// No-op when suppressed (see [`otel_suppression_reason`]) and best-effort +/// otherwise: an exporter that cannot be built logs a warning and the node +/// starts anyway. Metrics export must never be a startup dependency. +/// +/// After this returns, instrumentation anywhere in the crate is just +/// `opentelemetry::global::meter("freenet")` — there is deliberately no wrapper +/// type or registry to keep in sync. +pub fn init(config: &OtelConfig, local_peer_id: String) { + if let Some(reason) = otel_suppression_reason( + config, + cfg!(test), + super::telemetry::running_under_cargo_test(), + ) { + tracing::debug!(?reason, "OTel metrics exporter not started"); + return; + } + + let endpoint = resolve_metrics_endpoint( + config.endpoint.as_deref(), + std::env::var(opentelemetry_otlp::OTEL_EXPORTER_OTLP_METRICS_ENDPOINT) + .ok() + .as_deref(), + std::env::var(opentelemetry_otlp::OTEL_EXPORTER_OTLP_ENDPOINT) + .ok() + .as_deref(), + ); + + match build_provider(endpoint.as_deref(), local_peer_id) { + Ok(provider) => { + // ponytail: no shutdown hook. `set_meter_provider` holds a + // reference for the process lifetime and PeriodicReader exports + // every 60s (OTEL_METRIC_EXPORT_INTERVAL), so at most one partial + // interval is lost at exit. If that tail ever matters, keep the + // provider in a OnceLock and call `shutdown()` from the graceful + // shutdown path in `bin/freenet.rs`. + global::set_meter_provider(provider); + register_process_metrics(); + tracing::info!( + endpoint = endpoint + .as_deref() + .unwrap_or(""), + "OTel metrics exporter started" + ); + } + Err(error) => { + tracing::warn!( + %error, + "OTel metrics exporter failed to start; node continues without metrics" + ); + } + } +} + +/// Build the OTLP/HTTP exporter and meter provider. +/// +/// `endpoint` is `None` when the standard env vars should win — see +/// [`resolve_metrics_endpoint`] for why calling `with_endpoint` at all would +/// override them. +pub(crate) fn build_provider( + endpoint: Option<&str>, + local_peer_id: String, +) -> Result { + let mut builder = MetricExporter::builder().with_http(); + if let Some(endpoint) = endpoint { + builder = builder.with_endpoint(endpoint); + } + let exporter = builder.build()?; + + // `service.name` is overridden by OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES + // when the operator sets them; the SDK reads those itself. + let resource = Resource::builder() + .with_service_name("freenet-peer") + .with_attribute(KeyValue::new("peer.id", local_peer_id)) + .build(); + + Ok(SdkMeterProvider::builder() + .with_periodic_exporter(exporter) + .with_resource(resource) + .build()) +} + +/// Register the instruments this crate owns. +/// +/// Must run AFTER `global::set_meter_provider`: `global::meter` binds to +/// whatever provider is installed at call time. +fn register_process_metrics() { + let meter = global::meter(METER_NAME); + // The handle is dropped on purpose — the callback is registered into the + // pipeline at `build()` and observed on every collection cycle regardless. + let _rss = meter + .u64_observable_gauge("freenet.process.memory.rss") + .with_unit("By") + .with_description("Resident set size of the freenet process") + .with_callback(|observer| { + if let Some(rss) = crate::node::resource_metrics::rss_bytes() { + observer.observe(rss, &[]); + } + }) + .build(); +} + #[cfg(test)] mod tests { use super::*; @@ -178,4 +286,20 @@ mod tests { "a blank endpoint is not a configuration" ); } + + #[tokio::test] + async fn provider_builds_inside_a_tokio_runtime() { + // Two things under test. First, exporter construction must not panic + // when it happens inside an async context — the OTLP HTTP exporter uses + // reqwest's BLOCKING client because PeriodicReader exports from its own + // thread. Second, an unreachable collector must not surface as a build + // error: export failures are asynchronous and must never fail node + // startup. Port 1 is chosen because nothing can be listening there. + let provider = build_provider( + Some("http://127.0.0.1:1/v1/metrics"), + "peer-under-test".to_string(), + ) + .expect("exporter build must succeed against an unreachable collector"); + provider.shutdown().expect("clean shutdown"); + } } From 4e3a4f57be34229c2246146a5ddd607845365266 Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Sat, 1 Aug 2026 17:51:16 -0500 Subject: [PATCH 05/17] docs(otel): document the two independent telemetry pipelines The isolation between telemetry-enabled and otel-telemetry-enabled is a design constraint, not an accident, so it belongs where the next person reads before touching either. --- AGENTS.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 187d4463c6..5bd313110a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -325,6 +325,35 @@ until a runtime path actually reads it. Operator-facing documentation (encryption model, migration matrix, `freenet secrets` CLI) lives in [`docs/secrets-at-rest.md`](docs/secrets-at-rest.md). +## Two independent telemetry pipelines + +`telemetry-enabled` / `telemetry-endpoint` feed the project's central +dashboard through a hand-rolled OTLP-JSON log POST (`tracing/telemetry.rs`). + +`otel-telemetry-enabled` / `otel-endpoint` are a **separate, unrelated** +OpenTelemetry SDK metrics pipeline (`tracing/otel.rs`). The two share no +config, no endpoint, and no fallback in either direction — enabling or +disabling one has no effect on the other, and `otel-endpoint` must never +default to the dashboard collector. + +The otel pipeline honors the standard variables, which take priority over +`otel-endpoint` in `config.toml`: +`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, `OTEL_EXPORTER_OTLP_ENDPOINT`, +`OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_SERVICE_NAME`, +`OTEL_RESOURCE_ATTRIBUTES`, `OTEL_METRIC_EXPORT_INTERVAL`. Without any of +them it exports to `http://localhost:4318`. + +The proof-of-life gauge `freenet.process.memory.rss` is sourced from +`node::resource_metrics::rss_bytes()`, which is implemented for Linux only. +On macOS and Windows the gauge registers but reports no datapoints — an +empty series there is expected, not a broken pipeline. + +Adding an instrument is one call at the site — no registry, no wrapper: + + opentelemetry::global::meter("freenet").u64_counter("freenet.some.thing").build() + +Design: [`docs/design/otel-metrics-exporter.md`](docs/design/otel-metrics-exporter.md). + ## External Resources - API docs: https://docs.rs/freenet From d6ce2814662ebda557d1335e293511f730433a50 Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Sat, 1 Aug 2026 18:22:51 -0500 Subject: [PATCH 06/17] fix(otel): flatten otel config keys and trim the OTLP feature set Config::otel used serde(default) instead of serde(flatten), so the documented flat config.toml keys (otel-telemetry-enabled, otel-endpoint) were silently ignored. opentelemetry-otlp also had no TLS backend, so an https:// collector would never export, and its default features pulled in the trace/logs exporters despite metrics being the only goal. --- Cargo.lock | 102 ++++++++++++++++++++++++++- Cargo.toml | 2 +- crates/core/Cargo.toml | 20 ++++-- crates/core/src/config.rs | 51 +++++++++++++- docs/design/otel-metrics-exporter.md | 14 ++-- 5 files changed, 177 insertions(+), 12 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index a9bb0ee823..271b4dc0d7 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -279,6 +279,28 @@ version = "1.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" +[[package]] +name = "aws-lc-rs" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5ec2f1fc3ec205783a5da9a7e6c1509cc69dedf09a1949e412c1e18469326d00" +dependencies = [ + "aws-lc-sys", + "zeroize", +] + +[[package]] +name = "aws-lc-sys" +version = "0.41.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a2f9779ce85b93ab6170dd940ad0169b5766ff848247aff13bb788b832fe3f4" +dependencies = [ + "cc", + "cmake", + "dunce", + "fs_extra", +] + [[package]] name = "axum" version = "0.8.9" @@ -783,6 +805,15 @@ version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" +[[package]] +name = "cmake" +version = "0.1.58" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678" +dependencies = [ + "cc", +] + [[package]] name = "cmov" version = "0.5.4" @@ -1330,7 +1361,7 @@ checksum = "79fc3b6dd0b87ba36e565715bf9a2ced221311db47bd18011676f24a6066edbc" dependencies = [ "curl-sys", "libc", - "openssl-probe", + "openssl-probe 0.1.6", "openssl-sys", "schannel", "socket2", @@ -2338,6 +2369,12 @@ dependencies = [ "serde", ] +[[package]] +name = "fs_extra" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c" + [[package]] name = "fsevent-sys" version = "4.1.0" @@ -4593,6 +4630,12 @@ version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d05e27ee213611ffe7d6348b942e8f942b37114c00cc03cec254295a4a17852e" +[[package]] +name = "openssl-probe" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" + [[package]] name = "openssl-sys" version = "0.9.116" @@ -5317,6 +5360,7 @@ version = "0.11.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560" dependencies = [ + "aws-lc-rs", "bytes 1.12.1", "getrandom 0.4.2", "lru-slab", @@ -5691,13 +5735,19 @@ dependencies = [ "http-body", "http-body-util", "hyper", + "hyper-rustls", "hyper-util", "js-sys", "log", "percent-encoding", "pin-project-lite", + "quinn", + "rustls", + "rustls-pki-types", + "rustls-platform-verifier", "sync_wrapper", "tokio", + "tokio-rustls", "tower", "tower-http 0.6.11", "tower-service", @@ -5817,6 +5867,7 @@ version = "0.23.40" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b" dependencies = [ + "aws-lc-rs", "log", "once_cell", "ring", @@ -5826,6 +5877,18 @@ dependencies = [ "zeroize", ] +[[package]] +name = "rustls-native-certs" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d" +dependencies = [ + "openssl-probe 0.2.1", + "rustls-pki-types", + "schannel", + "security-framework 3.7.0", +] + [[package]] name = "rustls-pki-types" version = "1.14.1" @@ -5836,12 +5899,40 @@ dependencies = [ "zeroize", ] +[[package]] +name = "rustls-platform-verifier" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0" +dependencies = [ + "core-foundation 0.10.1", + "core-foundation-sys", + "jni 0.22.4", + "log", + "once_cell", + "rustls", + "rustls-native-certs", + "rustls-platform-verifier-android", + "rustls-webpki", + "security-framework 3.7.0", + "security-framework-sys", + "webpki-root-certs", + "windows-sys 0.52.0", +] + +[[package]] +name = "rustls-platform-verifier-android" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f" + [[package]] name = "rustls-webpki" version = "0.103.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" dependencies = [ + "aws-lc-rs", "ring", "rustls-pki-types", "untrusted", @@ -8248,6 +8339,15 @@ dependencies = [ "system-deps", ] +[[package]] +name = "webpki-root-certs" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31141ce3fc3e300ae89b78c0dd67f9708061d1d2eda54b8209346fd6be9a92c" +dependencies = [ + "rustls-pki-types", +] + [[package]] name = "webpki-roots" version = "0.26.11" diff --git a/Cargo.toml b/Cargo.toml index 954db07222..fde077ff86 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -90,7 +90,7 @@ rand = "0.9" # Observability opentelemetry = "0.32" opentelemetry-jaeger = "0.22" -opentelemetry-otlp = "0.32.0" +opentelemetry-otlp = { version = "0.32.0", default-features = false } opentelemetry_sdk = { version = "0.32", features = ["rt-tokio"] } tracing = "0.1" # Bumping this? Re-read `rolling.rs`'s `impl io::Write for diff --git a/crates/core/Cargo.toml b/crates/core/Cargo.toml index 909a81bd21..f34f23d12e 100644 --- a/crates/core/Cargo.toml +++ b/crates/core/Cargo.toml @@ -127,11 +127,21 @@ tracing-opentelemetry = { workspace = true, optional = true } tracing-subscriber = { workspace = true, features = ["json"], optional = true } tracing-appender = { workspace = true, optional = true } # Non-optional: the SDK metrics pipeline (tracing::otel) ships in every build. -# Default features already give us metrics + http-proto + reqwest-blocking-client. -# The blocking client is load-bearing, not incidental: PeriodicReader exports -# from a dedicated thread via futures_executor::block_on, where an async reqwest -# client would have no tokio reactor. -opentelemetry-otlp = { workspace = true } +# default-features = false + explicit list keeps the trace/logs exporters (and +# their prost/opentelemetry-proto deps) out of the default build; only metrics +# is a goal here (see docs/design/otel-metrics-exporter.md, non-goals). +# reqwest-blocking-client is load-bearing, not incidental: PeriodicReader +# exports from a dedicated thread via futures_executor::block_on, where an +# async reqwest client would have no tokio reactor. reqwest-rustls adds TLS +# so https:// collector endpoints actually work (otherwise every export to an +# https endpoint silently fails at connect time). +opentelemetry-otlp = { workspace = true, features = [ + "http-proto", + "metrics", + "reqwest-blocking-client", + "reqwest-rustls", + "internal-logs", +] } opentelemetry_sdk = { workspace = true } hkdf = { workspace = true } diff --git a/crates/core/src/config.rs b/crates/core/src/config.rs index 2c1d45bf46..8ecfdfa4d5 100644 --- a/crates/core/src/config.rs +++ b/crates/core/src/config.rs @@ -1731,7 +1731,7 @@ pub struct Config { /// OpenTelemetry SDK metrics exporter settings. Strictly isolated from /// `telemetry` above — see `docs/design/otel-metrics-exporter.md`. - #[serde(default)] + #[serde(flatten)] pub otel: OtelConfig, /// Maximum seconds to wait on graceful shutdown for in-flight @@ -6556,6 +6556,55 @@ shutdown-drain-secs = 42 ); } + /// C1 regression: the round-trip guard test above only round-trips the + /// serializer's OWN output, so a key-shape mismatch (nested `[otel]` + /// table vs. the flat keys the design spec and AGENTS.md document) is + /// invisible to it. Write the literal documented `config.toml` text and + /// confirm the flat keys actually parse into `Config::otel`. + #[tokio::test] + async fn otel_flat_config_toml_keys_are_honored() { + let temp_dir = tempfile::tempdir().unwrap(); + + // Base build to create the on-disk secrets + a valid config.toml for + // every OTHER field (all of them are `#[serde(flatten)]`d scalars, so + // this baseline has no `[table]` headers at all). + clap_bare_args(temp_dir.path()).build().await.unwrap(); + let base = tokio::fs::read_to_string(temp_dir.path().join("config.toml")) + .await + .unwrap(); + + // Strip whatever otel shape build() just wrote (pre-fix: a nested + // `[otel]` header + its two keys; post-fix: the two flat keys) so the + // literal lines appended below are unambiguous root-level keys. + let base: String = base + .lines() + .filter(|line| { + *line != "[otel]" + && !line.starts_with("otel-telemetry-enabled") + && !line.starts_with("otel-endpoint") + }) + .map(|line| format!("{line}\n")) + .collect(); + + // The literal config.toml the design spec (Configuration table) and + // AGENTS.md document: flat keys at the file root, no `[otel]` table. + let literal = format!( + "{base}otel-telemetry-enabled = true\notel-endpoint = \"http://collector.example:4318\"\n" + ); + std::fs::write(temp_dir.path().join("config.toml"), literal).unwrap(); + + let rebuilt = clap_bare_args(temp_dir.path()).build().await.unwrap(); + assert!( + rebuilt.otel.enabled, + "documented flat `otel-telemetry-enabled` key must be honored" + ); + assert_eq!( + rebuilt.otel.endpoint.as_deref(), + Some("http://collector.example:4318"), + "documented flat `otel-endpoint` key must be honored" + ); + } + #[test] fn otel_endpoint_never_defaults_to_the_dashboard_collector() { // Hard isolation requirement: the two pipelines share no backend. diff --git a/docs/design/otel-metrics-exporter.md b/docs/design/otel-metrics-exporter.md index 4e24df4ef5..e2212e3a51 100644 --- a/docs/design/otel-metrics-exporter.md +++ b/docs/design/otel-metrics-exporter.md @@ -185,10 +185,16 @@ The provider is registered globally; nothing is stored on `Node`. ## Risks -- Making the two crates non-optional grows the default build. Both already - compile in any `trace-ot` build and share reqwest with the existing HTTP - client, so the marginal cost is small — confirm cross-compile targets still - build (`.github/workflows/cross-compile.yml`, +- Making the two crates non-optional grows the default build. They do NOT + share reqwest with the existing HTTP client: `opentelemetry-otlp` pulls + reqwest 0.13, a separate major version from the workspace's reqwest 0.12. + `opentelemetry-otlp` is trimmed to `default-features = false` plus exactly + the metrics/http-proto/reqwest-blocking-client/reqwest-rustls/internal-logs + features, so the trace/logs exporters (explicit non-goals) and their + prost/opentelemetry-proto deps stay out of the default build. The bounded + remaining cost is a second reqwest major version plus the metrics-only OTLP + exporter — confirm cross-compile targets still build + (`.github/workflows/cross-compile.yml`, `crates/core/tests/cross_compile_feature_split.rs`). - `global::set_meter_provider` is process-global. A `trace-ot` build also sets OTel globals (tracer provider, not meter provider) — confirm no conflict. From 36000bd23f14e1eb67c5816dfb65aede8e42312f Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Sat, 1 Aug 2026 18:34:43 -0500 Subject: [PATCH 07/17] docs(otel): correct the OTLP feature-trim claim The trim drops the logs exporter only. http-proto mandates trace, prost and opentelemetry-proto, so the comment and the design doc were both wrong about what stays out of the default build. Also record the aws-lc-rs/CMake cost that reqwest-rustls adds to the release cross-compile. --- crates/core/Cargo.toml | 7 ++++--- docs/design/otel-metrics-exporter.md | 15 +++++++++++---- 2 files changed, 15 insertions(+), 7 deletions(-) diff --git a/crates/core/Cargo.toml b/crates/core/Cargo.toml index f34f23d12e..81fdd807d1 100644 --- a/crates/core/Cargo.toml +++ b/crates/core/Cargo.toml @@ -127,9 +127,10 @@ tracing-opentelemetry = { workspace = true, optional = true } tracing-subscriber = { workspace = true, features = ["json"], optional = true } tracing-appender = { workspace = true, optional = true } # Non-optional: the SDK metrics pipeline (tracing::otel) ships in every build. -# default-features = false + explicit list keeps the trace/logs exporters (and -# their prost/opentelemetry-proto deps) out of the default build; only metrics -# is a goal here (see docs/design/otel-metrics-exporter.md, non-goals). +# default-features = false + explicit list drops the logs exporter; only metrics +# is a goal here (see docs/design/otel-metrics-exporter.md, non-goals). It does +# NOT drop trace: `http-proto` mandates `trace`, `prost` and +# opentelemetry-proto, so those stay in the graph regardless. # reqwest-blocking-client is load-bearing, not incidental: PeriodicReader # exports from a dedicated thread via futures_executor::block_on, where an # async reqwest client would have no tokio reactor. reqwest-rustls adds TLS diff --git a/docs/design/otel-metrics-exporter.md b/docs/design/otel-metrics-exporter.md index e2212e3a51..132d1d1687 100644 --- a/docs/design/otel-metrics-exporter.md +++ b/docs/design/otel-metrics-exporter.md @@ -190,12 +190,19 @@ The provider is registered globally; nothing is stored on `Node`. reqwest 0.13, a separate major version from the workspace's reqwest 0.12. `opentelemetry-otlp` is trimmed to `default-features = false` plus exactly the metrics/http-proto/reqwest-blocking-client/reqwest-rustls/internal-logs - features, so the trace/logs exporters (explicit non-goals) and their - prost/opentelemetry-proto deps stay out of the default build. The bounded - remaining cost is a second reqwest major version plus the metrics-only OTLP - exporter — confirm cross-compile targets still build + features. That drops the logs exporter only: `http-proto` mandates `trace`, + `prost` and `opentelemetry-proto`, so those remain in the default build and + the trim is narrower than "metrics-only". The remaining cost is a second + reqwest major version plus the trace+metrics OTLP exporter — confirm + cross-compile targets still build (`.github/workflows/cross-compile.yml`, `crates/core/tests/cross_compile_feature_split.rs`). +- `reqwest-rustls` pulls `aws-lc-rs`/`aws-lc-sys`, which builds C sources via + CMake, and rustls 0.23 unification turns it on workspace-wide. Prebuilt musl + bindings ship for both release targets and the musl jobs are native-arch, so + this is expected to work — but `cross-compile.yml` never runs on PRs and does + not install `cmake`, so a failure would land after merge on the release path. + Verify with a `workflow_dispatch` run on the branch before merging. - `global::set_meter_provider` is process-global. A `trace-ot` build also sets OTel globals (tracer provider, not meter provider) — confirm no conflict. - `reqwest/blocking` gets enabled workspace-wide by feature unification, which From 991a48b74de203011af6f175969e5761c4c1329a Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Sun, 2 Aug 2026 01:09:03 -0500 Subject: [PATCH 08/17] feat(otel): export node, ring, transport, and queue metrics Adds the instruments behind the dashboard's connection-status tiles plus transport wire counters and RTT/cwnd distributions. Observable callbacks read state that already existed for the local dashboard, so nothing new lands on the hot path except two packet atomics. Identity moves from a PeerId to the transport public key fingerprint: PeerId renders as {pub_key}@{addr}, so the previous peer.id resource attribute exported this node's socket address and re-identified the node on every address change. No instrument carries an attribute identifying the remote end of a connection. Claude-Session: https://claude.ai/code/session_015Entifnvj528KjPErWsRyJ --- AGENTS.md | 32 +- crates/core/src/node.rs | 6 +- crates/core/src/node/network_status.rs | 37 +++ crates/core/src/tracing/otel.rs | 429 ++++++++++++++++++++++++- crates/core/src/transport/metrics.rs | 30 ++ docs/design/otel-metrics-exporter.md | 49 ++- 6 files changed, 564 insertions(+), 19 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5bd313110a..ce8f846231 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -343,15 +343,39 @@ The otel pipeline honors the standard variables, which take priority over `OTEL_RESOURCE_ATTRIBUTES`, `OTEL_METRIC_EXPORT_INTERVAL`. Without any of them it exports to `http://localhost:4318`. +All instruments are registered in `tracing/otel.rs::register_metrics`. Two +kinds, and the choice is not stylistic: + +- **Observable** (gauges / `observable_counter`) own a callback that reads + existing state at collection time — the transport's cumulative counters, or + `network_status::otel_metrics_snapshot()`. Nothing is added to the hot path. + Read cumulative, never-reset values only: the `TransportSnapshot` fields are + period accumulators that `take_snapshot` zeroes for the legacy telemetry + worker, so observing one as a counter yields a non-monotonic series whenever + `telemetry-enabled` is also on. +- **Synchronous** (`Histogram` / `Counter`) are held in the `INSTRUMENTS` + `OnceLock` and recorded via the `record_*` helpers. They need a stored handle + because an instrument built before `global::set_meter_provider` binds to the + no-op provider forever; when the exporter is off the helpers are one atomic + load and a branch. + +Every histogram is base-2 exponential via a single `with_view` in +`build_provider` — do not add explicit bucket boundaries per instrument. + +No instrument carries an attribute identifying the remote end of a connection. +Per-datapoint attributes cost per series and multiply by bucket count on +histograms; identifying THIS node is a resource attribute +(`service.instance.id`), which rides once per export batch. That id is the +transport public key fingerprint, NOT a `PeerId` — `PeerId` renders as +`{pub_key}@{addr}` and would export our socket address and re-identify the node +on every address change. Note also that "peer" means the other end of a +connection; metrics about ourselves use `freenet.node.*`. + The proof-of-life gauge `freenet.process.memory.rss` is sourced from `node::resource_metrics::rss_bytes()`, which is implemented for Linux only. On macOS and Windows the gauge registers but reports no datapoints — an empty series there is expected, not a broken pipeline. -Adding an instrument is one call at the site — no registry, no wrapper: - - opentelemetry::global::meter("freenet").u64_counter("freenet.some.thing").build() - Design: [`docs/design/otel-metrics-exporter.md`](docs/design/otel-metrics-exporter.md). ## External Resources diff --git a/crates/core/src/node.rs b/crates/core/src/node.rs index 17c363698b..e34b18e0df 100644 --- a/crates/core/src/node.rs +++ b/crates/core/src/node.rs @@ -823,7 +823,11 @@ impl NodeConfig { // collector. It is not a NetEventRegister — it installs a global // meter provider that instrumentation reaches via // `opentelemetry::global::meter`. - crate::tracing::otel::init(&self.config.otel, self.local_peer_id_string()); + // Public-key fingerprint, NOT `local_peer_id_string()`: a PeerId + // renders as `{pub_key}@{addr}`, which would put this node's socket + // address on every exported batch and re-identify the node on every + // address change. + crate::tracing::otel::init(&self.config.otel, self.key_pair.public().to_string()); (DynamicRegister::new(registers), flush_handle) }; diff --git a/crates/core/src/node/network_status.rs b/crates/core/src/node/network_status.rs index 54d55d9326..06652e9e46 100644 --- a/crates/core/src/node/network_status.rs +++ b/crates/core/src/node/network_status.rs @@ -111,6 +111,34 @@ pub struct RingStatsSnapshot { pub lattice_probe_improvements: u64, } +/// Scalar-only view for the OTel metrics callbacks. +/// +/// Deliberately NOT [`get_snapshot`]: that builds per-peer and per-contract +/// vectors and formats failure HTML, and the SDK has no batch-callback API in +/// 0.32 — every observable instrument gets its own callback, so the exporter +/// would pay that cost once per instrument per collection cycle. +#[derive(Debug, Clone, Default)] +pub(crate) struct OtelMetricsSnapshot { + pub connection_attempts: u32, + pub ring: RingStatsSnapshot, + pub fair_queue: crate::contract::FairQueueStats, +} + +/// Read the scalars the OTel exporter observes, or `None` before the node has +/// registered its status (metrics simply report nothing until then). +pub(crate) fn otel_metrics_snapshot() -> Option { + let connection_attempts = NETWORK_STATUS.get()?.read().ok()?.connection_attempts; + Some(OtelMetricsSnapshot { + connection_attempts, + ring: RING_STATS_PROVIDER + .read() + .as_ref() + .map(|provider| provider()) + .unwrap_or_default(), + fair_queue: crate::contract::fair_queue_stats(), + }) +} + static GOVERNANCE_PROVIDER: parking_lot::RwLock> = parking_lot::RwLock::new(None); @@ -826,6 +854,15 @@ pub fn record_peer_disconnected(addr: SocketAddr) { /// Audit: `grep -rn "record_op_result" crates/core/src/operations/` /// must show coverage for every op type with a driver. pub fn record_op_result(op_type: OpType, success: bool) { + crate::tracing::otel::record_op_result( + match op_type { + OpType::Get => "get", + OpType::Put => "put", + OpType::Update => "update", + OpType::Subscribe => "subscribe", + }, + success, + ); if let Some(status) = NETWORK_STATUS.get() { if let Ok(mut s) = status.write() { let counter = match op_type { diff --git a/crates/core/src/tracing/otel.rs b/crates/core/src/tracing/otel.rs index 9a4d46242c..72a73df144 100644 --- a/crates/core/src/tracing/otel.rs +++ b/crates/core/src/tracing/otel.rs @@ -74,9 +74,14 @@ pub(crate) fn resolve_metrics_endpoint( Some(format!("{}/v1/metrics", base.trim_end_matches('/'))) } +use opentelemetry::metrics::{Counter, Histogram}; use opentelemetry::{KeyValue, global}; use opentelemetry_otlp::{ExporterBuildError, MetricExporter, WithExportConfig}; -use opentelemetry_sdk::{Resource, metrics::SdkMeterProvider}; +use opentelemetry_sdk::{ + Resource, + metrics::{Aggregation, Instrument, InstrumentKind, SdkMeterProvider, Stream}, +}; +use std::sync::OnceLock; /// Instrumentation scope name for every instrument this crate registers. const METER_NAME: &str = "freenet"; @@ -88,10 +93,9 @@ const METER_NAME: &str = "freenet"; /// otherwise: an exporter that cannot be built logs a warning and the node /// starts anyway. Metrics export must never be a startup dependency. /// -/// After this returns, instrumentation anywhere in the crate is just -/// `opentelemetry::global::meter("freenet")` — there is deliberately no wrapper -/// type or registry to keep in sync. -pub fn init(config: &OtelConfig, local_peer_id: String) { +/// `instance_id` identifies THIS node and must not contain a network address: +/// see [`build_provider`]. +pub fn init(config: &OtelConfig, instance_id: String) { if let Some(reason) = otel_suppression_reason( config, cfg!(test), @@ -111,7 +115,7 @@ pub fn init(config: &OtelConfig, local_peer_id: String) { .as_deref(), ); - match build_provider(endpoint.as_deref(), local_peer_id) { + match build_provider(endpoint.as_deref(), instance_id) { Ok(provider) => { // ponytail: no shutdown hook. `set_meter_provider` holds a // reference for the process lifetime and PeriodicReader exports @@ -120,7 +124,7 @@ pub fn init(config: &OtelConfig, local_peer_id: String) { // provider in a OnceLock and call `shutdown()` from the graceful // shutdown path in `bin/freenet.rs`. global::set_meter_provider(provider); - register_process_metrics(); + register_metrics(); tracing::info!( endpoint = endpoint .as_deref() @@ -142,9 +146,15 @@ pub fn init(config: &OtelConfig, local_peer_id: String) { /// `endpoint` is `None` when the standard env vars should win — see /// [`resolve_metrics_endpoint`] for why calling `with_endpoint` at all would /// override them. +/// +/// `instance_id` MUST be the transport public key fingerprint, never a +/// `PeerId`: `PeerId`'s `Display` is `{pub_key}@{addr}`, so using it would put +/// this node's socket address in every exported batch AND make the identity +/// churn on every address change. The fingerprint is public by construction +/// (peers learn it on connect) and stable for the life of the keypair. pub(crate) fn build_provider( endpoint: Option<&str>, - local_peer_id: String, + instance_id: String, ) -> Result { let mut builder = MetricExporter::builder().with_http(); if let Some(endpoint) = endpoint { @@ -154,25 +164,153 @@ pub(crate) fn build_provider( // `service.name` is overridden by OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES // when the operator sets them; the SDK reads those itself. + // + // Resource attributes ride once per export batch, not per datapoint, so + // identifying THIS node here costs nothing per series — unlike a + // per-datapoint attribute, which is why no instrument below carries one + // identifying the remote end of a connection. let resource = Resource::builder() - .with_service_name("freenet-peer") - .with_attribute(KeyValue::new("peer.id", local_peer_id)) + .with_service_name("freenet-node") + .with_attribute(KeyValue::new("service.instance.id", instance_id)) + .with_attribute(KeyValue::new("service.version", env!("CARGO_PKG_VERSION"))) + .with_attribute(KeyValue::new("os.type", std::env::consts::OS)) + .with_attribute(KeyValue::new("host.arch", std::env::consts::ARCH)) .build(); Ok(SdkMeterProvider::builder() .with_periodic_exporter(exporter) .with_resource(resource) + // Every histogram this crate records is base-2 exponential rather than + // explicit-bucket: the SDK's default boundaries are tuned for + // millisecond latency and are useless for byte-scale instruments, and + // exponential buckets self-adjust instead of needing a hand-picked + // boundary set per instrument. + .with_view(|instrument: &Instrument| { + (instrument.kind() == InstrumentKind::Histogram) + .then(|| { + Stream::builder() + .with_aggregation(Aggregation::Base2ExponentialHistogram { + max_size: 160, + max_scale: 20, + record_min_max: true, + }) + .build() + .ok() + }) + .flatten() + }) .build()) } +/// Synchronous instruments, recorded from the code paths they measure. +/// +/// These need a handle held somewhere, unlike the observable instruments below +/// whose callbacks the pipeline owns. Kept behind a `OnceLock` set at the end +/// of [`init`] for two reasons: instruments built before +/// `global::set_meter_provider` would bind to the no-op provider forever, and +/// when the exporter is disabled the record helpers collapse to one relaxed +/// atomic load and a branch. +struct Instruments { + rtt: Histogram, + cwnd: Histogram, + transfers: Counter, + nat_traversal: Counter, + operations: Counter, +} + +static INSTRUMENTS: OnceLock = OnceLock::new(); + +/// Record a transport RTT sample. No-op until [`init`] installs the pipeline. +pub(crate) fn record_rtt_ms(rtt_ms: f64) { + if let Some(i) = INSTRUMENTS.get() { + i.rtt.record(rtt_ms, &[]); + } +} + +/// Record a congestion-window sample. +pub(crate) fn record_cwnd(cwnd_bytes: u64) { + if let Some(i) = INSTRUMENTS.get() { + i.cwnd.record(cwnd_bytes, &[]); + } +} + +/// Record a stream transfer outcome (`completed` / `failed`). +pub(crate) fn record_transfer(result: &'static str) { + if let Some(i) = INSTRUMENTS.get() { + i.transfers.add(1, &[KeyValue::new("result", result)]); + } +} + +/// Record a NAT traversal outcome (`attempt` / `established` / +/// `failed_error` / `failed_version`). +pub(crate) fn record_nat_traversal(result: &'static str) { + if let Some(i) = INSTRUMENTS.get() { + i.nat_traversal.add(1, &[KeyValue::new("result", result)]); + } +} + +/// Record an operation outcome. `op` is one of get/put/update/subscribe. +/// +/// ponytail: outcome only, no duration histogram — no driver measures its own +/// elapsed time today, and adding one means threading `TimeSource` through +/// every `op_ctx_task` (raw `Instant::now()` is banned in this crate). Add the +/// histogram when someone needs operation latency percentiles. +pub(crate) fn record_op_result(op: &'static str, success: bool) { + if let Some(i) = INSTRUMENTS.get() { + i.operations.add( + 1, + &[ + KeyValue::new("op", op), + KeyValue::new("result", if success { "success" } else { "failure" }), + ], + ); + } +} + /// Register the instruments this crate owns. /// /// Must run AFTER `global::set_meter_provider`: `global::meter` binds to /// whatever provider is installed at call time. -fn register_process_metrics() { +/// +/// Observable handles are dropped on purpose — the callback is registered into +/// the pipeline at `build()` and observed on every collection cycle regardless. +/// The SDK has no batch-callback API, so each one reads +/// [`network_status::otel_metrics_snapshot`] independently; that is why the +/// accessor is a cheap scalar read rather than the dashboard's `get_snapshot`. +fn register_metrics() { let meter = global::meter(METER_NAME); - // The handle is dropped on purpose — the callback is registered into the - // pipeline at `build()` and observed on every collection cycle regardless. + + let registered = INSTRUMENTS.set(Instruments { + rtt: meter + .f64_histogram("freenet.transport.rtt") + .with_unit("ms") + .with_description("Round-trip time observed on transport connections") + .build(), + cwnd: meter + .u64_histogram("freenet.transport.cwnd") + .with_unit("By") + .with_description("Congestion window samples") + .build(), + transfers: meter + .u64_counter("freenet.transport.transfers") + .with_description("Stream transfers by outcome") + .build(), + nat_traversal: meter + .u64_counter("freenet.transport.nat_traversal") + .with_description("Outbound NAT traversal attempts by outcome") + .build(), + operations: meter + .u64_counter("freenet.operation.results") + .with_description("Completed operations by type and outcome") + .build(), + }); + if registered.is_err() { + // A second `init` would leave the sync instruments bound to the first + // provider while the observable ones move to the new one — loud rather + // than silently half-migrated. + tracing::warn!("OTel instruments already registered; keeping the first set"); + } + let _rss = meter .u64_observable_gauge("freenet.process.memory.rss") .with_unit("By") @@ -183,6 +321,230 @@ fn register_process_metrics() { } }) .build(); + + register_transport_metrics(&meter); + register_ring_metrics(&meter); + register_queue_metrics(&meter); +} + +/// Wire-level counters, read from the cumulative (never-reset) transport +/// totals. +/// +/// Deliberately NOT read from `TransportSnapshot`: those fields are period +/// accumulators that `take_snapshot` zeroes for the legacy telemetry worker, so +/// observing them as counters would report a non-monotonic series whenever +/// `telemetry-enabled` is also on. +fn register_transport_metrics(meter: &opentelemetry::metrics::Meter) { + use crate::transport::TRANSPORT_METRICS; + + let _bytes = meter + .u64_observable_counter("freenet.transport.bytes") + .with_unit("By") + .with_description( + "Wire bytes. Sent is metered at the socket (includes keep-alives, ACKs and \ + NAT probes); received is metered post-authentication, so the two directions \ + are deliberately not symmetric.", + ) + .with_callback(|observer| { + observer.observe( + TRANSPORT_METRICS.cumulative_bytes_sent(), + &[KeyValue::new("direction", "sent")], + ); + observer.observe( + TRANSPORT_METRICS.cumulative_bytes_received(), + &[KeyValue::new("direction", "received")], + ); + }) + .build(); + + let _packets = meter + .u64_observable_counter("freenet.transport.packets") + .with_description("UDP datagrams, metered at the same sites as freenet.transport.bytes") + .with_callback(|observer| { + let (sent, received) = TRANSPORT_METRICS.cumulative_packets(); + observer.observe(sent, &[KeyValue::new("direction", "sent")]); + observer.observe(received, &[KeyValue::new("direction", "received")]); + }) + .build(); +} + +/// Ring / topology state, mirroring the dashboard's connection-status tiles. +fn register_ring_metrics(meter: &opentelemetry::metrics::Meter) { + use crate::node::network_status::otel_metrics_snapshot as snapshot; + + let _connections = meter + .u64_observable_gauge("freenet.ring.connections") + .with_description("Active ring connections") + .with_callback(|observer| { + if let Some(s) = snapshot() { + observer.observe(s.ring.connection_count as u64, &[]); + } + }) + .build(); + + let _hosted = meter + .u64_observable_gauge("freenet.node.contracts.hosted") + .with_description("Contracts currently hosted by this node") + .with_callback(|observer| { + if let Some(s) = snapshot() { + observer.observe(s.ring.hosted_contracts as u64, &[]); + } + }) + .build(); + + let _attempts = meter + .u64_observable_counter("freenet.connect.attempts") + .with_description("Connection attempts made since startup") + .with_callback(|observer| { + if let Some(s) = snapshot() { + observer.observe(s.connection_attempts as u64, &[]); + } + }) + .build(); + + let _lattice = meter + .u64_observable_gauge("freenet.ring.lattice.neighbor") + .with_description( + "1 when this node holds its closest connected ring neighbor on that side. \ + Held does not mean tight — compare distances across nodes.", + ) + .with_callback(|observer| { + if let Some(s) = snapshot() { + observer.observe( + s.ring.lattice_has_successor as u64, + &[KeyValue::new("position", "successor")], + ); + observer.observe( + s.ring.lattice_has_predecessor as u64, + &[KeyValue::new("position", "predecessor")], + ); + } + }) + .build(); + + let _distance = meter + .f64_observable_gauge("freenet.ring.lattice.neighbor.distance") + .with_description("Ring distance to each held lattice edge; absent when unheld") + .with_callback(|observer| { + if let Some(s) = snapshot() { + if let Some(d) = s.ring.lattice_successor_distance { + observer.observe(d, &[KeyValue::new("position", "successor")]); + } + if let Some(d) = s.ring.lattice_predecessor_distance { + observer.observe(d, &[KeyValue::new("position", "predecessor")]); + } + } + }) + .build(); + + let _probes = meter + .u64_observable_counter("freenet.ring.lattice.probes") + .with_description( + "Route-to-self probes fired, and lattice improvements observed. Counted \ + independently — an improvement lands some ticks after the probe that caused \ + it, so the ratio is a convergence gauge, not a success rate.", + ) + .with_callback(|observer| { + if let Some(s) = snapshot() { + observer.observe( + s.ring.lattice_probes_issued, + &[KeyValue::new("result", "issued")], + ); + observer.observe( + s.ring.lattice_probe_improvements, + &[KeyValue::new("result", "improvement")], + ); + } + }) + .build(); + + let _updates = meter + .u64_observable_counter("freenet.contract.updates") + .with_description("Relayed UPDATEs by admission outcome") + .with_callback(|observer| { + if let Some(s) = snapshot() { + observer.observe( + s.ring.updates_accepted, + &[KeyValue::new("result", "accepted")], + ); + observer.observe( + s.ring.updates_rate_limited, + &[KeyValue::new("result", "rate_limited")], + ); + observer.observe( + s.ring.updates_capacity_dropped, + &[KeyValue::new("result", "capacity_dropped")], + ); + } + }) + .build(); +} + +/// Executor fair-queue occupancy and admission outcomes. +fn register_queue_metrics(meter: &opentelemetry::metrics::Meter) { + use crate::node::network_status::otel_metrics_snapshot as snapshot; + + let _depth = meter + .u64_observable_gauge("freenet.contract.queue.depth") + .with_description("Current fair-queue occupancy") + .with_callback(|observer| { + if let Some(s) = snapshot() { + let q = &s.fair_queue; + for (tier, depth) in [ + ("total", q.depth_total), + ("client_local", q.depth_client_local), + ("network_relay", q.depth_network_relay), + ("background", q.depth_background), + ] { + observer.observe(depth as u64, &[KeyValue::new("queue", tier)]); + } + } + }) + .build(); + + // A gauge, not a counter: `high_water` is a running maximum, and a + // collector that saw it as a counter would read a plateau as "no traffic". + // It exists because a burst between two 60s collections leaves no trace in + // the instantaneous depth. + let _high_water = meter + .u64_observable_gauge("freenet.contract.queue.depth.high_water") + .with_description("Highest fair-queue occupancy reached since startup") + .with_callback(|observer| { + if let Some(s) = snapshot() { + observer.observe(s.fair_queue.high_water as u64, &[]); + } + }) + .build(); + + let _rejected = meter + .u64_observable_counter("freenet.contract.queue.rejected") + .with_description( + "Fair-queue admission rejections. global_capacity is node-wide saturation; \ + per_contract is one noisy contract hitting its own cap.", + ) + .with_callback(|observer| { + if let Some(s) = snapshot() { + observer.observe( + s.fair_queue.rejected_global_capacity, + &[KeyValue::new("reason", "global_capacity")], + ); + observer.observe( + s.fair_queue.rejected_per_contract, + &[KeyValue::new("reason", "per_contract")], + ); + } + }) + .build(); + + let _shed = meter + .u64_observable_counter("freenet.contract.queue.background_shed") + .with_description("Background events shed to make room for higher-priority work") + .with_callback(|observer| { + if let Some(s) = snapshot() { + observer.observe(s.fair_queue.background_shed, &[]); + } + }) + .build(); } #[cfg(test)] @@ -287,6 +649,47 @@ mod tests { ); } + #[test] + fn instance_id_carries_no_network_address() { + // The exporter identifies this node by its transport public key + // fingerprint. `PeerId` renders as `{pub_key}@{addr}`, so using it — as + // the exporter originally did — leaks our socket address into every + // batch and re-identifies the node whenever the address changes. + let keypair = crate::transport::TransportKeypair::new(); + let instance_id = keypair.public().to_string(); + + assert!(!instance_id.is_empty()); + assert!( + !instance_id.contains('@') && !instance_id.contains(':'), + "instance id must not embed an address, got {instance_id}" + ); + + let peer_id = crate::node::PeerId::new( + keypair.public().clone(), + "203.0.113.7:31337".parse().expect("valid addr"), + ); + assert!( + peer_id.to_string().contains("203.0.113.7"), + "guard is meaningless if PeerId stops embedding the address" + ); + } + + #[test] + fn record_helpers_are_inert_without_a_pipeline() { + // Every record helper is called from production paths that run whether + // or not the exporter is enabled, so an unset OnceLock must be a no-op + // rather than a panic or an implicit no-op-provider binding. + record_rtt_ms(12.5); + record_cwnd(4096); + record_transfer("completed"); + record_nat_traversal("attempt"); + record_op_result("get", true); + assert!( + INSTRUMENTS.get().is_none(), + "recording must not lazily bind instruments to the no-op provider" + ); + } + #[tokio::test] async fn provider_builds_inside_a_tokio_runtime() { // Two things under test. First, exporter construction must not panic diff --git a/crates/core/src/transport/metrics.rs b/crates/core/src/transport/metrics.rs index fd61c77ca2..dd374dfa68 100644 --- a/crates/core/src/transport/metrics.rs +++ b/crates/core/src/transport/metrics.rs @@ -73,6 +73,15 @@ pub struct TransportMetrics { cumulative_bytes_sent: AtomicU64, cumulative_bytes_received: AtomicU64, + // Cumulative wire-packet counters (never reset). Metered at exactly the + // same two call sites as the cumulative byte counters above, so the same + // send-at-socket / receive-post-auth asymmetry applies. Kept here rather + // than incremented as an OTel counter at the call site because these fire + // once per UDP datagram — an observable counter reading these costs + // nothing on the hot path. + cumulative_packets_sent: AtomicU64, + cumulative_packets_received: AtomicU64, + // Timing accumulators (for computing averages) total_transfer_time_ms: AtomicU64, @@ -163,6 +172,8 @@ impl TransportMetrics { bytes_received: AtomicU64::new(0), cumulative_bytes_sent: AtomicU64::new(0), cumulative_bytes_received: AtomicU64::new(0), + cumulative_packets_sent: AtomicU64::new(0), + cumulative_packets_received: AtomicU64::new(0), total_transfer_time_ms: AtomicU64::new(0), peak_throughput_bps: AtomicU64::new(0), peak_cwnd_bytes: AtomicU32::new(0), @@ -185,6 +196,7 @@ impl TransportMetrics { /// Record a completed outbound transfer. pub fn record_transfer_completed(&self, stats: &super::TransferStats) { + crate::tracing::otel::record_transfer("completed"); // Use saturating arithmetic to prevent overflow (though extremely unlikely // in practice - would require billions of transfers or exabytes of data) self.transfers_completed @@ -269,6 +281,7 @@ impl TransportMetrics { /// and the bytes it did put on the wire are already counted at the socket /// layer by `record_packet_sent`. pub fn record_transfer_failed(&self) { + crate::tracing::otel::record_transfer("failed"); // Saturating, matching `record_transfer_completed` — a pinned u32::MAX // is a better failure mode than wrapping to 0 and reporting a healthy // node. @@ -281,11 +294,13 @@ impl TransportMetrics { /// Record the start of an outbound NAT traversal attempt. pub fn record_nat_traversal_attempt(&self) { + crate::tracing::otel::record_nat_traversal("attempt"); self.nat_traversal_attempts.fetch_add(1, Ordering::Relaxed); } /// Record a NAT traversal attempt that established a connection. pub fn record_nat_traversal_established(&self) { + crate::tracing::otel::record_nat_traversal("established"); self.nat_traversal_established .fetch_add(1, Ordering::Relaxed); } @@ -293,6 +308,7 @@ impl TransportMetrics { /// Record a NAT traversal attempt that failed for a non-version reason /// (unreachable / symmetric-NAT / generic transport error). pub fn record_nat_traversal_failed_error(&self) { + crate::tracing::otel::record_nat_traversal("failed_error"); self.nat_traversal_failed_error .fetch_add(1, Ordering::Relaxed); } @@ -300,12 +316,14 @@ impl TransportMetrics { /// Record a NAT traversal attempt that failed due to a protocol version /// mismatch. pub fn record_nat_traversal_failed_version(&self) { + crate::tracing::otel::record_nat_traversal("failed_version"); self.nat_traversal_failed_version .fetch_add(1, Ordering::Relaxed); } /// Record a cwnd sample (called periodically or on transfer completion). pub(crate) fn record_cwnd_sample(&self, cwnd_bytes: u32) { + crate::tracing::otel::record_cwnd(cwnd_bytes as u64); self.cwnd_sum .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |v| { Some(v.saturating_add(cwnd_bytes as u64)) @@ -337,6 +355,7 @@ impl TransportMetrics { /// keep-alive path is what keeps RTT statistics populated for quiet, /// long-lived connections that rarely complete a stream transfer (#4000). pub(crate) fn record_rtt_sample(&self, rtt_us: u64) { + crate::tracing::otel::record_rtt_ms(rtt_us as f64 / 1000.0); self.rtt_sum_us .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |v| { Some(v.saturating_add(rtt_us)) @@ -412,6 +431,7 @@ impl TransportMetrics { pub fn record_packet_sent(&self, remote_addr: SocketAddr, bytes: u64) { self.cumulative_bytes_sent .fetch_add(bytes, Ordering::Relaxed); + self.cumulative_packets_sent.fetch_add(1, Ordering::Relaxed); self.record_per_peer(remote_addr, bytes, |s| &s.bytes_sent); } @@ -428,6 +448,8 @@ impl TransportMetrics { pub fn record_packet_received(&self, remote_addr: SocketAddr, bytes: u64) { self.cumulative_bytes_received .fetch_add(bytes, Ordering::Relaxed); + self.cumulative_packets_received + .fetch_add(1, Ordering::Relaxed); self.record_per_peer(remote_addr, bytes, |s| &s.bytes_received); } @@ -436,6 +458,14 @@ impl TransportMetrics { self.cumulative_bytes_received.load(Ordering::Relaxed) } + /// Read cumulative packets sent/received without resetting counters. + pub(crate) fn cumulative_packets(&self) -> (u64, u64) { + ( + self.cumulative_packets_sent.load(Ordering::Relaxed), + self.cumulative_packets_received.load(Ordering::Relaxed), + ) + } + /// Record per-peer bytes for the given direction. /// /// Bounded to [`MAX_TRACKED_PEERS`] entries with LRU eviction: when the diff --git a/docs/design/otel-metrics-exporter.md b/docs/design/otel-metrics-exporter.md index 132d1d1687..c429a2bac2 100644 --- a/docs/design/otel-metrics-exporter.md +++ b/docs/design/otel-metrics-exporter.md @@ -25,7 +25,54 @@ untouched. - Migrating existing events onto the new pipeline. - Logs and traces exporters. The endpoint resolution is designed to serve all three signals; only metrics ships now. -- Any instrumentation beyond the single proof-of-life gauge. +- Per-connection metrics. Identifying the remote end of a connection is a + per-datapoint attribute, multiplied by bucket count on histograms; the + aggregate signals below answer the operational questions without it. Transfer + and connection *events*, if wanted, belong on a log pipeline rather than as + metric series. +- Operation and contract-execution latency histograms. No driver measures its + own elapsed time today, and raw `Instant::now()` is banned in `crates/core/` + (`.claude/rules/testing.md`), so adding them means threading `TimeSource` + through every `op_ctx_task`. `freenet.operation.results` ships the outcome + counter now; the duration histogram is deferred until someone needs + percentiles. + +## Instruments + +Registered in `tracing/otel.rs::register_metrics`. + +| Instrument | Kind | Attributes | Source | +|---|---|---|---| +| `freenet.process.memory.rss` | gauge | — | `node::resource_metrics::rss_bytes` (Linux only) | +| `freenet.transport.bytes` | counter | `direction` | `cumulative_bytes_{sent,received}` | +| `freenet.transport.packets` | counter | `direction` | `cumulative_packets_{sent,received}` | +| `freenet.transport.transfers` | counter | `result` | `record_transfer_{completed,failed}` | +| `freenet.transport.nat_traversal` | counter | `result` | `record_nat_traversal_*` | +| `freenet.transport.rtt` | histogram | — | `record_rtt_sample` | +| `freenet.transport.cwnd` | histogram | — | `record_cwnd_sample` | +| `freenet.operation.results` | counter | `op`, `result` | `network_status::record_op_result` | +| `freenet.ring.connections` | gauge | — | `RingStatsSnapshot` | +| `freenet.node.contracts.hosted` | gauge | — | `RingStatsSnapshot` | +| `freenet.connect.attempts` | counter | — | `NetworkStatus` | +| `freenet.ring.lattice.neighbor` | gauge | `position` | `RingStatsSnapshot` | +| `freenet.ring.lattice.neighbor.distance` | gauge | `position` | `RingStatsSnapshot` | +| `freenet.ring.lattice.probes` | counter | `result` | `RingStatsSnapshot` | +| `freenet.contract.updates` | counter | `result` | `RingStatsSnapshot` | +| `freenet.contract.queue.depth` | gauge | `queue` | `FairQueueStats` | +| `freenet.contract.queue.depth.high_water` | gauge | — | `FairQueueStats` | +| `freenet.contract.queue.rejected` | counter | `reason` | `FairQueueStats` | +| `freenet.contract.queue.background_shed` | counter | — | `FairQueueStats` | + +Everything but the histograms and the four synchronous counters is an +observable callback over state that already existed for the local dashboard. + +Resource attributes: `service.instance.id` (transport public key fingerprint — +never a `PeerId`, which embeds our socket address), `service.version`, +`os.type`, `host.arch`, plus whatever `OTEL_RESOURCE_ATTRIBUTES` adds. + +Not instrumented: `TransportMetrics::slowdowns_triggered` is read and reset but +never incremented anywhere in the tree, so it is dead and was left out rather +than exported as a permanent zero. ## Isolation requirement (hard) From bf33c5d33c48bad1af5bbeb14e659f08539ca2ad Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Tue, 4 Aug 2026 04:56:32 -0500 Subject: [PATCH 09/17] feat(otel): authenticate collector exports with XEdDSA bearer tokens New otel-auth-mode config (default "freenet", or "disabled"). Each export request carries Authorization: Bearer freenet////, signed with the x25519 transport key itself (XEdDSA) so the collector verifies node identity against the same pubkey peers and UIs see. Resource attrs freenet.node.pubkey / freenet.node.fingerprint replace service.instance.id; the fingerprint is derivable from the pubkey, so the collector can validate both. Claude-Session: https://claude.ai/code/session_01QNvMzjXWyYsMeiJF5DrB2s --- AGENTS.md | 25 ++- Cargo.lock | 37 +++- Cargo.toml | 11 ++ crates/core/Cargo.toml | 18 +- crates/core/src/config.rs | 67 ++++++- crates/core/src/node.rs | 2 +- crates/core/src/tracing/otel.rs | 291 +++++++++++++++++++++++++--- crates/core/src/transport/crypto.rs | 20 ++ 8 files changed, 441 insertions(+), 30 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index ce8f846231..10b3d47aa3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -336,6 +336,20 @@ config, no endpoint, and no fallback in either direction — enabling or disabling one has no effect on the other, and `otel-endpoint` must never default to the dashboard collector. +The otel pipeline authenticates to the collector per `otel-auth-mode`: +`freenet` (default) sends a per-request +`Authorization: Bearer freenet////` +token — an XEdDSA (Signal construction, `xeddsa` crate) signature over +`freenet///` (epoch seconds, 16-byte nonce, all +base58), signed with the x25519 transport secret itself +(`TransportKeypair::auth_token_signer`). `` is the FULL base58 +x25519 transport public key — the node's one identity — so the collector +verifies by converting Montgomery→Edwards (sign bit 0) and running stock +Ed25519 verification; no shared secret, no second key, nothing assertable. +`disabled` sends no header. Tokens are built per request +(`tracing/otel.rs::bearer_token` via a custom `HttpClient`) so the timestamp +stays fresh; future auth methods get new enum variants. + The otel pipeline honors the standard variables, which take priority over `otel-endpoint` in `config.toml`: `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, `OTEL_EXPORTER_OTLP_ENDPOINT`, @@ -364,9 +378,14 @@ Every histogram is base-2 exponential via a single `with_view` in No instrument carries an attribute identifying the remote end of a connection. Per-datapoint attributes cost per series and multiply by bucket count on -histograms; identifying THIS node is a resource attribute -(`service.instance.id`), which rides once per export batch. That id is the -transport public key fingerprint, NOT a `PeerId` — `PeerId` renders as +histograms; identifying THIS node is done with two resource attributes, +riding once per export batch. `freenet.node.pubkey` +is the base58 ed25519 verifying key derived from the transport keypair — +byte-equal to the bearer token's `` field, so the collector +self-validates the node id against the signing key after verifying the +signature. `freenet.node.fingerprint` is the transport public key fingerprint +(what UIs and the legacy dashboard show), for cross-referencing only — +unverifiable by the collector. Neither is ever a `PeerId` — `PeerId` renders as `{pub_key}@{addr}` and would export our socket address and re-identify the node on every address change. Note also that "peer" means the other end of a connection; metrics about ourselves use `freenet.node.*`. diff --git a/Cargo.lock b/Cargo.lock index 271b4dc0d7..012c9a60fd 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -912,6 +912,15 @@ version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" +[[package]] +name = "convert_case" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "633458d4ef8c78b72454de2d54fd6ab2e60f9e02be22f3c6104cdc8a4e0fceb9" +dependencies = [ + "unicode-segmentation", +] + [[package]] name = "cookie" version = "0.18.1" @@ -1545,10 +1554,12 @@ version = "2.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "799a97264921d8623a957f6c3b9011f3b5492f557bbb7a5a19b7fa6d06ba8dcb" dependencies = [ + "convert_case", "proc-macro2", "quote", "rustc_version", "syn 2.0.119", + "unicode-xid", ] [[package]] @@ -2066,6 +2077,7 @@ dependencies = [ "anyhow", "arbitrary", "argon2", + "async-trait", "axum", "bincode", "blake3", @@ -2081,6 +2093,7 @@ dependencies = [ "cookie", "criterion", "ctrlc", + "curve25519-dalek 4.1.3", "dashmap", "delegate", "directories", @@ -2100,6 +2113,7 @@ dependencies = [ "hickory-resolver", "hkdf", "hostname", + "http 1.4.2", "httptest", "ipnet", "keyring", @@ -2109,6 +2123,7 @@ dependencies = [ "muda", "notify", "opentelemetry 0.32.0", + "opentelemetry-http 0.32.0", "opentelemetry-jaeger", "opentelemetry-otlp", "opentelemetry_sdk 0.32.1", @@ -2116,7 +2131,9 @@ dependencies = [ "pav_regression", "pin-project", "proptest", + "rand 0.10.1", "rand 0.9.4", + "rand_core 0.10.1", "redb", "regex", "renegade-ml", @@ -2164,6 +2181,7 @@ dependencies = [ "winres", "wry", "x25519-dalek 3.0.0", + "xeddsa", "xz2", "zeroize", "zip", @@ -5687,6 +5705,7 @@ checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147" dependencies = [ "base64 0.22.1", "bytes 1.12.1", + "futures-channel", "futures-core", "futures-util", "http 1.5.0", @@ -5917,7 +5936,7 @@ dependencies = [ "security-framework 3.7.0", "security-framework-sys", "webpki-root-certs", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -9091,6 +9110,22 @@ dependencies = [ "rustix", ] +[[package]] +name = "xeddsa" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4de76539e20b51353b2653cbf575bf987ab9fcef550fa59a2ff64562ed5b14cb" +dependencies = [ + "curve25519-dalek 4.1.3", + "derive_more", + "ed25519", + "ed25519-dalek", + "rand 0.10.1", + "sha2 0.11.0", + "x25519-dalek 2.0.1", + "zeroize", +] + [[package]] name = "xz2" version = "0.1.7" diff --git a/Cargo.toml b/Cargo.toml index fde077ff86..ee7f1c8cab 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -87,8 +87,19 @@ chrono = { version = "0.4", default-features = true } # Random rand = "0.9" +# XEdDSA (Signal's construction): signs with the x25519 transport key so the +# OTel collector can verify node identity against the SAME public key peers +# and UIs see. Its rand 0.10 dep is exposed as `rand10` for the sign nonce. +xeddsa = "1.1" +rand10 = { package = "rand", version = "0.10", default-features = false, features = ["sys_rng"] } +rand_core10 = { package = "rand_core", version = "0.10" } + # Observability +async-trait = "0.1" opentelemetry = "0.32" +# Trait/types only (HttpClient for the per-request auth wrapper); its reqwest +# integration targets reqwest 0.13 and the workspace is on 0.12, so no features. +opentelemetry-http = "0.32" opentelemetry-jaeger = "0.22" opentelemetry-otlp = { version = "0.32.0", default-features = false } opentelemetry_sdk = { version = "0.32", features = ["rt-tokio"] } diff --git a/crates/core/Cargo.toml b/crates/core/Cargo.toml index 81fdd807d1..e7e4a60e33 100644 --- a/crates/core/Cargo.toml +++ b/crates/core/Cargo.toml @@ -104,7 +104,9 @@ wasmtime = { workspace = true, optional = true } # PUT time must run regardless of which WASM execution backend is compiled in. wasmparser = { workspace = true } xz2 = { workspace = true } -reqwest = { workspace = true } +# `blocking` named explicitly (not just inherited via feature unification from +# opentelemetry-http) because tracing::otel constructs reqwest::blocking::Client. +reqwest = { workspace = true, features = ["blocking"] } # Hidden (no-echo) prompt for the `freenet secrets export/import` passphrase, so # secrets don't have to be passed on argv (where they leak via `ps`/history). rpassword = "7" @@ -136,6 +138,17 @@ tracing-appender = { workspace = true, optional = true } # async reqwest client would have no tokio reactor. reqwest-rustls adds TLS # so https:// collector endpoints actually work (otherwise every export to an # https endpoint silently fails at connect time). +# Direct dep so tracing::otel can implement HttpClient (per-request auth +# headers); the crate is already in the graph via opentelemetry-otlp. +opentelemetry-http = { workspace = true } +async-trait = { workspace = true } +http = { workspace = true } +# XEdDSA collector auth: sign with the x25519 transport key (see +# transport::crypto::TransportKeypair::auth_token_signer). rand10/rand_core10 +# exist only because xeddsa's rng bound is rand 0.10 (workspace rand is 0.9). +xeddsa = { workspace = true } +rand10 = { workspace = true } +rand_core10 = { workspace = true } opentelemetry-otlp = { workspace = true, features = [ "http-proto", "metrics", @@ -188,6 +201,9 @@ criterion = { workspace = true } freenet-stdlib = { features = ["net", "testing"], workspace = true } freenet-macros = { path = "../freenet-macros" } httptest = { workspace = true } +# Test-only: proves the collector can verify XEdDSA tokens with a stock +# ed25519 library after Montgomery->Edwards conversion (4.1.3 = xeddsa's). +curve25519-dalek = "4.1.3" libc = { workspace = true } # For sendmmsg syscall batching benchmarks parking_lot = { workspace = true, features = ["deadlock_detection"] } # Used by tests/in_process_restart.rs to directly probe the redb file lock as a diff --git a/crates/core/src/config.rs b/crates/core/src/config.rs index 8ecfdfa4d5..6a2b9211ca 100644 --- a/crates/core/src/config.rs +++ b/crates/core/src/config.rs @@ -981,6 +981,9 @@ impl ConfigArgs { if let Some(endpoint) = cfg.otel.endpoint { self.otel.endpoint.get_or_insert(endpoint); } + // Always emitted (non-Option in OtelConfig), so merge + // unconditionally; the CLI value still wins via get_or_insert. + self.otel.auth_mode.get_or_insert(cfg.otel.auth_mode); } // Validate the effective config (CLI + values merged from config.toml). @@ -1490,6 +1493,7 @@ impl ConfigArgs { otel: OtelConfig { enabled: self.otel.enabled, endpoint: self.otel.endpoint, + auth_mode: self.otel.auth_mode.unwrap_or_default(), // Same --id rule as telemetry: simulated networks and // integration tests must not ship data to a collector. is_test_environment: self.id.is_some(), @@ -2845,6 +2849,21 @@ fn default_iface_tx_enabled() -> bool { /// must always be an explicit operator choice. pub const DEFAULT_OTEL_ENDPOINT: &str = "http://localhost:4318"; +/// How the OTel exporter authenticates to the collector. +/// +/// `freenet` (the default) sends a per-request +/// `Authorization: Bearer freenet////` +/// token signed with a key derived from the node's transport keypair — see +/// `tracing::otel::bearer_token`. `disabled` sends no Authorization header. +/// Future methods get new variants. +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, clap::ValueEnum)] +#[serde(rename_all = "lowercase")] +pub enum OtelAuthMode { + #[default] + Freenet, + Disabled, +} + /// CLI/file args for the OpenTelemetry SDK metrics exporter. /// /// Strictly independent of [`TelemetryArgs`]: no shared field, no shared @@ -2882,6 +2901,14 @@ pub struct OtelArgs { #[arg(id = "otel_endpoint", long = "otel-endpoint")] #[serde(rename = "otel-endpoint", skip_serializing_if = "Option::is_none")] pub endpoint: Option, + + /// Collector authentication method. `Option` so `build()` can tell "not + /// given on the CLI" from an explicit choice and merge the config-file + /// value; resolves to [`OtelAuthMode::default`] (`freenet`) when neither + /// sets it. + #[arg(id = "otel_auth_mode", long = "otel-auth-mode", value_enum)] + #[serde(rename = "otel-auth-mode", skip_serializing_if = "Option::is_none")] + pub auth_mode: Option, } /// Resolved configuration for the OpenTelemetry SDK metrics exporter. @@ -2900,6 +2927,10 @@ pub struct OtelConfig { )] pub endpoint: Option, + /// Collector authentication method. + #[serde(default, rename = "otel-auth-mode")] + pub auth_mode: OtelAuthMode, + /// Whether this is a test environment (detected via `--id`). Mirrors /// [`TelemetryConfig::is_test_environment`]; suppresses export so test /// networks can't ship data to a collector. @@ -6528,6 +6559,34 @@ shutdown-drain-secs = 42 "otel-telemetry-enabled must default to false" ); assert_eq!(args.endpoint, None, "no implicit collector"); + assert_eq!( + args.auth_mode.unwrap_or_default(), + OtelAuthMode::Freenet, + "auth mode must default to the freenet bearer-token format" + ); + } + + #[test] + fn otel_auth_mode_parses_from_cli_and_file() { + use clap::Parser; + let none = ConfigArgs::try_parse_from(["freenet"]).expect("bare parse"); + assert_eq!(none.otel.auth_mode, None, "unset on the CLI stays None"); + let off = ConfigArgs::try_parse_from(["freenet", "--otel-auth-mode", "disabled"]) + .expect("disabled parse"); + assert_eq!(off.otel.auth_mode, Some(OtelAuthMode::Disabled)); + let on = ConfigArgs::try_parse_from(["freenet", "--otel-auth-mode", "freenet"]) + .expect("freenet parse"); + assert_eq!(on.otel.auth_mode, Some(OtelAuthMode::Freenet)); + + // The file spelling is the lowercase variant name. + let cfg: OtelConfig = toml::from_str("otel-auth-mode = \"disabled\"").unwrap(); + assert_eq!(cfg.auth_mode, OtelAuthMode::Disabled); + let cfg: OtelConfig = toml::from_str("").unwrap(); + assert_eq!( + cfg.auth_mode, + OtelAuthMode::Freenet, + "absent key -> default" + ); } #[test] @@ -7777,7 +7836,8 @@ shutdown-drain-secs = 42 otel: OtelConfig { enabled: true, endpoint: Some("http://example.invalid:4319".to_string()), - is_test_environment: false, // #[serde(skip)] — derived from --id + auth_mode: OtelAuthMode::Disabled, // non-default: default is Freenet + is_test_environment: false, // #[serde(skip)] — derived from --id }, shutdown_drain_secs: 77, disable_auto_update: true, // #[serde(skip)] — see destructure below @@ -7885,6 +7945,11 @@ shutdown-drain-secs = 42 "otel.endpoint — an operator's collector URL must survive the \ config.toml merge" ); + assert_eq!( + otel.auth_mode, seed.otel.auth_mode, + "otel.auth_mode — an operator's explicit `disabled` must survive \ + the config.toml merge, or auth silently re-enables on restart" + ); let NetworkApiConfig { address, diff --git a/crates/core/src/node.rs b/crates/core/src/node.rs index e34b18e0df..dbb1ee092b 100644 --- a/crates/core/src/node.rs +++ b/crates/core/src/node.rs @@ -827,7 +827,7 @@ impl NodeConfig { // renders as `{pub_key}@{addr}`, which would put this node's socket // address on every exported batch and re-identify the node on every // address change. - crate::tracing::otel::init(&self.config.otel, self.key_pair.public().to_string()); + crate::tracing::otel::init(&self.config.otel, &self.key_pair); (DynamicRegister::new(registers), flush_handle) }; diff --git a/crates/core/src/tracing/otel.rs b/crates/core/src/tracing/otel.rs index 72a73df144..f2158a4d17 100644 --- a/crates/core/src/tracing/otel.rs +++ b/crates/core/src/tracing/otel.rs @@ -76,13 +76,92 @@ pub(crate) fn resolve_metrics_endpoint( use opentelemetry::metrics::{Counter, Histogram}; use opentelemetry::{KeyValue, global}; -use opentelemetry_otlp::{ExporterBuildError, MetricExporter, WithExportConfig}; +use opentelemetry_http::{Bytes, HttpClient, HttpError, Request, Response}; +use opentelemetry_otlp::{ExporterBuildError, MetricExporter, WithExportConfig, WithHttpConfig}; use opentelemetry_sdk::{ Resource, metrics::{Aggregation, Instrument, InstrumentKind, SdkMeterProvider, Stream}, }; use std::sync::OnceLock; +/// Build one `freenet`-mode bearer token: +/// `freenet////`, where `` is +/// the XEdDSA signature over `freenet///`. +/// +/// `` is the base58 full x25519 transport public key — the node's one +/// real identity, the same key peers see and whose truncated fingerprint UIs +/// display. `` is seconds since the Unix epoch, `` is 16 +/// base58 random bytes for replay protection, `` is base58 too. +/// Freshly built per export request so the timestamp stays current. +/// +/// Collector-side verification needs no exotic library: convert the +/// Montgomery pubkey to Edwards (sign bit 0), then standard Ed25519 verify — +/// see `node_pubkey_is_verifiable_with_stock_ed25519` below. +pub(crate) fn bearer_token(signer: &xeddsa::xed25519::PrivateKey, pubkey_b58: &str) -> String { + use xeddsa::xeddsa::Sign; + // Wall-clock epoch seconds on purpose: the collector checks it against + // ITS clock, so simulation time would be meaningless here. + let timestamp = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or_default(); + let mut nonce_bytes = [0u8; 16]; + crate::config::GlobalRng::fill_bytes(&mut nonce_bytes); + let nonce = bs58::encode(nonce_bytes).into_string(); + let signed_payload = format!("freenet/{pubkey_b58}/{timestamp}/{nonce}"); + // OS entropy (SysRng), not GlobalRng: XEdDSA's Z randomness hedges the + // signature nonce, which is cryptographic material — the same exception + // documented in .claude/rules/code-style.md for keys/nonces. UnwrapErr is + // required because xeddsa's bound is the infallible rand 0.10 CryptoRng. + let signature: [u8; 64] = signer.sign( + signed_payload.as_bytes(), + rand_core10::UnwrapErr(rand10::rngs::SysRng), + ); + let signature = bs58::encode(signature).into_string(); + format!("{signed_payload}/{signature}") +} + +/// OTLP HTTP client that injects a fresh `Authorization: Bearer` token +/// (see [`bearer_token`]) into every export request, delegating the actual +/// send to the same blocking reqwest client the exporter would use anyway. +struct FreenetAuthClient { + inner: reqwest::blocking::Client, + signer: xeddsa::xed25519::PrivateKey, + pubkey_b58: String, +} + +// Manual impl: never print the signing key. +impl std::fmt::Debug for FreenetAuthClient { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("FreenetAuthClient").finish_non_exhaustive() + } +} + +#[async_trait::async_trait] +impl HttpClient for FreenetAuthClient { + async fn send_bytes(&self, mut request: Request) -> Result, HttpError> { + let token = bearer_token(&self.signer, &self.pubkey_b58); + request.headers_mut().insert( + http::header::AUTHORIZATION, + http::HeaderValue::from_str(&format!("Bearer {token}"))?, + ); + // Hand-rolled send, mirroring opentelemetry-http's blocking impl: + // that impl is on reqwest 0.13's client (opentelemetry-http's own + // dep), while the workspace is on 0.12, so we can't delegate to it. + // Blocking inside async is fine here for the same reason the SDK's + // default client is blocking: PeriodicReader exports via block_on on + // a dedicated thread. Fold both in when the workspace moves to 0.13. + let request: reqwest::blocking::Request = request.map(|body| body.to_vec()).try_into()?; + let mut response = self.inner.execute(request)?.error_for_status()?; + let headers = std::mem::take(response.headers_mut()); + let mut http_response = Response::builder() + .status(response.status()) + .body(response.bytes()?)?; + *http_response.headers_mut() = headers; + Ok(http_response) + } +} + /// Instrumentation scope name for every instrument this crate registers. const METER_NAME: &str = "freenet"; @@ -93,9 +172,12 @@ const METER_NAME: &str = "freenet"; /// otherwise: an exporter that cannot be built logs a warning and the node /// starts anyway. Metrics export must never be a startup dependency. /// -/// `instance_id` identifies THIS node and must not contain a network address: -/// see [`build_provider`]. -pub fn init(config: &OtelConfig, instance_id: String) { +/// `keypair` is the node's transport keypair: it yields the +/// `freenet.node.pubkey` / `freenet.node.fingerprint` resource attributes +/// (see [`build_provider`]) and, when `otel-auth-mode = "freenet"`, its +/// derived signing key authenticates every export request (see +/// [`bearer_token`]). +pub fn init(config: &OtelConfig, keypair: &crate::transport::TransportKeypair) { if let Some(reason) = otel_suppression_reason( config, cfg!(test), @@ -115,7 +197,19 @@ pub fn init(config: &OtelConfig, instance_id: String) { .as_deref(), ); - match build_provider(endpoint.as_deref(), instance_id) { + // `service.instance.id` IS the auth identity: the same base58 ed25519 + // verifying key the bearer token carries as ``, so the collector + // self-validates the node id against the signing key by string equality + // after verifying the signature. Derived from the keypair even when auth + // is disabled, so the id is stable across auth-mode changes. + let pubkey = bs58::encode(keypair.public_key_bytes()).into_string(); + let fingerprint = keypair.public().to_string(); + let auth_signer = match config.auth_mode { + crate::config::OtelAuthMode::Freenet => Some(keypair.auth_token_signer()), + crate::config::OtelAuthMode::Disabled => None, + }; + + match build_provider(endpoint.as_deref(), pubkey, fingerprint, auth_signer) { Ok(provider) => { // ponytail: no shutdown hook. `set_meter_provider` holds a // reference for the process lifetime and PeriodicReader exports @@ -147,19 +241,40 @@ pub fn init(config: &OtelConfig, instance_id: String) { /// [`resolve_metrics_endpoint`] for why calling `with_endpoint` at all would /// override them. /// -/// `instance_id` MUST be the transport public key fingerprint, never a -/// `PeerId`: `PeerId`'s `Display` is `{pub_key}@{addr}`, so using it would put -/// this node's socket address in every exported batch AND make the identity -/// churn on every address change. The fingerprint is public by construction -/// (peers learn it on connect) and stable for the life of the keypair. +/// Two identity resource attributes, both computed by [`init`] from the one +/// transport keypair: +/// +/// - `freenet.node.pubkey` — the base58 full x25519 transport public key, +/// byte-equal to the bearer token's `` field. The collector +/// verifies the token's XEdDSA signature against this key, so the identity +/// is self-validating and unforgeable. +/// - `freenet.node.fingerprint` — base58 of the FIRST 12 BYTES of the same +/// key (`TransportPublicKey::Display`, what UIs show). A pure public +/// function of `pubkey`, so the collector recomputes and checks it rather +/// than trusting it. +/// +/// Neither may ever be a `PeerId`: its `Display` is `{pub_key}@{addr}`, so +/// using it would put this node's socket address in every exported batch AND +/// make the identity churn on every address change. pub(crate) fn build_provider( endpoint: Option<&str>, - instance_id: String, + pubkey: String, + fingerprint: String, + auth_signer: Option, ) -> Result { let mut builder = MetricExporter::builder().with_http(); if let Some(endpoint) = endpoint { builder = builder.with_endpoint(endpoint); } + if let Some(signer) = auth_signer { + // Same blocking client the exporter defaults to (PeriodicReader + // exports off-runtime — see Cargo.toml), wrapped to sign each request. + builder = builder.with_http_client(FreenetAuthClient { + inner: reqwest::blocking::Client::new(), + signer, + pubkey_b58: pubkey.clone(), + }); + } let exporter = builder.build()?; // `service.name` is overridden by OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES @@ -171,7 +286,8 @@ pub(crate) fn build_provider( // identifying the remote end of a connection. let resource = Resource::builder() .with_service_name("freenet-node") - .with_attribute(KeyValue::new("service.instance.id", instance_id)) + .with_attribute(KeyValue::new("freenet.node.pubkey", pubkey)) + .with_attribute(KeyValue::new("freenet.node.fingerprint", fingerprint)) .with_attribute(KeyValue::new("service.version", env!("CARGO_PKG_VERSION"))) .with_attribute(KeyValue::new("os.type", std::env::consts::OS)) .with_attribute(KeyValue::new("host.arch", std::env::consts::ARCH)) @@ -556,10 +672,118 @@ mod tests { OtelConfig { enabled: true, endpoint: None, + auth_mode: Default::default(), is_test_environment: false, } } + /// One keypair plus its token, pre-split, for the verification tests. + fn token_fixture() -> (crate::transport::TransportKeypair, String) { + let keypair = crate::transport::TransportKeypair::new(); + let pubkey_b58 = bs58::encode(keypair.public_key_bytes()).into_string(); + let token = bearer_token(&keypair.auth_token_signer(), &pubkey_b58); + (keypair, token) + } + + #[test] + fn bearer_token_has_the_documented_shape_and_verifies() { + use xeddsa::xeddsa::Verify; + + let (keypair, token) = token_fixture(); + + let parts: Vec<&str> = token.split('/').collect(); + let [scheme, pubkey, timestamp, nonce, signature] = parts[..] else { + panic!("expected 5 /-separated parts, got {token}"); + }; + assert_eq!(scheme, "freenet"); + assert_eq!( + pubkey, + bs58::encode(keypair.public_key_bytes()).into_string(), + "pubkey part must be the full base58 x25519 transport public key" + ); + let ts: u64 = timestamp.parse().expect("timestamp is epoch seconds"); + assert!( + ts > 1_700_000_000, + "timestamp must be current epoch seconds" + ); + assert!(!nonce.is_empty()); + + // The signature covers everything before its own slash, and verifies + // against the token's OWN pubkey — the transport key itself. + let signed_payload = format!("freenet/{pubkey}/{timestamp}/{nonce}"); + let sig_bytes: [u8; 64] = bs58::decode(signature) + .into_vec() + .unwrap() + .try_into() + .expect("64-byte signature"); + xeddsa::xed25519::PublicKey(keypair.public_key_bytes()) + .verify(signed_payload.as_bytes(), &sig_bytes) + .expect("XEdDSA signature must verify against the transport pubkey"); + + // A forged payload with the same signature must fail. + assert!( + xeddsa::xed25519::PublicKey(keypair.public_key_bytes()) + .verify(b"freenet/forged", &sig_bytes) + .is_err() + ); + } + + #[test] + fn node_pubkey_is_verifiable_with_stock_ed25519() { + // The collector-side contract, spelled out: no xeddsa dependency + // needed there. Convert the Montgomery (x25519) pubkey to an Edwards + // point with sign bit 0, then run ordinary Ed25519 verification. + use ed25519_dalek::{Signature, Verifier, VerifyingKey}; + + let (keypair, token) = token_fixture(); + let (payload, sig_b58) = token.rsplit_once('/').unwrap(); + let sig_bytes: [u8; 64] = bs58::decode(sig_b58) + .into_vec() + .unwrap() + .try_into() + .unwrap(); + + let edwards = curve25519_dalek::montgomery::MontgomeryPoint(keypair.public_key_bytes()) + .to_edwards(0) + .expect("transport pubkey must map to Edwards") + .compress() + .to_bytes(); + VerifyingKey::from_bytes(&edwards) + .unwrap() + .verify(payload.as_bytes(), &Signature::from_bytes(&sig_bytes)) + .expect("stock ed25519 verify after Montgomery->Edwards conversion"); + } + + #[test] + fn fingerprint_attr_is_recomputable_from_the_pubkey_attr() { + // Requirement: a node cannot fake the UI-facing fingerprint. The + // collector derives it from the verified pubkey instead of trusting + // it: b58-decode pubkey, take the first 12 bytes, b58-encode. + let keypair = crate::transport::TransportKeypair::new(); + let pubkey_attr = bs58::encode(keypair.public_key_bytes()).into_string(); + let fingerprint_attr = keypair.public().to_string(); + + let decoded = bs58::decode(&pubkey_attr).into_vec().unwrap(); + assert_eq!( + bs58::encode(&decoded[..12]).into_string(), + fingerprint_attr, + "fingerprint must be a pure function of pubkey, or the collector \ + cannot validate the UI-facing id" + ); + } + + #[test] + fn bearer_tokens_are_unique_per_request() { + // Fresh nonce every call — a replayed token must be detectable. + let keypair = crate::transport::TransportKeypair::new(); + let pubkey = bs58::encode(keypair.public_key_bytes()).into_string(); + let signer = keypair.auth_token_signer(); + assert_ne!( + bearer_token(&signer, &pubkey), + bearer_token(&signer, &pubkey) + ); + } + #[test] fn production_shaped_input_is_not_suppressed() { assert_eq!( @@ -649,20 +873,37 @@ mod tests { ); } + #[test] + fn node_pubkey_attr_matches_the_bearer_token_pubkey() { + // The collector's self-validation contract: after verifying the token + // signature, `` must equal `freenet.node.pubkey` exactly. + let (keypair, token) = token_fixture(); + let pubkey_attr = bs58::encode(keypair.public_key_bytes()).into_string(); + assert_eq!( + token.split('/').nth(1), + Some(pubkey_attr.as_str()), + "token must equal freenet.node.pubkey, or the collector \ + cannot self-validate the node id against the signing key" + ); + } + #[test] fn instance_id_carries_no_network_address() { - // The exporter identifies this node by its transport public key - // fingerprint. `PeerId` renders as `{pub_key}@{addr}`, so using it — as - // the exporter originally did — leaks our socket address into every + // `PeerId` renders as `{pub_key}@{addr}`, so using it — as the + // exporter originally did — leaks our socket address into every // batch and re-identifies the node whenever the address changes. + // Both identity attributes must stay address-free. let keypair = crate::transport::TransportKeypair::new(); - let instance_id = keypair.public().to_string(); - - assert!(!instance_id.is_empty()); - assert!( - !instance_id.contains('@') && !instance_id.contains(':'), - "instance id must not embed an address, got {instance_id}" - ); + for instance_id in [ + bs58::encode(keypair.public_key_bytes()).into_string(), + keypair.public().to_string(), + ] { + assert!(!instance_id.is_empty()); + assert!( + !instance_id.contains('@') && !instance_id.contains(':'), + "identity attribute must not embed an address, got {instance_id}" + ); + } let peer_id = crate::node::PeerId::new( keypair.public().clone(), @@ -700,7 +941,11 @@ mod tests { // startup. Port 1 is chosen because nothing can be listening there. let provider = build_provider( Some("http://127.0.0.1:1/v1/metrics"), - "peer-under-test".to_string(), + "pubkey-under-test".to_string(), + "fingerprint-under-test".to_string(), + // Auth on: the signing client path must not panic in async + // context either. + Some(crate::transport::TransportKeypair::new().auth_token_signer()), ) .expect("exporter build must succeed against an unreachable collector"); provider.shutdown().expect("clean shutdown"); diff --git a/crates/core/src/transport/crypto.rs b/crates/core/src/transport/crypto.rs index 054b91fe5a..c93b45fbf9 100644 --- a/crates/core/src/transport/crypto.rs +++ b/crates/core/src/transport/crypto.rs @@ -99,6 +99,26 @@ impl TransportKeypair { } } +impl TransportKeypair { + /// XEdDSA signer for authenticating this node out-of-band (the OTel + /// collector bearer token — `tracing::otel::bearer_token`). + /// + /// XEdDSA (Signal's construction) signs with the x25519 transport secret + /// itself, so signatures verify against the SAME public key peers and + /// UIs already know: the verifier converts the Montgomery public key to + /// Edwards (sign bit 0) and runs stock Ed25519 verification. No second + /// identity, nothing to cross-certify. + pub(crate) fn auth_token_signer(&self) -> xeddsa::xed25519::PrivateKey { + xeddsa::xed25519::PrivateKey::from(self.secret.0.as_bytes()) + } + + /// The full 32-byte public key, e.g. for the base58 identity the OTel + /// exporter publishes (`Display` is a truncated fingerprint). + pub(crate) fn public_key_bytes(&self) -> [u8; 32] { + *self.public.0.as_bytes() + } +} + impl Default for TransportKeypair { fn default() -> Self { Self::new() From d233ca3a495369370c86927a162653bc7a4cdeb8 Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Tue, 4 Aug 2026 11:58:15 -0500 Subject: [PATCH 10/17] feat(otel): drop the nonce from the bearer token format The collector no longer accepts a nonce field; freshness is the timestamp alone. --- AGENTS.md | 6 +++--- crates/core/src/tracing/otel.rs | 24 ++++++++++-------------- 2 files changed, 13 insertions(+), 17 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 10b3d47aa3..c62bb2822f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -338,10 +338,10 @@ default to the dashboard collector. The otel pipeline authenticates to the collector per `otel-auth-mode`: `freenet` (default) sends a per-request -`Authorization: Bearer freenet////` +`Authorization: Bearer freenet///` token — an XEdDSA (Signal construction, `xeddsa` crate) signature over -`freenet///` (epoch seconds, 16-byte nonce, all -base58), signed with the x25519 transport secret itself +`freenet//` (epoch seconds, all base58), signed with +the x25519 transport secret itself (`TransportKeypair::auth_token_signer`). `` is the FULL base58 x25519 transport public key — the node's one identity — so the collector verifies by converting Montgomery→Edwards (sign bit 0) and running stock diff --git a/crates/core/src/tracing/otel.rs b/crates/core/src/tracing/otel.rs index f2158a4d17..7d8975ea86 100644 --- a/crates/core/src/tracing/otel.rs +++ b/crates/core/src/tracing/otel.rs @@ -85,13 +85,13 @@ use opentelemetry_sdk::{ use std::sync::OnceLock; /// Build one `freenet`-mode bearer token: -/// `freenet////`, where `` is -/// the XEdDSA signature over `freenet///`. +/// `freenet///`, where `` is +/// the XEdDSA signature over `freenet//`. /// /// `` is the base58 full x25519 transport public key — the node's one /// real identity, the same key peers see and whose truncated fingerprint UIs -/// display. `` is seconds since the Unix epoch, `` is 16 -/// base58 random bytes for replay protection, `` is base58 too. +/// display. `` is seconds since the Unix epoch, `` is +/// base58 too. /// Freshly built per export request so the timestamp stays current. /// /// Collector-side verification needs no exotic library: convert the @@ -105,10 +105,7 @@ pub(crate) fn bearer_token(signer: &xeddsa::xed25519::PrivateKey, pubkey_b58: &s .duration_since(std::time::UNIX_EPOCH) .map(|d| d.as_secs()) .unwrap_or_default(); - let mut nonce_bytes = [0u8; 16]; - crate::config::GlobalRng::fill_bytes(&mut nonce_bytes); - let nonce = bs58::encode(nonce_bytes).into_string(); - let signed_payload = format!("freenet/{pubkey_b58}/{timestamp}/{nonce}"); + let signed_payload = format!("freenet/{pubkey_b58}/{timestamp}"); // OS entropy (SysRng), not GlobalRng: XEdDSA's Z randomness hedges the // signature nonce, which is cryptographic material — the same exception // documented in .claude/rules/code-style.md for keys/nonces. UnwrapErr is @@ -692,8 +689,8 @@ mod tests { let (keypair, token) = token_fixture(); let parts: Vec<&str> = token.split('/').collect(); - let [scheme, pubkey, timestamp, nonce, signature] = parts[..] else { - panic!("expected 5 /-separated parts, got {token}"); + let [scheme, pubkey, timestamp, signature] = parts[..] else { + panic!("expected 4 /-separated parts, got {token}"); }; assert_eq!(scheme, "freenet"); assert_eq!( @@ -706,11 +703,9 @@ mod tests { ts > 1_700_000_000, "timestamp must be current epoch seconds" ); - assert!(!nonce.is_empty()); - // The signature covers everything before its own slash, and verifies // against the token's OWN pubkey — the transport key itself. - let signed_payload = format!("freenet/{pubkey}/{timestamp}/{nonce}"); + let signed_payload = format!("freenet/{pubkey}/{timestamp}"); let sig_bytes: [u8; 64] = bs58::decode(signature) .into_vec() .unwrap() @@ -774,7 +769,8 @@ mod tests { #[test] fn bearer_tokens_are_unique_per_request() { - // Fresh nonce every call — a replayed token must be detectable. + // XEdDSA's random Z makes each signature distinct even over an + // identical payload (same pubkey, same second). let keypair = crate::transport::TransportKeypair::new(); let pubkey = bs58::encode(keypair.public_key_bytes()).into_string(); let signer = keypair.auth_token_signer(); From 6cb17a7cb1981de8aff0bd81440d80f389c96146 Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Tue, 4 Aug 2026 20:39:17 -0500 Subject: [PATCH 11/17] fix(otel): build the exporter off the async runtime reqwest's blocking client owns a private tokio runtime; creating or dropping it inside an async context panics. build_provider now hops to a plain thread, and the test shuts down via spawn_blocking. --- crates/core/src/tracing/otel.rs | 40 ++++++++++++++++++++++++++++----- 1 file changed, 34 insertions(+), 6 deletions(-) diff --git a/crates/core/src/tracing/otel.rs b/crates/core/src/tracing/otel.rs index 7d8975ea86..30a9cd5ebc 100644 --- a/crates/core/src/tracing/otel.rs +++ b/crates/core/src/tracing/otel.rs @@ -258,6 +258,26 @@ pub(crate) fn build_provider( pubkey: String, fingerprint: String, auth_signer: Option, +) -> Result { + // The blocking reqwest clients below (ours and the exporter's default) + // each own a private tokio runtime. Creating one — or dropping one on the + // error path — inside an async context panics with "Cannot drop a runtime + // in a context where blocking is not allowed", and `init` runs inside the + // node's async build path. Hop to a plain thread so the whole build is + // async-context-free regardless of the caller. + std::thread::scope(|scope| { + scope + .spawn(move || build_provider_blocking(endpoint, pubkey, fingerprint, auth_signer)) + .join() + .expect("otel provider build thread panicked") + }) +} + +fn build_provider_blocking( + endpoint: Option<&str>, + pubkey: String, + fingerprint: String, + auth_signer: Option, ) -> Result { let mut builder = MetricExporter::builder().with_http(); if let Some(endpoint) = endpoint { @@ -930,11 +950,14 @@ mod tests { #[tokio::test] async fn provider_builds_inside_a_tokio_runtime() { // Two things under test. First, exporter construction must not panic - // when it happens inside an async context — the OTLP HTTP exporter uses - // reqwest's BLOCKING client because PeriodicReader exports from its own - // thread. Second, an unreachable collector must not surface as a build - // error: export failures are asynchronous and must never fail node - // startup. Port 1 is chosen because nothing can be listening there. + // when invoked from an async context — the blocking reqwest client + // owns a private tokio runtime, so `build_provider` hops to a plain + // thread internally; this asserts that hop works (on Linux, building + // inline panics with "Cannot drop a runtime in a context where + // blocking is not allowed"). Second, an unreachable collector must not + // surface as a build error: export failures are asynchronous and must + // never fail node startup. Port 1 is chosen because nothing can be + // listening there. let provider = build_provider( Some("http://127.0.0.1:1/v1/metrics"), "pubkey-under-test".to_string(), @@ -944,6 +967,11 @@ mod tests { Some(crate::transport::TransportKeypair::new().auth_token_signer()), ) .expect("exporter build must succeed against an unreachable collector"); - provider.shutdown().expect("clean shutdown"); + // Shutdown drops the exporter's blocking client and with it that + // private runtime — same hazard as construction, so it must happen + // where blocking is allowed, not on this async test thread. + tokio::task::spawn_blocking(move || provider.shutdown().expect("clean shutdown")) + .await + .expect("shutdown thread panicked"); } } From 2621cddfa32343db84a90827c99dce30caf5db2b Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Tue, 4 Aug 2026 20:56:03 -0500 Subject: [PATCH 12/17] fix(otel): address rule-review findings - correct stale nonce segment in OtelAuthMode's token-format doc - surface build-thread panics as ExporterBuildError instead of propagating into node startup - add wire-level test for FreenetAuthClient::send_bytes - consolidate module imports per code-style layout --- crates/core/src/config.rs | 5 +- crates/core/src/tracing/otel.rs | 121 +++++++++++++++++++++++++++++--- 2 files changed, 113 insertions(+), 13 deletions(-) diff --git a/crates/core/src/config.rs b/crates/core/src/config.rs index 6a2b9211ca..569a7dba6a 100644 --- a/crates/core/src/config.rs +++ b/crates/core/src/config.rs @@ -2852,8 +2852,9 @@ pub const DEFAULT_OTEL_ENDPOINT: &str = "http://localhost:4318"; /// How the OTel exporter authenticates to the collector. /// /// `freenet` (the default) sends a per-request -/// `Authorization: Bearer freenet////` -/// token signed with a key derived from the node's transport keypair — see +/// `Authorization: Bearer freenet///` +/// token — an XEdDSA signature over `freenet//`, signed +/// with the node's x25519 transport secret — see /// `tracing::otel::bearer_token`. `disabled` sends no Authorization header. /// Future methods get new variants. #[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, clap::ValueEnum)] diff --git a/crates/core/src/tracing/otel.rs b/crates/core/src/tracing/otel.rs index 30a9cd5ebc..a6f386fced 100644 --- a/crates/core/src/tracing/otel.rs +++ b/crates/core/src/tracing/otel.rs @@ -5,6 +5,17 @@ //! `DEFAULT_TELEMETRY_ENDPOINT`. The two features are independent by design — //! see `docs/design/otel-metrics-exporter.md`. +use std::sync::OnceLock; + +use opentelemetry::metrics::{Counter, Histogram}; +use opentelemetry::{KeyValue, global}; +use opentelemetry_http::{Bytes, HttpClient, HttpError, Request, Response}; +use opentelemetry_otlp::{ExporterBuildError, MetricExporter, WithExportConfig, WithHttpConfig}; +use opentelemetry_sdk::{ + Resource, + metrics::{Aggregation, Instrument, InstrumentKind, SdkMeterProvider, Stream}, +}; + use crate::config::OtelConfig; /// Why the OTel metrics exporter was not started. @@ -74,16 +85,6 @@ pub(crate) fn resolve_metrics_endpoint( Some(format!("{}/v1/metrics", base.trim_end_matches('/'))) } -use opentelemetry::metrics::{Counter, Histogram}; -use opentelemetry::{KeyValue, global}; -use opentelemetry_http::{Bytes, HttpClient, HttpError, Request, Response}; -use opentelemetry_otlp::{ExporterBuildError, MetricExporter, WithExportConfig, WithHttpConfig}; -use opentelemetry_sdk::{ - Resource, - metrics::{Aggregation, Instrument, InstrumentKind, SdkMeterProvider, Stream}, -}; -use std::sync::OnceLock; - /// Build one `freenet`-mode bearer token: /// `freenet///`, where `` is /// the XEdDSA signature over `freenet//`. @@ -269,7 +270,19 @@ pub(crate) fn build_provider( scope .spawn(move || build_provider_blocking(endpoint, pubkey, fingerprint, auth_signer)) .join() - .expect("otel provider build thread panicked") + // A panic in the builder must surface as a build error, not + // propagate: `init` runs on node startup and metrics export must + // never be a startup dependency. + .unwrap_or_else(|panic| { + let msg = panic + .downcast_ref::<&str>() + .map(ToString::to_string) + .or_else(|| panic.downcast_ref::().cloned()) + .unwrap_or_else(|| "non-string panic payload".to_owned()); + Err(ExporterBuildError::InternalFailure(format!( + "otel provider build thread panicked: {msg}" + ))) + }) }) } @@ -769,6 +782,92 @@ mod tests { .expect("stock ed25519 verify after Montgomery->Edwards conversion"); } + #[test] + fn send_bytes_puts_a_verifiable_bearer_header_on_the_wire() { + use std::io::{Read, Write}; + + use ed25519_dalek::{Signature, Verifier, VerifyingKey}; + + // The full auth transport path: header injection plus the + // http::Request -> reqwest::blocking::Request conversion, observed + // from the collector's side of a real socket. Plain #[test], not + // tokio: the blocking client must stay out of async contexts. + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let addr = listener.local_addr().unwrap(); + let server = std::thread::spawn(move || { + let (mut stream, _) = listener.accept().unwrap(); + let mut raw = Vec::new(); + let mut buf = [0u8; 1024]; + // Read until the body payload arrives; the client writes the + // whole request before waiting on the response. + while !raw.windows(14).any(|w| w == b"export-payload") { + let n = stream.read(&mut buf).unwrap(); + if n == 0 { + break; + } + raw.extend_from_slice(&buf[..n]); + } + stream + .write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 0\r\n\r\n") + .unwrap(); + String::from_utf8_lossy(&raw).into_owned() + }); + + let keypair = crate::transport::TransportKeypair::new(); + let pubkey_b58 = bs58::encode(keypair.public_key_bytes()).into_string(); + let client = FreenetAuthClient { + inner: reqwest::blocking::Client::new(), + signer: keypair.auth_token_signer(), + pubkey_b58: pubkey_b58.clone(), + }; + let request = Request::builder() + .method("POST") + .uri(format!("http://{addr}/v1/metrics")) + .body(Bytes::from_static(b"export-payload")) + .unwrap(); + // futures' executor, not tokio: send_bytes blocks internally. + let response = futures::executor::block_on(client.send_bytes(request)).unwrap(); + assert!(response.status().is_success()); + + let raw = server.join().unwrap(); + let auth = raw + .lines() + .find(|l| l.to_ascii_lowercase().starts_with("authorization:")) + .expect("Authorization header must reach the wire"); + let token = auth + .split_once(':') + .unwrap() + .1 + .trim() + .strip_prefix("Bearer ") + .expect("Bearer scheme"); + let (payload, sig_b58) = token.rsplit_once('/').unwrap(); + assert!( + payload.starts_with(&format!("freenet/{pubkey_b58}/")), + "wire token must carry this node's pubkey: {token}" + ); + // Verify exactly like a collector would — see + // node_pubkey_is_verifiable_with_stock_ed25519. + let sig_bytes: [u8; 64] = bs58::decode(sig_b58) + .into_vec() + .unwrap() + .try_into() + .unwrap(); + let edwards = curve25519_dalek::montgomery::MontgomeryPoint(keypair.public_key_bytes()) + .to_edwards(0) + .unwrap() + .compress() + .to_bytes(); + VerifyingKey::from_bytes(&edwards) + .unwrap() + .verify(payload.as_bytes(), &Signature::from_bytes(&sig_bytes)) + .expect("wire bearer token must verify with stock ed25519"); + assert!( + raw.contains("export-payload"), + "body must survive the http -> reqwest conversion" + ); + } + #[test] fn fingerprint_attr_is_recomputable_from_the_pubkey_attr() { // Requirement: a node cannot fake the UI-facing fingerprint. The From 4ce4cc3218b9378d9894b60f6b388a29a866530e Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Thu, 6 Aug 2026 15:36:51 -0500 Subject: [PATCH 13/17] fix(otel): shrink the dep graph and default collector auth off - always install our own HttpClient, so opentelemetry-otlp needs no reqwest/TLS feature: 11 new lockfile packages down to 2, no reqwest 0.13, no aws-lc-sys/cmake, no dual rustls provider - never overwrite an operator's Authorization header, honor the export timeout, keep the response body for OTLP error detail - otel-auth-mode defaults to disabled, and tokens bind the target collector's authority so they cannot be replayed elsewhere - --otel-telemetry-enabled=false now overrides config.toml - resource attributes no longer clobber OTEL_SERVICE_NAME - pin init's suppression check and the hot-path record_* mirrors - trim AGENTS.md, drop the stale plan, add operator docs --- AGENTS.md | 98 +- Cargo.lock | 143 +-- crates/core/Cargo.toml | 17 +- crates/core/src/config.rs | 142 ++- crates/core/src/tracing/otel.rs | 496 ++++++++-- docs/design/otel-metrics-exporter.md | 164 +++- docs/otel-metrics.md | 96 ++ .../plans/2026-08-01-otel-metrics-exporter.md | 869 ------------------ 8 files changed, 746 insertions(+), 1279 deletions(-) create mode 100644 docs/otel-metrics.md delete mode 100644 docs/superpowers/plans/2026-08-01-otel-metrics-exporter.md diff --git a/AGENTS.md b/AGENTS.md index c62bb2822f..678bb79378 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -327,75 +327,35 @@ Operator-facing documentation (encryption model, migration matrix, ## Two independent telemetry pipelines -`telemetry-enabled` / `telemetry-endpoint` feed the project's central -dashboard through a hand-rolled OTLP-JSON log POST (`tracing/telemetry.rs`). - -`otel-telemetry-enabled` / `otel-endpoint` are a **separate, unrelated** -OpenTelemetry SDK metrics pipeline (`tracing/otel.rs`). The two share no -config, no endpoint, and no fallback in either direction — enabling or -disabling one has no effect on the other, and `otel-endpoint` must never -default to the dashboard collector. - -The otel pipeline authenticates to the collector per `otel-auth-mode`: -`freenet` (default) sends a per-request -`Authorization: Bearer freenet///` -token — an XEdDSA (Signal construction, `xeddsa` crate) signature over -`freenet//` (epoch seconds, all base58), signed with -the x25519 transport secret itself -(`TransportKeypair::auth_token_signer`). `` is the FULL base58 -x25519 transport public key — the node's one identity — so the collector -verifies by converting Montgomery→Edwards (sign bit 0) and running stock -Ed25519 verification; no shared secret, no second key, nothing assertable. -`disabled` sends no header. Tokens are built per request -(`tracing/otel.rs::bearer_token` via a custom `HttpClient`) so the timestamp -stays fresh; future auth methods get new enum variants. - -The otel pipeline honors the standard variables, which take priority over -`otel-endpoint` in `config.toml`: -`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, `OTEL_EXPORTER_OTLP_ENDPOINT`, -`OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_SERVICE_NAME`, -`OTEL_RESOURCE_ATTRIBUTES`, `OTEL_METRIC_EXPORT_INTERVAL`. Without any of -them it exports to `http://localhost:4318`. - -All instruments are registered in `tracing/otel.rs::register_metrics`. Two -kinds, and the choice is not stylistic: - -- **Observable** (gauges / `observable_counter`) own a callback that reads - existing state at collection time — the transport's cumulative counters, or - `network_status::otel_metrics_snapshot()`. Nothing is added to the hot path. - Read cumulative, never-reset values only: the `TransportSnapshot` fields are - period accumulators that `take_snapshot` zeroes for the legacy telemetry - worker, so observing one as a counter yields a non-monotonic series whenever - `telemetry-enabled` is also on. -- **Synchronous** (`Histogram` / `Counter`) are held in the `INSTRUMENTS` - `OnceLock` and recorded via the `record_*` helpers. They need a stored handle - because an instrument built before `global::set_meter_provider` binds to the - no-op provider forever; when the exporter is off the helpers are one atomic - load and a branch. - -Every histogram is base-2 exponential via a single `with_view` in -`build_provider` — do not add explicit bucket boundaries per instrument. - -No instrument carries an attribute identifying the remote end of a connection. -Per-datapoint attributes cost per series and multiply by bucket count on -histograms; identifying THIS node is done with two resource attributes, -riding once per export batch. `freenet.node.pubkey` -is the base58 ed25519 verifying key derived from the transport keypair — -byte-equal to the bearer token's `` field, so the collector -self-validates the node id against the signing key after verifying the -signature. `freenet.node.fingerprint` is the transport public key fingerprint -(what UIs and the legacy dashboard show), for cross-referencing only — -unverifiable by the collector. Neither is ever a `PeerId` — `PeerId` renders as -`{pub_key}@{addr}` and would export our socket address and re-identify the node -on every address change. Note also that "peer" means the other end of a -connection; metrics about ourselves use `freenet.node.*`. - -The proof-of-life gauge `freenet.process.memory.rss` is sourced from -`node::resource_metrics::rss_bytes()`, which is implemented for Linux only. -On macOS and Windows the gauge registers but reports no datapoints — an -empty series there is expected, not a broken pipeline. - -Design: [`docs/design/otel-metrics-exporter.md`](docs/design/otel-metrics-exporter.md). +`telemetry-enabled` / `telemetry-endpoint` feed the project's central dashboard +(`tracing/telemetry.rs`). `otel-telemetry-enabled` / `otel-endpoint` are a +**separate, unrelated** OpenTelemetry SDK metrics pipeline (`tracing/otel.rs`): +no shared config, no shared endpoint, no fallback in either direction, and +`otel-endpoint` must never default to the dashboard collector. + +Rules when touching `tracing/otel.rs`: + +- Observable instruments must read **cumulative, never-reset** values. + `TransportSnapshot` fields are period accumulators that `take_snapshot` + zeroes for the legacy telemetry worker, so observing one as a counter yields + a non-monotonic series whenever `telemetry-enabled` is also on. +- Never export a `PeerId`, socket address, or any attribute identifying the + remote end of a connection. `PeerId` renders as `{pub_key}@{addr}`, which + leaks our address and re-identifies the node whenever it changes. This node's + identity is two resource attributes (`freenet.node.*`, one per export batch), + not a per-datapoint attribute. "Peer" means the *other* end of a connection. +- Histograms get their base-2 exponential aggregation from one `with_view` in + `build_provider`. Do not add explicit bucket boundaries per instrument. +- The exporter always installs its own `HttpClient`, so `opentelemetry-otlp` + needs none of its `reqwest-*`/TLS features — enabling one pulls a second + reqwest major, a second TLS stack, and a C/asm aws-lc build into every + release target. + +Everything else — the bearer-token format, endpoint precedence, the +`OTEL_*` variables honored, and per-instrument notes — is in +[`docs/design/otel-metrics-exporter.md`](docs/design/otel-metrics-exporter.md), +and operator-facing configuration is in +[`docs/otel-metrics.md`](docs/otel-metrics.md). ## External Resources diff --git a/Cargo.lock b/Cargo.lock index 012c9a60fd..90dad3498a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -279,28 +279,6 @@ version = "1.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" -[[package]] -name = "aws-lc-rs" -version = "1.17.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5ec2f1fc3ec205783a5da9a7e6c1509cc69dedf09a1949e412c1e18469326d00" -dependencies = [ - "aws-lc-sys", - "zeroize", -] - -[[package]] -name = "aws-lc-sys" -version = "0.41.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1a2f9779ce85b93ab6170dd940ad0169b5766ff848247aff13bb788b832fe3f4" -dependencies = [ - "cc", - "cmake", - "dunce", - "fs_extra", -] - [[package]] name = "axum" version = "0.8.9" @@ -805,15 +783,6 @@ version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" -[[package]] -name = "cmake" -version = "0.1.58" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678" -dependencies = [ - "cc", -] - [[package]] name = "cmov" version = "0.5.4" @@ -1370,7 +1339,7 @@ checksum = "79fc3b6dd0b87ba36e565715bf9a2ced221311db47bd18011676f24a6066edbc" dependencies = [ "curl-sys", "libc", - "openssl-probe 0.1.6", + "openssl-probe", "openssl-sys", "schannel", "socket2", @@ -2113,7 +2082,7 @@ dependencies = [ "hickory-resolver", "hkdf", "hostname", - "http 1.4.2", + "http 1.5.0", "httptest", "ipnet", "keyring", @@ -2137,7 +2106,7 @@ dependencies = [ "redb", "regex", "renegade-ml", - "reqwest 0.12.28", + "reqwest", "rpassword", "rstest", "semver", @@ -2274,7 +2243,7 @@ dependencies = [ "clap", "hex", "hmac", - "reqwest 0.12.28", + "reqwest", "semver", "serde", "serde_json", @@ -2387,12 +2356,6 @@ dependencies = [ "serde", ] -[[package]] -name = "fs_extra" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c" - [[package]] name = "fsevent-sys" version = "4.1.0" @@ -4208,7 +4171,7 @@ dependencies = [ "freenet-stdlib 0.8.5", "hex", "rand 0.9.4", - "reqwest 0.12.28", + "reqwest", "serde", "serde_json", "tempfile", @@ -4648,12 +4611,6 @@ version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d05e27ee213611ffe7d6348b942e8f942b37114c00cc03cec254295a4a17852e" -[[package]] -name = "openssl-probe" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" - [[package]] name = "openssl-sys" version = "0.9.116" @@ -4716,7 +4673,6 @@ dependencies = [ "bytes 1.12.1", "http 1.5.0", "opentelemetry 0.32.0", - "reqwest 0.13.3", ] [[package]] @@ -4750,7 +4706,6 @@ dependencies = [ "opentelemetry-proto", "opentelemetry_sdk 0.32.1", "prost", - "reqwest 0.13.3", "thiserror 2.0.19", ] @@ -5378,7 +5333,6 @@ version = "0.11.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560" dependencies = [ - "aws-lc-rs", "bytes 1.12.1", "getrandom 0.4.2", "lru-slab", @@ -5739,43 +5693,6 @@ dependencies = [ "webpki-roots 1.0.7", ] -[[package]] -name = "reqwest" -version = "0.13.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62e0021ea2c22aed41653bc7e1419abb2c97e038ff2c33d0e1309e49a97deec0" -dependencies = [ - "base64 0.22.1", - "bytes 1.12.1", - "futures-channel", - "futures-core", - "futures-util", - "http 1.5.0", - "http-body", - "http-body-util", - "hyper", - "hyper-rustls", - "hyper-util", - "js-sys", - "log", - "percent-encoding", - "pin-project-lite", - "quinn", - "rustls", - "rustls-pki-types", - "rustls-platform-verifier", - "sync_wrapper", - "tokio", - "tokio-rustls", - "tower", - "tower-http 0.6.11", - "tower-service", - "url", - "wasm-bindgen", - "wasm-bindgen-futures", - "web-sys", -] - [[package]] name = "resolv-conf" version = "0.7.6" @@ -5886,7 +5803,6 @@ version = "0.23.40" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b" dependencies = [ - "aws-lc-rs", "log", "once_cell", "ring", @@ -5896,18 +5812,6 @@ dependencies = [ "zeroize", ] -[[package]] -name = "rustls-native-certs" -version = "0.8.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d" -dependencies = [ - "openssl-probe 0.2.1", - "rustls-pki-types", - "schannel", - "security-framework 3.7.0", -] - [[package]] name = "rustls-pki-types" version = "1.14.1" @@ -5918,40 +5822,12 @@ dependencies = [ "zeroize", ] -[[package]] -name = "rustls-platform-verifier" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0" -dependencies = [ - "core-foundation 0.10.1", - "core-foundation-sys", - "jni 0.22.4", - "log", - "once_cell", - "rustls", - "rustls-native-certs", - "rustls-platform-verifier-android", - "rustls-webpki", - "security-framework 3.7.0", - "security-framework-sys", - "webpki-root-certs", - "windows-sys 0.61.2", -] - -[[package]] -name = "rustls-platform-verifier-android" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f" - [[package]] name = "rustls-webpki" version = "0.103.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" dependencies = [ - "aws-lc-rs", "ring", "rustls-pki-types", "untrusted", @@ -8358,15 +8234,6 @@ dependencies = [ "system-deps", ] -[[package]] -name = "webpki-root-certs" -version = "1.0.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f31141ce3fc3e300ae89b78c0dd67f9708061d1d2eda54b8209346fd6be9a92c" -dependencies = [ - "rustls-pki-types", -] - [[package]] name = "webpki-roots" version = "0.26.11" diff --git a/crates/core/Cargo.toml b/crates/core/Cargo.toml index e7e4a60e33..92b23aa7fa 100644 --- a/crates/core/Cargo.toml +++ b/crates/core/Cargo.toml @@ -133,11 +133,16 @@ tracing-appender = { workspace = true, optional = true } # is a goal here (see docs/design/otel-metrics-exporter.md, non-goals). It does # NOT drop trace: `http-proto` mandates `trace`, `prost` and # opentelemetry-proto, so those stay in the graph regardless. -# reqwest-blocking-client is load-bearing, not incidental: PeriodicReader -# exports from a dedicated thread via futures_executor::block_on, where an -# async reqwest client would have no tokio reactor. reqwest-rustls adds TLS -# so https:// collector endpoints actually work (otherwise every export to an -# https endpoint silently fails at connect time). +# NO reqwest-*-client / reqwest-rustls feature on purpose: tracing::otel always +# supplies its own HttpClient, so the exporter never builds one (it only does +# when none was given — opentelemetry-otlp src/exporter/http/mod.rs). Enabling +# them would pull opentelemetry-http's reqwest 0.13, whose `rustls` feature is +# hardwired to aws-lc-rs + rustls-platform-verifier: a second reqwest major, a +# second TLS root store, a C/asm build (aws-lc-sys via cc+cmake) on every +# release target, and two crypto providers in one rustls, which makes +# `CryptoProvider::get_default_or_install_from_crate_features` panic. +# Our client rides the workspace reqwest 0.12, which already has rustls-tls, so +# https:// collector endpoints work with nothing new in the graph. # Direct dep so tracing::otel can implement HttpClient (per-request auth # headers); the crate is already in the graph via opentelemetry-otlp. opentelemetry-http = { workspace = true } @@ -152,8 +157,6 @@ rand_core10 = { workspace = true } opentelemetry-otlp = { workspace = true, features = [ "http-proto", "metrics", - "reqwest-blocking-client", - "reqwest-rustls", "internal-logs", ] } opentelemetry_sdk = { workspace = true } diff --git a/crates/core/src/config.rs b/crates/core/src/config.rs index 569a7dba6a..bfc1f00aa2 100644 --- a/crates/core/src/config.rs +++ b/crates/core/src/config.rs @@ -971,13 +971,11 @@ impl ConfigArgs { if cfg.telemetry.iface_tx_enabled { self.telemetry.iface_tx_enabled = true; } - // otel-telemetry-enabled defaults to false via clap, so only the - // file-says-true direction needs handling — same one-directional - // override as reference-ping/iface-tx above. Kept separate from the - // telemetry merge on purpose: the two features are independent. - if cfg.otel.enabled { - self.otel.enabled = true; - } + // Kept separate from the telemetry merge above on purpose: the two + // features are independent. Unlike reference-ping/iface-tx this + // merge is bidirectional — `--otel-telemetry-enabled=false` parses + // to `Some(false)` and must override a config.toml that says true. + self.otel.enabled.get_or_insert(cfg.otel.enabled); if let Some(endpoint) = cfg.otel.endpoint { self.otel.endpoint.get_or_insert(endpoint); } @@ -1491,7 +1489,7 @@ impl ConfigArgs { iface_tx_enabled: self.telemetry.iface_tx_enabled, }, otel: OtelConfig { - enabled: self.otel.enabled, + enabled: self.otel.enabled.unwrap_or(false), endpoint: self.otel.endpoint, auth_mode: self.otel.auth_mode.unwrap_or_default(), // Same --id rule as telemetry: simulated networks and @@ -2840,28 +2838,25 @@ fn default_iface_tx_enabled() -> bool { false } -/// Default OTLP/HTTP endpoint for the SDK metrics pipeline, used when neither -/// the standard `OTEL_EXPORTER_OTLP_*` env vars nor `otel-endpoint` are set. -/// -/// Deliberately NOT `DEFAULT_TELEMETRY_ENDPOINT`: `otel-telemetry-enabled` and -/// `telemetry-enabled` are strictly isolated features that are not expected to -/// share a backend. Pointing this pipeline at the central dashboard collector -/// must always be an explicit operator choice. -pub const DEFAULT_OTEL_ENDPOINT: &str = "http://localhost:4318"; - /// How the OTel exporter authenticates to the collector. /// -/// `freenet` (the default) sends a per-request -/// `Authorization: Bearer freenet///` -/// token — an XEdDSA signature over `freenet//`, signed -/// with the node's x25519 transport secret — see -/// `tracing::otel::bearer_token`. `disabled` sends no Authorization header. -/// Future methods get new variants. +/// `freenet` sends a per-request `Authorization: Bearer +/// freenet////` token — an XEdDSA +/// signature over the preceding fields, signed with the node's x25519 +/// transport secret — see `tracing::otel::bearer_token`. Future methods get +/// new variants. +/// +/// `disabled` is the DEFAULT and sends no `Authorization` header: pointing the +/// exporter at your own collector must not ship a signed assertion of this +/// node's identity somewhere it was never asked to. Operators exporting to a +/// collector that verifies freenet tokens opt in explicitly; anyone else +/// carries their own auth in `OTEL_EXPORTER_OTLP_HEADERS`, which the exporter +/// never overwrites. #[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, clap::ValueEnum)] #[serde(rename_all = "lowercase")] pub enum OtelAuthMode { - #[default] Freenet, + #[default] Disabled, } @@ -2880,17 +2875,25 @@ pub struct OtelArgs { /// as true, so `FREENET_OTEL_TELEMETRY_ENABLED=false` would silently turn /// the exporter ON. This form accepts `--otel-telemetry-enabled`, /// `--otel-telemetry-enabled=false`, and a properly parsed env value. + /// + /// `Option` and NO `default_value`, unlike the sibling telemetry flags: + /// with a default, "unset" and "explicitly false" are indistinguishable + /// after parsing, so `build()` cannot let `--otel-telemetry-enabled=false` + /// override a `config.toml` that says true — i.e. the off switch would not + /// work. `None` means "not given"; `build()` resolves it to `false`. #[arg( id = "otel_telemetry_enabled", long = "otel-telemetry-enabled", env = "FREENET_OTEL_TELEMETRY_ENABLED", num_args = 0..=1, - default_value = "false", default_missing_value = "true", action = clap::ArgAction::Set )] - #[serde(rename = "otel-telemetry-enabled", default)] - pub enabled: bool, + #[serde( + rename = "otel-telemetry-enabled", + skip_serializing_if = "Option::is_none" + )] + pub enabled: Option, /// OTLP/HTTP collector base URL (e.g. `http://collector:4318`). /// @@ -6555,15 +6558,17 @@ shutdown-drain-secs = 42 // The new pipeline exports nothing yet, so shipping it on would be a // behavior change. Operators opt in explicitly. let args = OtelArgs::default(); - assert!( - !args.enabled, - "otel-telemetry-enabled must default to false" + assert_eq!( + args.enabled, None, + "otel-telemetry-enabled unset must stay None so an explicit \ + --otel-telemetry-enabled=false can override config.toml" ); assert_eq!(args.endpoint, None, "no implicit collector"); assert_eq!( args.auth_mode.unwrap_or_default(), - OtelAuthMode::Freenet, - "auth mode must default to the freenet bearer-token format" + OtelAuthMode::Disabled, + "auth must default off: pointing the exporter at a collector must \ + not ship a signed assertion of this node's identity unasked" ); } @@ -6585,7 +6590,7 @@ shutdown-drain-secs = 42 let cfg: OtelConfig = toml::from_str("").unwrap(); assert_eq!( cfg.auth_mode, - OtelAuthMode::Freenet, + OtelAuthMode::Disabled, "absent key -> default" ); } @@ -6594,16 +6599,24 @@ shutdown-drain-secs = 42 fn otel_flag_parses_from_cli() { use clap::Parser; let none = ConfigArgs::try_parse_from(["freenet"]).expect("bare parse"); - assert!(!none.otel.enabled, "no flag -> off"); + assert_eq!(none.otel.enabled, None, "no flag -> unset, not false"); let set = ConfigArgs::try_parse_from(["freenet", "--otel-telemetry-enabled"]) .expect("flag parse"); - assert!(set.otel.enabled, "--otel-telemetry-enabled -> on"); + assert_eq!( + set.otel.enabled, + Some(true), + "--otel-telemetry-enabled -> on" + ); // Explicit `=false` must parse and mean false. Without this form the flag // would be a bare ArgAction::SetTrue, and clap turns ANY value of the bound // env var — including "false" — into true. let off = ConfigArgs::try_parse_from(["freenet", "--otel-telemetry-enabled=false"]) .expect("explicit false parse"); - assert!(!off.otel.enabled, "--otel-telemetry-enabled=false -> off"); + assert_eq!( + off.otel.enabled, + Some(false), + "--otel-telemetry-enabled=false -> off" + ); let with_ep = ConfigArgs::try_parse_from([ "freenet", "--otel-endpoint", @@ -6665,14 +6678,37 @@ shutdown-drain-secs = 42 ); } - #[test] - fn otel_endpoint_never_defaults_to_the_dashboard_collector() { - // Hard isolation requirement: the two pipelines share no backend. - assert_ne!( - DEFAULT_OTEL_ENDPOINT, DEFAULT_TELEMETRY_ENDPOINT, - "otel must not default to the central dashboard collector" + #[tokio::test] + async fn otel_cli_false_overrides_a_config_file_that_says_true() { + // The off switch has to work: an operator who exports to a collector + // and then needs it stopped must be able to do it from the command + // line without editing config.toml. `Some(false)` from the CLI beats + // the file; `None` (flag absent) lets the file's `true` through. + let temp_dir = tempfile::tempdir().unwrap(); + clap_bare_args(temp_dir.path()).build().await.unwrap(); + let path = temp_dir.path().join("config.toml"); + let base = tokio::fs::read_to_string(&path).await.unwrap(); + let base: String = base + .lines() + .filter(|line| !line.starts_with("otel-telemetry-enabled")) + .map(|line| format!("{line}\n")) + .collect(); + std::fs::write(&path, format!("{base}otel-telemetry-enabled = true\n")).unwrap(); + + // Flag absent first: build() rewrites config.toml, so the negative + // case has to run last or it would overwrite the seed. + let inherited = clap_bare_args(temp_dir.path()).build().await.unwrap(); + assert!( + inherited.otel.enabled, + "with the flag absent, config.toml's `true` must still win" + ); + + let mut args = clap_bare_args(temp_dir.path()); + args.otel.enabled = Some(false); + assert!( + !args.build().await.unwrap().otel.enabled, + "--otel-telemetry-enabled=false must override config.toml" ); - assert_eq!(DEFAULT_OTEL_ENDPOINT, "http://localhost:4318"); } #[tokio::test] @@ -7837,8 +7873,8 @@ shutdown-drain-secs = 42 otel: OtelConfig { enabled: true, endpoint: Some("http://example.invalid:4319".to_string()), - auth_mode: OtelAuthMode::Disabled, // non-default: default is Freenet - is_test_environment: false, // #[serde(skip)] — derived from --id + auth_mode: OtelAuthMode::Freenet, // non-default: default is Disabled + is_test_environment: false, // #[serde(skip)] — derived from --id }, shutdown_drain_secs: 77, disable_auto_update: true, // #[serde(skip)] — see destructure below @@ -7940,16 +7976,22 @@ shutdown-drain-secs = 42 shutdown_drain_secs, seed.shutdown_drain_secs, "shutdown_drain_secs" ); - assert_eq!(otel.enabled, seed.otel.enabled, "otel.enabled"); + let OtelConfig { + enabled: otel_enabled, + endpoint: otel_endpoint, + auth_mode: otel_auth_mode, + is_test_environment: _, // serde-skip, derived from --id + } = otel; + assert_eq!(otel_enabled, seed.otel.enabled, "otel.enabled"); assert_eq!( - otel.endpoint, seed.otel.endpoint, + otel_endpoint, seed.otel.endpoint, "otel.endpoint — an operator's collector URL must survive the \ config.toml merge" ); assert_eq!( - otel.auth_mode, seed.otel.auth_mode, - "otel.auth_mode — an operator's explicit `disabled` must survive \ - the config.toml merge, or auth silently re-enables on restart" + otel_auth_mode, seed.otel.auth_mode, + "otel.auth_mode — an operator's explicit choice must survive the \ + config.toml merge, or auth silently reverts on restart" ); let NetworkApiConfig { diff --git a/crates/core/src/tracing/otel.rs b/crates/core/src/tracing/otel.rs index a6f386fced..6d78533b2e 100644 --- a/crates/core/src/tracing/otel.rs +++ b/crates/core/src/tracing/otel.rs @@ -86,19 +86,30 @@ pub(crate) fn resolve_metrics_endpoint( } /// Build one `freenet`-mode bearer token: -/// `freenet///`, where `` is -/// the XEdDSA signature over `freenet//`. +/// `freenet////`, where `` +/// is the XEdDSA signature over everything preceding it. /// /// `` is the base58 full x25519 transport public key — the node's one /// real identity, the same key peers see and whose truncated fingerprint UIs -/// display. `` is seconds since the Unix epoch, `` is -/// base58 too. +/// display. `` is the authority (`host[:port]`) of the collector the +/// request is going to, so a token is only valid at the collector it was +/// minted for: without it, any collector we export to could replay the token +/// to any other collector accepting this scheme and impersonate this node. +/// Authority rather than the full URL because it carries no `/`, which is the +/// token's field separator. `` is seconds since the Unix epoch, +/// `` is base58 too. /// Freshly built per export request so the timestamp stays current. /// /// Collector-side verification needs no exotic library: convert the /// Montgomery pubkey to Edwards (sign bit 0), then standard Ed25519 verify — -/// see `node_pubkey_is_verifiable_with_stock_ed25519` below. -pub(crate) fn bearer_token(signer: &xeddsa::xed25519::PrivateKey, pubkey_b58: &str) -> String { +/// see `node_pubkey_is_verifiable_with_stock_ed25519` below. The collector +/// must additionally check `` against its own hostname and +/// `` against its own clock. +pub(crate) fn bearer_token( + signer: &xeddsa::xed25519::PrivateKey, + pubkey_b58: &str, + audience: &str, +) -> String { use xeddsa::xeddsa::Sign; // Wall-clock epoch seconds on purpose: the collector checks it against // ITS clock, so simulation time would be meaningless here. @@ -106,7 +117,7 @@ pub(crate) fn bearer_token(signer: &xeddsa::xed25519::PrivateKey, pubkey_b58: &s .duration_since(std::time::UNIX_EPOCH) .map(|d| d.as_secs()) .unwrap_or_default(); - let signed_payload = format!("freenet/{pubkey_b58}/{timestamp}"); + let signed_payload = format!("freenet/{pubkey_b58}/{audience}/{timestamp}"); // OS entropy (SysRng), not GlobalRng: XEdDSA's Z randomness hedges the // signature nonce, which is cryptographic material — the same exception // documented in .claude/rules/code-style.md for keys/nonces. UnwrapErr is @@ -119,30 +130,47 @@ pub(crate) fn bearer_token(signer: &xeddsa::xed25519::PrivateKey, pubkey_b58: &s format!("{signed_payload}/{signature}") } -/// OTLP HTTP client that injects a fresh `Authorization: Bearer` token -/// (see [`bearer_token`]) into every export request, delegating the actual -/// send to the same blocking reqwest client the exporter would use anyway. -struct FreenetAuthClient { +/// The exporter's only HTTP transport, in every auth mode. +/// +/// Always installed, so `opentelemetry-otlp` never builds a client of its own +/// and none of its `reqwest-*` features have to be enabled — see the comment +/// on the dependency in `crates/core/Cargo.toml` for the dependency-graph +/// reason that matters. +/// +/// With `signer` set (`otel-auth-mode = "freenet"`) it adds a fresh +/// `Authorization: Bearer` token (see [`bearer_token`]) to each request; +/// with it unset it is a plain sender. +struct OtlpHttpClient { inner: reqwest::blocking::Client, - signer: xeddsa::xed25519::PrivateKey, + /// `None` in `disabled` auth mode. + signer: Option, pubkey_b58: String, } // Manual impl: never print the signing key. -impl std::fmt::Debug for FreenetAuthClient { +impl std::fmt::Debug for OtlpHttpClient { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("FreenetAuthClient").finish_non_exhaustive() + f.debug_struct("OtlpHttpClient").finish_non_exhaustive() } } #[async_trait::async_trait] -impl HttpClient for FreenetAuthClient { +impl HttpClient for OtlpHttpClient { async fn send_bytes(&self, mut request: Request) -> Result, HttpError> { - let token = bearer_token(&self.signer, &self.pubkey_b58); - request.headers_mut().insert( - http::header::AUTHORIZATION, - http::HeaderValue::from_str(&format!("Bearer {token}"))?, - ); + // Never clobber an operator-supplied header: the exporter applies + // OTEL_EXPORTER_OTLP_HEADERS before calling us, so a hosted collector + // configured with `Authorization: Basic ...` there must win. Our + // bearer token is one auth scheme among several, not the only one. + if let Some(signer) = self.signer.as_ref() { + if !request.headers().contains_key(http::header::AUTHORIZATION) { + let audience = request.uri().authority().map(|a| a.as_str()).unwrap_or(""); + let token = bearer_token(signer, &self.pubkey_b58, audience); + request.headers_mut().insert( + http::header::AUTHORIZATION, + http::HeaderValue::from_str(&format!("Bearer {token}"))?, + ); + } + } // Hand-rolled send, mirroring opentelemetry-http's blocking impl: // that impl is on reqwest 0.13's client (opentelemetry-http's own // dep), while the workspace is on 0.12, so we can't delegate to it. @@ -150,11 +178,14 @@ impl HttpClient for FreenetAuthClient { // default client is blocking: PeriodicReader exports via block_on on // a dedicated thread. Fold both in when the workspace moves to 0.13. let request: reqwest::blocking::Request = request.map(|body| body.to_vec()).try_into()?; - let mut response = self.inner.execute(request)?.error_for_status()?; + // Deliberately no `error_for_status()`: it discards the body, which is + // where OTLP puts rejection detail. Hand the whole response back and + // the SDK turns a non-2xx into an export error itself, logging the + // status, the URL and the body (`HttpClient.StatusError`). + let mut response = self.inner.execute(request)?; + let status = response.status(); let headers = std::mem::take(response.headers_mut()); - let mut http_response = Response::builder() - .status(response.status()) - .body(response.bytes()?)?; + let mut http_response = Response::builder().status(status).body(response.bytes()?)?; *http_response.headers_mut() = headers; Ok(http_response) } @@ -175,14 +206,23 @@ const METER_NAME: &str = "freenet"; /// (see [`build_provider`]) and, when `otel-auth-mode = "freenet"`, its /// derived signing key authenticates every export request (see /// [`bearer_token`]). -pub fn init(config: &OtelConfig, keypair: &crate::transport::TransportKeypair) { +/// +/// Returns the reason it did NOT start, or `None` when the pipeline is +/// installed. Callers ignore it; it exists so a test can assert that `init` +/// consults [`otel_suppression_reason`] and returns before building anything — +/// deleting the check would make an enabled config return `None` under +/// `cfg(test)`, i.e. ship a test network's metrics to a collector. +pub(crate) fn init( + config: &OtelConfig, + keypair: &crate::transport::TransportKeypair, +) -> Option { if let Some(reason) = otel_suppression_reason( config, cfg!(test), super::telemetry::running_under_cargo_test(), ) { tracing::debug!(?reason, "OTel metrics exporter not started"); - return; + return Some(reason); } let endpoint = resolve_metrics_endpoint( @@ -195,13 +235,7 @@ pub fn init(config: &OtelConfig, keypair: &crate::transport::TransportKeypair) { .as_deref(), ); - // `service.instance.id` IS the auth identity: the same base58 ed25519 - // verifying key the bearer token carries as ``, so the collector - // self-validates the node id against the signing key by string equality - // after verifying the signature. Derived from the keypair even when auth - // is disabled, so the id is stable across auth-mode changes. - let pubkey = bs58::encode(keypair.public_key_bytes()).into_string(); - let fingerprint = keypair.public().to_string(); + let (pubkey, fingerprint) = identity_attributes(keypair); let auth_signer = match config.auth_mode { crate::config::OtelAuthMode::Freenet => Some(keypair.auth_token_signer()), crate::config::OtelAuthMode::Disabled => None, @@ -209,7 +243,7 @@ pub fn init(config: &OtelConfig, keypair: &crate::transport::TransportKeypair) { match build_provider(endpoint.as_deref(), pubkey, fingerprint, auth_signer) { Ok(provider) => { - // ponytail: no shutdown hook. `set_meter_provider` holds a + // NOTE: no shutdown hook. `set_meter_provider` holds a // reference for the process lifetime and PeriodicReader exports // every 60s (OTEL_METRIC_EXPORT_INTERVAL), so at most one partial // interval is lost at exit. If that tail ever matters, keep the @@ -217,10 +251,29 @@ pub fn init(config: &OtelConfig, keypair: &crate::transport::TransportKeypair) { // shutdown path in `bin/freenet.rs`. global::set_meter_provider(provider); register_metrics(); + // Log where this node's signed identity is actually going, + // including when an inherited OTEL_* variable overrode the + // configured endpoint — an operator who cannot see that from the + // logs cannot tell their collector was bypassed. + let env_endpoint = + std::env::var(opentelemetry_otlp::OTEL_EXPORTER_OTLP_METRICS_ENDPOINT) + .or_else(|_| std::env::var(opentelemetry_otlp::OTEL_EXPORTER_OTLP_ENDPOINT)) + .ok(); + if let (Some(env_endpoint), Some(cfg_endpoint)) = + (env_endpoint.as_deref(), config.endpoint.as_deref()) + { + tracing::warn!( + %env_endpoint, + %cfg_endpoint, + "OTEL_EXPORTER_OTLP_* overrides the configured otel-endpoint" + ); + } tracing::info!( endpoint = endpoint .as_deref() - .unwrap_or(""), + .or(env_endpoint.as_deref()) + .unwrap_or("http://localhost:4318 (SDK default)"), + auth_mode = ?config.auth_mode, "OTel metrics exporter started" ); } @@ -231,6 +284,25 @@ pub fn init(config: &OtelConfig, keypair: &crate::transport::TransportKeypair) { ); } } + None +} + +/// The two resource attributes that identify THIS node, both derived from the +/// one transport keypair: `(freenet.node.pubkey, freenet.node.fingerprint)`. +/// +/// A function rather than two inline expressions in [`init`] so the guards in +/// this module's tests assert on what production actually attaches — building +/// the same strings in a test body would pass no matter what `init` does. +fn identity_attributes(keypair: &crate::transport::TransportKeypair) -> (String, String) { + // The full base58 x25519 transport public key. Byte-equal to the bearer + // token's `` field, so the collector self-validates the node id + // against the signing key after verifying the signature. Derived from the + // keypair even when auth is disabled, so the id is stable across auth-mode + // changes. NEVER a `PeerId`: its Display is `{pub_key}@{addr}`. + ( + bs58::encode(keypair.public_key_bytes()).into_string(), + keypair.public().to_string(), + ) } /// Build the OTLP/HTTP exporter and meter provider. @@ -286,6 +358,38 @@ pub(crate) fn build_provider( }) } +/// Per-export HTTP timeout, resolved exactly like `opentelemetry-otlp` would +/// (`OTEL_EXPORTER_OTLP_METRICS_TIMEOUT` > `OTEL_EXPORTER_OTLP_TIMEOUT` > +/// 10s, in milliseconds). The SDK applies its own resolution only to a client +/// it builds itself, and we always supply one, so it has to happen here. +fn export_timeout() -> std::time::Duration { + [ + opentelemetry_otlp::OTEL_EXPORTER_OTLP_METRICS_TIMEOUT, + opentelemetry_otlp::OTEL_EXPORTER_OTLP_TIMEOUT, + ] + .iter() + .find_map(|var| std::env::var(var).ok()?.trim().parse::().ok()) + .map(std::time::Duration::from_millis) + .unwrap_or(opentelemetry_otlp::OTEL_EXPORTER_OTLP_TIMEOUT_DEFAULT) +} + +/// Resource attribute keys the operator declared through the environment, +/// which must not be overwritten — see the merge note in +/// [`build_provider_blocking`]. +fn env_declared_resource_keys() -> Vec { + let mut keys = Vec::new(); + if std::env::var("OTEL_SERVICE_NAME").is_ok_and(|v| !v.trim().is_empty()) { + keys.push("service.name".to_owned()); + } + if let Ok(attrs) = std::env::var("OTEL_RESOURCE_ATTRIBUTES") { + keys.extend(attrs.split(',').filter_map(|pair| { + let key = pair.split('=').next()?.trim(); + (!key.is_empty()).then(|| key.to_owned()) + })); + } + keys +} + fn build_provider_blocking( endpoint: Option<&str>, pubkey: String, @@ -296,32 +400,48 @@ fn build_provider_blocking( if let Some(endpoint) = endpoint { builder = builder.with_endpoint(endpoint); } - if let Some(signer) = auth_signer { - // Same blocking client the exporter defaults to (PeriodicReader - // exports off-runtime — see Cargo.toml), wrapped to sign each request. - builder = builder.with_http_client(FreenetAuthClient { - inner: reqwest::blocking::Client::new(), - signer, - pubkey_b58: pubkey.clone(), - }); - } + // Always ours, in every auth mode — see `OtlpHttpClient`. Blocking on + // purpose: PeriodicReader exports off-runtime (see Cargo.toml). The + // timeout is the one the SDK would have applied to its own client, which + // it does not apply to a supplied one. + builder = builder.with_http_client(OtlpHttpClient { + inner: reqwest::blocking::Client::builder() + .timeout(export_timeout()) + .build() + .unwrap_or_else(|_| reqwest::blocking::Client::new()), + signer: auth_signer, + pubkey_b58: pubkey.clone(), + }); let exporter = builder.build()?; - // `service.name` is overridden by OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES - // when the operator sets them; the SDK reads those itself. - // // Resource attributes ride once per export batch, not per datapoint, so // identifying THIS node here costs nothing per series — unlike a // per-datapoint attribute, which is why no instrument below carries one // identifying the remote end of a connection. - let resource = Resource::builder() - .with_service_name("freenet-node") - .with_attribute(KeyValue::new("freenet.node.pubkey", pubkey)) - .with_attribute(KeyValue::new("freenet.node.fingerprint", fingerprint)) - .with_attribute(KeyValue::new("service.version", env!("CARGO_PKG_VERSION"))) - .with_attribute(KeyValue::new("os.type", std::env::consts::OS)) - .with_attribute(KeyValue::new("host.arch", std::env::consts::ARCH)) - .build(); + // + // `Resource::builder` seeds from OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES + // and then `with_attribute`/`with_service_name` MERGE OVER that seed, so + // setting a literal unconditionally silently discards the operator's + // value: two nodes on one host with distinct OTEL_SERVICE_NAMEs would both + // export `service.name=freenet-node`. Only fill in what the environment + // did not declare. + let declared = env_declared_resource_keys(); + let mut resource = Resource::builder(); + if !declared.iter().any(|k| k == "service.name") { + resource = resource.with_service_name("freenet-node"); + } + for (key, value) in [ + ("freenet.node.pubkey", pubkey), + ("freenet.node.fingerprint", fingerprint), + ("service.version", env!("CARGO_PKG_VERSION").to_owned()), + ("os.type", std::env::consts::OS.to_owned()), + ("host.arch", std::env::consts::ARCH.to_owned()), + ] { + if !declared.iter().any(|k| k == key) { + resource = resource.with_attribute(KeyValue::new(key, value)); + } + } + let resource = resource.build(); Ok(SdkMeterProvider::builder() .with_periodic_exporter(exporter) @@ -397,7 +517,7 @@ pub(crate) fn record_nat_traversal(result: &'static str) { /// Record an operation outcome. `op` is one of get/put/update/subscribe. /// -/// ponytail: outcome only, no duration histogram — no driver measures its own +/// NOTE: outcome only, no duration histogram — no driver measures its own /// elapsed time today, and adding one means threading `TimeSource` through /// every `op_ctx_task` (raw `Instant::now()` is banned in this crate). Add the /// histogram when someone needs operation latency percentiles. @@ -632,12 +752,14 @@ fn register_queue_metrics(meter: &opentelemetry::metrics::Meter) { let _depth = meter .u64_observable_gauge("freenet.contract.queue.depth") - .with_description("Current fair-queue occupancy") + .with_description( + "Current fair-queue occupancy, per tier. No `total` series: it would \ + double-count under `sum by (queue)` — sum the tiers instead.", + ) .with_callback(|observer| { if let Some(s) = snapshot() { let q = &s.fair_queue; for (tier, depth) in [ - ("total", q.depth_total), ("client_local", q.depth_client_local), ("network_relay", q.depth_network_relay), ("background", q.depth_background), @@ -698,6 +820,9 @@ mod tests { use super::*; use crate::config::OtelConfig; + /// The collector this module's tests mint tokens for. + const TEST_AUDIENCE: &str = "collector.example:4318"; + fn enabled_config() -> OtelConfig { OtelConfig { enabled: true, @@ -711,7 +836,7 @@ mod tests { fn token_fixture() -> (crate::transport::TransportKeypair, String) { let keypair = crate::transport::TransportKeypair::new(); let pubkey_b58 = bs58::encode(keypair.public_key_bytes()).into_string(); - let token = bearer_token(&keypair.auth_token_signer(), &pubkey_b58); + let token = bearer_token(&keypair.auth_token_signer(), &pubkey_b58, TEST_AUDIENCE); (keypair, token) } @@ -722,8 +847,8 @@ mod tests { let (keypair, token) = token_fixture(); let parts: Vec<&str> = token.split('/').collect(); - let [scheme, pubkey, timestamp, signature] = parts[..] else { - panic!("expected 4 /-separated parts, got {token}"); + let [scheme, pubkey, audience, timestamp, signature] = parts[..] else { + panic!("expected 5 /-separated parts, got {token}"); }; assert_eq!(scheme, "freenet"); assert_eq!( @@ -731,6 +856,11 @@ mod tests { bs58::encode(keypair.public_key_bytes()).into_string(), "pubkey part must be the full base58 x25519 transport public key" ); + assert_eq!( + audience, TEST_AUDIENCE, + "the token must name the collector it was minted for, or it can be \ + replayed to any other collector accepting this scheme" + ); let ts: u64 = timestamp.parse().expect("timestamp is epoch seconds"); assert!( ts > 1_700_000_000, @@ -738,7 +868,7 @@ mod tests { ); // The signature covers everything before its own slash, and verifies // against the token's OWN pubkey — the transport key itself. - let signed_payload = format!("freenet/{pubkey}/{timestamp}"); + let signed_payload = format!("freenet/{pubkey}/{audience}/{timestamp}"); let sig_bytes: [u8; 64] = bs58::decode(signature) .into_vec() .unwrap() @@ -782,16 +912,11 @@ mod tests { .expect("stock ed25519 verify after Montgomery->Edwards conversion"); } - #[test] - fn send_bytes_puts_a_verifiable_bearer_header_on_the_wire() { + /// One-shot collector on a real socket: returns its address and a handle + /// yielding the raw request text it received. + fn oneshot_collector() -> (std::net::SocketAddr, std::thread::JoinHandle) { use std::io::{Read, Write}; - use ed25519_dalek::{Signature, Verifier, VerifyingKey}; - - // The full auth transport path: header injection plus the - // http::Request -> reqwest::blocking::Request conversion, observed - // from the collector's side of a real socket. Plain #[test], not - // tokio: the blocking client must stay out of async contexts. let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); let addr = listener.local_addr().unwrap(); let server = std::thread::spawn(move || { @@ -812,39 +937,62 @@ mod tests { .unwrap(); String::from_utf8_lossy(&raw).into_owned() }); + (addr, server) + } - let keypair = crate::transport::TransportKeypair::new(); - let pubkey_b58 = bs58::encode(keypair.public_key_bytes()).into_string(); - let client = FreenetAuthClient { - inner: reqwest::blocking::Client::new(), - signer: keypair.auth_token_signer(), - pubkey_b58: pubkey_b58.clone(), - }; - let request = Request::builder() + /// The `Authorization` header value as the collector saw it, if any. + fn wire_auth_header(raw: &str) -> Option { + raw.lines() + .find(|l| l.to_ascii_lowercase().starts_with("authorization:")) + .map(|l| l.split_once(':').unwrap().1.trim().to_owned()) + } + + fn post_export(client: &OtlpHttpClient, addr: std::net::SocketAddr, auth: Option<&str>) { + let mut request = Request::builder() .method("POST") - .uri(format!("http://{addr}/v1/metrics")) - .body(Bytes::from_static(b"export-payload")) - .unwrap(); + .uri(format!("http://{addr}/v1/metrics")); + if let Some(auth) = auth { + request = request.header(http::header::AUTHORIZATION, auth); + } + let request = request.body(Bytes::from_static(b"export-payload")).unwrap(); // futures' executor, not tokio: send_bytes blocks internally. let response = futures::executor::block_on(client.send_bytes(request)).unwrap(); assert!(response.status().is_success()); + } + + fn test_client(keypair: &crate::transport::TransportKeypair, signed: bool) -> OtlpHttpClient { + OtlpHttpClient { + inner: reqwest::blocking::Client::new(), + signer: signed.then(|| keypair.auth_token_signer()), + pubkey_b58: bs58::encode(keypair.public_key_bytes()).into_string(), + } + } + + #[test] + fn send_bytes_puts_a_verifiable_bearer_header_on_the_wire() { + use ed25519_dalek::{Signature, Verifier, VerifyingKey}; + + // The full auth transport path: header injection plus the + // http::Request -> reqwest::blocking::Request conversion, observed + // from the collector's side of a real socket. Plain #[test], not + // tokio: the blocking client must stay out of async contexts. + let (addr, server) = oneshot_collector(); + let keypair = crate::transport::TransportKeypair::new(); + let pubkey_b58 = bs58::encode(keypair.public_key_bytes()).into_string(); + post_export(&test_client(&keypair, true), addr, None); let raw = server.join().unwrap(); - let auth = raw - .lines() - .find(|l| l.to_ascii_lowercase().starts_with("authorization:")) - .expect("Authorization header must reach the wire"); - let token = auth - .split_once(':') - .unwrap() - .1 - .trim() + let token = wire_auth_header(&raw) + .expect("Authorization header must reach the wire") .strip_prefix("Bearer ") - .expect("Bearer scheme"); + .expect("Bearer scheme") + .to_owned(); let (payload, sig_b58) = token.rsplit_once('/').unwrap(); + let audience = addr.to_string(); assert!( - payload.starts_with(&format!("freenet/{pubkey_b58}/")), - "wire token must carry this node's pubkey: {token}" + payload.starts_with(&format!("freenet/{pubkey_b58}/{audience}/")), + "wire token must carry this node's pubkey and the collector's own \ + authority as the audience: {token}" ); // Verify exactly like a collector would — see // node_pubkey_is_verifiable_with_stock_ed25519. @@ -868,14 +1016,49 @@ mod tests { ); } + #[test] + fn an_operator_supplied_authorization_header_is_never_overwritten() { + // OTEL_EXPORTER_OTLP_HEADERS is applied by the exporter before it + // calls us. An operator pointing at a hosted collector that wants + // `Authorization: Basic ...` would otherwise get a 401 on every + // export with no way to turn our token off but `otel-auth-mode`. + let (addr, server) = oneshot_collector(); + let keypair = crate::transport::TransportKeypair::new(); + post_export( + &test_client(&keypair, true), + addr, + Some("Basic b3BlcmF0b3I="), + ); + assert_eq!( + wire_auth_header(&server.join().unwrap()).as_deref(), + Some("Basic b3BlcmF0b3I="), + "the operator's own credentials must reach the collector" + ); + } + + #[test] + fn disabled_auth_mode_sends_no_authorization_header() { + // The default mode: exporting to your own collector must not ship a + // signed assertion of this node's identity there. + let (addr, server) = oneshot_collector(); + let keypair = crate::transport::TransportKeypair::new(); + post_export(&test_client(&keypair, false), addr, None); + assert_eq!( + wire_auth_header(&server.join().unwrap()), + None, + "auth-mode disabled must put no Authorization header on the wire" + ); + } + #[test] fn fingerprint_attr_is_recomputable_from_the_pubkey_attr() { // Requirement: a node cannot fake the UI-facing fingerprint. The // collector derives it from the verified pubkey instead of trusting // it: b58-decode pubkey, take the first 12 bytes, b58-encode. + // Asserts on `identity_attributes` — the function production uses — + // not on strings rebuilt here, which would pass whatever init does. let keypair = crate::transport::TransportKeypair::new(); - let pubkey_attr = bs58::encode(keypair.public_key_bytes()).into_string(); - let fingerprint_attr = keypair.public().to_string(); + let (pubkey_attr, fingerprint_attr) = identity_attributes(&keypair); let decoded = bs58::decode(&pubkey_attr).into_vec().unwrap(); assert_eq!( @@ -894,11 +1077,113 @@ mod tests { let pubkey = bs58::encode(keypair.public_key_bytes()).into_string(); let signer = keypair.auth_token_signer(); assert_ne!( - bearer_token(&signer, &pubkey), - bearer_token(&signer, &pubkey) + bearer_token(&signer, &pubkey, TEST_AUDIENCE), + bearer_token(&signer, &pubkey, TEST_AUDIENCE) + ); + } + + #[test] + fn a_token_for_one_collector_does_not_verify_at_another() { + // The replay bound: a collector we export to must not be able to + // present our token at a different collector and impersonate us. + use xeddsa::xeddsa::Verify; + + let keypair = crate::transport::TransportKeypair::new(); + let (pubkey, _) = identity_attributes(&keypair); + let token = bearer_token(&keypair.auth_token_signer(), &pubkey, TEST_AUDIENCE); + let (payload, sig_b58) = token.rsplit_once('/').unwrap(); + let sig_bytes: [u8; 64] = bs58::decode(sig_b58) + .into_vec() + .unwrap() + .try_into() + .unwrap(); + let replayed = payload.replace(TEST_AUDIENCE, "other-collector:4318"); + assert_ne!(replayed, payload, "audience must appear in the payload"); + assert!( + xeddsa::xed25519::PublicKey(keypair.public_key_bytes()) + .verify(replayed.as_bytes(), &sig_bytes) + .is_err(), + "a token re-aimed at another collector must fail verification" + ); + } + + #[test] + fn init_refuses_to_start_from_a_test_process() { + // The suppression check lives in `init`, and `init` is unreachable + // under cfg(test) — so without this, deleting the whole block leaves + // every test green and a `--id` test network ships to a collector. + // Under cfg(test) an ENABLED config must still come back suppressed; + // if the check were gone, init would build a pipeline and return None. + assert_eq!( + init( + &enabled_config(), + &crate::transport::TransportKeypair::new() + ), + Some(OtelSuppression::TestHarness), + "init must consult otel_suppression_reason and return before \ + building anything" + ); + assert!( + INSTRUMENTS.get().is_none(), + "a suppressed init must not have registered instruments" ); } + /// Cross-file pin (a same-file scrape can be satisfied by its own literal + /// — see .claude/rules/bug-prevention-patterns.md): every hot-path mirror + /// that feeds a synchronous instrument must still exist. Delete one and + /// its counter reports zero forever with nothing else failing. + /// + /// Ceiling: presence in the file, not in the right function. A mirror + /// moved to the wrong call site still passes; a deleted one does not. + #[test] + fn every_sync_instrument_still_has_its_hot_path_mirror() { + for (source, call) in [ + ( + include_str!("../transport/metrics.rs"), + "crate::tracing::otel::record_rtt_ms(", + ), + ( + include_str!("../transport/metrics.rs"), + "crate::tracing::otel::record_cwnd(", + ), + ( + include_str!("../transport/metrics.rs"), + "crate::tracing::otel::record_transfer(\"completed\")", + ), + ( + include_str!("../transport/metrics.rs"), + "crate::tracing::otel::record_transfer(\"failed\")", + ), + ( + include_str!("../transport/metrics.rs"), + "crate::tracing::otel::record_nat_traversal(\"attempt\")", + ), + ( + include_str!("../transport/metrics.rs"), + "crate::tracing::otel::record_nat_traversal(\"established\")", + ), + ( + include_str!("../transport/metrics.rs"), + "crate::tracing::otel::record_nat_traversal(\"failed_error\")", + ), + ( + include_str!("../transport/metrics.rs"), + "crate::tracing::otel::record_nat_traversal(\"failed_version\")", + ), + ( + include_str!("../node/network_status.rs"), + "crate::tracing::otel::record_op_result(", + ), + ] { + assert!( + source.contains(call), + "missing hot-path mirror `{call}`: the instrument it feeds \ + would report zero forever" + ); + } + } + #[test] fn production_shaped_input_is_not_suppressed() { assert_eq!( @@ -1046,6 +1331,23 @@ mod tests { ); } + #[tokio::test] + async fn provider_builds_with_auth_disabled() { + // The default auth mode, and the only path where no signer is + // installed. It still gets our HttpClient — the exporter has no + // reqwest feature enabled and would fail with NoHttpClient otherwise. + let provider = build_provider( + Some("http://127.0.0.1:1/v1/metrics"), + "pubkey-under-test".to_string(), + "fingerprint-under-test".to_string(), + None, + ) + .expect("exporter build must succeed with auth disabled"); + tokio::task::spawn_blocking(move || provider.shutdown().expect("clean shutdown")) + .await + .expect("shutdown thread panicked"); + } + #[tokio::test] async fn provider_builds_inside_a_tokio_runtime() { // Two things under test. First, exporter construction must not panic diff --git a/docs/design/otel-metrics-exporter.md b/docs/design/otel-metrics-exporter.md index c429a2bac2..795278edc3 100644 --- a/docs/design/otel-metrics-exporter.md +++ b/docs/design/otel-metrics-exporter.md @@ -1,7 +1,7 @@ # Design: OpenTelemetry metrics exporter (isolated from existing telemetry) -Status: proposed. Implementation plan: -[`docs/superpowers/plans/2026-08-01-otel-metrics-exporter.md`](../superpowers/plans/2026-08-01-otel-metrics-exporter.md). +Status: implemented (`crates/core/src/tracing/otel.rs`). +Operator-facing configuration: [`docs/otel-metrics.md`](../otel-metrics.md). ## Problem @@ -66,9 +66,18 @@ Registered in `tracing/otel.rs::register_metrics`. Everything but the histograms and the four synchronous counters is an observable callback over state that already existed for the local dashboard. -Resource attributes: `service.instance.id` (transport public key fingerprint — -never a `PeerId`, which embeds our socket address), `service.version`, -`os.type`, `host.arch`, plus whatever `OTEL_RESOURCE_ATTRIBUTES` adds. +Resource attributes: `freenet.node.pubkey` (the full base58 **x25519** transport +public key — byte-equal to the bearer token's `` field, so a collector +that verified the signature has also verified the node id), +`freenet.node.fingerprint` (the truncated form UIs show, recomputable from +`freenet.node.pubkey`, so the collector derives rather than trusts it), +`service.name`, `service.version`, `os.type`, `host.arch`. Never a `PeerId`, +which renders as `{pub_key}@{addr}` and would export our socket address. + +The literals are applied only for keys the operator did **not** declare through +`OTEL_SERVICE_NAME` / `OTEL_RESOURCE_ATTRIBUTES`: `ResourceBuilder` seeds from +the environment and then merges `with_attribute` over that seed, so setting one +unconditionally would silently discard the operator's value. Not instrumented: `TransportMetrics::slowdowns_triggered` is read and reset but never incremented anywhere in the tree, so it is dead and was left out rather @@ -96,10 +105,42 @@ New `OtelArgs` (clap + serde) and `OtelConfig` (resolved), beside |---|---|---|---| | `otel-telemetry-enabled` | `FREENET_OTEL_TELEMETRY_ENABLED` | `false` | Enable the SDK metrics exporter | | `otel-endpoint` | (see below) | none | OTLP/HTTP collector base URL | +| `otel-auth-mode` | — | `disabled` | `freenet` (bearer token) or `disabled` (no header) | Default is `false`: nothing is exported yet, so off is the no-behavior-change default. Operators opt in. +`otel-telemetry-enabled` is `Option` in `OtelArgs` and has no clap +`default_value`, unlike its `telemetry-*` siblings. With a default, "unset" and +"explicitly false" are indistinguishable after parsing, so +`--otel-telemetry-enabled=false` could not override a `config.toml` that says +`true` — the off switch would not work. + +### Collector authentication + +`otel-auth-mode = "freenet"` puts a per-request + +``` +Authorization: Bearer freenet//// +``` + +on every export. `` is an XEdDSA (Signal construction, `xeddsa` +crate) signature over everything preceding it, made with the x25519 transport +secret itself (`TransportKeypair::auth_token_signer`) — the node's one +identity, no second key to cross-certify. All fields are base58 except +`` (the target collector's `host[:port]`, taken from the request URI) +and `` (epoch seconds). A collector verifies with a stock Ed25519 +library after converting the Montgomery public key to Edwards with sign bit 0, +then checks `` against its own hostname and `` against its +own clock. The audience binding is what stops a collector we export to from +replaying our token at a different collector. + +The default is `disabled` — no `Authorization` header. Pointing the exporter at +your own collector must not ship a signed assertion of this node's identity +somewhere that never asked for one; `freenet` mode is for collectors that +actually verify these tokens. Either way, an `Authorization` header supplied +through `OTEL_EXPORTER_OTLP_HEADERS` is never overwritten. + ### Endpoint precedence `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` > `OTEL_EXPORTER_OTLP_ENDPOINT` > @@ -127,20 +168,33 @@ No code for them. ## Dependencies `opentelemetry` is already non-optional in `crates/core/Cargo.toml`. -`opentelemetry_sdk` and `opentelemetry-otlp` are optional and reachable only -through the `trace-ot` feature; both are already in `Cargo.lock`. Making them -non-optional is the whole dependency change — their default features already -cover what is needed: - -- `opentelemetry-otlp` defaults: `http-proto`, `reqwest-blocking-client`, - `metrics`, `trace`, `logs`, `internal-logs`. -- `opentelemetry_sdk` defaults: `metrics`, `trace`, `logs` (workspace decl adds - `rt-tokio`). - -`reqwest-blocking-client` is load-bearing, not incidental: `PeriodicReader` runs -exports on a dedicated thread through `futures_executor::block_on` -(`periodic_reader.rs:419`). An async reqwest client on that thread has no tokio -reactor and would fail at export time. +`opentelemetry_sdk` and `opentelemetry-otlp` were optional and reachable only +through the `trace-ot` feature; both were already in `Cargo.lock`. Making them +non-optional, with `opentelemetry-otlp` trimmed to `http-proto` + `metrics` + +`internal-logs`, is the dependency change. Net new packages in `Cargo.lock`: +`xeddsa` and its `convert_case`. + +**No `reqwest-*-client` and no `reqwest-rustls` feature.** `tracing::otel` +always supplies its own `HttpClient`, and `opentelemetry-otlp` builds a client +only when none was given, so those features would be dead weight with a real +cost: they pull `opentelemetry-http`'s reqwest **0.13**, whose `rustls` feature +is hardwired to `aws-lc-rs` + `rustls-platform-verifier`. That is a second +reqwest major, a second TLS root store, a C/assembly `aws-lc-sys` build (via +`cc`+`cmake`, on musl and Windows release targets that install neither), and +two crypto providers inside one rustls — which makes +`CryptoProvider::get_default_or_install_from_crate_features` return `None` and +panic for any caller that does not pass a provider explicitly. Note that +switching to `reqwest-rustls-webpki-roots` does **not** avoid this: it still +enables `reqwest/default-tls`, i.e. the same aws-lc stack, and merely adds +webpki roots on top. + +Our client is blocking on purpose: `PeriodicReader` runs exports on a dedicated +thread through `futures_executor::block_on` (`periodic_reader.rs:419`), where +an async reqwest client has no tokio reactor. It rides the workspace reqwest +0.12, which already carries `rustls-tls`, so `https://` endpoints work with +nothing new in the graph. It also applies the export timeout +(`OTEL_EXPORTER_OTLP_METRICS_TIMEOUT` > `OTEL_EXPORTER_OTLP_TIMEOUT` > 10s) +itself, because the SDK only resolves that for a client it built. Because a feature array may not name a non-optional dependency, `trace-ot` drops `"opentelemetry-otlp"` from its list. @@ -168,19 +222,28 @@ inside a test process, which by construction trips signals 3 and 4. `crates/core/src/tracing/otel.rs`: ``` -MetricExporter::builder().with_http()[.with_endpoint(resolved)].build() +MetricExporter::builder().with_http()[.with_endpoint(resolved)] + .with_http_client(OtlpHttpClient { .. }) // always ours; signs when auth is on + .build() → SdkMeterProvider::builder() .with_periodic_exporter(exporter) .with_resource(Resource::builder() - .with_service_name("freenet-peer") - .with_attribute(KeyValue::new("peer.id", local_peer_id)) + .with_service_name("freenet-node") // only if OTEL_* did not set it + .with_attribute(KeyValue::new("freenet.node.pubkey", pubkey)) + .with_attribute(KeyValue::new("freenet.node.fingerprint", fingerprint)) .build()) + .with_view(/* base-2 exponential for every histogram */) .build() - → opentelemetry::global::set_meter_provider(provider.clone()) + → opentelemetry::global::set_meter_provider(provider) ``` -Exporter build failure logs a WARN and returns `None`. Metrics export must never -fail node startup. +Exporter build failure logs a WARN and the node starts anyway. Metrics export +must never fail node startup. + +The whole build runs on a plain `std::thread`: `reqwest::blocking::Client` owns +a private tokio runtime, and creating or dropping one inside an async context +panics with "Cannot drop a runtime in a context where blocking is not allowed". +`init` is called from the node's async build path. No wrapper type, no registry, no facade. Future instrumentation is `opentelemetry::global::meter("freenet").u64_counter(…)` at the call site. @@ -197,7 +260,7 @@ at in a collector to confirm the pipeline works. Not wired. `global::set_meter_provider` holds a reference for the process lifetime, and `PeriodicReader` exports every 60s, so at most one partial interval is lost at exit. Flushing on the signal path is plumbing this does not need yet; -the code carries a `ponytail:` comment naming the ceiling and the upgrade path. +the code carries a `NOTE:` comment naming the ceiling and the upgrade path. ## Wire-up @@ -205,10 +268,15 @@ the code carries a `ponytail:` comment naming the ceiling and the upgrade path. `TelemetryReporter::new` call (`node.rs:754`): ```rust -otel::init(&self.config.otel, self.local_peer_id_string()); +crate::tracing::otel::init(&self.config.otel, &self.key_pair); ``` -The provider is registered globally; nothing is stored on `Node`. +The transport keypair, not a `PeerId`: it yields both identity resource +attributes and, in `freenet` auth mode, the token signing key. The provider is +registered globally; nothing is stored on `Node`. `init` returns the +suppression reason (or `None` when it started) so a test can prove it consults +that check before building anything — `init` is otherwise unreachable under +`cfg(test)`. ## Testing @@ -223,7 +291,17 @@ The provider is registered globally; nothing is stored on `Node`. widening of the harness detector). - CLI/env parsing: `--otel-telemetry-enabled=false` parses as false. A bare clap flag with an `env` binding treats any value of the variable as true, - which would turn the exporter on for an operator trying to turn it off. + which would turn the exporter on for an operator trying to turn it off. It + also overrides a `config.toml` that says `true`, asserted through + `ConfigArgs::build()` rather than at parse level. +- Auth: token shape and stock-Ed25519 verification; a token re-aimed at another + collector fails to verify; the header reaches a real socket in `freenet` + mode, is absent in `disabled` mode, and never replaces an operator-supplied + `Authorization`. +- Pins that would otherwise be vacuous: `init` returns a suppression reason + under `cfg(test)` (deleting the check makes it build a pipeline instead), and + a cross-file scrape asserts every hot-path `record_*` mirror still exists in + `transport/metrics.rs` / `node/network_status.rs`. - Provider construction succeeds inside a tokio runtime against an unreachable endpoint (guards the reqwest-blocking-in-async-context concern; export failure is asynchronous and must not surface at build time). @@ -232,26 +310,14 @@ The provider is registered globally; nothing is stored on `Node`. ## Risks -- Making the two crates non-optional grows the default build. They do NOT - share reqwest with the existing HTTP client: `opentelemetry-otlp` pulls - reqwest 0.13, a separate major version from the workspace's reqwest 0.12. - `opentelemetry-otlp` is trimmed to `default-features = false` plus exactly - the metrics/http-proto/reqwest-blocking-client/reqwest-rustls/internal-logs - features. That drops the logs exporter only: `http-proto` mandates `trace`, - `prost` and `opentelemetry-proto`, so those remain in the default build and - the trim is narrower than "metrics-only". The remaining cost is a second - reqwest major version plus the trace+metrics OTLP exporter — confirm - cross-compile targets still build - (`.github/workflows/cross-compile.yml`, - `crates/core/tests/cross_compile_feature_split.rs`). -- `reqwest-rustls` pulls `aws-lc-rs`/`aws-lc-sys`, which builds C sources via - CMake, and rustls 0.23 unification turns it on workspace-wide. Prebuilt musl - bindings ship for both release targets and the musl jobs are native-arch, so - this is expected to work — but `cross-compile.yml` never runs on PRs and does - not install `cmake`, so a failure would land after merge on the release path. - Verify with a `workflow_dispatch` run on the branch before merging. +- Making the two crates non-optional grows the default build: the trim drops + the logs exporter only, since `http-proto` mandates `trace`, `prost` and + `opentelemetry-proto`. No new native toolchain requirement — see + Dependencies — so the musl/Windows release targets in + `.github/workflows/cross-compile.yml` (which never runs on PRs) build the + same way they did before. - `global::set_meter_provider` is process-global. A `trace-ot` build also sets - OTel globals (tracer provider, not meter provider) — confirm no conflict. + OTel globals (tracer provider, not meter provider). - `reqwest/blocking` gets enabled workspace-wide by feature unification, which pulls in a background runtime thread for blocking clients. @@ -259,4 +325,4 @@ The provider is registered globally; nothing is stored on `Node`. This is a feature, not a bug fix. Per [CONTRIBUTING.md](../../CONTRIBUTING.md) it needs a maintainer-approved issue -before implementation starts. +before implementation starts — see #5046. diff --git a/docs/otel-metrics.md b/docs/otel-metrics.md new file mode 100644 index 0000000000..2e9729c6e4 --- /dev/null +++ b/docs/otel-metrics.md @@ -0,0 +1,96 @@ +# Exporting node metrics to an OpenTelemetry collector + +A freenet node can export its own metrics — transport, ring, contract queue, +process RSS — to any OTLP/HTTP collector. It is **off by default** and +completely separate from `telemetry-enabled`, which feeds the project's central +dashboard: turning one on or off has no effect on the other. + +Design notes and the full instrument list are in +[`docs/design/otel-metrics-exporter.md`](design/otel-metrics-exporter.md). + +## Turning it on + +In `config.toml`: + +```toml +otel-telemetry-enabled = true +otel-endpoint = "http://collector.example:4318" +``` + +or on the command line: + +```bash +freenet network --otel-telemetry-enabled --otel-endpoint http://collector.example:4318 +``` + +`--otel-telemetry-enabled=false` turns it off again without editing the file. +`FREENET_OTEL_TELEMETRY_ENABLED` works too, and unlike a plain flag it honors +`=false`. + +Nodes started with `--id` (test networks, the integration harness) never +export, regardless of configuration. + +## Settings + +| `config.toml` key | Default | Meaning | +|---|---|---| +| `otel-telemetry-enabled` | `false` | Enable the exporter | +| `otel-endpoint` | none | Collector base URL, e.g. `http://collector:4318`. `/v1/metrics` is appended for you | +| `otel-auth-mode` | `disabled` | `disabled` sends no `Authorization` header; `freenet` sends a signed bearer token (below) | + +The standard OpenTelemetry environment variables take priority over +`otel-endpoint` and are handled by the SDK: +`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, `OTEL_EXPORTER_OTLP_ENDPOINT`, +`OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_TIMEOUT`, +`OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`, `OTEL_METRIC_EXPORT_INTERVAL` +(default 60s). With none of them and no `otel-endpoint`, the exporter uses +`http://localhost:4318`. + +When an environment variable overrides a configured `otel-endpoint`, the node +logs a warning at startup naming both, and the "OTel metrics exporter started" +line always names the endpoint actually in use. + +## Authentication + +For most setups leave `otel-auth-mode` at `disabled` and carry whatever +credentials your collector wants in `OTEL_EXPORTER_OTLP_HEADERS`: + +```bash +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(printf 'user:pass' | base64)" +``` + +An `Authorization` header set that way is never overwritten by the node. + +`otel-auth-mode = "freenet"` is for collectors that verify freenet node +identities. It adds + +``` +Authorization: Bearer freenet//// +``` + +to each export, where `` is an XEdDSA signature over the preceding +fields made with the node's transport key, `` is that key in base58, +and `` is the collector's `host:port`. It proves the metrics came +from the node they claim to, and the audience field means a token sent to one +collector cannot be replayed at another. Do not enable it for a collector that +does not check these tokens — it ships a signed assertion of your node's +identity to whatever it is pointed at. + +## Identifying a node + +Every export batch carries two resource attributes: + +- `freenet.node.pubkey` — the node's full transport public key, base58. This is + the value a collector verifies the bearer token against. +- `freenet.node.fingerprint` — the short form shown in UIs and the local + dashboard, for cross-referencing. + +Neither contains an address, so a node keeps the same identity across IP +changes. + +## Notes + +- `freenet.process.memory.rss` is Linux-only. On macOS and Windows the series + is empty; that is expected, not a broken pipeline. +- Export failures never affect the node: a collector that is down or rejecting + batches produces a log line and nothing else. diff --git a/docs/superpowers/plans/2026-08-01-otel-metrics-exporter.md b/docs/superpowers/plans/2026-08-01-otel-metrics-exporter.md deleted file mode 100644 index a559eb7eee..0000000000 --- a/docs/superpowers/plans/2026-08-01-otel-metrics-exporter.md +++ /dev/null @@ -1,869 +0,0 @@ -# OpenTelemetry Metrics Exporter Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a standards-configured OpenTelemetry SDK metrics pipeline to `freenet`, strictly isolated from the existing `telemetry-enabled` reporter, so real metrics can be exported to any OTLP collector. - -**Architecture:** A new `OtelArgs`/`OtelConfig` config pair (sibling of, never nested in, `TelemetryArgs`/`TelemetryConfig`) gates a new `crates/core/src/tracing/otel.rs` module. That module builds an OTLP/HTTP `MetricExporter` and an `SdkMeterProvider`, registers it as the process-global meter provider, and registers one observable RSS gauge as proof of life. The endpoint is resolved env-first, which requires working *around* `opentelemetry-otlp`'s own precedence. Everything in `telemetry.rs` is untouched. - -**Tech Stack:** Rust, clap + serde config, `opentelemetry` / `opentelemetry_sdk` / `opentelemetry-otlp` 0.32, OTLP over HTTP/protobuf with the blocking reqwest client. - -**Design spec:** [`docs/design/otel-metrics-exporter.md`](../../design/otel-metrics-exporter.md) - -## Global Constraints - -- **This is a feature, not a bug fix.** Per `CONTRIBUTING.md` a maintainer-approved issue MUST exist before implementation starts. Do not open a PR without it. -- Branch name: `feat/otel-metrics-exporter`. -- Conventional-commit subjects, under 72 chars, body explains WHY (`.claude/rules/git-workflow.md`). -- Before every commit: `cargo fmt` and `cargo clippy -p freenet -- -D warnings`. CI treats any warning as failure. -- No behavior change to `TelemetryReporter`, `to_otlp_logs`, the `/v1/logs` path, `telemetry-enabled`, or `telemetry-endpoint`. If a diff touches those, it is wrong. -- `otel::init` and everything it calls must never read `TelemetryConfig`. -- `otel-endpoint` must NEVER fall back to `DEFAULT_TELEMETRY_ENDPOINT` (`http://nova.locut.us:4318`). Its default is `http://localhost:4318`. -- Default of `otel-telemetry-enabled` is `false`. -- Crate under test is `freenet` (`crates/core`). Unit tests run with `cargo test -p freenet --lib `. -- Production code in `crates/core/` uses `TimeSource` for time and `GlobalRng` for randomness (`.claude/rules/code-style.md`). Neither is needed here — do not introduce `Instant::now()` or `rand::random()`. -- Any deliberate simplification gets a `ponytail:` comment naming the ceiling and the upgrade path. - ---- - -### Task 1: Config — `OtelArgs` / `OtelConfig` - -**Files:** -- Modify: `crates/core/src/config.rs` (add structs after `TelemetryConfig`'s helpers ~line 2237; add `ConfigArgs` field ~line 236; add `ConfigArgs::default()` entry ~line 294; add merge block in `build()` ~line 629; add `Config` field ~line 1312; add `Config` construction in `build()` ~line 1109) -- Test: `crates/core/src/config.rs` (the `mod tests` block — new tests plus the existing guard at line 5241) - -**Interfaces:** -- Consumes: nothing from earlier tasks. -- Produces: - - `pub const DEFAULT_OTEL_ENDPOINT: &str = "http://localhost:4318";` - - `pub struct OtelArgs { pub enabled: bool, pub endpoint: Option }` - - `pub struct OtelConfig { pub enabled: bool, pub endpoint: Option, pub is_test_environment: bool }` - - `Config::otel: OtelConfig`, `ConfigArgs::otel: OtelArgs` - -- [ ] **Step 1: Write the failing tests** - -Add to the `mod tests` block in `crates/core/src/config.rs`: - -```rust -#[test] -fn otel_args_default_is_off_and_endpointless() { - // The new pipeline exports nothing yet, so shipping it on would be a - // behavior change. Operators opt in explicitly. - let args = OtelArgs::default(); - assert!(!args.enabled, "otel-telemetry-enabled must default to false"); - assert_eq!(args.endpoint, None, "no implicit collector"); -} - -#[test] -fn otel_flag_parses_from_cli() { - use clap::Parser; - let none = ConfigArgs::try_parse_from(["freenet"]).expect("bare parse"); - assert!(!none.otel.enabled, "no flag -> off"); - let set = ConfigArgs::try_parse_from(["freenet", "--otel-telemetry-enabled"]) - .expect("flag parse"); - assert!(set.otel.enabled, "--otel-telemetry-enabled -> on"); - // Explicit `=false` must parse and mean false. Without this form the flag - // would be a bare ArgAction::SetTrue, and clap turns ANY value of the bound - // env var — including "false" — into true. - let off = ConfigArgs::try_parse_from(["freenet", "--otel-telemetry-enabled=false"]) - .expect("explicit false parse"); - assert!(!off.otel.enabled, "--otel-telemetry-enabled=false -> off"); - let with_ep = ConfigArgs::try_parse_from([ - "freenet", - "--otel-endpoint", - "http://collector.example:4318", - ]) - .expect("endpoint parse"); - assert_eq!( - with_ep.otel.endpoint.as_deref(), - Some("http://collector.example:4318") - ); -} - -#[test] -fn otel_endpoint_never_defaults_to_the_dashboard_collector() { - // Hard isolation requirement: the two pipelines share no backend. - assert_ne!( - DEFAULT_OTEL_ENDPOINT, DEFAULT_TELEMETRY_ENDPOINT, - "otel must not default to the central dashboard collector" - ); - assert_eq!(DEFAULT_OTEL_ENDPOINT, "http://localhost:4318"); -} -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `cargo test -p freenet --lib config::tests::otel_ -- --nocapture` -Expected: FAIL — `cannot find type OtelArgs in this scope`, `cannot find value DEFAULT_OTEL_ENDPOINT`. - -- [ ] **Step 3: Add the config types** - -Insert after `fn default_iface_tx_enabled()` (~line 2237) in `crates/core/src/config.rs`: - -```rust -/// Default OTLP/HTTP endpoint for the SDK metrics pipeline, used when neither -/// the standard `OTEL_EXPORTER_OTLP_*` env vars nor `otel-endpoint` are set. -/// -/// Deliberately NOT `DEFAULT_TELEMETRY_ENDPOINT`: `otel-telemetry-enabled` and -/// `telemetry-enabled` are strictly isolated features that are not expected to -/// share a backend. Pointing this pipeline at the central dashboard collector -/// must always be an explicit operator choice. -pub const DEFAULT_OTEL_ENDPOINT: &str = "http://localhost:4318"; - -/// CLI/file args for the OpenTelemetry SDK metrics exporter. -/// -/// Strictly independent of [`TelemetryArgs`]: no shared field, no shared -/// default, no fallback in either direction. -#[derive(clap::Parser, Debug, Clone, Default, Serialize, Deserialize)] -pub struct OtelArgs { - /// Enable the OpenTelemetry SDK metrics exporter. Independent of - /// `telemetry-enabled`; enabling or disabling one has no effect on the - /// other. - /// - /// `num_args`/`default_missing_value` rather than a bare flag: with an - /// `env` binding, clap's `SetTrue` action treats ANY value of the variable - /// as true, so `FREENET_OTEL_TELEMETRY_ENABLED=false` would silently turn - /// the exporter ON. This form accepts `--otel-telemetry-enabled`, - /// `--otel-telemetry-enabled=false`, and a properly parsed env value. - #[arg( - long = "otel-telemetry-enabled", - env = "FREENET_OTEL_TELEMETRY_ENABLED", - num_args = 0..=1, - default_value = "false", - default_missing_value = "true", - action = clap::ArgAction::Set - )] - #[serde(rename = "otel-telemetry-enabled", default)] - pub enabled: bool, - - /// OTLP/HTTP collector base URL (e.g. `http://collector:4318`). - /// - /// No clap `env =` binding on purpose. The standard - /// `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` - /// variables must take priority over this file-level value, and binding - /// them here would merge them into the config layer and invert that - /// precedence. They are resolved in `tracing::otel` instead. - #[arg(long = "otel-endpoint")] - #[serde(rename = "otel-endpoint", skip_serializing_if = "Option::is_none")] - pub endpoint: Option, -} - -/// Resolved configuration for the OpenTelemetry SDK metrics exporter. -#[derive(Debug, Clone, Default, Serialize, Deserialize)] -pub struct OtelConfig { - /// Whether the SDK metrics exporter is enabled. - #[serde(default, rename = "otel-telemetry-enabled")] - pub enabled: bool, - - /// Operator-configured OTLP/HTTP collector base URL, if any. `None` means - /// "let the SDK resolve it" — see `tracing::otel::resolve_metrics_endpoint`. - #[serde(default, rename = "otel-endpoint", skip_serializing_if = "Option::is_none")] - pub endpoint: Option, - - /// Whether this is a test environment (detected via `--id`). Mirrors - /// [`TelemetryConfig::is_test_environment`]; suppresses export so test - /// networks can't ship data to a collector. - #[serde(skip)] - pub is_test_environment: bool, -} -``` - -- [ ] **Step 4: Hang the args off `ConfigArgs`** - -In `crates/core/src/config.rs`, after the `pub telemetry: TelemetryArgs,` field (~line 236): - -```rust - #[command(flatten)] - pub otel: OtelArgs, -``` - -And in `impl Default for ConfigArgs`, after `telemetry: Default::default(),` (~line 294): - -```rust - otel: Default::default(), -``` - -- [ ] **Step 5: Run the tests to verify they pass** - -Run: `cargo test -p freenet --lib config::tests::otel_ -- --nocapture` -Expected: PASS (3 tests). - -- [ ] **Step 6: Write the failing round-trip test** - -In `crates/core/src/config.rs`, extend the existing guard test -`all_persisted_config_fields_round_trip_through_build` (line 5241). - -In the `seed` literal, after the `telemetry: TelemetryConfig { … },` block (~line 5323): - -```rust - otel: OtelConfig { - enabled: true, - endpoint: Some("http://example.invalid:4319".to_string()), - is_test_environment: false, // #[serde(skip)] — derived from --id - }, -``` - -In the exhaustive `let Config { … } = rebuilt;` destructure, after `telemetry,` (~line 5357): - -```rust - otel, -``` - -And after the `shutdown_drain_secs` assertion (~line 5403): - -```rust - assert_eq!(otel.enabled, seed.otel.enabled, "otel.enabled"); - assert_eq!( - otel.endpoint, seed.otel.endpoint, - "otel.endpoint — an operator's collector URL must survive the \ - config.toml merge" - ); -``` - -- [ ] **Step 7: Run it to verify it fails** - -Run: `cargo test -p freenet --lib config::tests::all_persisted_config_fields_round_trip_through_build` -Expected: FAIL to COMPILE — `struct Config has no field named otel`. - -- [ ] **Step 8: Add the `Config` field, the merge, and the construction** - -Three edits in `crates/core/src/config.rs`. - -(a) On `pub struct Config`, after `pub telemetry: TelemetryConfig,` (~line 1312): - -```rust - /// OpenTelemetry SDK metrics exporter settings. Strictly isolated from - /// `telemetry` above — see `docs/design/otel-metrics-exporter.md`. - #[serde(default)] - pub otel: OtelConfig, -``` - -(b) In `ConfigArgs::build()`, inside the `if let Some(cfg) = …` merge block, after the `iface_tx_enabled` merge (~line 629): - -```rust - // otel-telemetry-enabled defaults to false via clap, so only the - // file-says-true direction needs handling — same one-directional - // override as reference-ping/iface-tx above. Kept separate from the - // telemetry merge on purpose: the two features are independent. - if cfg.otel.enabled { - self.otel.enabled = true; - } - if let Some(endpoint) = cfg.otel.endpoint { - self.otel.endpoint.get_or_insert(endpoint); - } -``` - -(c) In the `Config { … }` literal in `build()`, after the `telemetry: TelemetryConfig { … },` block (~line 1109): - -```rust - otel: OtelConfig { - enabled: self.otel.enabled, - endpoint: self.otel.endpoint, - // Same --id rule as telemetry: simulated networks and - // integration tests must not ship data to a collector. - is_test_environment: self.id.is_some(), - }, -``` - -- [ ] **Step 9: Run the round-trip test to verify it passes** - -Run: `cargo test -p freenet --lib config::tests::all_persisted_config_fields_round_trip_through_build` -Expected: PASS. - -- [ ] **Step 10: Run the whole config module and lint** - -Run: `cargo test -p freenet --lib config::` -Expected: PASS, no regressions. - -Run: `cargo fmt && cargo clippy -p freenet -- -D warnings` -Expected: clean. - -- [ ] **Step 11: Commit** - -```bash -git add crates/core/src/config.rs -git commit -m "feat(otel): add isolated otel-telemetry config - -New OtelArgs/OtelConfig sit beside TelemetryArgs/TelemetryConfig rather -than inside them: the SDK metrics pipeline and the dashboard reporter are -independent features that are not expected to share a backend, so neither -enable-flag nor endpoint may fall back to the other." -``` - ---- - -### Task 2: Pure decision functions in `tracing::otel` - -**Files:** -- Create: `crates/core/src/tracing/otel.rs` -- Modify: `crates/core/src/tracing.rs` (add module declaration next to `pub mod telemetry;`, ~line 42) -- Modify: `crates/core/src/tracing/telemetry.rs:615` (widen `running_under_cargo_test` to `pub(crate)`) -- Test: `crates/core/src/tracing/otel.rs` (inline `mod tests`) - -**Interfaces:** -- Consumes: `crate::config::OtelConfig` from Task 1. -- Produces: - - `pub(crate) enum OtelSuppression { Disabled, TestEnvironmentFlag, TestHarness }` - - `pub(crate) fn otel_suppression_reason(cfg: &OtelConfig, is_test_build: bool, running_under_cargo_test: bool) -> Option` - - `pub(crate) fn resolve_metrics_endpoint(cfg_endpoint: Option<&str>, metrics_env: Option<&str>, generic_env: Option<&str>) -> Option` — `None` means "let the SDK resolve it". - -- [ ] **Step 1: Write the failing tests** - -Create `crates/core/src/tracing/otel.rs` containing ONLY this test module for now: - -```rust -#[cfg(test)] -mod tests { - use super::*; - use crate::config::OtelConfig; - - fn enabled_config() -> OtelConfig { - OtelConfig { - enabled: true, - endpoint: None, - is_test_environment: false, - } - } - - #[test] - fn production_shaped_input_is_not_suppressed() { - assert_eq!( - otel_suppression_reason(&enabled_config(), false, false), - None, - "a real release binary with the flag on must export" - ); - } - - #[test] - fn every_test_signal_suppresses() { - let disabled = OtelConfig { - enabled: false, - ..enabled_config() - }; - assert_eq!( - otel_suppression_reason(&disabled, false, false), - Some(OtelSuppression::Disabled) - ); - - let test_env = OtelConfig { - is_test_environment: true, - ..enabled_config() - }; - assert_eq!( - otel_suppression_reason(&test_env, false, false), - Some(OtelSuppression::TestEnvironmentFlag) - ); - - assert_eq!( - otel_suppression_reason(&enabled_config(), true, false), - Some(OtelSuppression::TestHarness), - "cfg(test) build" - ); - assert_eq!( - otel_suppression_reason(&enabled_config(), false, true), - Some(OtelSuppression::TestHarness), - "running from a cargo deps/ harness" - ); - } - - #[test] - fn metrics_env_wins_over_generic_env_and_config() { - // Both env forms mean "let the SDK resolve it", because - // opentelemetry-otlp gives a programmatic endpoint priority over the - // env vars — passing one would invert the required precedence. - assert_eq!( - resolve_metrics_endpoint( - Some("http://from-config:4318"), - Some("http://from-metrics-env:4318/v1/metrics"), - Some("http://from-generic-env:4318"), - ), - None - ); - assert_eq!( - resolve_metrics_endpoint( - Some("http://from-config:4318"), - None, - Some("http://from-generic-env:4318"), - ), - None - ); - } - - #[test] - fn config_endpoint_gets_the_signal_path_appended() { - // The SDK appends /v1/metrics only on the env-var path; a programmatic - // endpoint is used verbatim, so we append it ourselves. - assert_eq!( - resolve_metrics_endpoint(Some("http://collector:4318"), None, None), - Some("http://collector:4318/v1/metrics".to_string()) - ); - assert_eq!( - resolve_metrics_endpoint(Some("http://collector:4318/"), None, None), - Some("http://collector:4318/v1/metrics".to_string()), - "trailing slash must not double up" - ); - } - - #[test] - fn nothing_configured_defers_to_the_sdk_default() { - assert_eq!(resolve_metrics_endpoint(None, None, None), None); - assert_eq!( - resolve_metrics_endpoint(Some(" "), None, None), - None, - "a blank endpoint is not a configuration" - ); - } -} -``` - -- [ ] **Step 2: Declare the module and run the tests to verify they fail** - -In `crates/core/src/tracing.rs`, after `pub use telemetry::TelemetryReporter;` (~line 43): - -```rust -/// Standards-configured OpenTelemetry SDK metrics pipeline. Strictly isolated -/// from `telemetry` above — see `docs/design/otel-metrics-exporter.md`. -pub mod otel; -``` - -Run: `cargo test -p freenet --lib tracing::otel` -Expected: FAIL — `cannot find function otel_suppression_reason in this scope`. - -- [ ] **Step 3: Write the implementation** - -Prepend to `crates/core/src/tracing/otel.rs`, above the test module: - -```rust -//! Standards-configured OpenTelemetry SDK metrics pipeline. -//! -//! Strictly isolated from [`super::telemetry`]: nothing here reads -//! `TelemetryConfig`, and the endpoint never falls back to -//! `DEFAULT_TELEMETRY_ENDPOINT`. The two features are independent by design — -//! see `docs/design/otel-metrics-exporter.md`. - -use crate::config::OtelConfig; - -/// Why the OTel metrics exporter was not started. -/// -/// Mirrors `telemetry::TelemetrySuppression` so both pipelines refuse to ship -/// data from a test process, but the decision is computed from `OtelConfig` -/// alone — the two flags never consult each other. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub(crate) enum OtelSuppression { - /// Operator left `otel-telemetry-enabled` off (the default). - Disabled, - /// `--id` test environment (integration/CLI harness sets `is_test_environment`). - TestEnvironmentFlag, - /// A `cfg(test)` build or a binary running under a cargo test/bench harness. - TestHarness, -} - -/// Decide whether the metrics exporter should be suppressed. -/// -/// Pure and side-effect free: callers pass `cfg!(test)` and the result of -/// `telemetry::running_under_cargo_test()` so this is testable for a -/// production release binary (must NOT suppress) from inside a test process, -/// which by construction trips both test signals. -/// -/// Suppression is keyed only on signals a real release binary never matches, -/// and deliberately NOT on `cfg!(feature = "testing")` — that flag leaks onto -/// the shipped binary through Cargo feature unification with `fdev` and -/// silently disabled telemetry across the fleet once already (#4366, the -/// 0.2.81 blackout). See `telemetry::telemetry_suppression_reason`. -pub(crate) fn otel_suppression_reason( - config: &OtelConfig, - is_test_build: bool, - running_under_cargo_test: bool, -) -> Option { - if !config.enabled { - return Some(OtelSuppression::Disabled); - } - if config.is_test_environment { - return Some(OtelSuppression::TestEnvironmentFlag); - } - if is_test_build || running_under_cargo_test { - return Some(OtelSuppression::TestHarness); - } - None -} - -/// Endpoint to hand to `MetricExporter`'s builder, or `None` to let the SDK -/// resolve it. -/// -/// Required precedence is env > config file > SDK default, but -/// `opentelemetry-otlp` 0.32 inverts the first two: `resolve_http_endpoint` -/// (`src/exporter/http/mod.rs`) checks the programmatic value FIRST and only -/// then `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` / `OTEL_EXPORTER_OTLP_ENDPOINT`. -/// So whenever either variable is set we return `None` and stay out of the -/// way. It also appends the `/v1/metrics` signal path only on the env-var -/// path, so a config-file value gets the path appended here. -pub(crate) fn resolve_metrics_endpoint( - cfg_endpoint: Option<&str>, - metrics_env: Option<&str>, - generic_env: Option<&str>, -) -> Option { - let is_set = |v: Option<&str>| v.is_some_and(|s| !s.trim().is_empty()); - if is_set(metrics_env) || is_set(generic_env) { - return None; - } - let base = cfg_endpoint.map(str::trim).filter(|s| !s.is_empty())?; - Some(format!("{}/v1/metrics", base.trim_end_matches('/'))) -} -``` - -- [ ] **Step 4: Run the tests to verify they pass** - -Run: `cargo test -p freenet --lib tracing::otel` -Expected: PASS (5 tests). - -- [ ] **Step 5: Widen the shared harness detector** - -In `crates/core/src/tracing/telemetry.rs`, change line 615 from: - -```rust -fn running_under_cargo_test() -> bool { -``` - -to: - -```rust -pub(crate) fn running_under_cargo_test() -> bool { -``` - -Leave its doc comment unchanged. This is the only code shared between the two -pipelines — a free function with no config in it. - -- [ ] **Step 6: Verify the crate still builds and lints** - -Run: `cargo test -p freenet --lib tracing:: && cargo fmt && cargo clippy -p freenet -- -D warnings` -Expected: PASS, clean. - -- [ ] **Step 7: Commit** - -```bash -git add crates/core/src/tracing.rs crates/core/src/tracing/otel.rs crates/core/src/tracing/telemetry.rs -git commit -m "feat(otel): add suppression and endpoint-precedence logic - -Both decisions are pure functions so the production direction is testable -from inside a test process. Endpoint resolution deliberately returns None -when a standard OTEL_* var is set: opentelemetry-otlp gives a programmatic -endpoint priority over the env vars, which is the opposite of the -precedence operators expect." -``` - ---- - -### Task 3: Exporter, meter provider, RSS gauge, and node wire-up - -**Files:** -- Modify: `crates/core/Cargo.toml` (deps ~lines 129-130, `trace-ot` feature ~line 233) -- Modify: `crates/core/src/tracing/otel.rs` (add `init`, `build_provider`, `register_process_metrics`) -- Modify: `crates/core/src/node.rs` (~line 754, beside the `TelemetryReporter::new` call) -- Test: `crates/core/src/tracing/otel.rs` (inline `mod tests`) - -**Interfaces:** -- Consumes: `otel_suppression_reason`, `resolve_metrics_endpoint` (Task 2); `OtelConfig` (Task 1); `crate::node::resource_metrics::rss_bytes() -> Option` (existing, `node/resource_metrics.rs:79`); `NodeConfig::local_peer_id_string() -> String` (existing, `node.rs:441`). -- Produces: - - `pub fn init(config: &OtelConfig, local_peer_id: String)` - - `pub(crate) fn build_provider(endpoint: Option<&str>, local_peer_id: String) -> Result` - -- [ ] **Step 1: Make the OTel crates non-optional** - -In `crates/core/Cargo.toml`, change lines 129-130 from: - -```toml -opentelemetry-otlp = { workspace = true, optional = true } -opentelemetry_sdk = { workspace = true, optional = true } -``` - -to: - -```toml -# Non-optional: the SDK metrics pipeline (tracing::otel) ships in every build. -# Default features already give us metrics + http-proto + reqwest-blocking-client. -# The blocking client is load-bearing, not incidental: PeriodicReader exports -# from a dedicated thread via futures_executor::block_on, where an async reqwest -# client would have no tokio reactor. -opentelemetry-otlp = { workspace = true } -opentelemetry_sdk = { workspace = true } -``` - -And on line 233, drop `"opentelemetry-otlp"` from the feature list (a feature -array may not name a non-optional dependency): - -```toml -trace-ot = ["opentelemetry-jaeger", "trace", "tracing-opentelemetry"] -``` - -- [ ] **Step 2: Verify the crate still builds** - -Run: `cargo build -p freenet` -Expected: SUCCESS. - -Run: `cargo test -p freenet --test cross_compile_feature_split` -Expected: PASS. - -- [ ] **Step 3: Write the failing test** - -Add to the `mod tests` block in `crates/core/src/tracing/otel.rs`: - -```rust - #[tokio::test] - async fn provider_builds_inside_a_tokio_runtime() { - // Two things under test. First, exporter construction must not panic - // when it happens inside an async context — the OTLP HTTP exporter uses - // reqwest's BLOCKING client because PeriodicReader exports from its own - // thread. Second, an unreachable collector must not surface as a build - // error: export failures are asynchronous and must never fail node - // startup. Port 1 is chosen because nothing can be listening there. - let provider = build_provider( - Some("http://127.0.0.1:1/v1/metrics"), - "peer-under-test".to_string(), - ) - .expect("exporter build must succeed against an unreachable collector"); - provider.shutdown().expect("clean shutdown"); - } -``` - -- [ ] **Step 4: Run it to verify it fails** - -Run: `cargo test -p freenet --lib tracing::otel::tests::provider_builds_inside_a_tokio_runtime` -Expected: FAIL — `cannot find function build_provider in this scope`. - -- [ ] **Step 5: Write the implementation** - -Add to `crates/core/src/tracing/otel.rs`, after `resolve_metrics_endpoint` and -before the test module: - -```rust -use opentelemetry::{KeyValue, global}; -use opentelemetry_otlp::{ExporterBuildError, MetricExporter, WithExportConfig}; -use opentelemetry_sdk::{Resource, metrics::SdkMeterProvider}; - -/// Instrumentation scope name for every instrument this crate registers. -const METER_NAME: &str = "freenet"; - -/// Start the OpenTelemetry SDK metrics pipeline and install it as the -/// process-global meter provider. -/// -/// No-op when suppressed (see [`otel_suppression_reason`]) and best-effort -/// otherwise: an exporter that cannot be built logs a warning and the node -/// starts anyway. Metrics export must never be a startup dependency. -/// -/// After this returns, instrumentation anywhere in the crate is just -/// `opentelemetry::global::meter("freenet")` — there is deliberately no wrapper -/// type or registry to keep in sync. -pub fn init(config: &OtelConfig, local_peer_id: String) { - if let Some(reason) = otel_suppression_reason( - config, - cfg!(test), - super::telemetry::running_under_cargo_test(), - ) { - tracing::debug!(?reason, "OTel metrics exporter not started"); - return; - } - - let endpoint = resolve_metrics_endpoint( - config.endpoint.as_deref(), - std::env::var(opentelemetry_otlp::OTEL_EXPORTER_OTLP_METRICS_ENDPOINT) - .ok() - .as_deref(), - std::env::var(opentelemetry_otlp::OTEL_EXPORTER_OTLP_ENDPOINT) - .ok() - .as_deref(), - ); - - match build_provider(endpoint.as_deref(), local_peer_id) { - Ok(provider) => { - // ponytail: no shutdown hook. `set_meter_provider` holds a - // reference for the process lifetime and PeriodicReader exports - // every 60s (OTEL_METRIC_EXPORT_INTERVAL), so at most one partial - // interval is lost at exit. If that tail ever matters, keep the - // provider in a OnceLock and call `shutdown()` from the graceful - // shutdown path in `bin/freenet.rs`. - global::set_meter_provider(provider); - register_process_metrics(); - tracing::info!( - endpoint = endpoint.as_deref().unwrap_or(""), - "OTel metrics exporter started" - ); - } - Err(error) => { - tracing::warn!( - %error, - "OTel metrics exporter failed to start; node continues without metrics" - ); - } - } -} - -/// Build the OTLP/HTTP exporter and meter provider. -/// -/// `endpoint` is `None` when the standard env vars should win — see -/// [`resolve_metrics_endpoint`] for why calling `with_endpoint` at all would -/// override them. -pub(crate) fn build_provider( - endpoint: Option<&str>, - local_peer_id: String, -) -> Result { - let mut builder = MetricExporter::builder().with_http(); - if let Some(endpoint) = endpoint { - builder = builder.with_endpoint(endpoint); - } - let exporter = builder.build()?; - - // `service.name` is overridden by OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES - // when the operator sets them; the SDK reads those itself. - let resource = Resource::builder() - .with_service_name("freenet-peer") - .with_attribute(KeyValue::new("peer.id", local_peer_id)) - .build(); - - Ok(SdkMeterProvider::builder() - .with_periodic_exporter(exporter) - .with_resource(resource) - .build()) -} - -/// Register the instruments this crate owns. -/// -/// Must run AFTER `global::set_meter_provider`: `global::meter` binds to -/// whatever provider is installed at call time. -fn register_process_metrics() { - let meter = global::meter(METER_NAME); - // The handle is dropped on purpose — the callback is registered into the - // pipeline at `build()` and observed on every collection cycle regardless. - let _rss = meter - .u64_observable_gauge("freenet.process.memory.rss") - .with_unit("By") - .with_description("Resident set size of the freenet process") - .with_callback(|observer| { - if let Some(rss) = crate::node::resource_metrics::rss_bytes() { - observer.observe(rss, &[]); - } - }) - .build(); -} -``` - -- [ ] **Step 6: Run the test to verify it passes** - -Run: `cargo test -p freenet --lib tracing::otel` -Expected: PASS (6 tests). - -- [ ] **Step 7: Wire it into node startup** - -In `crates/core/src/node.rs`, in `build_with_flush_handle`, immediately after the -`if let Some(telemetry) = TelemetryReporter::new(…) { … }` block (~line 758) and -before `(DynamicRegister::new(registers), flush_handle)`: - -```rust - // Independent of the TelemetryReporter above: a separate opt-in - // (`otel-telemetry-enabled`), a separate endpoint, and a separate - // collector. It is not a NetEventRegister — it installs a global - // meter provider that instrumentation reaches via - // `opentelemetry::global::meter`. - crate::tracing::otel::init(&self.config.otel, self.local_peer_id_string()); -``` - -- [ ] **Step 8: Verify the node still builds and its tests pass** - -Run: `cargo build -p freenet && cargo test -p freenet --lib node::` -Expected: SUCCESS, PASS. - -- [ ] **Step 9: Lint** - -Run: `cargo fmt && cargo clippy -p freenet -- -D warnings` -Expected: clean. - -- [ ] **Step 10: Commit** - -```bash -git add crates/core/Cargo.toml crates/core/src/tracing/otel.rs crates/core/src/node.rs -git commit -m "feat(otel): export metrics through the OpenTelemetry SDK - -Installs a global meter provider backed by an OTLP/HTTP exporter, plus one -RSS gauge so the pipeline carries a real datapoint end to end. Future -instrumentation is a global::meter call at the site, with no registry to -keep in sync. The OTel crates become non-optional because the pipeline -ships in every build, not just trace-ot ones." -``` - ---- - -### Task 4: Operator documentation - -**Files:** -- Modify: `AGENTS.md` (new section after "Delegate secrets-at-rest") -- Reference: `docs/design/otel-metrics-exporter.md` (already written; do not restate it) - -**Interfaces:** -- Consumes: the config keys from Task 1 and the env-var precedence from Task 2. -- Produces: nothing code depends on. - -- [ ] **Step 1: Add the section** - -In `AGENTS.md`, after the `## Delegate secrets-at-rest` section, insert: - -```markdown -## Two independent telemetry pipelines - -`telemetry-enabled` / `telemetry-endpoint` feed the project's central -dashboard through a hand-rolled OTLP-JSON log POST (`tracing/telemetry.rs`). - -`otel-telemetry-enabled` / `otel-endpoint` are a **separate, unrelated** -OpenTelemetry SDK metrics pipeline (`tracing/otel.rs`). The two share no -config, no endpoint, and no fallback in either direction — enabling or -disabling one has no effect on the other, and `otel-endpoint` must never -default to the dashboard collector. - -The otel pipeline honors the standard variables, which take priority over -`otel-endpoint` in `config.toml`: -`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, `OTEL_EXPORTER_OTLP_ENDPOINT`, -`OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_SERVICE_NAME`, -`OTEL_RESOURCE_ATTRIBUTES`, `OTEL_METRIC_EXPORT_INTERVAL`. Without any of -them it exports to `http://localhost:4318`. - -Adding an instrument is one call at the site — no registry, no wrapper: - - opentelemetry::global::meter("freenet").u64_counter("freenet.some.thing").build() - -Design: [`docs/design/otel-metrics-exporter.md`](docs/design/otel-metrics-exporter.md). -``` - -- [ ] **Step 2: Verify the design doc links resolve** - -Run: `ls docs/design/otel-metrics-exporter.md docs/superpowers/plans/2026-08-01-otel-metrics-exporter.md` -Expected: both listed. - -- [ ] **Step 3: Commit** - -```bash -git add AGENTS.md -git commit -m "docs(otel): document the two independent telemetry pipelines - -The isolation between telemetry-enabled and otel-telemetry-enabled is a -design constraint, not an accident, so it belongs where the next person -reads before touching either." -``` - ---- - -## Verification - -After Task 4, before opening the PR: - -- [ ] `cargo fmt --check` — clean -- [ ] `cargo clippy -p freenet -- -D warnings` — clean -- [ ] `cargo test -p freenet --lib config:: tracing::` — pass -- [ ] `cargo test -p freenet --test cross_compile_feature_split` — pass -- [ ] `cargo build -p freenet --features trace-ot` — the feature-list edit in Task 3 did not break the jaeger path -- [ ] `git diff main --stat` — `tracing/telemetry.rs` shows exactly one changed line (the `pub(crate)` widening). Anything more means the isolation constraint was violated. - -Manual smoke check (optional, needs a collector): - -```bash -docker run --rm -p 4318:4318 otel/opentelemetry-collector:latest -FREENET_OTEL_TELEMETRY_ENABLED=true cargo run -p freenet --bin freenet -- network -# collector log should show freenet.process.memory.rss within ~60s -``` From 4a78d3b99155bec41c0107553b300e8a8ed846c4 Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Thu, 6 Aug 2026 16:01:04 -0500 Subject: [PATCH 14/17] fix(otel): bind the bearer token to a hash of the full target URL MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The audience was the URI authority, which carries userinfo: an endpoint of https://user:secret@collector/ would have signed the operator's password into a token sent over the wire and logged by the collector. Now base58(SHA-256(canonical URL)[..16]) — credentials stripped, and hashed because a URL contains the token's '/' separator. Binding the path too distinguishes two collectors behind one hostname. --- crates/core/src/tracing/otel.rs | 175 +++++++++++++++++++++++---- docs/design/otel-metrics-exporter.md | 38 ++++-- docs/otel-metrics.md | 27 ++++- 3 files changed, 206 insertions(+), 34 deletions(-) diff --git a/crates/core/src/tracing/otel.rs b/crates/core/src/tracing/otel.rs index 6d78533b2e..6e53a94a3c 100644 --- a/crates/core/src/tracing/otel.rs +++ b/crates/core/src/tracing/otel.rs @@ -91,20 +91,18 @@ pub(crate) fn resolve_metrics_endpoint( /// /// `` is the base58 full x25519 transport public key — the node's one /// real identity, the same key peers see and whose truncated fingerprint UIs -/// display. `` is the authority (`host[:port]`) of the collector the -/// request is going to, so a token is only valid at the collector it was -/// minted for: without it, any collector we export to could replay the token -/// to any other collector accepting this scheme and impersonate this node. -/// Authority rather than the full URL because it carries no `/`, which is the -/// token's field separator. `` is seconds since the Unix epoch, -/// `` is base58 too. +/// display. `` names the collector the request is going to (see +/// [`audience_of`]), so a token is only valid there: without it, any collector +/// we export to could replay the token to any other collector accepting this +/// scheme and impersonate this node. `` is seconds since the Unix +/// epoch, `` is base58 too. /// Freshly built per export request so the timestamp stays current. /// /// Collector-side verification needs no exotic library: convert the /// Montgomery pubkey to Edwards (sign bit 0), then standard Ed25519 verify — /// see `node_pubkey_is_verifiable_with_stock_ed25519` below. The collector -/// must additionally check `` against its own hostname and -/// `` against its own clock. +/// must additionally check `` against the hash of each URL it +/// answers at (see [`audience_of`]) and `` against its own clock. pub(crate) fn bearer_token( signer: &xeddsa::xed25519::PrivateKey, pubkey_b58: &str, @@ -130,6 +128,54 @@ pub(crate) fn bearer_token( format!("{signed_payload}/{signature}") } +/// The `` field of a bearer token: base58 of the first 16 bytes of +/// `SHA-256(canonical target URL)`. +/// +/// A hash rather than the URL itself for two reasons: a URL contains `/`, +/// which is the token's field separator, and the full URL is longer than the +/// binding needs to be. 16 bytes is 128 bits — an attacker looking to reuse a +/// token elsewhere needs a *second meaningful collector URL* colliding with +/// the first, which this is far past sufficient for. +/// +/// The collector recomputes this from the URL(s) it expects to be reached at +/// and compares, so both sides must canonicalize identically. The rules, +/// exactly: +/// +/// - `{scheme}://{host}:{port}{path}`, e.g. +/// `http://collector.example:4318/v1/metrics`. +/// - scheme and host lowercased (both are case-insensitive). +/// - port always explicit, defaulting to 80 for `http` and 443 for `https`, +/// so `https://c.example/x` and `https://c.example:443/x` agree. +/// - path verbatim — no trailing-slash or dot-segment normalization. +/// - **userinfo stripped**, and query/fragment dropped (an OTLP export URL has +/// neither). Stripping userinfo is not cosmetic: hashing an endpoint of +/// `https://user:secret@collector/` would make the value unreproducible for +/// a collector that does not know the password, and it keeps credentials out +/// of the signed input entirely. +/// +/// Cost of hashing: a rejected token tells the collector nothing about what +/// the sender aimed at. The node logs its resolved endpoint at startup, and +/// `docs/otel-metrics.md` documents this computation so a mismatch can be +/// worked out by hand. +fn audience_of(uri: &http::Uri) -> String { + use sha2::{Digest, Sha256}; + + let Some(host) = uri.host() else { + return String::new(); + }; + let scheme = uri.scheme_str().unwrap_or("http").to_ascii_lowercase(); + let port = uri.port_u16().unwrap_or(match scheme.as_str() { + "https" => 443, + _ => 80, + }); + let canonical = format!( + "{scheme}://{}:{port}{}", + host.to_ascii_lowercase(), + uri.path() + ); + bs58::encode(&Sha256::digest(canonical.as_bytes())[..16]).into_string() +} + /// The exporter's only HTTP transport, in every auth mode. /// /// Always installed, so `opentelemetry-otlp` never builds a client of its own @@ -163,8 +209,7 @@ impl HttpClient for OtlpHttpClient { // bearer token is one auth scheme among several, not the only one. if let Some(signer) = self.signer.as_ref() { if !request.headers().contains_key(http::header::AUTHORIZATION) { - let audience = request.uri().authority().map(|a| a.as_str()).unwrap_or(""); - let token = bearer_token(signer, &self.pubkey_b58, audience); + let token = bearer_token(signer, &self.pubkey_b58, &audience_of(request.uri())); request.headers_mut().insert( http::header::AUTHORIZATION, http::HeaderValue::from_str(&format!("Bearer {token}"))?, @@ -820,8 +865,15 @@ mod tests { use super::*; use crate::config::OtelConfig; - /// The collector this module's tests mint tokens for. - const TEST_AUDIENCE: &str = "collector.example:4318"; + /// The collector this module's tests mint tokens for, as the audience + /// hash of `http://collector.example:4318/v1/metrics`. + fn test_audience() -> String { + audience_of( + &"http://collector.example:4318/v1/metrics" + .parse::() + .unwrap(), + ) + } fn enabled_config() -> OtelConfig { OtelConfig { @@ -836,7 +888,7 @@ mod tests { fn token_fixture() -> (crate::transport::TransportKeypair, String) { let keypair = crate::transport::TransportKeypair::new(); let pubkey_b58 = bs58::encode(keypair.public_key_bytes()).into_string(); - let token = bearer_token(&keypair.auth_token_signer(), &pubkey_b58, TEST_AUDIENCE); + let token = bearer_token(&keypair.auth_token_signer(), &pubkey_b58, &test_audience()); (keypair, token) } @@ -857,7 +909,8 @@ mod tests { "pubkey part must be the full base58 x25519 transport public key" ); assert_eq!( - audience, TEST_AUDIENCE, + audience, + test_audience(), "the token must name the collector it was minted for, or it can be \ replayed to any other collector accepting this scheme" ); @@ -988,11 +1041,16 @@ mod tests { .expect("Bearer scheme") .to_owned(); let (payload, sig_b58) = token.rsplit_once('/').unwrap(); - let audience = addr.to_string(); + // What the collector computes from the URL it answers at. + let audience = audience_of( + &format!("http://{addr}/v1/metrics") + .parse::() + .unwrap(), + ); assert!( payload.starts_with(&format!("freenet/{pubkey_b58}/{audience}/")), - "wire token must carry this node's pubkey and the collector's own \ - authority as the audience: {token}" + "wire token must carry this node's pubkey and the audience hash of \ + the URL it was actually sent to: {token}" ); // Verify exactly like a collector would — see // node_pubkey_is_verifiable_with_stock_ed25519. @@ -1077,8 +1135,8 @@ mod tests { let pubkey = bs58::encode(keypair.public_key_bytes()).into_string(); let signer = keypair.auth_token_signer(); assert_ne!( - bearer_token(&signer, &pubkey, TEST_AUDIENCE), - bearer_token(&signer, &pubkey, TEST_AUDIENCE) + bearer_token(&signer, &pubkey, &test_audience()), + bearer_token(&signer, &pubkey, &test_audience()) ); } @@ -1090,14 +1148,19 @@ mod tests { let keypair = crate::transport::TransportKeypair::new(); let (pubkey, _) = identity_attributes(&keypair); - let token = bearer_token(&keypair.auth_token_signer(), &pubkey, TEST_AUDIENCE); + let token = bearer_token(&keypair.auth_token_signer(), &pubkey, &test_audience()); let (payload, sig_b58) = token.rsplit_once('/').unwrap(); let sig_bytes: [u8; 64] = bs58::decode(sig_b58) .into_vec() .unwrap() .try_into() .unwrap(); - let replayed = payload.replace(TEST_AUDIENCE, "other-collector:4318"); + let other = audience_of( + &"http://other-collector:4318/v1/metrics" + .parse::() + .unwrap(), + ); + let replayed = payload.replace(&test_audience(), &other); assert_ne!(replayed, payload, "audience must appear in the payload"); assert!( xeddsa::xed25519::PublicKey(keypair.public_key_bytes()) @@ -1107,6 +1170,74 @@ mod tests { ); } + #[test] + fn the_audience_hash_is_reproducible_from_the_documented_canonical_url() { + // The collector recomputes this from the URL it expects, so the + // canonicalization is a wire contract. Recompute it here the way the + // doc comment (and docs/otel-metrics.md) describe, independently of + // audience_of's own string building. + use sha2::{Digest, Sha256}; + let expect = |canonical: &str| { + bs58::encode(&Sha256::digest(canonical.as_bytes())[..16]).into_string() + }; + + for (uri, canonical) in [ + ( + "http://collector.example:4318/v1/metrics", + "http://collector.example:4318/v1/metrics", + ), + // Default port filled in, scheme and host lowercased. + ( + "https://Collector.Example/v1/metrics", + "https://collector.example:443/v1/metrics", + ), + ( + "http://collector.example/v1/metrics", + "http://collector.example:80/v1/metrics", + ), + // Credentials stripped: a collector that does not know the + // password must still be able to reproduce the hash. + ( + "https://user:secret@collector.example:4318/v1/metrics", + "https://collector.example:4318/v1/metrics", + ), + ( + "http://[::1]:4318/v1/metrics", + "http://[::1]:4318/v1/metrics", + ), + // Everything at once: credentials, port, multi-segment path. + ( + "http://user:pass@host:1234/path/here", + "http://host:1234/path/here", + ), + ] { + let audience = audience_of(&uri.parse::().unwrap()); + assert_eq!(audience, expect(canonical), "audience of {uri}"); + assert!( + !audience.contains('/'), + "audience must not contain the token's field separator" + ); + } + + // The binding has to be sensitive to the parts it claims to cover. + let of = |u: &str| audience_of(&u.parse::().unwrap()); + assert_ne!( + of("https://c.example:4318/v1/metrics"), + of("http://c.example:4318/v1/metrics"), + "scheme is bound" + ); + assert_ne!( + of("http://c.example:4318/v1/metrics"), + of("http://c.example:4319/v1/metrics"), + "port is bound" + ); + assert_ne!( + of("http://c.example:4318/tenant-a/v1/metrics"), + of("http://c.example:4318/tenant-b/v1/metrics"), + "path is bound — two collectors behind one host must differ" + ); + } + #[test] fn init_refuses_to_start_from_a_test_process() { // The suppression check lives in `init`, and `init` is unreachable diff --git a/docs/design/otel-metrics-exporter.md b/docs/design/otel-metrics-exporter.md index 795278edc3..cc33777e4b 100644 --- a/docs/design/otel-metrics-exporter.md +++ b/docs/design/otel-metrics-exporter.md @@ -127,13 +127,37 @@ Authorization: Bearer freenet//// on every export. `` is an XEdDSA (Signal construction, `xeddsa` crate) signature over everything preceding it, made with the x25519 transport secret itself (`TransportKeypair::auth_token_signer`) — the node's one -identity, no second key to cross-certify. All fields are base58 except -`` (the target collector's `host[:port]`, taken from the request URI) -and `` (epoch seconds). A collector verifies with a stock Ed25519 -library after converting the Montgomery public key to Edwards with sign bit 0, -then checks `` against its own hostname and `` against its -own clock. The audience binding is what stops a collector we export to from -replaying our token at a different collector. +identity, no second key to cross-certify. `` and `` are +base58, `` is epoch seconds. + +`` binds the token to the collector it was minted for: without it, +any collector we export to could replay our token at any other collector +accepting this scheme and impersonate the node. It is +**base58 of the first 16 bytes of `SHA-256(canonical target URL)`**, hashed +rather than sent literally because a URL contains `/`, the token's field +separator. The canonical form both sides must agree on: + +- `{scheme}://{host}:{port}{path}`, e.g. `http://collector.example:4318/v1/metrics` +- scheme and host lowercased +- port always explicit, defaulting to 80 for `http` and 443 for `https` +- path verbatim, no normalization +- userinfo stripped, query and fragment dropped + +Stripping userinfo is load-bearing twice over: it keeps operator credentials +out of a signed, wire-visible field, and a collector that does not know the +password could not otherwise reproduce the hash. + +Hashing the full URL rather than just the authority means two collectors +behind one hostname on different paths (`/tenant-a` vs `/tenant-b`) get +distinct audiences. The cost is diagnosability — a rejected token tells the +collector nothing about where the sender thought it was pointing — so the node +logs its resolved endpoint at startup and `docs/otel-metrics.md` documents the +computation for hand-checking a mismatch. + +A collector verifies with a stock Ed25519 library after converting the +Montgomery public key to Edwards with sign bit 0, then checks `` +against the hash of each URL it answers at and `` against its own +clock. The default is `disabled` — no `Authorization` header. Pointing the exporter at your own collector must not ship a signed assertion of this node's identity diff --git a/docs/otel-metrics.md b/docs/otel-metrics.md index 2e9729c6e4..40c1d726ac 100644 --- a/docs/otel-metrics.md +++ b/docs/otel-metrics.md @@ -70,11 +70,28 @@ Authorization: Bearer freenet//// to each export, where `` is an XEdDSA signature over the preceding fields made with the node's transport key, `` is that key in base58, -and `` is the collector's `host:port`. It proves the metrics came -from the node they claim to, and the audience field means a token sent to one -collector cannot be replayed at another. Do not enable it for a collector that -does not check these tokens — it ships a signed assertion of your node's -identity to whatever it is pointed at. +and `` identifies the exact URL the export was sent to. It proves the +metrics came from the node they claim to, and the audience field means a token +sent to one collector cannot be replayed at another. Do not enable it for a +collector that does not check these tokens — it ships a signed assertion of +your node's identity to whatever it is pointed at. + +`` is base58 of the first 16 bytes of `SHA-256` over the canonical +target URL, which is `{scheme}://{host}:{port}{path}` with scheme and host +lowercased, the port always explicit (80 for `http`, 443 for `https` when the +URL omits it), the path verbatim, and any `user:password@` stripped. So an +endpoint of `http://collector.example:4318` produces the audience for +`http://collector.example:4318/v1/metrics`, which you can reproduce with: + +```bash +printf 'http://collector.example:4318/v1/metrics' \ + | openssl dgst -sha256 -binary | head -c 16 | base58 +``` + +If your collector rejects tokens with an audience mismatch, compare that value +against the endpoint in the node's "OTel metrics exporter started" log line — +the two must name the same URL, including the path your ingress finally +delivers to. ## Identifying a node From f94e0a78021245e04da85072121463de6c9f4f0b Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Thu, 6 Aug 2026 16:09:06 -0500 Subject: [PATCH 15/17] fix(otel): drop the scheme from the audience canonical form MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hash host:port/path, not scheme://host:port/path. The scheme names a transport, not a party, so binding it never narrows which collector may use a token — it only forces a collector reachable over both http and https to be configured twice. Userinfo was already stripped. --- crates/core/src/tracing/otel.rs | 65 ++++++++++++++++------------ docs/design/otel-metrics-exporter.md | 11 +++-- docs/otel-metrics.md | 12 ++--- 3 files changed, 51 insertions(+), 37 deletions(-) diff --git a/crates/core/src/tracing/otel.rs b/crates/core/src/tracing/otel.rs index 6e53a94a3c..fcc16769c0 100644 --- a/crates/core/src/tracing/otel.rs +++ b/crates/core/src/tracing/otel.rs @@ -141,11 +141,10 @@ pub(crate) fn bearer_token( /// and compares, so both sides must canonicalize identically. The rules, /// exactly: /// -/// - `{scheme}://{host}:{port}{path}`, e.g. -/// `http://collector.example:4318/v1/metrics`. -/// - scheme and host lowercased (both are case-insensitive). -/// - port always explicit, defaulting to 80 for `http` and 443 for `https`, -/// so `https://c.example/x` and `https://c.example:443/x` agree. +/// - `{host}:{port}{path}`, e.g. `collector.example:4318/v1/metrics`. +/// - host lowercased (case-insensitive). +/// - port always explicit, defaulting to 80 for an `http` URL and 443 for an +/// `https` one, so `https://c.example/x` and `https://c.example:443/x` agree. /// - path verbatim — no trailing-slash or dot-segment normalization. /// - **userinfo stripped**, and query/fragment dropped (an OTLP export URL has /// neither). Stripping userinfo is not cosmetic: hashing an endpoint of @@ -153,6 +152,13 @@ pub(crate) fn bearer_token( /// a collector that does not know the password, and it keeps credentials out /// of the signed input entirely. /// +/// The scheme is deliberately NOT part of the hashed string. It identifies a +/// transport, not a party, so binding it would not narrow "which collector may +/// use this token" at all — while forcing every collector reachable over both +/// http and https to configure the same URL twice. Note it still leaks in +/// indirectly through the default-port rule above, which is why +/// `http://c.example/x` and `https://c.example/x` do not collide. +/// /// Cost of hashing: a rejected token tells the collector nothing about what /// the sender aimed at. The node logs its resolved endpoint at startup, and /// `docs/otel-metrics.md` documents this computation so a mismatch can be @@ -163,16 +169,11 @@ fn audience_of(uri: &http::Uri) -> String { let Some(host) = uri.host() else { return String::new(); }; - let scheme = uri.scheme_str().unwrap_or("http").to_ascii_lowercase(); - let port = uri.port_u16().unwrap_or(match scheme.as_str() { - "https" => 443, + let port = uri.port_u16().unwrap_or(match uri.scheme_str() { + Some("https") => 443, _ => 80, }); - let canonical = format!( - "{scheme}://{}:{port}{}", - host.to_ascii_lowercase(), - uri.path() - ); + let canonical = format!("{}:{port}{}", host.to_ascii_lowercase(), uri.path()); bs58::encode(&Sha256::digest(canonical.as_bytes())[..16]).into_string() } @@ -1184,31 +1185,28 @@ mod tests { for (uri, canonical) in [ ( "http://collector.example:4318/v1/metrics", - "http://collector.example:4318/v1/metrics", + "collector.example:4318/v1/metrics", ), - // Default port filled in, scheme and host lowercased. + // Default port filled in from the scheme, host lowercased. ( "https://Collector.Example/v1/metrics", - "https://collector.example:443/v1/metrics", + "collector.example:443/v1/metrics", ), ( "http://collector.example/v1/metrics", - "http://collector.example:80/v1/metrics", + "collector.example:80/v1/metrics", ), // Credentials stripped: a collector that does not know the // password must still be able to reproduce the hash. ( "https://user:secret@collector.example:4318/v1/metrics", - "https://collector.example:4318/v1/metrics", - ), - ( - "http://[::1]:4318/v1/metrics", - "http://[::1]:4318/v1/metrics", + "collector.example:4318/v1/metrics", ), + ("http://[::1]:4318/v1/metrics", "[::1]:4318/v1/metrics"), // Everything at once: credentials, port, multi-segment path. ( "http://user:pass@host:1234/path/here", - "http://host:1234/path/here", + "host:1234/path/here", ), ] { let audience = audience_of(&uri.parse::().unwrap()); @@ -1221,11 +1219,6 @@ mod tests { // The binding has to be sensitive to the parts it claims to cover. let of = |u: &str| audience_of(&u.parse::().unwrap()); - assert_ne!( - of("https://c.example:4318/v1/metrics"), - of("http://c.example:4318/v1/metrics"), - "scheme is bound" - ); assert_ne!( of("http://c.example:4318/v1/metrics"), of("http://c.example:4319/v1/metrics"), @@ -1236,6 +1229,22 @@ mod tests { of("http://c.example:4318/tenant-b/v1/metrics"), "path is bound — two collectors behind one host must differ" ); + // ...and NOT to the scheme, which names a transport rather than a + // party: one collector reachable both ways must not need two + // audience entries. Deliberate, so pin it rather than leave it to + // be "fixed" later. + assert_eq!( + of("https://c.example:4318/v1/metrics"), + of("http://c.example:4318/v1/metrics"), + "scheme is deliberately not bound" + ); + // It does still reach the hash through the default-port rule, so + // these two do not collide. + assert_ne!( + of("https://c.example/v1/metrics"), + of("http://c.example/v1/metrics"), + "an omitted port defaults per scheme: 443 vs 80" + ); } #[test] diff --git a/docs/design/otel-metrics-exporter.md b/docs/design/otel-metrics-exporter.md index cc33777e4b..e94d5caa51 100644 --- a/docs/design/otel-metrics-exporter.md +++ b/docs/design/otel-metrics-exporter.md @@ -137,11 +137,16 @@ accepting this scheme and impersonate the node. It is rather than sent literally because a URL contains `/`, the token's field separator. The canonical form both sides must agree on: -- `{scheme}://{host}:{port}{path}`, e.g. `http://collector.example:4318/v1/metrics` -- scheme and host lowercased -- port always explicit, defaulting to 80 for `http` and 443 for `https` +- `{host}:{port}{path}`, e.g. `collector.example:4318/v1/metrics` +- host lowercased +- port always explicit, defaulting to 80 for an `http` URL and 443 for an `https` one - path verbatim, no normalization - userinfo stripped, query and fragment dropped +- **scheme not included**: it names a transport, not a party, so binding it + would not narrow which collector may use the token, while forcing every + collector reachable over both http and https to be configured twice. It does + still reach the hash through the default-port rule, so `http://c/x` and + `https://c/x` differ. Stripping userinfo is load-bearing twice over: it keeps operator credentials out of a signed, wire-visible field, and a collector that does not know the diff --git a/docs/otel-metrics.md b/docs/otel-metrics.md index 40c1d726ac..1cb450f0c8 100644 --- a/docs/otel-metrics.md +++ b/docs/otel-metrics.md @@ -77,14 +77,14 @@ collector that does not check these tokens — it ships a signed assertion of your node's identity to whatever it is pointed at. `` is base58 of the first 16 bytes of `SHA-256` over the canonical -target URL, which is `{scheme}://{host}:{port}{path}` with scheme and host -lowercased, the port always explicit (80 for `http`, 443 for `https` when the -URL omits it), the path verbatim, and any `user:password@` stripped. So an -endpoint of `http://collector.example:4318` produces the audience for -`http://collector.example:4318/v1/metrics`, which you can reproduce with: +target, which is `{host}:{port}{path}` — host lowercased, port always explicit +(filled in from the scheme as 80 or 443 when the URL omits it), path verbatim, +any `user:password@` stripped, and no scheme. So an endpoint of +`http://collector.example:4318` produces the audience for +`collector.example:4318/v1/metrics`, which you can reproduce with: ```bash -printf 'http://collector.example:4318/v1/metrics' \ +printf 'collector.example:4318/v1/metrics' \ | openssl dgst -sha256 -binary | head -c 16 | base58 ``` From f891539c4b309421862fbc49d472c92f79f3c990 Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Thu, 6 Aug 2026 22:49:49 -0500 Subject: [PATCH 16/17] fix(otel): make export failures and identity spoofing visible Findings from a full review pass on this branch. Export outcomes were invisible: opentelemetry-otlp logs network errors and non-2xx at DEBUG, justified by a comment claiming PeriodicReader re-logs them via otel_error!. That holds for the batch log/span processors and NOT for metrics, where the reader logs its export result with otel_debug! and the only otel_error! is thread creation. A dead collector produced no output at all while startup still said "started". send_bytes now warns once per failing streak and logs the recovery edge. OTEL_RESOURCE_ATTRIBUTES could shadow freenet.node.pubkey, exporting an identity that does not match the key the bearer token was signed with, silently breaking the collector's self-validation. The two identity attributes are now always emitted; descriptive attributes still defer. Also: reject endpoints reqwest cannot send (http::Uri accepts host:port as a schemeless authority, so the exporter built and every export then failed); propagate the HTTP client build error instead of falling back to a client with no timeout, which could stall the reader thread forever; warn when the exporter is enabled but suppressed, and on unparseable or millisecond-confused export timeouts; return None rather than defaults when the ring provider is unregistered, so a gauge is absent rather than reporting a real zero; publish the fair-queue counters with fetch_max, since they are now exported as observable counters where a decrease reads as a reset. Docs corrected against the pinned crate sources: init's return value, the two endpoint env vars not being interchangeable, timeout ownership, OTEL_EXPORTER_OTLP_COMPRESSION being unsupported rather than free, slowdowns_triggered not being dead code, the auth-mode default, and the instrument counts. Fixed a guard test that rebuilt the identity strings locally instead of calling identity_attributes, so it would have passed had production switched back to PeerId. --- AGENTS.md | 12 +- crates/core/src/config.rs | 2 +- crates/core/src/contract/fair_queue.rs | 14 +- crates/core/src/node.rs | 9 +- crates/core/src/node/network_status.rs | 16 +- crates/core/src/tracing/otel.rs | 387 ++++++++++++++++++++----- docs/design/otel-metrics-exporter.md | 52 +++- docs/otel-metrics.md | 39 ++- 8 files changed, 430 insertions(+), 101 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 678bb79378..020313a5b5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -345,7 +345,17 @@ Rules when touching `tracing/otel.rs`: identity is two resource attributes (`freenet.node.*`, one per export batch), not a per-datapoint attribute. "Peer" means the *other* end of a connection. - Histograms get their base-2 exponential aggregation from one `with_view` in - `build_provider`. Do not add explicit bucket boundaries per instrument. + `build_provider_blocking`. Do not add explicit bucket boundaries per + instrument. +- Export outcomes must be logged by `OtlpHttpClient::send_bytes`. The SDK will + not do it: `opentelemetry-otlp` logs network errors and non-2xx at DEBUG on + the stated grounds that `PeriodicReader` re-logs them at error level, which + is true for the batch log/span processors and false for metrics. Deleting + that logging makes a dead collector produce no output at all. +- The `freenet.node.*` resource attributes are always emitted and must never + be made deferrable to `OTEL_RESOURCE_ATTRIBUTES`. They are what the + collector checks the bearer-token signature against; an override would + export an identity that does not match the signing key. - The exporter always installs its own `HttpClient`, so `opentelemetry-otlp` needs none of its `reqwest-*`/TLS features — enabling one pulls a second reqwest major, a second TLS stack, and a C/asm aws-lc build into every diff --git a/crates/core/src/config.rs b/crates/core/src/config.rs index bfc1f00aa2..10f49033ce 100644 --- a/crates/core/src/config.rs +++ b/crates/core/src/config.rs @@ -2908,7 +2908,7 @@ pub struct OtelArgs { /// Collector authentication method. `Option` so `build()` can tell "not /// given on the CLI" from an explicit choice and merge the config-file - /// value; resolves to [`OtelAuthMode::default`] (`freenet`) when neither + /// value; resolves to [`OtelAuthMode::default`] (`disabled`) when neither /// sets it. #[arg(id = "otel_auth_mode", long = "otel-auth-mode", value_enum)] #[serde(rename = "otel-auth-mode", skip_serializing_if = "Option::is_none")] diff --git a/crates/core/src/contract/fair_queue.rs b/crates/core/src/contract/fair_queue.rs index 0f797cad11..ff48f52f5c 100644 --- a/crates/core/src/contract/fair_queue.rs +++ b/crates/core/src/contract/fair_queue.rs @@ -429,10 +429,18 @@ impl FairEventQueue { gauge::DEPTH_CLIENT_LOCAL.store(s.depth_client_local, Ordering::Relaxed); gauge::DEPTH_NETWORK_RELAY.store(s.depth_network_relay, Ordering::Relaxed); gauge::DEPTH_BACKGROUND.store(s.depth_background, Ordering::Relaxed); + // `fetch_max`, not `store`, for the same reason as HIGH_WATER: these + // four are exported as OTel observable *counters* + // (`tracing::otel::register_queue_metrics`), where a value that ever + // decreases is read as a counter reset. `self.counters` only grows, so + // this is equivalent today — there is one `FairEventQueue` per process + // — but a second queue, or a respawned `contract_handling` loop, would + // publish its own counters from zero and drag the global series + // backwards. gauge::HIGH_WATER.fetch_max(s.high_water, Ordering::Relaxed); - gauge::REJECTED_GLOBAL_CAPACITY.store(s.rejected_global_capacity, Ordering::Relaxed); - gauge::REJECTED_PER_CONTRACT.store(s.rejected_per_contract, Ordering::Relaxed); - gauge::BACKGROUND_SHED.store(s.background_shed, Ordering::Relaxed); + gauge::REJECTED_GLOBAL_CAPACITY.fetch_max(s.rejected_global_capacity, Ordering::Relaxed); + gauge::REJECTED_PER_CONTRACT.fetch_max(s.rejected_per_contract, Ordering::Relaxed); + gauge::BACKGROUND_SHED.fetch_max(s.background_shed, Ordering::Relaxed); for band in DEPTH_WARN_BANDS { if previous_high_water < band && s.high_water >= band { diff --git a/crates/core/src/node.rs b/crates/core/src/node.rs index dbb1ee092b..b2429eb43a 100644 --- a/crates/core/src/node.rs +++ b/crates/core/src/node.rs @@ -823,10 +823,11 @@ impl NodeConfig { // collector. It is not a NetEventRegister — it installs a global // meter provider that instrumentation reaches via // `opentelemetry::global::meter`. - // Public-key fingerprint, NOT `local_peer_id_string()`: a PeerId - // renders as `{pub_key}@{addr}`, which would put this node's socket - // address on every exported batch and re-identify the node on every - // address change. + // The transport keypair, NOT a `PeerId`: it yields both identity + // resource attributes and, in `freenet` auth mode, the token + // signing key. A PeerId renders as `{pub_key}@{addr}`, which would + // put this node's socket address on every exported batch and + // re-identify the node on every address change. crate::tracing::otel::init(&self.config.otel, &self.key_pair); (DynamicRegister::new(registers), flush_handle) diff --git a/crates/core/src/node/network_status.rs b/crates/core/src/node/network_status.rs index 06652e9e46..bb413e24a8 100644 --- a/crates/core/src/node/network_status.rs +++ b/crates/core/src/node/network_status.rs @@ -126,15 +126,21 @@ pub(crate) struct OtelMetricsSnapshot { /// Read the scalars the OTel exporter observes, or `None` before the node has /// registered its status (metrics simply report nothing until then). +/// +/// Every unavailable source yields `None` for the whole snapshot rather than a +/// default: an observable instrument that skips a collection cycle exports +/// nothing, which reads as "not known yet", while a zero is a real datapoint — +/// `freenet.ring.connections = 0` before the ring provider registers is +/// indistinguishable from a node that has lost every connection. pub(crate) fn otel_metrics_snapshot() -> Option { let connection_attempts = NETWORK_STATUS.get()?.read().ok()?.connection_attempts; + let ring = RING_STATS_PROVIDER + .read() + .as_ref() + .map(|provider| provider())?; Some(OtelMetricsSnapshot { connection_attempts, - ring: RING_STATS_PROVIDER - .read() - .as_ref() - .map(|provider| provider()) - .unwrap_or_default(), + ring, fair_queue: crate::contract::fair_queue_stats(), }) } diff --git a/crates/core/src/tracing/otel.rs b/crates/core/src/tracing/otel.rs index fcc16769c0..1bc55c31b7 100644 --- a/crates/core/src/tracing/otel.rs +++ b/crates/core/src/tracing/otel.rs @@ -6,6 +6,7 @@ //! see `docs/design/otel-metrics-exporter.md`. use std::sync::OnceLock; +use std::sync::atomic::{AtomicBool, Ordering}; use opentelemetry::metrics::{Counter, Histogram}; use opentelemetry::{KeyValue, global}; @@ -85,6 +86,46 @@ pub(crate) fn resolve_metrics_endpoint( Some(format!("{}/v1/metrics", base.trim_end_matches('/'))) } +/// The endpoint the standard environment declares, in SDK precedence order, +/// or `None` when neither variable is set to a non-blank value. +/// +/// Blank-filtered on purpose: `env::var` returns `Ok("")` for a variable set +/// to the empty string, which [`resolve_metrics_endpoint`] correctly treats as +/// unset — so reading it raw would report an override that is not happening. +fn env_endpoint() -> Option { + [ + opentelemetry_otlp::OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, + opentelemetry_otlp::OTEL_EXPORTER_OTLP_ENDPOINT, + ] + .iter() + .find_map(|var| { + let value = std::env::var(var).ok()?; + let value = value.trim(); + (!value.is_empty()).then(|| value.to_owned()) + }) +} + +/// Why an OTLP endpoint will not work, or `None` when it is usable. +/// +/// Both failure modes are otherwise near-invisible. `http::Uri` accepts +/// `collector:4318` as an authority with no scheme, so the exporter builds and +/// then every export dies converting to a `reqwest::Request` ("relative URL"); +/// and an endpoint the SDK cannot parse at all is swallowed with `.ok()`, +/// falling back to `http://localhost:4318` while the startup log still names +/// the operator's URL. +fn endpoint_problem(endpoint: &str) -> Option<&'static str> { + match endpoint.parse::() { + Err(_) => Some( + "not a valid URL; the SDK will silently fall back to http://localhost:4318. \ + Include the scheme, e.g. http://collector:4318", + ), + Ok(uri) if !matches!(uri.scheme_str(), Some("http" | "https")) => { + Some("missing an http:// or https:// scheme; every export will fail to build a request") + } + Ok(_) => None, + } +} + /// Build one `freenet`-mode bearer token: /// `freenet////`, where `` /// is the XEdDSA signature over everything preceding it. @@ -201,6 +242,46 @@ impl std::fmt::Debug for OtlpHttpClient { } } +/// Whether the last export attempt failed, so failures log on the first +/// occurrence and on each failing<->recovering transition rather than every +/// 60s forever. +static EXPORT_FAILING: AtomicBool = AtomicBool::new(false); + +/// Report a failed export at WARN, once per failing streak. +/// +/// The SDK will not do this for us, despite `opentelemetry-otlp` logging both +/// network errors and non-2xx responses at DEBUG on the stated grounds that +/// "PeriodicReader already logs the returned error via `otel_error!`". That is +/// true for the batch log/span processors and FALSE for metrics: the metrics +/// `PeriodicReader` logs its export result via `otel_debug!` +/// ("PeriodReaderInvokedExport"), and the only `otel_error!` in that file is +/// thread-creation failure. Without this, a collector that is down, rejecting +/// our token, or 413-ing our batches produces NO output at the default log +/// level while startup still says "OTel metrics exporter started". +fn report_export_failure(uri: &http::Uri, detail: &str) { + if !EXPORT_FAILING.swap(true, Ordering::Relaxed) { + tracing::warn!( + %uri, + detail, + "OTel metrics export is failing; metrics are not reaching the collector" + ); + } +} + +/// Report a successful export, logging only the failing -> recovered edge. +fn report_export_success() { + if EXPORT_FAILING.swap(false, Ordering::Relaxed) { + tracing::info!("OTel metrics export recovered"); + } +} + +/// First 256 bytes of an error body — OTLP puts rejection detail there, but a +/// collector may echo back headers (including our bearer token), so it is +/// truncated rather than logged whole. +fn truncated_body(body: &[u8]) -> String { + String::from_utf8_lossy(&body[..body.len().min(256)]).into_owned() +} + #[async_trait::async_trait] impl HttpClient for OtlpHttpClient { async fn send_bytes(&self, mut request: Request) -> Result, HttpError> { @@ -217,21 +298,44 @@ impl HttpClient for OtlpHttpClient { ); } } + let uri = request.uri().clone(); // Hand-rolled send, mirroring opentelemetry-http's blocking impl: // that impl is on reqwest 0.13's client (opentelemetry-http's own // dep), while the workspace is on 0.12, so we can't delegate to it. // Blocking inside async is fine here for the same reason the SDK's // default client is blocking: PeriodicReader exports via block_on on // a dedicated thread. Fold both in when the workspace moves to 0.13. - let request: reqwest::blocking::Request = request.map(|body| body.to_vec()).try_into()?; + // + // Every outcome is reported through `report_export_*`: the SDK logs + // them at DEBUG only, so this is the only place a failing export + // becomes visible to an operator. + let request: reqwest::blocking::Request = request + .map(|body| body.to_vec()) + .try_into() + .inspect_err(|error| { + // Reached when the endpoint has no scheme: reqwest requires an + // absolute URL, while `http::Uri` accepts `host:port` as an + // authority. `init` warns about that at startup too. + report_export_failure(&uri, &format!("invalid request URL: {error}")) + })?; // Deliberately no `error_for_status()`: it discards the body, which is // where OTLP puts rejection detail. Hand the whole response back and - // the SDK turns a non-2xx into an export error itself, logging the - // status, the URL and the body (`HttpClient.StatusError`). - let mut response = self.inner.execute(request)?; + // the SDK turns a non-2xx into an export error itself. + let mut response = self + .inner + .execute(request) + .inspect_err(|error| report_export_failure(&uri, &format!("{error}")))?; let status = response.status(); let headers = std::mem::take(response.headers_mut()); - let mut http_response = Response::builder().status(status).body(response.bytes()?)?; + let body = response + .bytes() + .inspect_err(|error| report_export_failure(&uri, &format!("{error}")))?; + if status.is_success() { + report_export_success(); + } else { + report_export_failure(&uri, &format!("HTTP {status}: {}", truncated_body(&body))); + } + let mut http_response = Response::builder().status(status).body(body)?; *http_response.headers_mut() = headers; Ok(http_response) } @@ -253,11 +357,12 @@ const METER_NAME: &str = "freenet"; /// derived signing key authenticates every export request (see /// [`bearer_token`]). /// -/// Returns the reason it did NOT start, or `None` when the pipeline is -/// installed. Callers ignore it; it exists so a test can assert that `init` -/// consults [`otel_suppression_reason`] and returns before building anything — -/// deleting the check would make an enabled config return `None` under -/// `cfg(test)`, i.e. ship a test network's metrics to a collector. +/// Returns `Some(reason)` when the exporter was SUPPRESSED and `None` +/// otherwise — including when the pipeline failed to build, which is logged +/// but is not a suppression. Callers ignore the value; it exists so a test can +/// assert that `init` consults [`otel_suppression_reason`] and returns before +/// building anything — deleting the check would make an enabled config return +/// `None` under `cfg(test)`, i.e. ship a test network's metrics to a collector. pub(crate) fn init( config: &OtelConfig, keypair: &crate::transport::TransportKeypair, @@ -267,10 +372,21 @@ pub(crate) fn init( cfg!(test), super::telemetry::running_under_cargo_test(), ) { - tracing::debug!(?reason, "OTel metrics exporter not started"); + // Being off by default is unremarkable; being switched ON and then + // suppressed anyway is something the operator has to be told about, + // or the exporter looks enabled and silently ships nothing. + if config.enabled && reason != OtelSuppression::Disabled { + tracing::warn!( + ?reason, + "otel-telemetry-enabled is set but the OTel metrics exporter was suppressed" + ); + } else { + tracing::debug!(?reason, "OTel metrics exporter not started"); + } return Some(reason); } + let env_endpoint = env_endpoint(); let endpoint = resolve_metrics_endpoint( config.endpoint.as_deref(), std::env::var(opentelemetry_otlp::OTEL_EXPORTER_OTLP_METRICS_ENDPOINT) @@ -281,6 +397,29 @@ pub(crate) fn init( .as_deref(), ); + // Log where this node's signed identity is actually going, including when + // an inherited OTEL_* variable overrode the configured endpoint — an + // operator who cannot see that from the logs cannot tell their collector + // was bypassed. + if let (Some(env_endpoint), Some(cfg_endpoint)) = + (env_endpoint.as_deref(), config.endpoint.as_deref()) + { + tracing::warn!( + %env_endpoint, + %cfg_endpoint, + "OTEL_EXPORTER_OTLP_* overrides the configured otel-endpoint" + ); + } + // Validate before building: an endpoint the SDK accepts but reqwest does + // not produces a per-export failure rather than a build error, and an + // unparseable env value is swallowed by the SDK, which then silently + // exports to localhost while the log line below names the operator's URL. + if let Some(effective) = endpoint.as_deref().or(env_endpoint.as_deref()) { + if let Some(problem) = endpoint_problem(effective) { + tracing::warn!(endpoint = effective, problem, "OTLP endpoint is unusable"); + } + } + let (pubkey, fingerprint) = identity_attributes(keypair); let auth_signer = match config.auth_mode { crate::config::OtelAuthMode::Freenet => Some(keypair.auth_token_signer()), @@ -297,23 +436,6 @@ pub(crate) fn init( // shutdown path in `bin/freenet.rs`. global::set_meter_provider(provider); register_metrics(); - // Log where this node's signed identity is actually going, - // including when an inherited OTEL_* variable overrode the - // configured endpoint — an operator who cannot see that from the - // logs cannot tell their collector was bypassed. - let env_endpoint = - std::env::var(opentelemetry_otlp::OTEL_EXPORTER_OTLP_METRICS_ENDPOINT) - .or_else(|_| std::env::var(opentelemetry_otlp::OTEL_EXPORTER_OTLP_ENDPOINT)) - .ok(); - if let (Some(env_endpoint), Some(cfg_endpoint)) = - (env_endpoint.as_deref(), config.endpoint.as_deref()) - { - tracing::warn!( - %env_endpoint, - %cfg_endpoint, - "OTEL_EXPORTER_OTLP_* overrides the configured otel-endpoint" - ); - } tracing::info!( endpoint = endpoint .as_deref() @@ -409,19 +531,46 @@ pub(crate) fn build_provider( /// 10s, in milliseconds). The SDK applies its own resolution only to a client /// it builds itself, and we always supply one, so it has to happen here. fn export_timeout() -> std::time::Duration { - [ + for var in [ opentelemetry_otlp::OTEL_EXPORTER_OTLP_METRICS_TIMEOUT, opentelemetry_otlp::OTEL_EXPORTER_OTLP_TIMEOUT, - ] - .iter() - .find_map(|var| std::env::var(var).ok()?.trim().parse::().ok()) - .map(std::time::Duration::from_millis) - .unwrap_or(opentelemetry_otlp::OTEL_EXPORTER_OTLP_TIMEOUT_DEFAULT) + ] { + let Ok(raw) = std::env::var(var) else { + continue; + }; + let raw = raw.trim(); + if raw.is_empty() { + continue; + } + match raw.parse::() { + Ok(ms) => { + // The spec unit is MILLISECONDS, so `=10` meaning "10 seconds" + // yields a 10ms timeout and every export times out instead. + if ms < 100 { + tracing::warn!( + var, + ms, + "OTLP export timeout is in MILLISECONDS and this value is very small; \ + a seconds value was probably intended" + ); + } + return std::time::Duration::from_millis(ms); + } + // Fall through to the next variable, as the SDK's own `.ok()` + // resolution would — but say so rather than ignoring it silently. + Err(error) => tracing::warn!( + var, + raw, + %error, + "ignoring unparseable OTLP export timeout (expected whole milliseconds)" + ), + } + } + opentelemetry_otlp::OTEL_EXPORTER_OTLP_TIMEOUT_DEFAULT } -/// Resource attribute keys the operator declared through the environment, -/// which must not be overwritten — see the merge note in -/// [`build_provider_blocking`]. +/// Resource attribute keys the operator declared through the environment — +/// see [`resource_attributes`] for which of them win. fn env_declared_resource_keys() -> Vec { let mut keys = Vec::new(); if std::env::var("OTEL_SERVICE_NAME").is_ok_and(|v| !v.trim().is_empty()) { @@ -436,6 +585,53 @@ fn env_declared_resource_keys() -> Vec { keys } +/// The resource attributes to attach, given the keys the operator declared +/// through `OTEL_SERVICE_NAME` / `OTEL_RESOURCE_ATTRIBUTES`. +/// +/// Descriptive attributes defer to the environment: `Resource::builder` seeds +/// from those variables and `with_attribute` merges OVER that seed, so setting +/// a literal unconditionally would silently discard the operator's value — two +/// nodes on one host with distinct `OTEL_SERVICE_NAME`s would both export +/// `service.name=freenet-node`. +/// +/// The two `freenet.node.*` identity attributes are the exception and are +/// ALWAYS emitted. They are not description, they are the collector's proof of +/// which node sent the batch: `freenet.node.pubkey` must equal the bearer +/// token's `` field, which is verified against the signature. Letting +/// `OTEL_RESOURCE_ATTRIBUTES=freenet.node.pubkey=...` shadow them would export +/// an identity that does not match the key everything was signed with, and +/// break that self-validation silently on both sides. +fn resource_attributes( + pubkey: String, + fingerprint: String, + declared: &[String], +) -> Vec<(&'static str, String)> { + let mut attributes = vec![ + ("freenet.node.pubkey", pubkey), + ("freenet.node.fingerprint", fingerprint), + ]; + for (key, _) in &attributes { + if declared.iter().any(|declared| declared == key) { + tracing::warn!( + key, + "the node identity resource attribute cannot be overridden by \ + OTEL_RESOURCE_ATTRIBUTES; the signed value is exported instead" + ); + } + } + attributes.extend( + [ + ("service.name", "freenet-node".to_owned()), + ("service.version", env!("CARGO_PKG_VERSION").to_owned()), + ("os.type", std::env::consts::OS.to_owned()), + ("host.arch", std::env::consts::ARCH.to_owned()), + ] + .into_iter() + .filter(|(key, _)| !declared.iter().any(|declared| declared == key)), + ); + attributes +} + fn build_provider_blocking( endpoint: Option<&str>, pubkey: String, @@ -450,11 +646,19 @@ fn build_provider_blocking( // purpose: PeriodicReader exports off-runtime (see Cargo.toml). The // timeout is the one the SDK would have applied to its own client, which // it does not apply to a supplied one. + // + // A build failure is propagated rather than falling back to + // `Client::new()`: that fallback has NO timeout, so a collector that + // accepts the connection and never answers would block the PeriodicReader + // thread indefinitely and stop all metric collection. + let http_client = reqwest::blocking::Client::builder() + .timeout(export_timeout()) + .build() + .map_err(|error| { + ExporterBuildError::InternalFailure(format!("otel http client build failed: {error}")) + })?; builder = builder.with_http_client(OtlpHttpClient { - inner: reqwest::blocking::Client::builder() - .timeout(export_timeout()) - .build() - .unwrap_or_else(|_| reqwest::blocking::Client::new()), + inner: http_client, signer: auth_signer, pubkey_b58: pubkey.clone(), }); @@ -464,28 +668,11 @@ fn build_provider_blocking( // identifying THIS node here costs nothing per series — unlike a // per-datapoint attribute, which is why no instrument below carries one // identifying the remote end of a connection. - // - // `Resource::builder` seeds from OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES - // and then `with_attribute`/`with_service_name` MERGE OVER that seed, so - // setting a literal unconditionally silently discards the operator's - // value: two nodes on one host with distinct OTEL_SERVICE_NAMEs would both - // export `service.name=freenet-node`. Only fill in what the environment - // did not declare. - let declared = env_declared_resource_keys(); + // Which of these defer to the environment and which do not is + // [`resource_attributes`]'s decision. let mut resource = Resource::builder(); - if !declared.iter().any(|k| k == "service.name") { - resource = resource.with_service_name("freenet-node"); - } - for (key, value) in [ - ("freenet.node.pubkey", pubkey), - ("freenet.node.fingerprint", fingerprint), - ("service.version", env!("CARGO_PKG_VERSION").to_owned()), - ("os.type", std::env::consts::OS.to_owned()), - ("host.arch", std::env::consts::ARCH.to_owned()), - ] { - if !declared.iter().any(|k| k == key) { - resource = resource.with_attribute(KeyValue::new(key, value)); - } + for (key, value) in resource_attributes(pubkey, fingerprint, &env_declared_resource_keys()) { + resource = resource.with_attribute(KeyValue::new(key, value)); } let resource = resource.build(); @@ -1432,12 +1619,13 @@ mod tests { // `PeerId` renders as `{pub_key}@{addr}`, so using it — as the // exporter originally did — leaks our socket address into every // batch and re-identifies the node whenever the address changes. - // Both identity attributes must stay address-free. + // Both identity attributes must stay address-free. Asserts on + // `identity_attributes` — the function production uses — because + // rebuilding the strings here would pass whatever `init` actually + // attaches, including a `PeerId`. let keypair = crate::transport::TransportKeypair::new(); - for instance_id in [ - bs58::encode(keypair.public_key_bytes()).into_string(), - keypair.public().to_string(), - ] { + let (pubkey_attr, fingerprint_attr) = identity_attributes(&keypair); + for instance_id in [pubkey_attr, fingerprint_attr] { assert!(!instance_id.is_empty()); assert!( !instance_id.contains('@') && !instance_id.contains(':'), @@ -1471,6 +1659,71 @@ mod tests { ); } + #[test] + fn env_declared_attributes_cannot_shadow_the_node_identity() { + // The collector verifies the bearer token's signature and then trusts + // `freenet.node.pubkey` as the sender's identity. If + // OTEL_RESOURCE_ATTRIBUTES could override that attribute, a node would + // export an identity that does not match the key it signed with, and + // neither side would notice. + let declared = [ + "freenet.node.pubkey".to_owned(), + "freenet.node.fingerprint".to_owned(), + "service.name".to_owned(), + ]; + let attributes = resource_attributes("REAL-PK".into(), "REAL-FP".into(), &declared); + let value = |key: &str| { + attributes + .iter() + .find(|(k, _)| *k == key) + .map(|(_, v)| v.as_str()) + }; + + assert_eq!( + value("freenet.node.pubkey"), + Some("REAL-PK"), + "the signed pubkey must be exported even when the environment declares it" + ); + assert_eq!(value("freenet.node.fingerprint"), Some("REAL-FP")); + // Descriptive attributes still defer: the operator's own service.name + // has to survive, or two nodes on one host collapse into one series. + assert_eq!( + value("service.name"), + None, + "a declared service.name must be left to the environment" + ); + assert!( + value("service.version").is_some(), + "undeclared descriptive attributes are still filled in" + ); + } + + #[test] + fn unusable_endpoints_are_diagnosed() { + // `http::Uri` accepts `host:port` as an authority with no scheme, so + // the exporter builds and every export then dies converting to a + // reqwest request — the failure is per-export, not at startup. + assert!( + endpoint_problem("collector.example:4318").is_some(), + "a schemeless authority must be reported" + ); + // Not parseable at all: the SDK swallows this and falls back to + // localhost while the startup log names the operator's URL. + assert!(endpoint_problem("collector.example:4318/v1/metrics").is_some()); + assert!(endpoint_problem("not a url").is_some()); + // A non-HTTP scheme parses fine but reqwest cannot send it. + assert!(endpoint_problem("ftp://collector.example:4318/v1/metrics").is_some()); + + assert_eq!( + endpoint_problem("http://collector.example:4318/v1/metrics"), + None + ); + assert_eq!( + endpoint_problem("https://collector.example/v1/metrics"), + None + ); + } + #[tokio::test] async fn provider_builds_with_auth_disabled() { // The default auth mode, and the only path where no signer is diff --git a/docs/design/otel-metrics-exporter.md b/docs/design/otel-metrics-exporter.md index e94d5caa51..d368aa58ee 100644 --- a/docs/design/otel-metrics-exporter.md +++ b/docs/design/otel-metrics-exporter.md @@ -63,7 +63,7 @@ Registered in `tracing/otel.rs::register_metrics`. | `freenet.contract.queue.rejected` | counter | `reason` | `FairQueueStats` | | `freenet.contract.queue.background_shed` | counter | — | `FairQueueStats` | -Everything but the histograms and the four synchronous counters is an +Everything but the two histograms and the three synchronous counters is an observable callback over state that already existed for the local dashboard. Resource attributes: `freenet.node.pubkey` (the full base58 **x25519** transport @@ -77,11 +77,18 @@ which renders as `{pub_key}@{addr}` and would export our socket address. The literals are applied only for keys the operator did **not** declare through `OTEL_SERVICE_NAME` / `OTEL_RESOURCE_ATTRIBUTES`: `ResourceBuilder` seeds from the environment and then merges `with_attribute` over that seed, so setting one -unconditionally would silently discard the operator's value. - -Not instrumented: `TransportMetrics::slowdowns_triggered` is read and reset but -never incremented anywhere in the tree, so it is dead and was left out rather -than exported as a permanent zero. +unconditionally would silently discard the operator's value. This deference +applies to the descriptive attributes only — the two `freenet.node.*` identity +attributes are always emitted, because they are what the collector checks the +bearer-token signature against, and an operator-supplied override would export +an identity that does not match the signing key. + +Not instrumented: `TransportMetrics::slowdowns_triggered` is a period +accumulator that `take_snapshot` zeroes for the legacy telemetry worker, so +observing it as a counter would report a non-monotonic series whenever +`telemetry-enabled` is also on — the same hazard as the transport byte and +packet counters, which is why those are read from the cumulative totals +instead. ## Isolation requirement (hard) @@ -177,7 +184,7 @@ through `OTEL_EXPORTER_OTLP_HEADERS` is never overwritten. This is deliberately *not* what the SDK does by itself. In `opentelemetry-otlp` 0.32, `resolve_http_endpoint` -(`src/exporter/http/mod.rs:719-749`) gives a programmatic `with_endpoint` value +(`src/exporter/http/mod.rs:720-750`) gives a programmatic `with_endpoint` value **priority over both env vars**, and uses it **verbatim** — `build_endpoint_uri` appends `/v1/metrics` only on the env-var path. So to get env-wins precedence the code must: @@ -188,11 +195,26 @@ code must: `otel-endpoint` therefore has no clap `env =` binding — binding it would merge the standard variable into the config layer and invert the precedence. -Every other standard variable — `OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`, -`OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_TIMEOUT`, -`OTEL_EXPORTER_OTLP_COMPRESSION`, `OTEL_METRIC_EXPORT_INTERVAL` (default 60s, -`opentelemetry_sdk/src/metrics/periodic_reader.rs:24-43`) — is read by the SDK. -No code for them. +Note the two endpoint variables are NOT interchangeable. `resolve_http_endpoint` +uses the signal-specific `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` **verbatim** +(`// per signal env var is not modified`); only the generic +`OTEL_EXPORTER_OTLP_ENDPOINT` and the built-in default go through +`build_endpoint_uri`. So the signal-specific variable must carry the full +`/v1/metrics` path and the generic one must not. + +Most other standard variables — `OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`, +`OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_METRIC_EXPORT_INTERVAL` (default 60s, +`opentelemetry_sdk/src/metrics/periodic_reader.rs:24-43`) — are read by the SDK. +No code for them. Two exceptions: + +- `OTEL_EXPORTER_OTLP_TIMEOUT` / `OTEL_EXPORTER_OTLP_METRICS_TIMEOUT` are + resolved by `otel::export_timeout`, not the SDK: the SDK applies its resolved + timeout only to a client it builds itself, and we always supply one. +- `OTEL_EXPORTER_OTLP_COMPRESSION` is **not supported**. The exporter validates + it at build time and hard-errors unless the `gzip-http` / `zstd-http` feature + is enabled, which we deliberately do not enable. Setting it therefore fails + the exporter build and the node runs with no metrics (with a WARN naming the + cause). Enable the matching feature if compression is ever wanted. ## Dependencies @@ -230,8 +252,8 @@ Because a feature array may not name a non-optional dependency, `trace-ot` drops ## Suppression -`otel::init` returns `None` — no provider, no exporter, no global registration — -when any of these hold, mirroring `telemetry_suppression_reason` +`otel::init` returns `Some(reason)` — no provider, no exporter, no global +registration — when any of these hold, mirroring `telemetry_suppression_reason` (`telemetry.rs:660-679`): 1. `!cfg.enabled` @@ -294,7 +316,7 @@ the code carries a `NOTE:` comment naming the ceiling and the upgrade path. ## Wire-up `crates/core/src/node.rs`, in `build_with_flush_handle` beside the -`TelemetryReporter::new` call (`node.rs:754`): +`TelemetryReporter::new` call (`node.rs:815`): ```rust crate::tracing::otel::init(&self.config.otel, &self.key_pair); diff --git a/docs/otel-metrics.md b/docs/otel-metrics.md index 1cb450f0c8..a32e3f9671 100644 --- a/docs/otel-metrics.md +++ b/docs/otel-metrics.md @@ -39,13 +39,36 @@ export, regardless of configuration. | `otel-auth-mode` | `disabled` | `disabled` sends no `Authorization` header; `freenet` sends a signed bearer token (below) | The standard OpenTelemetry environment variables take priority over -`otel-endpoint` and are handled by the SDK: -`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, `OTEL_EXPORTER_OTLP_ENDPOINT`, -`OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_TIMEOUT`, +`otel-endpoint`: `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, +`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, +`OTEL_EXPORTER_OTLP_TIMEOUT`, `OTEL_EXPORTER_OTLP_METRICS_TIMEOUT`, `OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`, `OTEL_METRIC_EXPORT_INTERVAL` (default 60s). With none of them and no `otel-endpoint`, the exporter uses `http://localhost:4318`. +**The two endpoint variables are not interchangeable.** The signal-specific +`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` is used exactly as written, so it must +include the full path: + +```bash +OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://collector:4318/v1/metrics # full path +OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318 # base URL only +``` + +Getting this backwards produces a 404 on every export — and, in `freenet` auth +mode, an audience hash over the wrong path. Include the scheme in either form: +a bare `collector:4318` is accepted as a URL but cannot be sent, and the node +warns about it at startup. + +The timeout variables are in **milliseconds** (`OTEL_EXPORTER_OTLP_TIMEOUT=10` +is 10ms, not 10 seconds, and every export will time out). The node warns on a +suspiciously small value. + +`OTEL_EXPORTER_OTLP_COMPRESSION` is **not supported** — the compression +features are deliberately not compiled in, and setting the variable makes the +exporter fail to start, leaving the node with no metrics at all. The startup +warning names the cause. + When an environment variable overrides a configured `otel-endpoint`, the node logs a warning at startup naming both, and the "OTel metrics exporter started" line always names the endpoint actually in use. @@ -84,8 +107,12 @@ any `user:password@` stripped, and no scheme. So an endpoint of `collector.example:4318/v1/metrics`, which you can reproduce with: ```bash +# base58 has no standard CLI; this uses Python, which is universally available. printf 'collector.example:4318/v1/metrics' \ - | openssl dgst -sha256 -binary | head -c 16 | base58 + | openssl dgst -sha256 -binary | head -c 16 \ + | python3 -c 'import sys;d=sys.stdin.buffer.read();n=int.from_bytes(d,"big");a="123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz";s="" +while n: n,r = divmod(n,58); s = a[r]+s +print("1"*(len(d)-len(d.lstrip(b"\0")))+s)' ``` If your collector rejects tokens with an audience mismatch, compare that value @@ -110,4 +137,6 @@ changes. - `freenet.process.memory.rss` is Linux-only. On macOS and Windows the series is empty; that is expected, not a broken pipeline. - Export failures never affect the node: a collector that is down or rejecting - batches produces a log line and nothing else. + batches produces a `WARN` naming the endpoint and the reason, and nothing + else. The warning is logged once per failing streak, with an `INFO` when + exports recover — not once per 60s interval. From cf6fb39f6aebf1198e3e5eb361562a541cfeab7a Mon Sep 17 00:00:00 2001 From: Michael Graff Date: Sat, 8 Aug 2026 17:09:22 -0500 Subject: [PATCH 17/17] feat(otel): break out hosted contracts by why they are held MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The hosting cache already records why each contract is there (access type, local-client flag, abandonment, subscriber maps) but nothing outside hosting.rs could see it, so a node's hosted count was a single opaque number. Adds HostingReason as a partition over the hosted set and exports count plus state bytes per reason. Partition, not flags: the signals overlap, and an overlapping breakdown makes sum-by-reason lie. Its own provider rather than RingStatsSnapshot — that one runs on every dashboard request and this is an O(hosted) walk under the cache lock. --- crates/core/src/node/network_status.rs | 24 +++ crates/core/src/node/p2p_impl.rs | 9 ++ crates/core/src/ring.rs | 12 +- crates/core/src/ring/hosting.rs | 198 +++++++++++++++++++++++++ crates/core/src/ring/hosting/cache.rs | 11 ++ crates/core/src/tracing/otel.rs | 36 ++++- docs/design/otel-metrics-exporter.md | 32 +++- 7 files changed, 317 insertions(+), 5 deletions(-) diff --git a/crates/core/src/node/network_status.rs b/crates/core/src/node/network_status.rs index bb413e24a8..c12717d64d 100644 --- a/crates/core/src/node/network_status.rs +++ b/crates/core/src/node/network_status.rs @@ -122,6 +122,11 @@ pub(crate) struct OtelMetricsSnapshot { pub connection_attempts: u32, pub ring: RingStatsSnapshot, pub fair_queue: crate::contract::FairQueueStats, + /// Hosted contracts partitioned by why they are held. Deliberately NOT a + /// `RingStatsSnapshot` field: that provider runs on every dashboard HTTP + /// request, and this is an O(hosted) walk under the hosting-cache read + /// lock. Its own provider keeps the cost on the OTel collection cadence. + pub hosting_reasons: crate::ring::HostingReasonStats, } /// Read the scalars the OTel exporter observes, or `None` before the node has @@ -138,13 +143,32 @@ pub(crate) fn otel_metrics_snapshot() -> Option { .read() .as_ref() .map(|provider| provider())?; + let hosting_reasons = HOSTING_REASON_PROVIDER + .read() + .as_ref() + .map(|provider| provider())?; Some(OtelMetricsSnapshot { connection_attempts, ring, fair_queue: crate::contract::fair_queue_stats(), + hosting_reasons, }) } +/// Source of the per-reason hosted-contract breakdown +/// (`Ring::hosted_by_reason`). OTel-only; see [`OtelMetricsSnapshot`]. +pub type HostingReasonProvider = + Arc crate::ring::HostingReasonStats + Send + Sync + 'static>; + +static HOSTING_REASON_PROVIDER: parking_lot::RwLock> = + parking_lot::RwLock::new(None); + +/// Register the hosting-reason data source. Replaces any previously-registered +/// provider. +pub fn set_hosting_reason_provider(provider: HostingReasonProvider) { + *HOSTING_REASON_PROVIDER.write() = Some(provider); +} + static GOVERNANCE_PROVIDER: parking_lot::RwLock> = parking_lot::RwLock::new(None); diff --git a/crates/core/src/node/p2p_impl.rs b/crates/core/src/node/p2p_impl.rs index fdab84c584..7a2804aba2 100644 --- a/crates/core/src/node/p2p_impl.rs +++ b/crates/core/src/node/p2p_impl.rs @@ -524,6 +524,15 @@ impl NodeP2P { hosting_ring.dashboard_hosting_snapshot() })); + // Per-reason hosted-contract breakdown for the OTel exporter only + // (count + state bytes per `HostingReason`). Kept off the ring-stats + // provider above because that one runs on every dashboard HTTP + // request and this is an O(hosted) walk under the cache read lock. + let reason_ring = self.op_manager.ring.clone(); + super::network_status::set_hosting_reason_provider(std::sync::Arc::new(move || { + reason_ring.hosted_by_reason() + })); + // Wire live ring stats for the dashboard: connection count + // hosted contracts + own public key, read on every homepage // request. diff --git a/crates/core/src/ring.rs b/crates/core/src/ring.rs index 9bc4dbce60..715548ec67 100644 --- a/crates/core/src/ring.rs +++ b/crates/core/src/ring.rs @@ -25,8 +25,8 @@ use parking_lot::{Mutex, RwLock}; use tokio_util::sync::CancellationToken; pub use hosting::{ - AddClientSubscriptionResult, AddSubscriberOutcome, ClientDisconnectResult, SubscribeResult, - SubscribedContractSnapshot, + AddClientSubscriptionResult, AddSubscriberOutcome, ClientDisconnectResult, HostingReason, + HostingReasonStats, SubscribeResult, SubscribedContractSnapshot, }; use crate::message::TransactionType; @@ -4189,6 +4189,14 @@ impl Ring { self.hosting_manager.hosting_contracts_count() } + /// The same hosted set as [`Self::hosting_contracts_count`], partitioned by + /// WHY each contract is held, with state bytes per bucket. Backs the + /// `freenet.node.contracts.hosted{,.bytes}` OTel gauges. See + /// [`HostingReason`]. + pub fn hosted_by_reason(&self) -> HostingReasonStats { + self.hosting_manager.hosted_by_reason() + } + /// Number of active network subscription leases this node currently holds. /// /// Together with [`hosting_contracts_count`](Self::hosting_contracts_count) diff --git a/crates/core/src/ring/hosting.rs b/crates/core/src/ring/hosting.rs index c712495b23..d365d65231 100644 --- a/crates/core/src/ring/hosting.rs +++ b/crates/core/src/ring/hosting.rs @@ -202,6 +202,93 @@ pub(crate) enum PhantomRepair { Drop(ContractKey), } +/// Why this node is holding a contract it hosts. +/// +/// This is a PARTITION, not a set of flags: the classifier in +/// [`HostingManager::hosted_by_reason`] evaluates the variants in declaration +/// order and assigns each hosted contract to the FIRST one that matches, so +/// the per-reason counts sum to the hosting-cache size and the per-reason +/// bytes sum to its used bytes. That is the whole point — the underlying +/// signals overlap (a contract can be locally accessed AND have downstream +/// subscribers), and an overlapping breakdown makes `sum by (reason)` lie. +/// +/// Ordering is strongest-claim-first: a reason further down the list only +/// applies when every reason above it is absent. `LocalClient` outranks +/// `Downstream` for the same reason eviction does (`local_and_downstream_counts` +/// — this node's own user beats forwarded demand), and everything outranks +/// `Routed`, which is the residual "no demand signal at all" bucket. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum HostingReason { + /// A local client (WebSocket/HTTP) holds a subscription. This node's own + /// user wants the contract. + LocalClient, + /// A downstream peer subscribes to us for this contract — we are a relay + /// in someone else's update mesh. + Downstream, + /// We hold an unexpired network subscription but nothing local or + /// downstream reads it: hosted on the network's behalf. + Subscribed, + /// No subscription of any kind, but a local client GET/PUT touched it + /// (`local_client_access`). The read-only / PUT-only local-demand class + /// (River UI containers and friends). + LocalAccess, + /// Was in use and no longer is (`abandoned_at`) — the eviction candidate + /// pool. Distinguished from `Routed` because a rising `abandoned` count is + /// churn, while a rising `routed` count is ordinary transit caching. + Abandoned, + /// Residual: arrived through a routed GET/PUT and never acquired any + /// demand signal. + Routed, +} + +impl HostingReason { + /// Every variant, in classifier (and export) order. + pub const ALL: [HostingReason; 6] = [ + HostingReason::LocalClient, + HostingReason::Downstream, + HostingReason::Subscribed, + HostingReason::LocalAccess, + HostingReason::Abandoned, + HostingReason::Routed, + ]; + + /// Stable attribute value. These strings are a metrics contract — a + /// collector-side dashboard filters on them, so renaming one silently + /// empties a panel. Add variants rather than repurposing these. + pub fn as_str(self) -> &'static str { + match self { + HostingReason::LocalClient => "local_client", + HostingReason::Downstream => "downstream", + HostingReason::Subscribed => "subscribed", + HostingReason::LocalAccess => "local_access", + HostingReason::Abandoned => "abandoned", + HostingReason::Routed => "routed", + } + } +} + +/// Hosted-contract count and state bytes per [`HostingReason`], indexed by +/// `reason as usize`. Both arrays partition the hosting cache (see +/// [`HostingReason`]), so `counts.iter().sum()` is the hosted-contract count +/// and `bytes.iter().sum()` is the cache's used bytes. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct HostingReasonStats { + counts: [u64; HostingReason::ALL.len()], + bytes: [u64; HostingReason::ALL.len()], +} + +impl HostingReasonStats { + /// Contracts held for `reason`. + pub fn count(&self, reason: HostingReason) -> u64 { + self.counts[reason as usize] + } + + /// Contract state bytes held for `reason`. + pub fn bytes(&self, reason: HostingReason) -> u64 { + self.bytes[reason as usize] + } +} + /// Result of adding a client subscription. #[derive(Debug)] pub struct AddClientSubscriptionResult { @@ -897,6 +984,40 @@ impl HostingManager { Some(tracker.stats()) } + /// Count and state bytes of hosted contracts, partitioned by WHY each one + /// is held (see [`HostingReason`]). Fixed cardinality — six buckets, no + /// contract identity survives the walk — so it is safe to export as + /// metric attributes. + /// + /// One O(hosted) pass under the hosting-cache read lock. The subscription + /// lookups inside the closure read only the `client_subscriptions` / + /// `downstream_subscribers` / `active_subscriptions` DashMaps, never the + /// hosting cache, so there is no re-lock — the same discipline + /// [`Self::cost_eligibility_stats`] relies on. + pub(crate) fn hosted_by_reason(&self) -> HostingReasonStats { + let mut stats = HostingReasonStats::default(); + self.hosting_cache.read().for_each_reason_row(|key, entry| { + let (local, downstream) = self.local_and_downstream_counts(key); + let reason = if local > 0 { + HostingReason::LocalClient + } else if downstream > 0 { + HostingReason::Downstream + } else if self.is_subscribed(key) { + HostingReason::Subscribed + } else if entry.local_client_access { + HostingReason::LocalAccess + } else if entry.abandoned_at.is_some() { + HostingReason::Abandoned + } else { + HostingReason::Routed + }; + let bucket = reason as usize; + stats.counts[bucket] = stats.counts[bucket].saturating_add(1); + stats.bytes[bucket] = stats.bytes[bucket].saturating_add(entry.size_bytes); + }); + stats + } + pub(crate) fn cost_eligibility_stats( &self, cost_axes: &[CostAxisPressure], @@ -4252,6 +4373,83 @@ mod tests { assert!(used.in_use, "a client subscription is real demand → in_use"); } + /// `hosted_by_reason` must PARTITION the hosting cache: one bucket per + /// contract, counts summing to the cache size and bytes to its used bytes. + /// The classification is priority-ordered, so each case below is set up + /// with every HIGHER-priority signal deliberately absent — a contract with + /// both a local client subscription and downstream subscribers must land in + /// `local_client` only, never be counted twice. + #[test] + fn hosted_by_reason_partitions_the_hosting_cache() { + let manager = HostingManager::new(DEFAULT_HOSTING_BUDGET_BYTES); + + // Empty cache: every bucket zero (a real datapoint, not absence). + let empty = manager.hosted_by_reason(); + for reason in HostingReason::ALL { + assert_eq!(empty.count(reason), 0, "{reason:?} on an empty cache"); + assert_eq!(empty.bytes(reason), 0, "{reason:?} on an empty cache"); + } + + // One contract per reason, distinct sizes so a mis-bucketed contract + // shows up in the bytes assertions too. + let local_client = make_contract_key(1); + let downstream = make_contract_key(2); + let subscribed = make_contract_key(3); + let local_access = make_contract_key(4); + let abandoned = make_contract_key(5); + let routed = make_contract_key(6); + for (key, size) in [ + (local_client, 100), + (downstream, 200), + (subscribed, 400), + (local_access, 800), + (abandoned, 1_600), + (routed, 3_200), + ] { + manager.record_contract_access(key, size, AccessType::Get); + } + + // `local_client` ALSO gets a downstream subscriber and a network + // subscription: priority must keep it in exactly one bucket. + manager.add_client_subscription(local_client.id(), crate::client_events::ClientId::next()); + manager.add_downstream_subscriber(&local_client, make_peer_key(10)); + manager.subscribe(local_client); + + // `downstream` also holds a network subscription — downstream wins. + manager.add_downstream_subscriber(&downstream, make_peer_key(11)); + manager.subscribe(downstream); + + manager.subscribe(subscribed); + manager.mark_local_client_access(&local_access); + + // Abandonment is a transition, not a flag: subscribe a downstream peer + // and take it away again. + manager.add_downstream_subscriber(&abandoned, make_peer_key(12)); + manager.remove_downstream_subscriber(&abandoned, &make_peer_key(12)); + + // `routed` gets nothing beyond the GET that seeded it. + + let stats = manager.hosted_by_reason(); + for (reason, size) in [ + (HostingReason::LocalClient, 100), + (HostingReason::Downstream, 200), + (HostingReason::Subscribed, 400), + (HostingReason::LocalAccess, 800), + (HostingReason::Abandoned, 1_600), + (HostingReason::Routed, 3_200), + ] { + assert_eq!(stats.count(reason), 1, "{reason:?} count"); + assert_eq!(stats.bytes(reason), size, "{reason:?} bytes"); + } + + // The partition property itself — what makes `sum by (reason)` valid. + let total_count: u64 = HostingReason::ALL.iter().map(|r| stats.count(*r)).sum(); + let total_bytes: u64 = HostingReason::ALL.iter().map(|r| stats.bytes(*r)).sum(); + let cache = manager.hosting_cache_stats(); + assert_eq!(total_count, cache.contract_count, "counts must partition"); + assert_eq!(total_bytes, cache.current_bytes, "bytes must partition"); + } + /// `is_eviction_eligible` gates the dashboard's "next to evict" badge on the /// sweep skip filter, which since 2026-07-08 is `!contract_in_use` ONLY (the /// `min_ttl` age gate was dropped — invariant 3). A freshly-accessed, not-in- diff --git a/crates/core/src/ring/hosting/cache.rs b/crates/core/src/ring/hosting/cache.rs index ed9a50c321..c24e2f35e1 100644 --- a/crates/core/src/ring/hosting/cache.rs +++ b/crates/core/src/ring/hosting/cache.rs @@ -1787,6 +1787,17 @@ impl HostingCache { } } + /// Current hosted rows for the fixed-cardinality hosting-REASON telemetry + /// (`HostingManager::hosted_by_reason`). Same contract-identity discipline + /// as [`Self::for_each_cost_eligibility_row`]: keys are visible to the + /// manager's classifier (which needs them to look up subscription state) + /// but are aggregated away before anything is exported. + pub(crate) fn for_each_reason_row(&self, mut visit: impl FnMut(&ContractKey, &HostedContract)) { + for (key, entry) in &self.contracts { + visit(key, entry); + } + } + /// Per-contract rows for the local dashboard, in the cache-side EVICTION /// order: ascending `(recency_seq, key)` — least-recent real GET/PUT first, the /// same order [`Self::keys_eviction_order`] uses. This reflects the order diff --git a/crates/core/src/tracing/otel.rs b/crates/core/src/tracing/otel.rs index 1bc55c31b7..8a4dbb191e 100644 --- a/crates/core/src/tracing/otel.rs +++ b/crates/core/src/tracing/otel.rs @@ -870,6 +870,7 @@ fn register_transport_metrics(meter: &opentelemetry::metrics::Meter) { /// Ring / topology state, mirroring the dashboard's connection-status tiles. fn register_ring_metrics(meter: &opentelemetry::metrics::Meter) { use crate::node::network_status::otel_metrics_snapshot as snapshot; + use crate::ring::HostingReason; let _connections = meter .u64_observable_gauge("freenet.ring.connections") @@ -881,12 +882,43 @@ fn register_ring_metrics(meter: &opentelemetry::metrics::Meter) { }) .build(); + // Both hosted-contract gauges are attributed by `reason` and carry NO + // un-attributed total: emitting both on one instrument would make + // `sum by (reason)` double-count. `HostingReason` partitions the hosted + // set, so the total is `sum(freenet.node.contracts.hosted)`. let _hosted = meter .u64_observable_gauge("freenet.node.contracts.hosted") - .with_description("Contracts currently hosted by this node") + .with_description( + "Contracts currently hosted by this node, partitioned by why each one is held", + ) .with_callback(|observer| { if let Some(s) = snapshot() { - observer.observe(s.ring.hosted_contracts as u64, &[]); + for reason in HostingReason::ALL { + observer.observe( + s.hosting_reasons.count(reason), + &[KeyValue::new("reason", reason.as_str())], + ); + } + } + }) + .build(); + + let _hosted_bytes = meter + .u64_observable_gauge("freenet.node.contracts.hosted.bytes") + .with_unit("By") + .with_description( + "Contract state bytes hosted by this node, partitioned by why each contract is \ + held. State only — WASM code blobs and database overhead are excluded, matching \ + what the hosting cache's byte budget measures.", + ) + .with_callback(|observer| { + if let Some(s) = snapshot() { + for reason in HostingReason::ALL { + observer.observe( + s.hosting_reasons.bytes(reason), + &[KeyValue::new("reason", reason.as_str())], + ); + } } }) .build(); diff --git a/docs/design/otel-metrics-exporter.md b/docs/design/otel-metrics-exporter.md index d368aa58ee..e8e511be45 100644 --- a/docs/design/otel-metrics-exporter.md +++ b/docs/design/otel-metrics-exporter.md @@ -52,7 +52,8 @@ Registered in `tracing/otel.rs::register_metrics`. | `freenet.transport.cwnd` | histogram | — | `record_cwnd_sample` | | `freenet.operation.results` | counter | `op`, `result` | `network_status::record_op_result` | | `freenet.ring.connections` | gauge | — | `RingStatsSnapshot` | -| `freenet.node.contracts.hosted` | gauge | — | `RingStatsSnapshot` | +| `freenet.node.contracts.hosted` | gauge | `reason` | `HostingReasonStats` | +| `freenet.node.contracts.hosted.bytes` | gauge | `reason` | `HostingReasonStats` | | `freenet.connect.attempts` | counter | — | `NetworkStatus` | | `freenet.ring.lattice.neighbor` | gauge | `position` | `RingStatsSnapshot` | | `freenet.ring.lattice.neighbor.distance` | gauge | `position` | `RingStatsSnapshot` | @@ -66,6 +67,35 @@ Registered in `tracing/otel.rs::register_metrics`. Everything but the two histograms and the three synchronous counters is an observable callback over state that already existed for the local dashboard. +### `reason` on the hosted-contract gauges + +`freenet.node.contracts.hosted` and `.hosted.bytes` answer "how much are we +holding, and *why*". The `reason` values come from `ring::HostingReason` and +are a **partition**: the classifier assigns each hosted contract to the first +matching bucket in priority order, so `sum by (reason)` is the hosted-contract +count and the byte gauge sums to the hosting cache's used bytes. Neither gauge +emits an un-attributed total — that would double-count under `sum`. + +| `reason` | held because | +|---|---| +| `local_client` | a local client (WebSocket/HTTP) holds a subscription | +| `downstream` | a downstream peer subscribes to us — we relay its updates | +| `subscribed` | unexpired network subscription, no local or downstream reader | +| `local_access` | no subscription, but a local client GET/PUT touched it | +| `abandoned` | was in use and no longer is — the eviction-candidate pool | +| `routed` | residual: arrived via a routed GET/PUT, no demand signal | + +The strings are a metrics contract — collector-side dashboards filter on them, +so add variants rather than repurpose existing values. + +The breakdown has its own provider (`set_hosting_reason_provider`) rather than +riding `RingStatsSnapshot`: that provider runs on every dashboard HTTP request, +and this is an O(hosted) walk under the hosting-cache read lock. Its own +provider confines the cost to the OTel collection cadence. The bytes gauge +counts contract **state** only — no WASM blobs, no database overhead — matching +what the hosting cache's byte budget measures, so it is comparable against the +budget but not against on-disk usage. + Resource attributes: `freenet.node.pubkey` (the full base58 **x25519** transport public key — byte-equal to the bearer token's `` field, so a collector that verified the signature has also verified the node id),