feat(otel): export node metrics via the OpenTelemetry SDK

[skandragon]

Problem

Node operators have no way to monitor a node over time — memory, ring health, transport throughput, queue depths — beyond point-in-time snapshots. The existing telemetry pipeline (telemetry-enabled / telemetry-endpoint) is a hand-rolled OTLP-JSON log POST that feeds the project's central dashboard; it isn't a general-purpose metrics exporter and operators shouldn't have to piggyback on it.

Approach

Adds a second, fully independent pipeline built on the OpenTelemetry SDK (tracing/otel.rs), gated by otel-telemetry-enabled / otel-endpoint. The two pipelines share no config, no endpoint, and no fallback — enabling one has no effect on the other, and otel-endpoint never defaults to the dashboard collector.

Key decisions:

• Standard OTEL env vars win over config (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME, etc.); without any, exports go to http://localhost:4318.
• Observable instruments for existing state (ring, transport, queue gauges read at collection time — nothing added to the hot path); synchronous instruments (Histogram/Counter) held in a OnceLock so they bind to the real provider, recorded via record_* helpers that cost one atomic load when the exporter is off.
• All histograms are base-2 exponential via a single with_view — no per-instrument bucket boundaries.
• Collector auth (otel-auth-mode, default disabled): in freenet mode, a per-request Authorization: Bearer freenet/<pubkey>/<audience>/<timestamp>/<signature> — an XEdDSA signature over the token prefix, signed with the node's x25519 transport secret. The collector verifies with stock Ed25519 after Montgomery→Edwards conversion; no shared secret, no second key. <audience> is the target collector's host:port, so a token cannot be replayed at a different collector. Default is disabled (no header) so that pointing the exporter at your own collector never ships a signed identity assertion unasked; an Authorization supplied via OTEL_EXPORTER_OTLP_HEADERS is never overwritten in either mode.
• No new native build dependency: the exporter always installs its own HttpClient (workspace reqwest 0.12), so none of opentelemetry-otlp's reqwest-*/TLS features are enabled. Net new packages in Cargo.lock: xeddsa + convert_case.
• No per-connection identifying attributes (series-cost multiplication); the node identifies itself via two resource attributes, freenet.node.pubkey (verifiable against the bearer signature) and freenet.node.fingerprint (dashboard cross-reference). Never a PeerId, which would leak the socket address.

Design doc: docs/design/otel-metrics-exporter.md. Operator docs: docs/otel-metrics.md.

Testing

• Unit tests covering config parsing/precedence (flat keys, env-var priority, endpoint isolation), provider construction inside a tokio runtime, bearer-token format and XEdDSA sign/verify round-trip, suppression when disabled, and resource-attribute derivation.
• cargo test --release -p freenet --lib otel (18 passed), config:: (139 passed), transport:: (694 passed) all green locally.
• Verified end-to-end against a local OTLP collector on :4318.

Note: freenet.process.memory.rss is Linux-only by design; on macOS/Windows the gauge registers but emits no datapoints.

Fixes

Refs #5046 — that issue also asks for log export, which is an explicit non-goal here, so it stays open.

[AI-assisted - Claude]

[github-actions]

---

Rule review against .claude/rules/. WARNING findings block merge.

[sanity]

Review

Thanks for this. The craft here is high, and a few things in it are genuinely careful in ways that are easy to get wrong: the OnceLock reasoning about instruments binding to the no-op provider, the std::thread::scope hop to build the exporter off the async runtime, dodging the clap SetTrue trap on --otel-telemetry-enabled, and adding new cumulative packet counters rather than observing the period accumulators that take_snapshot zeroes. That last one is exactly the trap the AGENTS.md note warns about, and you avoided it deliberately.

The problems below are mostly about packaging, not correctness. The biggest one has a small fix.

Review ran five independent lenses (supply chain / security, cryptography, code-first correctness, test quality, and architecture), plus direct verification of the load-bearing claims against crate sources.

---

Blocking

1. The OTel stack is now unconditional, and it pulls a C build dependency into every build.

crates/core/Cargo.toml moves opentelemetry-otlp and opentelemetry_sdk out of the optional trace-ot feature, so every freenet binary compiles them whether or not the operator ever sets otel-telemetry-enabled. Via reqwest-rustls that pulls rustls-platform-verifier, which pulls aws-lc-rs and aws-lc-sys. aws-lc-sys builds AWS-LC from C and assembly through cc + cmake.

Eleven new lockfile packages, ten of them downstream of that chain, plus a second reqwest major (0.13 alongside the workspace's 0.12) and a second TLS root store.

Suggested fix: put the pipeline behind a default-off cargo feature, or switch reqwest-rustls to reqwest-rustls-webpki-roots, which skips the platform verifier and therefore aws-lc entirely. Worth noting the default auth_mode = "freenet" path uses the workspace reqwest 0.12 (which already has TLS) through FreenetAuthClient, so the 0.13 TLS stack is dead weight in the default configuration.

2. The release build path was never exercised, and it is where this is most likely to break.

.github/workflows/cross-compile.yml explicitly skips PRs ("Skip PRs - only build on main and releases to reduce CI costs"). It builds x86_64-unknown-linux-musl and aarch64-unknown-linux-musl (the arm64 job installs only musl-tools musl-dev), a macOS cross-compile, and Windows MSVC. No workflow in the repo installs cmake or nasm.

So aws-lc-sys can pass every PR check and fail on main, on the release path, after merge. docs/design/otel-metrics-exporter.md lines 246-251 identifies this risk and says to verify with a workflow_dispatch run on the branch before merging. Please do that and post the result.

3. Dual rustls crypto provider.

On main, rustls 0.23.40 resolves with ring only. With this PR it carries both ring and aws-lc-rs. In rustls-0.23.40/src/crypto/mod.rs:265-286, from_crate_features() has one cfg block per provider and each requires not(feature = <the other>), so with both enabled it returns None, and get_default_or_install_from_crate_features() panics with "Could not automatically determine the process-level CryptoProvider". There is no CryptoProvider::install_default() anywhere in crates/.

To be fair about severity: this does not panic today. Every reachable caller passes a provider explicitly (reqwest-0.12.28/src/async_impl/client.rs:770 hardcodes ring, reqwest 0.13 hardcodes aws_lc_rs, sqlx uses tls-rustls-ring). What the PR removes is the property that made it impossible. The bare ClientConfig::builder() path does exist in tokio-tungstenite-0.27.0/src/tls.rs:126 for wss:// connects, and that crate also backs freenet-stdlib's WebApi, so the exposure reaches downstream consumers. This codebase has hit the dual-provider panic before, and CI did not catch it that time either. Fixing (1) removes this.

4. rule-review/warnings is red on the untested OtelAuthMode::Disabled path. That gate blocks merge. (The macOS Service Unit failure was an unrelated rustc SIGSEGV compiling wasmprinter; it passed on re-run.)

---

Functional bugs worth fixing before merge

5. OTEL_SERVICE_NAME is silently ignored, which contradicts the PR description's "Standard OTEL env vars win over config".

Resource::builder() seeds from EnvResourceDetector into self (opentelemetry_sdk-0.32.1/src/resource/mod.rs:62-68). ResourceBuilder::with_attributes then does self.resource.merge(&Resource::new(kvs)), and merge inserts other over self (mod.rs:179-182). So .with_service_name("freenet-node") overwrites the env value. Same shape for service.version, os.type, and host.arch against OTEL_RESOURCE_ATTRIBUTES.

Effect: two nodes on one host with distinct OTEL_SERVICE_NAME values both export service.name=freenet-node, and any dashboard keyed on it merges them. Fix: apply the literal only when the env var is absent.

6. The bearer token clobbers an operator's own Authorization header.

tracing/otel.rs:142 does an unconditional headers_mut().insert(AUTHORIZATION, ...), and the exporter applies OTEL_EXPORTER_OTLP_HEADERS before calling send_bytes. An operator pointing at a hosted collector that requires Authorization: Basic ... gets a 401 on every export, and the only workaround (otel-auth-mode = disabled) is not documented anywhere operator-facing.

This one bites the use case in #5046 directly, since the ask was to export to the operator's own collector. Fix: if !request.headers().contains_key(AUTHORIZATION).

Related: FreenetAuthClient uses reqwest::blocking::Client::new(), where the client it replaces is built with .timeout(resolve_timeout(...)), so OTEL_EXPORTER_OTLP_TIMEOUT is ignored.

7. The bearer token has no audience binding.

Commit 7035ae4a4 removed the nonce, so the signed payload is freenet/<pubkey>/<timestamp> and does not name the collector it is for. A collector you export to can replay your token to any other collector accepting this scheme and impersonate your node, bounded only by clock skew. The default endpoint is plain http, so a passive observer on that path can do the same.

Two thoughts. First, if the auth scheme stays, sign the target endpoint into the payload. Second, and worth considering more seriously: auth_mode defaults to freenet, so an operator pointing at their own collector ships a signed assertion of their node identity there every 60 seconds without asking for it, and there is no collector today that verifies these tokens. Defaulting to disabled and letting OTEL_EXPORTER_OTLP_HEADERS carry auth would cover #5046's actual request. The XEdDSA scheme could then land separately, alongside the collector that consumes it.

To be clear, the cryptography itself checks out. I read xeddsa 1.1.0's source: it clamps the scalar, forces the Edwards sign bit to zero by negating when needed, and derives the nonce as H(padding || a || M || Z), so even a failing RNG cannot leak the key. auth_token_signer() passes the x25519 secret correctly, and the interop test verifies with genuinely independent curve25519-dalek + ed25519-dalek rather than the code that signed. The concern is protocol design and defaults, not the primitive.

8. AGENTS.md describes the wrong key type. It says freenet.node.pubkey is "the base58 ed25519 verifying key derived from the transport keypair". The code emits the x25519 Montgomery public key (transport/crypto.rs:117-119). Those are different byte strings, and a collector implementer following that line would call VerifyingKey::from_bytes on it and fail. The module rustdoc in tracing/otel.rs gets it right. Same paragraph calls freenet.node.fingerprint "unverifiable by the collector", while otel.rs and its test say the collector recomputes it.

9. The startup log hides the destination in the one case it matters. When the endpoint comes from OTEL_EXPORTER_OTLP_*, the log prints the literal <resolved by OTEL_* env or SDK default> instead of the URL. Env-over-config is the documented and intended precedence, so that part is fine, but an operator whose config endpoint was overridden by an inherited env var cannot tell from the logs where their node's signed identity is being sent. Logging the resolved value, and warning when env overrides a config endpoint, is cheap.

---

Tests

Several of the new tests cannot fail for the reason they name:

• Deleting the entire suppression block in init() leaves every test passing. otel_suppression_reason is well covered but init() is its only caller and is unreachable under cfg(test), so a --id test network would ship to a collector with no test going red. A source-scrape pin that init calls it and returns before build_provider would close this.
• The identity-attribute guards build their expected strings in the test body and never call init or build_provider. Changing init to use peer_id.to_string(), the exact regression the comment names, leaves them green. Extracting an identity_attributes(keypair) helper and asserting on its output would fix it.
• OtelConfig is not destructured in the config round-trip guard, unlike TelemetryConfig and its other siblings. Per .claude/rules/code-style.md the guard's value is that a new field fails to compile until classified; a fourth OtelConfig field will compile clean and silently never merge. This is the fix(config): merge allowed-source-cidrs and allowed-host from config file #3890 / fix: service start silently drops ExecStart CLI flags #4275 class.
• DEFAULT_OTEL_ENDPOINT has no production reader; its only consumers are the two assertions in the test that pins it.
• The eight new record_* mirrors in transport/metrics.rs have no pin. Delete any one and the counter reports zero forever with nothing failing. This is verbatim the "manually-mirrored telemetry counters" row in .claude/rules/bug-prevention-patterns.md (refactor(dashboard): read subscribed contracts from canonical ring state #4009 / Dashboard op_stats counters for SUBSCRIBE and UPDATE silently stuck #4010 precedents).
• --otel-telemetry-enabled=false cannot override a config-file true (config.rs:922 is one-directional, and clap's default_value = "false" makes "unset" and "explicitly false" indistinguishable). The test that looks like it covers this asserts at parse level, before the merge. Making the field Option<bool> with get_or_insert, matching endpoint and auth_mode, would fix both. I realize this matches the existing reference-ping / iface-tx pattern, so it is not a regression, but "the off switch does not work" reads differently for a network export than for a ping toggle.

---

Smaller

• docs/superpowers/ is a new directory whose only content is an 869-line agent work plan, already stale against what shipped (it describes a three-field OtelConfig; the shipped struct has four). docs/design/otel-metrics-exporter.md links to it as the canonical plan, so a reader gets pointed at the stale copy. Suggest deleting it.
• docs/design/otel-metrics-exporter.md still says "Status: proposed" and disagrees with the code in several places: it documents otel::init(&self.config.otel, self.local_peer_id_string()) where the code passes &self.key_pair, documents service.instance.id and peer.id attributes that are never set, and its configuration table omits otel-auth-mode entirely.
• The ponytail: comment convention is defined only in this PR's own docs. Future readers of crates/core/ will not know what it means. A plain NOTE: with the same text, or a tracked issue reference, would carry better.
• "Closes feat(telemetry): actual Otel metrics export #5046" is a little strong given log export is an explicit non-goal; the issue asks for "logs for the Freenet server as well as metrics". Maybe leave it open.
• No operator-facing docs for otel-telemetry-enabled / otel-endpoint / otel-auth-mode. The requester is a node operator, and these currently appear only in AGENTS.md and docs/design/. Compare docs/secrets-at-rest.md.
• A poisoned NETWORK_STATUS lock takes down ten of the nineteen instruments permanently, including the fair-queue stats, which are pure atomics and do not need that lock at all.
• freenet.contract.queue.depth emits queue="total" alongside the three tiers, so sum by (queue) double-counts.
• error_for_status() discards the collector's response body, which is where OTLP partial-success and error detail live.
• AGENTS.md gains 72 lines for an off-by-default exporter. That file is loaded into every agent session in this repo. The genuinely rule-shaped parts (do not add per-instrument bucket boundaries, never export a PeerId, read cumulative rather than period counters) are about eight lines plus a link to the design doc.

---

Checked and clean

Recording these so nobody re-runs them: no CI, workflow, build script, or .claude/rules changes. The transport secret is never logged or serialized (the manual Debug impl on FreenetAuthClient emits nothing), and no attacker-influenced bytes reach sign(). Every exported attribute is either a node-derived constant or a low-cardinality literal; no PeerId, socket address, IP, or contract key appears anywhere, and the Resource builder attaches no hostname or process detector. No deadlock or hot-path stall from the observable callbacks: fair_queue_stats() and the transport counters are lock-free atomic loads, and otel_metrics_snapshot() takes only short read locks that already have the homepage handler as a caller. No task or thread leak from the missing shutdown hook, and it cannot block process exit. Endpoint precedence matches resolve_http_endpoint exactly, including the /v1/metrics asymmetry. Isolation from the existing dashboard pipeline is structurally enforced rather than asserted.

---

Suggested path

Items 1 through 4 are what I would want resolved before merge. Feature-gating (1) default-off takes care of (3) as a side effect and shrinks (2) to a formality. Items 5, 6, and 8 are small and worth doing in the same pass. Item 7 is the one real design question, and splitting the auth scheme into its own change would let the metrics exporter land sooner.

Happy to re-review once those are in.

[AI-assisted - Claude]

[skandragon]

Thanks — most of this landed. Pushed as faa6a1e8.

1–3 (deps, release build, dual crypto provider): fixed, but not the suggested way

The reqwest-rustls → reqwest-rustls-webpki-roots swap does not drop aws-lc:

opentelemetry-http/reqwest-rustls-webpki-roots = ["reqwest/default-tls", "reqwest/webpki-roots"]
reqwest 0.13: default-tls = ["rustls"]; rustls = ["__rustls-aws-lc-rs", "dep:rustls-platform-verifier", …]

It adds webpki roots on top of the same stack.

What actually removes it: FreenetAuthClient (now OtlpHttpClient) is installed in every auth mode, not just when a signer exists. opentelemetry-otlp builds a client only when none was supplied (exporter/http/mod.rs:223-267), so its reqwest-blocking-client / reqwest-rustls features are now off entirely. Our client rides the workspace reqwest 0.12, which already has rustls-tls, so https:// endpoints still work.

Cargo.lock delta versus main went from 11 new packages to 2 — xeddsa and its convert_case. No reqwest 0.13, no rustls-platform-verifier, no aws-lc-rs/aws-lc-sys, no cmake, and rustls resolves with ring alone again. That also means (2) no longer has a native-toolchain risk to verify, and (3) is structurally back to impossible.

I preferred this to a default-off cargo feature: feature-gating would mean an operator can't enable metrics without building their own binary, which defeats the point of the issue.

4 (rule-review warning): fixed — provider_builds_with_auth_disabled.

5, 6, 8, 9: fixed

• Resource literals are applied only for keys OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES did not declare.
• Authorization is set only when absent, so OTEL_EXPORTER_OTLP_HEADERS wins. Also honors OTEL_EXPORTER_OTLP_{METRICS_,}TIMEOUT (the SDK only resolves that for a client it builds itself), and no longer calls error_for_status(), so the body survives for the SDK's HttpClient.StatusError log.
• AGENTS.md corrected (x25519, not ed25519) and cut from +72 lines to +29: the rule-shaped parts plus links.
• Startup log names the resolved endpoint and warns when an OTEL_* variable overrode a configured one.

7 (audience binding + default): both done

Default otel-auth-mode is now disabled. Tokens are freenet/<pubkey>/<audience>/<timestamp>/<signature> where <audience> is the target collector's host:port, taken from the request URI — a token sent to one collector no longer verifies at another (a_token_for_one_collector_does_not_verify_at_another).

One correction on the premise, though: a collector that verifies these tokens does exist and works. It's an OTel collector extension — https://github.com/cardinalhq/cardinalhq-otel-collector/tree/main/extension/fbnauthextension — which is what the scheme was built against, and it verified end-to-end before this PR opened. I'll update it to match whatever format we settle on here, so treat the wire format as ours to choose. That's also why I'd rather keep the scheme in this PR than split it out.

Tests

Fixed the vacuous ones you named:

• init now returns its suppression reason, so init_refuses_to_start_from_a_test_process asserts on init's own behavior rather than on otel_suppression_reason in isolation. Mutation-tested: deleting the suppression block makes it fail.
• Identity attributes come from an identity_attributes(keypair) helper that production calls; the guards assert on its output.
• OtelConfig is destructured in the config round-trip guard.
• Cross-file scrape pins all nine hot-path record_* mirrors (cross-file, so it can't be satisfied by its own literal). Mutation-tested: deleting one fails.
• DEFAULT_OTEL_ENDPOINT deleted — you're right that it had no production reader.
• otel-telemetry-enabled is Option<bool> now, and otel_cli_false_overrides_a_config_file_that_says_true asserts through build(), not at parse level.

Smaller

docs/superpowers/ deleted. Design doc updated to match what shipped (status, auth-mode row, wire-up signature, dependency section). ponytail: → NOTE:. queue="total" dropped. Closes #5046 → Refs, since log export is a non-goal. New operator page at docs/otel-metrics.md.

Not changed: the poisoned-NETWORK_STATUS-lock point is fair but it's a pre-existing property of otel_metrics_snapshot's callers, and splitting the fair-queue reads out of it is a separate change.