docs(tracing): complete architecture, pipeline tuning, and operational runbooks for OTel ClickHouse

[blue4209211]

Summary

This PR overhauls the OTel ClickHouse tracing documentation (clickhouse-tracing.md), expanding the initial 55-line stub into a comprehensive, code-grounded architectural and operational manual grounded in k8s-agent and node-agent.

Key Improvements:

• End-to-End Pipeline Architecture: Maps the 4-tier pipeline (node-agent eBPF probes + application OTel SDKs $\rightarrow$ opentelemetry-collector :4317/:4318 $\rightarrow$ clickhouse :9000/:8123 $\rightarrow$ runner).
• Collector Processing Pipeline Tuning: Details the strict ordering of processors (memory_limiter, filter/drop_namespaces, filter/drop_health_check, probabilistic_sampler, batch), highlighting Helm list override gotchas and memory backpressure mechanisms.
• ClickHouse Storage & Schema: Documents the otel_traces MergeTree table schema, TTL retention (168h), internal log suppression (extraOverrides dropping query_log/part_log), and the 13 materialized columns managed by runner.EnsureMaterializedColumns for fast-path queries.
• Production Troubleshooting Runbooks:
  • Collector span drops and OOMKilled diagnostics via internal metrics (otelcol_processor_dropped_spans_total).
  • ClickHouse "Too Many Parts" error (code 252) root cause and batching remediation.
  • Disk full diagnostics, TTL reduction, partition optimization, and online PVC expansion.
  • HTTP 401 Unauthorized / secret synchronization troubleshooting.
  • End-to-end trace flow verification queries using clickhouse-client.
• Safe Resource Reclamation: Clarifies the safe disabling sequence for ClickHouse and the collector while explicitly preserving nodeAgent.enabled: true for essential kernel network metrics.

Verification:

• Verified against k8s-agent/charts/nudgebee-agent/values.yaml, templates/_helpers.tpl, and runner/pkg/clickhouse/.
• Validated with clean local build (npm run build) in doc-server with 0 broken links or anchors.

[gemini-code-assist]

Code Review

This pull request significantly expands the documentation for OTel ClickHouse Tracing, adding a comprehensive pipeline architecture diagram, component references, tuning invariants, schema details, and troubleshooting runbooks. The review feedback highlights two important corrections: updating the OTel Collector namespace filter to correctly target resource-level attributes (resource.attributes instead of attributes) and fixing a naming inconsistency in the Mermaid diagram regarding the ClickHouseStatus telemetry field.

[gemini-code-assist]

In OpenTelemetry, k8s.namespace.name is a resource-level attribute rather than a span-level attribute. To correctly filter out spans from the kube-system namespace, you should reference it via resource.attributes instead of attributes.

Suggested change

	- attributes["k8s.namespace.name"] == "kube-system"
	- resource.attributes["k8s.namespace.name"] == "kube-system"

[gemini-code-assist]

There is an inconsistency in the heartbeat telemetry field name. Line 274 documents the key as ClickHouseStatus, but the Mermaid diagram here refers to it as clickhouse_status. Update this to ClickHouseStatus for consistency with the actual telemetry payload.

Suggested change

	RUN["runner Deployment<br/>• Initializes Materialized Columns<br/>• Executes analytical SQL via HTTP :8123<br/>• Heartbeat telemetry reports clickhouse_status"]
	RUN["runner Deployment<br/>• Initializes Materialized Columns<br/>• Executes analytical SQL via HTTP :8123<br/>• Heartbeat telemetry reports ClickHouseStatus"]

References

1. API documentation must match the implementation, including any typos. Do not correct typos in documentation if they exist in the code, as this can break user copy-paste workflows.