SK-289 // feat: scrape cadvisor metrics

[drmorr0]

Description and Rationale

• Scrape cadvisor metrics in the tracer

How

• follow the same pattern as dyn_obj_watcher/pod_watcher; the trace manager starts up the metrics collector, sets up a channel for communication, and then forwards on the relevant metrics for processing by the store
• Each node in the cluster gets its own individual scraper which is started as a separate async task. When the node goes away, the collector drops the task, which (thanks to the AbortOnDropHandler) automatically cleans up the scraper. The MetricsScraper struct is supposedly generic-enough to handle other custom metrics in the future.
• tracer needs some additional RBAC perms to see the node-level metrics

Test Steps

• make test, added a bunch of new unit tests for the new logic
• manual testing, confirmed that the trace store is receiving the metrics data and printing it out.

Other Notes

• Metrics endpoints are secured over TLS which means we need to add a TLS feature to reqwest; using default-tls or rustls pulls in the new aws-lc-rs backend, whereas some other stuff we're using depends on the ring backend, and then it conflicts and panics. I think when we update all our dependencies maybe this problem will go away, but for right now I'm having to set the -no-provider feature for reqwest and then (in our tests) manually set the ring backend which is annoying.

• the danger_accept_invalid_certs thing is because kubelet uses a (different) self-signed cert for each node, so we just have to live with it. It's fine, this is what prometheus-collector does too, what could possibly go wrong.

• fixed a random bug in the makefile to allow commands like ARTIFACTS=sk-tracer make to actually work.

• I certify that this PR does not contain any code that has been generated with GitHub Copilot or any other AI-based code generation tool, in accordance with this project's policies.

[drmorr0]

Kubernetes Object DAG

%%{init: {'themeVariables': {'mainBkg': '#ddd'}}}%%
graph LR

classDef default color:#000
subgraph global
  direction LR
  global/simkube[<b>Namespace</b><br>simkube]
%% DELETED OBJECTS START
%% DELETED OBJECTS END
end

subgraph sk-tracer
  direction LR
  simkube/sk-tracer-svc[<b>Service</b><br>sk-tracer-svc]
  simkube/sk-tracer-depl[<b>Deployment</b><br>sk-tracer-depl]
  simkube/sk-tracer-sa[<b>ServiceAccount</b><br>sk-tracer-sa]
  sk-tracer/sk-tracer-crb[<b>ClusterRoleBinding</b><br>sk-tracer-crb]
  simkube/sk-tracer-tracer-config[<b>ConfigMap</b><br>sk-tracer-tracer-config]
  simkube/sk-tracer-sa--->simkube/sk-tracer-depl
  sk-tracer/sk-tracer-crb--->simkube/sk-tracer-depl
  simkube/sk-tracer-tracer-config--->simkube/sk-tracer-depl
%% DELETED OBJECTS START
%% DELETED OBJECTS END
end

subgraph sk-ctrl
  direction LR
  simkube/sk-ctrl-depl[<b>Deployment</b><br>sk-ctrl-depl]
  simkube/sk-ctrl-sa[<b>ServiceAccount</b><br>sk-ctrl-sa]
  sk-ctrl/sk-ctrl-crb[<b>ClusterRoleBinding</b><br>sk-ctrl-crb]
  simkube/sk-ctrl-sa--->simkube/sk-ctrl-depl
  sk-ctrl/sk-ctrl-crb--->simkube/sk-ctrl-depl
%% DELETED OBJECTS START
%% DELETED OBJECTS END
end

global--->sk-tracer
global--->sk-ctrl

%% STYLE DEFINITIONS START
%% STYLE DEFINITIONS END

Loading

New object
Deleted object
Updated object
Updated object (causes pod recreation)

Detailed Diff

[linear-code]

SK-289

SK-290

SK-291

[codecov]

Codecov Report

❌ Patch coverage is 83.76623% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.53%. Comparing base (9c88229) to head (712dbcb).

Files with missing lines	Patch %	Lines
sk-tracer/src/metrics/collector.rs	89.61%	8 Missing ⚠️
sk-tracer/src/metrics/scraper.rs	82.22%	8 Missing ⚠️
sk-tracer/src/manager.rs	68.42%	6 Missing ⚠️
sk-tracer/src/store.rs	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #273      +/-   ##
==========================================
+ Coverage   79.38%   79.53%   +0.14%     
==========================================
  Files          62       65       +3     
  Lines        4021     4172     +151     
==========================================
+ Hits         3192     3318     +126     
- Misses        829      854      +25     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.