Skip to content

feat: correlate slow requests with symbolised stack frames - #505

Merged
gma1k merged 1 commit into
mainfrom
feat/correlate-slow-request-stacks
Sep 14, 2026
Merged

gma1k merged 1 commit into
mainfrom
feat/correlate-slow-request-stacks

Conversation

@gma1k

@gma1k gma1k commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Summary

Stack capture, symbolisation and --profiling were each broken in a way that produced no error, so the profiling correlation section rendered raw hex addresses.

Changes

Stack capture

  • BPF_F_USER_STACK was defined as 8 in the bpf/common.h fallback, the shift amount instead of the value. 8 is BPF_F_REUSE_STACKID, so bpf_get_stack walked the kernel stack and every captured "user" stack was kernel text. The #ifndef guard cannot catch this: it tests for a macro and vmlinux.h supplies an enum. Now 256, pinned by a _Static_assert.

Profiling wiring

  • The --preresolved-pod ref carried no pod IP, so the spawned pod found no target, skipped pprof discovery, and the entire profiling path including the correlator never ran. The ref now carries the IP as an optional fifth field; four-field refs from an older binary still parse.

Pid namespace

  • Events were stamped with bpf_get_current_pid_tgid(), which returns the init-namespace pid, while /host/proc shows the node's.
  • agent_ns_tgid() is __noinline: inlined at that many sites the object grew from 2.36MB to 3.09MB and approached the verifier's complexity limit.
  • Left raw deliberately: cgroup_skb programs, which have no current task, and BPF-internal map keys, which only need to be self-consistent.
  • nghttp3 needed both: its rec->pid is a reported pid and half of a correlation key that crosses into userspace, while the other half stayed raw.

Symbolisation

  • /proc/<pid>/exe resolves inside the target's mount namespace, so it is reached through /proc/<pid>/root/..., the same route uprobe attachment takes.
  • The runtime image ships no addr2line. Symbols now come from the binary itself, Go .gopclntab first, then ELF .symtab, cached per executable, with addr2line kept as a workstation fallback.

Correlator

  • Frames are symbolised instead of printed as hex with a note telling the reader to run addr2line themselves.
  • Addresses inside one function merge, because several addresses in one function are one hot function.
  • Ranking happens before symbolisation, so resolution cost is bounded by the number of frames reported rather than the number of distinct addresses captured.
  • Stack depth 3 to 16: at sched_switch the top frames are the scheduler path and the application frame sits below them.
  • Frame ordering is deterministic; ties break on pid then address rather than map iteration order.

Signed-off-by: Ghassan Malke <ghassan+github@malke.nl>
@github-actions github-actions Bot added feat conventional-commit type area/bpf labels Sep 14, 2026
@gma1k
gma1k merged commit 613d490 into main Sep 14, 2026
28 checks passed
@gma1k
gma1k deleted the feat/correlate-slow-request-stacks branch September 14, 2026 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/bpf feat conventional-commit type

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant