feat: correlate slow requests with symbolised stack frames - #505
Merged
Merged
Conversation
Signed-off-by: Ghassan Malke <ghassan+github@malke.nl>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stack capture, symbolisation and
--profilingwere each broken in a way that produced no error, so the profiling correlation section rendered raw hex addresses.Changes
Stack capture
BPF_F_USER_STACKwas defined as8in thebpf/common.hfallback, the shift amount instead of the value.8isBPF_F_REUSE_STACKID, sobpf_get_stackwalked the kernel stack and every captured "user" stack was kernel text. The#ifndefguard cannot catch this: it tests for a macro andvmlinux.hsupplies an enum. Now256, pinned by a_Static_assert.Profiling wiring
--preresolved-podref carried no pod IP, so the spawned pod found no target, skipped pprof discovery, and the entire profiling path including the correlator never ran. The ref now carries the IP as an optional fifth field; four-field refs from an older binary still parse.Pid namespace
bpf_get_current_pid_tgid(), which returns the init-namespace pid, while/host/procshows the node's.agent_ns_tgid()is__noinline: inlined at that many sites the object grew from 2.36MB to 3.09MB and approached the verifier's complexity limit.cgroup_skbprograms, which have no current task, and BPF-internal map keys, which only need to be self-consistent.nghttp3needed both: itsrec->pidis a reported pid and half of a correlation key that crosses into userspace, while the other half stayed raw.Symbolisation
/proc/<pid>/exeresolves inside the target's mount namespace, so it is reached through/proc/<pid>/root/..., the same route uprobe attachment takes.addr2line. Symbols now come from the binary itself, Go.gopclntabfirst, then ELF.symtab, cached per executable, withaddr2linekept as a workstation fallback.Correlator
addr2linethemselves.sched_switchthe top frames are the scheduler path and the application frame sits below them.