fix(profiler): make the JVM and Python profilers actually produce a profile - #128
Merged
Merged
Conversation
Two independent reasons Java flamegraphs could never work. The build: async-profiler 2.9's Makefile compiles its Java helper with a hardcoded `-source 7`, which javac rejects from JDK 20 onwards, so `make` died on its first target under the JDK 26 builder. A trailing `;` swallowed that failure and `COPY` happily copied the resulting empty build/ into the image, so every published tag since 2026-05-14 shipped no libasyncProfiler.so, no jattach and no fdtransfer - profiles failed at runtime with "/tmp/async-profiler/build/fdtransfer: not found". Pin the builder to JDK 17, drop the `native` target (upstream's release-packaging step, which expects prebuilt tarballs in the tree and always failed here), and assert the three artefacts exist before shipping a layer. The library: libasyncProfiler.so is dlopen'd by the *target* JVM, not by anything in our image, so a musl build fails on every glibc JVM with "libc.musl-x86_64.so.1: cannot open shared object file". Build the library twice and ship both; the agent now picks the one matching the target's libc. Also: py-spy 0.4.0 cannot read CPython 3.14 and reports it as "No python processes found in process <pid>" - bump to 0.4.2. And that stage no longer builds at all, because Debian 11 is past EOL and its security suite has moved to archive; move it to bookworm. Finally, build the images in code-verify. They were only ever built by release.yml, after merge, which is how a broken image shipped for two months.
There was a problem hiding this comment.
Code Review
This pull request updates the JVM and Python Dockerfiles to fix build failures and support a wider range of target environments. Specifically, it pins the JVM builder to JDK 17 to avoid compilation errors, builds both glibc and musl versions of libasyncProfiler.so, and dynamically swaps them at runtime based on the target container's libc. It also updates the Python base image to Debian Bookworm and upgrades py-spy to version 0.4.2. Feedback on the changes highlights a potential false positive in targetUsesMusl when targetFs is empty, which could cause a glibc-based target to be incorrectly identified as musl-based, and suggests adding a defensive check.
RamanKharchee
previously approved these changes
Sep 18, 2026
An empty targetFs made the loader glob relative to our own working directory — and this image is alpine, so every target would have been reported as musl-based.
RamanKharchee
previously approved these changes
Sep 18, 2026
mayankpande88
enabled auto-merge (squash)
September 18, 2026 08:37
Two defects in the austin path, both verified on the dev cluster. Python memory profiles came back as a zero-byte artefact reported as a success. austin writes its samples to the -o file, but the non-flamegraph branch then overwrote that file with the command's stdout - which is empty, because austin prints nothing on stdout when -o is given. Stop overwriting it, and fail loudly when austin genuinely collected nothing rather than publishing an empty file. austin 3.7.0 cannot read CPython 3.14; it reports the target as "not a Python process", the same class of gap py-spy 0.4.0 had. Bump to austin 4, which reads 3.14 but only writes the binary MOJO format, so convert it back with mojo2austin to the text format the raw artefact and flamegraph.pl are built around. austin 4 also ends every exposure-limited run by returning -SIGINT (254), which is not a failure.
blue4209211
approved these changes
Sep 18, 2026
4 tasks
mayankpande88
added a commit
to nudgebee/k8s-agent
that referenced
this pull request
Sep 18, 2026
The pinned 4f4b455 ships an empty async-profiler/build (no libasyncProfiler.so, no jattach, no fdtransfer), a py-spy that cannot read CPython 3.14, and an austin that returns an empty artefact. ca8a3b3 is the first tag with nudgebee/application-profiler#128, which fixes all of those and adds the glibc build of libasyncProfiler.so the agent now selects for non-musl targets.
mayankpande88
added a commit
to nudgebee/k8s-agent
that referenced
this pull request
Sep 18, 2026
) The pinned 4f4b455 ships an empty async-profiler/build (no libasyncProfiler.so, no jattach, no fdtransfer), a py-spy that cannot read CPython 3.14, and an austin that returns an empty artefact. ca8a3b3 is the first tag with nudgebee/application-profiler#128, which fixes all of those and adds the glibc build of libasyncProfiler.so the agent now selects for non-musl targets.
This was referenced Sep 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Five defects, all verified by running the images rather than by reading them. Java flamegraphs and Python memory profiles have never produced a usable artefact.
1. The JVM image shipped an empty
async-profiler/build/. async-profiler 2.9's Makefile compiles its Java helper with a hardcoded-source 7, which javac rejects from JDK 20 onwards, somakedied on its first target under the JDK 26 builder. The trailing;swallowed that, andCOPYof a directory is happy to copy nothing — so every tag published since 2026-05-14 (including4f4b455, which every environment runs) contains nolibasyncProfiler.so, nojattachand nofdtransfer. Profiles fail with/tmp/async-profiler/build/fdtransfer: not found.2. Even with the files present, the library can't load.
libasyncProfiler.sois dlopen'd by the target JVM, not by anything in our image, so the musl build fails on every glibc JVM (temurin, corretto, distroless) withlibc.musl-x86_64.so.1: cannot open shared object file. The library is now built twice and both are shipped; the agent picks the one matching the target's libc.jattachandfdtransferstay musl — those execute inside our alpine image.3. Python memory profiles came back empty — and said they succeeded. austin writes its samples to the
-ofile, then the non-flamegraph branch overwrote that file with the command's stdout, which is empty because austin prints nothing on stdout when-ois given. Every such profile published a zero-byte artefact. Fixed, plus an explicit failure when austin genuinely collected nothing.4. py-spy and austin could not read CPython 3.14, both reporting it as if the target weren't Python. py-spy 0.4.0 → 0.4.2; austin 3.7.0 → 4.0.0. austin 4 only writes the binary MOJO format, so
mojo2austinconverts it back to the text format the raw artefact andflamegraph.plare built around, and its-SIGINT(254) exit — which ends every exposure-limited run — is no longer treated as a failure. The Python stage also no longer built at all: Debian 11 is past EOL and its security suite has moved to archive, soapt-get install git404s; moved to bookworm.5. The Dockerfiles were only ever built by
release.yml, after merge. That is why a broken image shipped for two months.code-verifynow builds all five on every PR.Type of change
Test plan
Java — attach to a real glibc JVM (eclipse-temurin 21) with the rebuilt image:
The same attach against the published
4f4b455image fails withlibc.musl-x86_64.so.1: cannot open shared object file, and before that withfdtransfer: not found.Python 3.14 — the rebuilt image, end to end:
On the published image: py-spy 0.4.0 →
Failed to find python version from target process; austin 3.7.0 →It looks like you are trying to profile a process that is not a Python process.The empty-artefact bug, on the dev cluster (
agent_taskresponse for a python/memory profile of a Python 3.11 pod) —contents_base64decodes to a gzip of 0 bytes, with"success": true.go build ./...andgo test ./...pass. New unit tests cover musl/glibc target detection, the library swap, the MOJO conversion, and austin's signal exit. (TestAsyncProfiler_CleanUpfails on macOS before and after this change — pre-existing, passes on Linux CI.)Checklist
go build ./...andgo test ./...pass locallyRelated issues
Companion fix in the agent: nudgebee/k8s-agent#619