Skip to content

fix(profiler): make the JVM and Python profilers actually produce a profile - #128

Merged
mayankpande88 merged 3 commits into
mainfrom
fix/jvm-image-empty-async-profiler-build
Sep 18, 2026
Merged

mayankpande88 merged 3 commits into
mainfrom
fix/jvm-image-empty-async-profiler-build

Conversation

@mayankpande88

@mayankpande88 mayankpande88 commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Five defects, all verified by running the images rather than by reading them. Java flamegraphs and Python memory profiles have never produced a usable artefact.

1. The JVM image shipped an empty async-profiler/build/. async-profiler 2.9's Makefile compiles its Java helper with a hardcoded -source 7, which javac rejects from JDK 20 onwards, so make died on its first target under the JDK 26 builder. The trailing ; swallowed that, and COPY of a directory is happy to copy nothing — so every tag published since 2026-05-14 (including 4f4b455, which every environment runs) contains no libasyncProfiler.so, no jattach and no fdtransfer. Profiles fail with /tmp/async-profiler/build/fdtransfer: not found.

2. Even with the files present, the library can't load. libasyncProfiler.so is dlopen'd by the target JVM, not by anything in our image, so the musl build fails on every glibc JVM (temurin, corretto, distroless) with libc.musl-x86_64.so.1: cannot open shared object file. The library is now built twice and both are shipped; the agent picks the one matching the target's libc. jattach and fdtransfer stay musl — those execute inside our alpine image.

3. Python memory profiles came back empty — and said they succeeded. austin writes its samples to the -o file, then the non-flamegraph branch overwrote that file with the command's stdout, which is empty because austin prints nothing on stdout when -o is given. Every such profile published a zero-byte artefact. Fixed, plus an explicit failure when austin genuinely collected nothing.

4. py-spy and austin could not read CPython 3.14, both reporting it as if the target weren't Python. py-spy 0.4.0 → 0.4.2; austin 3.7.0 → 4.0.0. austin 4 only writes the binary MOJO format, so mojo2austin converts it back to the text format the raw artefact and flamegraph.pl are built around, and its -SIGINT (254) exit — which ends every exposure-limited run — is no longer treated as a failure. The Python stage also no longer built at all: Debian 11 is past EOL and its security suite has moved to archive, so apt-get install git 404s; moved to bookworm.

5. The Dockerfiles were only ever built by release.yml, after merge. That is why a broken image shipped for two months. code-verify now builds all five on every PR.

Type of change

  • Bug fix
  • CI / build

Test plan

Java — attach to a real glibc JVM (eclipse-temurin 21) with the rebuilt image:

$ jattach 29 load .../libasyncProfiler.so true "start,event=itimer,file=out.txt,collapsed"
Connected to remote JVM
JVM response code = 0
return code: 0
$ head -2 out.txt
L.main;java/lang/Thread.sleep;java/lang/Thread.sleep0 1
L.main;java/lang/Thread.sleep;java/lang/Thread.sleep0;JVM_Sleep;JavaThread::sleep_nanos(long);... 2

The same attach against the published 4f4b455 image fails with libc.musl-x86_64.so.1: cannot open shared object file, and before that with fdtransfer: not found.

Python 3.14 — the rebuilt image, end to end:

austin: austin 4.0.0   py-spy: py-spy 0.4.2   Python 3.14.0
austin exit=254  bytes=795        # MOJO
after conversion: bytes=1558, sample lines=32
P12;T0:12;/probe/load.py:<module>:6 163840
flamegraph.pl -> exit=0 svg bytes=14508

On the published image: py-spy 0.4.0 → Failed to find python version from target process; austin 3.7.0 → It looks like you are trying to profile a process that is not a Python process.

The empty-artefact bug, on the dev cluster (agent_task response for a python/memory profile of a Python 3.11 pod) — contents_base64 decodes to a gzip of 0 bytes, with "success": true.

go build ./... and go test ./... pass. New unit tests cover musl/glibc target detection, the library swap, the MOJO conversion, and austin's signal exit. (TestAsyncProfiler_CleanUp fails on macOS before and after this change — pre-existing, passes on Linux CI.)

Checklist

  • go build ./... and go test ./... pass locally
  • No breaking changes to the agent's stdout JSON envelope

Related issues

Companion fix in the agent: nudgebee/k8s-agent#619

Two independent reasons Java flamegraphs could never work.

The build: async-profiler 2.9's Makefile compiles its Java helper with a
hardcoded `-source 7`, which javac rejects from JDK 20 onwards, so `make`
died on its first target under the JDK 26 builder. A trailing `;` swallowed
that failure and `COPY` happily copied the resulting empty build/ into the
image, so every published tag since 2026-05-14 shipped no libasyncProfiler.so,
no jattach and no fdtransfer - profiles failed at runtime with
"/tmp/async-profiler/build/fdtransfer: not found". Pin the builder to JDK 17,
drop the `native` target (upstream's release-packaging step, which expects
prebuilt tarballs in the tree and always failed here), and assert the three
artefacts exist before shipping a layer.

The library: libasyncProfiler.so is dlopen'd by the *target* JVM, not by
anything in our image, so a musl build fails on every glibc JVM with
"libc.musl-x86_64.so.1: cannot open shared object file". Build the library
twice and ship both; the agent now picks the one matching the target's libc.

Also: py-spy 0.4.0 cannot read CPython 3.14 and reports it as "No python
processes found in process <pid>" - bump to 0.4.2. And that stage no longer
builds at all, because Debian 11 is past EOL and its security suite has moved
to archive; move it to bookworm.

Finally, build the images in code-verify. They were only ever built by
release.yml, after merge, which is how a broken image shipped for two months.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the JVM and Python Dockerfiles to fix build failures and support a wider range of target environments. Specifically, it pins the JVM builder to JDK 17 to avoid compilation errors, builds both glibc and musl versions of libasyncProfiler.so, and dynamically swaps them at runtime based on the target container's libc. It also updates the Python base image to Debian Bookworm and upgrades py-spy to version 0.4.2. Feedback on the changes highlights a potential false positive in targetUsesMusl when targetFs is empty, which could cause a glibc-based target to be incorrectly identified as musl-based, and suggests adding a defensive check.

Comment thread internal/agent/profiler/jvm/async_profiler.go
RamanKharchee
RamanKharchee previously approved these changes Sep 18, 2026
An empty targetFs made the loader glob relative to our own working
directory — and this image is alpine, so every target would have been
reported as musl-based.
RamanKharchee
RamanKharchee previously approved these changes Sep 18, 2026
@mayankpande88
mayankpande88 enabled auto-merge (squash) September 18, 2026 08:37
Two defects in the austin path, both verified on the dev cluster.

Python memory profiles came back as a zero-byte artefact reported as a
success. austin writes its samples to the -o file, but the non-flamegraph
branch then overwrote that file with the command's stdout - which is
empty, because austin prints nothing on stdout when -o is given. Stop
overwriting it, and fail loudly when austin genuinely collected nothing
rather than publishing an empty file.

austin 3.7.0 cannot read CPython 3.14; it reports the target as "not a
Python process", the same class of gap py-spy 0.4.0 had. Bump to austin
4, which reads 3.14 but only writes the binary MOJO format, so convert it
back with mojo2austin to the text format the raw artefact and
flamegraph.pl are built around. austin 4 also ends every exposure-limited
run by returning -SIGINT (254), which is not a failure.
@mayankpande88 mayankpande88 changed the title fix(jvm): ship an async-profiler that builds, and one that loads fix(profiler): make the JVM and Python profilers actually produce a profile Sep 18, 2026
@mayankpande88
mayankpande88 merged commit ca8a3b3 into main Sep 18, 2026
11 checks passed
@mayankpande88
mayankpande88 deleted the fix/jvm-image-empty-async-profiler-build branch September 18, 2026 09:57
mayankpande88 added a commit to nudgebee/k8s-agent that referenced this pull request Sep 18, 2026
The pinned 4f4b455 ships an empty async-profiler/build (no
libasyncProfiler.so, no jattach, no fdtransfer), a py-spy that cannot read
CPython 3.14, and an austin that returns an empty artefact. ca8a3b3 is the
first tag with nudgebee/application-profiler#128, which fixes all of those
and adds the glibc build of libasyncProfiler.so the agent now selects for
non-musl targets.
mayankpande88 added a commit to nudgebee/k8s-agent that referenced this pull request Sep 18, 2026
)

The pinned 4f4b455 ships an empty async-profiler/build (no
libasyncProfiler.so, no jattach, no fdtransfer), a py-spy that cannot read
CPython 3.14, and an austin that returns an empty artefact. ca8a3b3 is the
first tag with nudgebee/application-profiler#128, which fixes all of those
and adds the glibc build of libasyncProfiler.so the agent now selects for
non-musl targets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants