Write a heap dump when the JVM dies of an OOM - #30
Merged
Merged
Conversation
Challenge evaluation in readingbat-core retains roughly 1.3 MB of heap per eval and never releases it (readingbat/readingbat-core#128). An OOM here is a question about what was holding memory, and only a dump taken at the moment it happens answers it. The flags cost nothing until then. The image sets JAVA_TOOL_OPTIONS to dump into /app/dumps, and both docker-compose.yml and machines/content/run.sh mount a volume there so the dump outlives the container that produced it -- run.sh uses --rm, which would otherwise discard it immediately. Named volumes rather than bind mounts, which is not cosmetic: Docker creates a missing bind-mount directory as root and this image runs as uid 1000, so the JVM could not have written the dump, silently, at exactly the moment it is needed. A named volume is initialized from the image's own /app/dumps and inherits its readingbat ownership. Verified by forcing a real OOM in a container built the same way: the dump lands on the volume owned by 1000. One volume per composed service, because each container is pid 1 in its own namespace -- the dump is named java_pid1.hprof, so a shared volume would have all three overwriting each other. A dump is a byte-for-byte copy of live memory, so it carries session secrets, database credentials and user data. README says to treat the file as a secret, and dumps/ is gitignored in case anyone bind-mounts one locally. It is also roughly the size of the live heap, which MaxRAMPercentage=75 allows to be most of the machine's RAM, so it is worth watching disk. Takes effect on the next image build; setting JAVA_TOOL_OPTIONS in docker_env_vars overrides the image default in the meantime, since env_file wins over ENV. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Dockerfile comment and changelog entry justified heap dumps with readingbat-core#128's figure of about 1.3 MB retained per evaluation. That was measured on a many-jar Gradle classpath. The root cause, found since, is a classloader per evaluation that holds every jar on kotlin.script.classpath open -- and this image sets -Dkotlin.script.classpath to the single server.jar. Measured against that uberjar, the cost here is about 20 KB per eval. So #128 is not the likely cause of an OOM in this deployment, and the comment now says so. The case for heap dumps stands on its own: they are the only record of what was holding memory when one happens, whatever the cause. Comments and changelog only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Dockerfile comment and changelog entry justified heap dumps by pointing at readingbat-core#128, a per-evaluation classloader leak. That leak turned out not to exist: a GC-root trace showed the retained classloaders were held by Kover's coverage agent, which only runs inside instrumented test tasks. With the agent off, nothing is retained, and it never runs in this image. Heap dumps stand on their own -- a dump is the only record of what was holding memory when an OOM happens, whatever the cause, and the flags cost nothing until then -- so the comment now says that and nothing more. Comments and changelog only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A heap dump taken at the moment of an OOM is the only record of what was holding memory, and the flags cost nothing until then. This arms them for every deployment path.
DockerfilesetsJAVA_TOOL_OPTIONS="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/app/dumps"and creates/app/dumpsowned by thereadingbatuser.docker-compose.ymlmounts a named volume per service at/app/dumps;machines/content/run.shmounts one too, since its--rmwould otherwise discard the dump with the container. The retrieval command is in a comment there and in the README..gitignoreignoresdumps/, in case anyone bind-mounts locally.Two details that are not cosmetic:
/app/dumpsand inherits its ownership.java_pid1.hprof; a shared volume would have all three overwriting each other.A dump is a copy of live memory, so it carries session secrets, database credentials and user data — the README says to treat the file as a secret. It is also roughly the size of the live heap, which
MaxRAMPercentage=75allows to be most of the machine's RAM.Takes effect on the next image build (
make release). Until then, adding the sameJAVA_TOOL_OPTIONSline todocker_env_varsarms it immediately, sinceenv_fileoverridesENV. Once deployed, readingbat-core's startup log reports whether dumps are enabled (readingbat/readingbat-core#129).Test plan
JAVA_TOOL_OPTIONSabove:Dumping heap to /app/dumps/java_pid1.hprof, and the 23 MB dump landed on the named volume owned by uid 1000.uid=1000(readingbat)and that a named volume mounted at/app/dumpsis writable by it.docker compose configresolves a distinct named volume per service.🤖 Generated with Claude Code