Stream image layers instead of extracting to disk - #6
Merged
Conversation
The scanner unpacked the entire docker-save archive to a temp dir and then walked the layers a second time to find the apk database — paying for full extraction and a separate read pass. Read the one file we need (lib/apk/db/installed, as merged across layers) directly from the saved tar instead: lower disk footprint, no unpacked tree, and a single in-memory walk guarded by the same tar-bomb size caps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Scanning unpacked the whole
docker savearchive to a temp directory and then walked the layers a second time to locate the Alpine package database — two full passes and a fully materialized image tree on disk for one file's worth of data. This lands Phase 1 #3 from the roadmap: read that file directly from the saved tar, so a multi-GB image no longer has to be exploded to disk, and most scans need no unpacked tree at all.Approach
The analyzer only needs
lib/apk/db/installedas it exists in the final merged filesystem. We keep the singleimage.tardocker savewrites but stop unpacking it:extractor.SaveImagereplacesExtractImage— pulls + saves, returns the tar path (no untar).extractor.StreamWalkerwalks a tar stream applying the same per-entry and whole-image cumulative size caps and traversal guard asUntar, via a callback instead of writing to disk.analyzer.BuildSBOMstreams: pass 1 locatesmanifest.json(position in the archive isn't guaranteed), pass 2 walks layers, scans each for the apk DB, and resolves the winner in manifest order — including OCI whiteout handling so a deleted DB in a later layer doesn't surface a stale one.The bounded-extraction guards from the prior tar-bomb work are preserved on the streaming path, and
Untaris retained as a fallback.Tests
StreamWalkertests (guards + the shared cumulative budget spanning nested walks).analyzer_test.go(the package's first): nested in-memory tar fixtures covering gzip/plain layers, manifest-vs-tar ordering, last-write-wins, whiteout + opaque-whiteout deletion, re-add after whiteout, empty-DB → not-found, and the size cap. All race-clean, no network.A high-effort review pass caught and fixed an empty/corrupt-DB case that returned an empty SBOM instead of an error.
🤖 Generated with Claude Code