ci,local: one package set for local-stack binaries, and warm CI caches - #3454
Merged
Conversation
The Dekaf CI job ran 34-45 minutes on a cache miss and 22-26 on a hit, nearly all of it compiling. Three causes, plus two undersized timeouts. Cargo resolves features over the set of packages named on the command line, so `-p agent -p dekaf` and `-p dekaf` resolve differently, and the difference reaches each dependency's `-C metadata` hash. Builds naming different sets share no artifacts and recompile each other's dependencies. Each flow-*@.service ran its own `cargo build -p <one>` and `ci:gnu-dev` another combination ahead of them, so the same dependency graph was paid for repeatedly, serialized behind the single lock on CARGO_TARGET_DIR. Counted on a cold target directory, that arrangement ran 1154 crate compilations to produce these binaries, 655 of them inside ExecStartPre with a port-waiter blocked on them. build:local-rust builds all seven as one set, once, leaving none in any ExecStartPre after it; anything wanting them must run it rather than its own `cargo build -p <subset>`. Since they all now build `bindings`, they all now need librocksdb, which the script checks for. A test build is the one thing it cannot absorb: `cargo test` unifies dev-dependency features and `cargo build` does not, so ci:dekaf-e2e-build resolves its own way whatever packages it names. Second, GitHub scopes a cache to the ref that wrote it. Neither workflow ran on master, so nothing ever wrote a master-scoped entry for these jobs; platform-build does write five, but rust-cache keys on $GITHUB_JOB, so those cannot serve another job whatever it builds. Sampled over 14 recent runs each, 71% of dekaf-test, 71% of platform-test and 57% of generated-artifacts-check runs missed both the primary and the restore key, at 13-15 minutes apiece. What they track is eviction, and the 45% of dekaf-test runs that are cancelled and so save nothing. Both workflows now also run on master, no longer cancel in-progress runs there, and save with cache-on-failure, since master's is the only entry and the job's last step is its flakiest. Given that, pull requests no longer save at all. The cache key ends in a hash of Cargo.lock and the Cargo.tomls, so a pull request that leaves dependencies alone computes the key master already holds and its save is a private copy. Their volume is what drives eviction: of 69 v0-rust entries, holding 94.5GB of a 136.7GB store, 64 were pull-request copies across just 20 distinct keys, one key holding 8 interchangeable copies. Nothing in the store had survived 18.5 hours against a nominal 7-day retention, so the shared entry was routinely evicted before it could be restored. Third, dead time. ci:dekaf-e2e-build compiles the test binary before the stack starts rather than against a fully-running idle one, and Supabase's container startup now overlaps the build instead of following it. The e2e task splits into -build and -run halves, matching the ci:nextest-build / ci:nextest-run precedent. Two readiness budgets were undersized. The Kafka wait counted loop iterations rather than seconds, and each pass also costs however long the probe takes to fail -- which is what let a nominal 60 survive the 1m40s to 4m32s cold `docker pull` of confluentinc/cp-kafka observed on fresh runners. It now measures the clock, against 300s. Separately, flow-supabase@.service has failed at 300s with the machine idle, and its ~8 image pulls now overlap a full cargo build; raised to 900s. Validated end to end on a local stack: bring-up 52s with 13/13 services accepting connections, and 26/26 e2e tests passing in 124s. That host is aarch64 while CI is x86_64, and wall-clock varied by roughly 40% across runs, so the docs cite crate counts rather than seconds. Running both workflows on master costs one run of each per merge, against ~2.1 master pushes per day. Cache figures observed 2026-09-01.
jgraettinger
force-pushed
the
johnny/dekaf-ci-performance-c4ca21
branch
from
September 2, 2026 14:32
33114ae to
a7ebd1e
Compare
github-actions
Bot
added
pending:agent
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The commit message carries the full reasoning and measurements. Notes for
review, and for after merge:
The cache benefit does not appear on this PR. No master-scoped entry
exists for
dekaf-test,platform-testorgenerated-artifacts-checktoday. The first master run after merge writes them; pull requests opened
after that are the ones that restore.
The first build on any checkout is a full rebuild. Naming a different
package set changes every dependency's
-C metadatahash, so foldingdata-plane-controllerinto the shared set invalidates what a warm targetdirectory already holds. One time, locally and in CI.
local:supabase --no-blockand thesave-if/pushpairing each have amatched half elsewhere in the file. Removing the
pushtrigger withoutremoving
save-ifwrites no cache at all.Validated on a local stack: 13/13 services accepting connections after 52s,
26/26 Dekaf e2e tests passing in 124s. CI on this PR exercises the reordered
Dekaf job, still on a cold cache.