[APMSVLS-501] feat(bottlecap): add bottlecap-test-mode binary - #1216
[APMSVLS-501] feat(bottlecap): add bottlecap-test-mode binary#1216lucaspimentel wants to merge 16 commits into
bottlecap-test-mode binary#1216Conversation
149d9d4 to
f8da804
Compare
5f17230 to
14cfb71
Compare
|
bottlecap-test-mode binary
b7bfcb6 to
abb35a3
Compare
4efbc0d to
0145da0
Compare
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0145da0538
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
abb35a3 to
211d68c
Compare
0145da0 to
b0f9bcc
Compare
901ea64 to
e852fe4
Compare
b0f9bcc to
f7dbb9d
Compare
e49b50d to
1976b21
Compare
f7dbb9d to
c559152
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Concurrent flushes can report success or exit while an earlier destructive flush remains incomplete.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a feature-gated, long-running APM test-mode server while sharing the production trace pipeline.
Changes:
- Adds trace ingestion, manual/periodic flushing, and graceful shutdown.
- Extracts shared trace-agent startup and adds ingestion barriers.
- Adds flush failure reporting and CI coverage for
test-mode.
File summaries
| File | Description |
|---|---|
.github/workflows/rs_ci.yml |
Checks the test-mode feature. |
.gitlab/templates/pipeline.yaml.tpl |
Checks the test-mode feature. |
bottlecap/Cargo.toml |
Registers the binary and Tokio signal support. |
bottlecap/src/bin/bottlecap-test-mode/main.rs |
Implements the test-mode server. |
bottlecap/src/bin/bottlecap/main.rs |
Uses shared startup assembly. |
bottlecap/src/flushing/service.rs |
Reports undelivered blocking flushes. |
bottlecap/src/lib.rs |
Exposes the startup module. |
bottlecap/src/startup.rs |
Builds and starts the shared trace pipeline. |
bottlecap/src/traces/trace_agent.rs |
Adds trace/stats ingestion barriers. |
Review details
- Files reviewed: 9/9 changed files
- Comments generated: 5
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
@codex review |
|
Codex Review: Didn't find any major issues. Nice work! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
🟡 Changes recommended
Critical flush and ingest-barrier issues, lifecycle defects, and missing feature-enabled CI tests remain unresolved.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (4)
Previously missed (2) — in code that hasn't changed since the last review.
bottlecap/src/traces/trace_agent.rs:245
- With
biased, this branch is always preferred whenevertrace_rxis ready. Under sustained trace traffic the payload channel can remain ready indefinitely, starving the barrier branch;/flushcan then wait until its 30-second timeout despite a healthy forwarder. Use an ordering/drain protocol that preserves all payloads before the barrier while guaranteeing barrier progress.
bottlecap/src/traces/trace_agent.rs:275 - With
biased, this branch is always preferred wheneverstats_rxis ready. Under sustained stats traffic the payload channel can remain ready indefinitely, starving the barrier branch;/flushcan then wait until its 30-second timeout despite a healthy forwarder. Use an ordering/drain protocol that preserves all payloads before the barrier while guaranteeing barrier progress.
.github/workflows/rs_ci.yml:86
- This CI change only compiles the new required-features binary with Clippy. The test job at
rs_ci.yml:118-147still runscargo nextest run --workspacewithout--features test-mode, so the new binary's#[cfg(test)]tests—including the/flush204/502/500/504 cases—are skipped in CI. Add a feature-enabled test invocation or matrix so these tests actually run.
# The test-mode feature gates the bottlecap-test-mode binary via
# required-features, so no other job compiles it.
bottlecap/src/bin/bottlecap-test-mode/main.rs:273
FlushControl::get_flush_interval()returns a Tokio interval whose first tick is immediate. Unlike the production flush loops (main.rs:493-495andmain.rs:675), this loop does not discard that tick, soperiodically,<n>triggers an extra flush at startup instead of waiting for the configured interval. Consume the initial tick before entering the loop.
tokio::spawn(async move {
loop {
- Files reviewed: 11/11 changed files
- Comments generated: 3
- Review effort level: Lite
| // expects on the metrics aggregator handle, so a dead aggregator | ||
| // task would otherwise panic the connection task instead of | ||
| // returning a status the harness can act on. | ||
| let mut task = tokio::task::spawn(async move { flush_op().await }); |
| if let Err(e) = aggregator_handle.insert_payload(tracer_payload_info) { | ||
| error!("TRACE_AGENT | Failed to insert payload into aggregator: {e}"); | ||
| } |
| match tokio::time::timeout(SHUTDOWN_TIMEOUT, listener_task).await { | ||
| Ok(Ok(Ok(()))) => {} | ||
| Ok(Ok(Err(e))) => error!("Trace agent shut down with an error: {e:?}"), | ||
| Ok(Err(e)) => error!("Trace agent task failed: {e:?}"), | ||
| Err(_) => error!( | ||
| "Trace agent did not shut down within {}s, draining anyway", | ||
| SHUTDOWN_TIMEOUT.as_secs() | ||
| ), | ||
| } |
Moves start_trace_agent out of the Lambda binary into bottlecap::startup so both [[bin]] targets can share it, and splits it into: - build_trace_agent, which returns an unspawned TraceAgent plus a TraceAgentPipeline handle struct - start_trace_agent, a thin wrapper that spawns the agent bottlecap-test-mode needs the unspawned agent so it can attach its /flush RouterExtension before spawning; the Lambda binary keeps calling start_trace_agent and its call site is unchanged. Placed at the crate root rather than under traces/ because it wires trace, stats, proxy, lifecycle, tags, appsec, and flushing together.
A second [[bin]] target that runs the APM trace-processing surface as a long-lived HTTP server with no Lambda lifecycle. Listens on 127.0.0.1:8126 and exposes the standard tracer endpoints (/v0.4/traces, /v0.5/traces, /v0.6/stats, /info) plus POST /flush for deterministic harness-driven flushing. Configured by the same DD_* env vars the Lambda binary reads. Optional periodic flushing via DD_SERVERLESS_FLUSH_STRATEGY (decoupled from managed-instance mode). Gated behind the `test-mode` cargo feature (required-features), so it is not built in default or fips builds. Build with `cargo build --bin bottlecap-test-mode --features test-mode`. Intended for the cross-agent parity harness (APMSVLS-496) and for local dev workflows that need a tracer endpoint without standing up a Lambda. APMSVLS-501 🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
flush_blocking_final expects on the metrics aggregator handle, so a dead aggregator task panicked the connection task and the harness saw a dropped connection instead of a status it could act on. The five flushers also bound only their individual HTTP calls, so stacked retries could leave a request outstanding far longer than a harness should wait. Runs the flush in a spawned task and caps it at 30s: 204 on success, 500 if the task panics, 504 after aborting a timed-out flush. Restores the hardening that previously lived in the trace agent's hardcoded /flush handler, now on the consumer side where the route lives.
The bottlecap-test-mode binary is gated behind required-features, so no existing CI job compiles it and a change to library code could break it without any job failing. Add a clippy pass with the test-mode feature enabled to both the GitHub Actions and GitLab pipelines.
Trace and stats requests only await the hand-off into an intermediate channel, so a request can be answered while its payload is still queued ahead of the aggregator. A flush issued right after an accepted request could therefore miss it, leaving the payload buffered until the next flush and making the parity harness nondeterministic. Add an ingest barrier over both forwarder tasks and await it in POST /flush and in the shutdown drain. 🤖
DD_APM_DD_URL only moved the trace intake, so a harness pointing traces at a local fake-intake still sent stats toward Datadog with the stub API key, leaving the binary's advertised /v0.6/stats path unexercised. Derive the stats endpoint from the resolved trace intake in test mode. With DD_APM_DD_URL unset this reproduces the site-derived default, and production routing is unchanged. 🤖
The blocking flush path discarded every flusher result, so payloads that could not be delivered were dropped while POST /flush still answered 204. The harness read that as a successful drain. Report undelivered payloads from the blocking flush, log which domains dropped data, and answer 502 instead of 204 when any did. 🤖
Cancelling the shutdown token only signals graceful shutdown, so the final drain could run while a request handler was still processing and its payload would be dropped when the process exited. Await the listener task, bounded so a lingering connection cannot wedge shutdown. 🤖
A failure to bind port 8126 was only logged, leaving a live process with no listener while main waited on ctrl-c. The harness saw a healthy process, or connected to whatever already held the port. Propagate the listener error and race it against ctrl-c so startup failures end the process with a non-zero exit. 🤖
Each flusher drains its aggregator before awaiting network delivery, so the periodic driver, POST /flush, and the shutdown drain could overlap and let one report success on empty queues while another's send was still in flight. All three now run the barrier-plus-flush sequence under a shared lock, which also makes the final drain wait for an in-flight periodic flush. 🤖
A closed barrier channel or a dropped acknowledgement means the forwarder task is gone and the payloads queued ahead of it were lost, not drained. The barrier now returns an error instead of treating that as success, and POST /flush reports 502 rather than 204. 🤖
The endpoint table listed only the five core paths, but the binary also serves every proxy route the trace agent's router registers. 🤖
The handler now takes the flush work as an injectable operation, so the 204, 502, 500 and 504 branches can be driven directly. Without this a regression in the status the harness reads would pass CI. 🤖
The ingest barrier held only its own acknowledgement channels, while every payload sender for the stats forwarder lived inside the trace agent. On the normal test-mode shutdown path the agent is dropped before the final drain runs, so the stats forwarder had already exited (after draining its queue) and the barrier reported it as dead, logging that accepted payloads were lost on every clean shutdown even though nothing was lost. The barrier now keeps a payload sender for each forwarder, so a live barrier keeps both forwarders running and a barrier error again means a forwarder genuinely died. The stats route is also no longer passed in to the router builder, so it cannot be wired to a channel outside the barrier's coverage. 🤖
Only SIGINT was handled, so a container runtime or harness stopping the binary with SIGTERM killed it outright and the final drain never ran, losing the last accepted payloads. The signal driver is now pulled in by the test-mode feature rather than unconditionally, so the shipped extension no longer carries it. Also corrects the periodic flush comment: the `end` strategy yields the placeholder interval that means "never race a flush", so it is periodic only in name. 🤖
The path the config crate appends to the trace intake URL was declared twice, once for data streams and once for the test-mode binary. Both now use a single definition. 🤖
bf758e6 to
ec8ff3b
Compare
Part of a PR stack:
bottlecap-test-modebinary #1216 👈🏽 this PROverview
Adds a second
[[bin]]target,bottlecap-test-mode: the APM trace-processing surface as a long-lived HTTP server with no Lambda Extension lifecycle. Same tracer endpoints on127.0.0.1:8126, plusPOST /flushfor deterministic harness-driven flushing. Gated behind thetest-modecargo feature, so it is not built bydefault,fips, orcargo build --workspace.It reuses
TraceAgent,FlushingService, and the trace/stats/proxy flushers, so it exercises the same code paths the Lambda binary does. Consumers: the APM parity harness (drives fixture payloads through three agents and diffs what reaches the fake-intake from #1194) and local tracer debugging without Lambda RIE.Endpoints
/v0.4/traces,/v0.5/traces,/v0.6/stats,/infocome unchanged fromTraceAgent's router.POST /flushis registered by aFlushRouterExtensionattached viaTraceAgent::with_router_extension(...)(the seam from #1344). It drains the ingest barrier, then runsflush_blocking_final()in a task bounded at 30s:204on full delivery,502when a flusher gave up on payloads,504on timeout,500on panic. The spawn matters becauseflush_blocking_finalexpect()s on the metrics aggregator handle: inline, a dead aggregator would drop the connection instead of returning a status the harness can act on.Configuration
Same
DD_*env vars as the Lambda binary.DD_APM_DD_URLredirects the intake (harness points it at the fake-intake),DD_SERVERLESS_FLUSH_STRATEGYopts into a periodic flush ticker,DD_TESTMODE_FUNCTION_ARNoverrides the stub ARN used for tag generation. The API key is hardcoded to"stub-key", keeping the secrets resolver out of a test-only binary.bottlecap::startupextractionstart_trace_agentmoves out ofsrc/bin/bottlecap/main.rsinto a new top-level library module and splits intobuild_trace_agent(returns an unspawnedTraceAgentplus aTraceAgentPipelinestruct of named handles) andstart_trace_agent(thin wrapper that spawns; the Lambda call site is unchanged). Test-mode needs the unspawned agent to attach/flushbefore spawning, which is the whole reason for the split. It sits at the crate root rather than undertraces/because it wires trace, stats, proxy, lifecycle, tags, appsec, and flushing together.This was originally in #1344 and moved here after review: with no second binary,
build_trace_agenthad no caller and the public module looked unmotivated.Determinism fixes in the library crate
Four ways the harness could have observed a wrong answer. Production behavior is unchanged in all four.
traces/trace_agent.rs) — request handlers only awaited the hand-off into an intermediate channel, so a200could be returned while the payload was still queued and an immediately following/flushwould miss it. Each forwarder loop now selects on its payload channel before a barrier channel, so it can only acknowledge on an iteration where the payload channel was empty;/flushand the shutdown drain awaitTraceAgent::ingest_barrier()first. A closed barrier channel or dropped acknowledgement means the forwarder task is gone and its queued payloads are lost, sowaitreturns an error rather than reporting success. Costs one idleselect!branch per payload.startup.rs) —DD_APM_DD_URLmoved only the trace endpoint, so a redirected harness still sent stats to Datadog with the stub key, leaving/v0.6/statsunexercised.build_trace_agenttakes astats_url_override; the Lambda binary passesNonefor the site-derived default.flushing/service.rs) — the blocking flush path discarded all five flusher results, so/flushanswered204after dropping payloads.flush_blocking/flush_blocking_finalnow return whether any flusher gave up, and log which domains dropped data. Lambda call sites ignore the value.bin/bottlecap-test-mode/main.rs) — the periodic driver,POST /flush, and the shutdown drain shared oneFlushingService, and every flusher drains its aggregator before awaiting delivery, so two overlapping flushes let one report success on empty queues while the other's send was still in flight. All three now run barrier-plus-flush under a shared lock, which also makes the final drain wait for an in-flight periodic flush.Shutdown
ctrl_c()races the listener task, so a listener that never bound (port taken) exits non-zero instead of idling with nothing to connect to. On ctrl-c: cancel the token, await the listener so in-flight handlers finish (bounded at 10s), then take the flush lock, await the ingest barrier, and runflush_blocking_final(). The periodic flush task selects on the same token so it doesn't leak.Why a second binary
The overlap with the Lambda binary is ~20 lines (
init_ustr, logging, config load); everything else is intentionally absent (no telemetry listener, LWA, logs agent, proxy, DogStatsD, event-bus lifecycle). A separate[[bin]]makes that surface compiler-enforced rather than a runtime branch. Rejected alternatives (env-var branch, auto-detect, CLI flag, mode gating in one binary) are in the design doc:bottlecap-test-mode.md(local; happy to land it in-repo if reviewers prefer).Testing
cargo clippy --workspace --all-targets --features default -- -D warnings— cleancargo clippy --workspace --all-targets --features default,test-mode -- -D warnings— cleancargo fmt --all -- --check— cleancargo test -p bottlecap --lib— cleancargo test -p bottlecap --features test-mode --bin bottlecap-test-mode— cleanNew tests:
ingest_barrier_waits_for_queued_payloads_to_reach_the_aggregator— confirmed to fail (0 of 3 payloads) with the barrier wait removed.stats_url_follows_the_overridden_trace_intakeandstats_url_matches_the_site_default_when_not_overridden.POST /flushstatus contract, driven through the router with an injectable flush operation:204when nothing was lost,502when payloads were lost,500when the flush panics, and504on timeout (paused Tokio time).Manual verification against a local fake intake on
:8200:POST /v0.4/traces(real msgpack span) → 200, buffered, periodic flush fired at 2s, intake receivedPOST /api/v0.2/traces(485 bytes,DD-API-KEY: stub-key). Same trace_id+span_id sent twice: second dropped by the existing dedup service.POST /v0.5/traces(malformed) → 500 from the existing deserializer.GET /info→ 200.POST /flush→ 204 (predates the timeout/panic guard; the 502/500/504 paths are now covered by handler tests rather than manual runs).Error: trace agent failed: Address already in use (os error 98), exit 1. SIGINT → clean drain and exit.End-to-end coverage lands with the parity harness (#1194 and the future
apm-agent-parity-rsrepo).Follow-ups
--features test-mode— existing jobs run--features defaultand--no-default-features --features fips, andcargo build --allskipsrequired-featurestargets. Added acargo clippy --workspace --features default,test-modestep to both the GitHub Actions and GitLab pipelines.POST /flushhas no committed test. The handler now takes its flush work as an injectable operation, so the 204/500/502/504 branches are driven directly through the router.🤖