Skip to content

[APMSVLS-501] feat(bottlecap): add bottlecap-test-mode binary - #1216

Draft
lucaspimentel wants to merge 16 commits into
mainfrom
lpimentel/bottlecap-testmode-binary
Draft

[APMSVLS-501] feat(bottlecap): add bottlecap-test-mode binary#1216
lucaspimentel wants to merge 16 commits into
mainfrom
lpimentel/bottlecap-testmode-binary

Conversation

@lucaspimentel

@lucaspimentel lucaspimentel commented Apr 29, 2026

Copy link
Copy Markdown
Member

Part of a PR stack:

  1. [APMSVLS-501] refactor(bottlecap): preparatory work for a new "test-mode" binary #1344
  2. [APMSVLS-501] feat(bottlecap): add bottlecap-test-mode binary #1216 👈🏽 this PR

Overview

Adds a second [[bin]] target, bottlecap-test-mode: the APM trace-processing surface as a long-lived HTTP server with no Lambda Extension lifecycle. Same tracer endpoints on 127.0.0.1:8126, plus POST /flush for deterministic harness-driven flushing. Gated behind the test-mode cargo feature, so it is not built by default, fips, or cargo build --workspace.

cargo run --bin bottlecap-test-mode --features test-mode

It reuses TraceAgent, FlushingService, and the trace/stats/proxy flushers, so it exercises the same code paths the Lambda binary does. Consumers: the APM parity harness (drives fixture payloads through three agents and diffs what reaches the fake-intake from #1194) and local tracer debugging without Lambda RIE.

Endpoints

/v0.4/traces, /v0.5/traces, /v0.6/stats, /info come unchanged from TraceAgent's router. POST /flush is registered by a FlushRouterExtension attached via TraceAgent::with_router_extension(...) (the seam from #1344). It drains the ingest barrier, then runs flush_blocking_final() in a task bounded at 30s: 204 on full delivery, 502 when a flusher gave up on payloads, 504 on timeout, 500 on panic. The spawn matters because flush_blocking_final expect()s on the metrics aggregator handle: inline, a dead aggregator would drop the connection instead of returning a status the harness can act on.

Configuration

Same DD_* env vars as the Lambda binary. DD_APM_DD_URL redirects the intake (harness points it at the fake-intake), DD_SERVERLESS_FLUSH_STRATEGY opts into a periodic flush ticker, DD_TESTMODE_FUNCTION_ARN overrides the stub ARN used for tag generation. The API key is hardcoded to "stub-key", keeping the secrets resolver out of a test-only binary.

bottlecap::startup extraction

start_trace_agent moves out of src/bin/bottlecap/main.rs into a new top-level library module and splits into build_trace_agent (returns an unspawned TraceAgent plus a TraceAgentPipeline struct of named handles) and start_trace_agent (thin wrapper that spawns; the Lambda call site is unchanged). Test-mode needs the unspawned agent to attach /flush before spawning, which is the whole reason for the split. It sits at the crate root rather than under traces/ because it wires trace, stats, proxy, lifecycle, tags, appsec, and flushing together.

This was originally in #1344 and moved here after review: with no second binary, build_trace_agent had no caller and the public module looked unmotivated.

Determinism fixes in the library crate

Four ways the harness could have observed a wrong answer. Production behavior is unchanged in all four.

  • Ingest barrier (traces/trace_agent.rs) — request handlers only awaited the hand-off into an intermediate channel, so a 200 could be returned while the payload was still queued and an immediately following /flush would miss it. Each forwarder loop now selects on its payload channel before a barrier channel, so it can only acknowledge on an iteration where the payload channel was empty; /flush and the shutdown drain await TraceAgent::ingest_barrier() first. A closed barrier channel or dropped acknowledgement means the forwarder task is gone and its queued payloads are lost, so wait returns an error rather than reporting success. Costs one idle select! branch per payload.
  • Stats intake follows the trace intake (startup.rs) — DD_APM_DD_URL moved only the trace endpoint, so a redirected harness still sent stats to Datadog with the stub key, leaving /v0.6/stats unexercised. build_trace_agent takes a stats_url_override; the Lambda binary passes None for the site-derived default.
  • Flush failures are reported (flushing/service.rs) — the blocking flush path discarded all five flusher results, so /flush answered 204 after dropping payloads. flush_blocking/flush_blocking_final now return whether any flusher gave up, and log which domains dropped data. Lambda call sites ignore the value.
  • Flushes are serialized (bin/bottlecap-test-mode/main.rs) — the periodic driver, POST /flush, and the shutdown drain shared one FlushingService, and every flusher drains its aggregator before awaiting delivery, so two overlapping flushes let one report success on empty queues while the other's send was still in flight. All three now run barrier-plus-flush under a shared lock, which also makes the final drain wait for an in-flight periodic flush.

Shutdown

ctrl_c() races the listener task, so a listener that never bound (port taken) exits non-zero instead of idling with nothing to connect to. On ctrl-c: cancel the token, await the listener so in-flight handlers finish (bounded at 10s), then take the flush lock, await the ingest barrier, and run flush_blocking_final(). The periodic flush task selects on the same token so it doesn't leak.

Why a second binary

The overlap with the Lambda binary is ~20 lines (init_ustr, logging, config load); everything else is intentionally absent (no telemetry listener, LWA, logs agent, proxy, DogStatsD, event-bus lifecycle). A separate [[bin]] makes that surface compiler-enforced rather than a runtime branch. Rejected alternatives (env-var branch, auto-detect, CLI flag, mode gating in one binary) are in the design doc: bottlecap-test-mode.md (local; happy to land it in-repo if reviewers prefer).

Testing

  • cargo clippy --workspace --all-targets --features default -- -D warnings — clean
  • cargo clippy --workspace --all-targets --features default,test-mode -- -D warnings — clean
  • cargo fmt --all -- --check — clean
  • cargo test -p bottlecap --lib — clean
  • cargo test -p bottlecap --features test-mode --bin bottlecap-test-mode — clean

New tests:

  • ingest_barrier_waits_for_queued_payloads_to_reach_the_aggregator — confirmed to fail (0 of 3 payloads) with the barrier wait removed.
  • stats_url_follows_the_overridden_trace_intake and stats_url_matches_the_site_default_when_not_overridden.
  • POST /flush status contract, driven through the router with an injectable flush operation: 204 when nothing was lost, 502 when payloads were lost, 500 when the flush panics, and 504 on timeout (paused Tokio time).

Manual verification against a local fake intake on :8200:

  • POST /v0.4/traces (real msgpack span) → 200, buffered, periodic flush fired at 2s, intake received POST /api/v0.2/traces (485 bytes, DD-API-KEY: stub-key). Same trace_id+span_id sent twice: second dropped by the existing dedup service.
  • POST /v0.5/traces (malformed) → 500 from the existing deserializer. GET /info → 200. POST /flush → 204 (predates the timeout/panic guard; the 502/500/504 paths are now covered by handler tests rather than manual runs).
  • Port 8126 already bound → Error: trace agent failed: Address already in use (os error 98), exit 1. SIGINT → clean drain and exit.

End-to-end coverage lands with the parity harness (#1194 and the future apm-agent-parity-rs repo).

Follow-ups

  • CI did not exercise --features test-mode — existing jobs run --features default and --no-default-features --features fips, and cargo build --all skips required-features targets. Added a cargo clippy --workspace --features default,test-mode step to both the GitHub Actions and GitLab pipelines.
  • POST /flush has no committed test. The handler now takes its flush work as an injectable operation, so the 204/500/502/504 branches are driven directly through the router.

🤖

@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Aug 26, 2026

Copy link
Copy Markdown

Pipelines  Tests

Unblock PR with BitsAI

⚠️ Warnings

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 2 Pipeline jobs failed

DataDog/datadog-lambda-extension | e2e-test-status (amd64) — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/datadog-lambda-extension | publish layer e2e sandbox (amd64, fips)

View more details · View in GitLab

ℹ️ Info

🔄 Datadog auto-retried 1 job - 0 passed on retry View in Datadog

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ec8ff3b | Docs | View more details | Give us feedback!

@lucaspimentel lucaspimentel changed the title [APMSVLS-501] feat(bottlecap): add bottlecap-test-mode binary [APMSVLS-501] feat(bottlecap): add bottlecap-test-mode binary Aug 27, 2026
@lucaspimentel
lucaspimentel force-pushed the lpimentel/bottlecap-test-mode branch from b7bfcb6 to abb35a3 Compare September 2, 2026 20:29
@lucaspimentel
lucaspimentel force-pushed the lpimentel/bottlecap-testmode-binary branch from 4efbc0d to 0145da0 Compare September 2, 2026 20:56
@lucaspimentel

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-11T17:51:12.645139Z bf758e6 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0145da0538

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bottlecap/src/bin/bottlecap-test-mode/main.rs Outdated
Comment thread bottlecap/src/startup.rs Outdated
Comment thread bottlecap/src/bin/bottlecap-test-mode/main.rs Outdated
Comment thread bottlecap/src/bin/bottlecap-test-mode/main.rs Outdated
Comment thread bottlecap/src/bin/bottlecap-test-mode/main.rs Outdated
@lucaspimentel
lucaspimentel force-pushed the lpimentel/bottlecap-test-mode branch from abb35a3 to 211d68c Compare September 3, 2026 21:20
@lucaspimentel
lucaspimentel force-pushed the lpimentel/bottlecap-testmode-binary branch from 0145da0 to b0f9bcc Compare September 3, 2026 21:22
@lucaspimentel
lucaspimentel force-pushed the lpimentel/bottlecap-test-mode branch 2 times, most recently from 901ea64 to e852fe4 Compare September 9, 2026 15:22
@lucaspimentel
lucaspimentel force-pushed the lpimentel/bottlecap-testmode-binary branch from b0f9bcc to f7dbb9d Compare September 9, 2026 16:31
@lucaspimentel
lucaspimentel force-pushed the lpimentel/bottlecap-test-mode branch from e49b50d to 1976b21 Compare September 10, 2026 15:30
Base automatically changed from lpimentel/bottlecap-test-mode to main September 10, 2026 17:51
@lucaspimentel
lucaspimentel force-pushed the lpimentel/bottlecap-testmode-binary branch from f7dbb9d to c559152 Compare September 10, 2026 18:49
@lucaspimentel
lucaspimentel requested a balanced review from Copilot September 10, 2026 20:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Concurrent flushes can report success or exit while an earlier destructive flush remains incomplete.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a feature-gated, long-running APM test-mode server while sharing the production trace pipeline.

Changes:

  • Adds trace ingestion, manual/periodic flushing, and graceful shutdown.
  • Extracts shared trace-agent startup and adds ingestion barriers.
  • Adds flush failure reporting and CI coverage for test-mode.
File summaries
File Description
.github/workflows/rs_ci.yml Checks the test-mode feature.
.gitlab/templates/pipeline.yaml.tpl Checks the test-mode feature.
bottlecap/Cargo.toml Registers the binary and Tokio signal support.
bottlecap/src/bin/bottlecap-test-mode/main.rs Implements the test-mode server.
bottlecap/src/bin/bottlecap/main.rs Uses shared startup assembly.
bottlecap/src/flushing/service.rs Reports undelivered blocking flushes.
bottlecap/src/lib.rs Exposes the startup module.
bottlecap/src/startup.rs Builds and starts the shared trace pipeline.
bottlecap/src/traces/trace_agent.rs Adds trace/stats ingestion barriers.
Review details
  • Files reviewed: 9/9 changed files
  • Comments generated: 5
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread bottlecap/src/bin/bottlecap-test-mode/main.rs Outdated
Comment thread bottlecap/src/traces/trace_agent.rs Outdated
Comment thread bottlecap/src/bin/bottlecap-test-mode/main.rs Outdated
Comment thread bottlecap/src/bin/bottlecap-test-mode/main.rs Outdated
Comment thread bottlecap/src/bin/bottlecap-test-mode/main.rs
@lucaspimentel

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: bf758e621f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical flush and ingest-barrier issues, lifecycle defects, and missing feature-enabled CI tests remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (4)

Previously missed (2) — in code that hasn't changed since the last review.

bottlecap/src/traces/trace_agent.rs:245

  • With biased, this branch is always preferred whenever trace_rx is ready. Under sustained trace traffic the payload channel can remain ready indefinitely, starving the barrier branch; /flush can then wait until its 30-second timeout despite a healthy forwarder. Use an ordering/drain protocol that preserves all payloads before the barrier while guaranteeing barrier progress.
    bottlecap/src/traces/trace_agent.rs:275
  • With biased, this branch is always preferred whenever stats_rx is ready. Under sustained stats traffic the payload channel can remain ready indefinitely, starving the barrier branch; /flush can then wait until its 30-second timeout despite a healthy forwarder. Use an ordering/drain protocol that preserves all payloads before the barrier while guaranteeing barrier progress.

.github/workflows/rs_ci.yml:86

  • This CI change only compiles the new required-features binary with Clippy. The test job at rs_ci.yml:118-147 still runs cargo nextest run --workspace without --features test-mode, so the new binary's #[cfg(test)] tests—including the /flush 204/502/500/504 cases—are skipped in CI. Add a feature-enabled test invocation or matrix so these tests actually run.
      # The test-mode feature gates the bottlecap-test-mode binary via
      # required-features, so no other job compiles it.

bottlecap/src/bin/bottlecap-test-mode/main.rs:273

  • FlushControl::get_flush_interval() returns a Tokio interval whose first tick is immediate. Unlike the production flush loops (main.rs:493-495 and main.rs:675), this loop does not discard that tick, so periodically,<n> triggers an extra flush at startup instead of waiting for the configured interval. Consume the initial tick before entering the loop.
    tokio::spawn(async move {
        loop {
  • Files reviewed: 11/11 changed files
  • Comments generated: 3
  • Review effort level: Lite

// expects on the metrics aggregator handle, so a dead aggregator
// task would otherwise panic the connection task instead of
// returning a status the harness can act on.
let mut task = tokio::task::spawn(async move { flush_op().await });
Comment on lines +247 to +249
if let Err(e) = aggregator_handle.insert_payload(tracer_payload_info) {
error!("TRACE_AGENT | Failed to insert payload into aggregator: {e}");
}
Comment on lines +211 to +219
match tokio::time::timeout(SHUTDOWN_TIMEOUT, listener_task).await {
Ok(Ok(Ok(()))) => {}
Ok(Ok(Err(e))) => error!("Trace agent shut down with an error: {e:?}"),
Ok(Err(e)) => error!("Trace agent task failed: {e:?}"),
Err(_) => error!(
"Trace agent did not shut down within {}s, draining anyway",
SHUTDOWN_TIMEOUT.as_secs()
),
}
lucaspimentel and others added 3 commits September 11, 2026 19:16
Moves start_trace_agent out of the Lambda binary into bottlecap::startup
so both [[bin]] targets can share it, and splits it into:

- build_trace_agent, which returns an unspawned TraceAgent plus a
  TraceAgentPipeline handle struct
- start_trace_agent, a thin wrapper that spawns the agent

bottlecap-test-mode needs the unspawned agent so it can attach its
/flush RouterExtension before spawning; the Lambda binary keeps calling
start_trace_agent and its call site is unchanged.

Placed at the crate root rather than under traces/ because it wires
trace, stats, proxy, lifecycle, tags, appsec, and flushing together.
A second [[bin]] target that runs the APM trace-processing surface as a
long-lived HTTP server with no Lambda lifecycle. Listens on
127.0.0.1:8126 and exposes the standard tracer endpoints
(/v0.4/traces, /v0.5/traces, /v0.6/stats, /info) plus POST /flush for
deterministic harness-driven flushing. Configured by the same DD_* env
vars the Lambda binary reads. Optional periodic flushing via
DD_SERVERLESS_FLUSH_STRATEGY (decoupled from managed-instance mode).

Gated behind the `test-mode` cargo feature (required-features), so it is
not built in default or fips builds. Build with
`cargo build --bin bottlecap-test-mode --features test-mode`.

Intended for the cross-agent parity harness (APMSVLS-496) and for local
dev workflows that need a tracer endpoint without standing up a Lambda.

APMSVLS-501

🤖 Co-Authored-By: Claude Code <noreply@anthropic.com>
flush_blocking_final expects on the metrics aggregator handle, so a dead
aggregator task panicked the connection task and the harness saw a
dropped connection instead of a status it could act on. The five flushers
also bound only their individual HTTP calls, so stacked retries could
leave a request outstanding far longer than a harness should wait.

Runs the flush in a spawned task and caps it at 30s: 204 on success, 500
if the task panics, 504 after aborting a timed-out flush.

Restores the hardening that previously lived in the trace agent's
hardcoded /flush handler, now on the consumer side where the route lives.
The bottlecap-test-mode binary is gated behind required-features, so
no existing CI job compiles it and a change to library code could
break it without any job failing. Add a clippy pass with the test-mode
feature enabled to both the GitHub Actions and GitLab pipelines.
Trace and stats requests only await the hand-off into an intermediate
channel, so a request can be answered while its payload is still queued
ahead of the aggregator. A flush issued right after an accepted request
could therefore miss it, leaving the payload buffered until the next
flush and making the parity harness nondeterministic.

Add an ingest barrier over both forwarder tasks and await it in POST
/flush and in the shutdown drain.

🤖
DD_APM_DD_URL only moved the trace intake, so a harness pointing traces
at a local fake-intake still sent stats toward Datadog with the stub API
key, leaving the binary's advertised /v0.6/stats path unexercised.

Derive the stats endpoint from the resolved trace intake in test mode.
With DD_APM_DD_URL unset this reproduces the site-derived default, and
production routing is unchanged.

🤖
The blocking flush path discarded every flusher result, so payloads that
could not be delivered were dropped while POST /flush still answered 204.
The harness read that as a successful drain.

Report undelivered payloads from the blocking flush, log which domains
dropped data, and answer 502 instead of 204 when any did.

🤖
Cancelling the shutdown token only signals graceful shutdown, so the
final drain could run while a request handler was still processing and
its payload would be dropped when the process exited.

Await the listener task, bounded so a lingering connection cannot wedge
shutdown.

🤖
A failure to bind port 8126 was only logged, leaving a live process with
no listener while main waited on ctrl-c. The harness saw a healthy
process, or connected to whatever already held the port.

Propagate the listener error and race it against ctrl-c so startup
failures end the process with a non-zero exit.

🤖
Each flusher drains its aggregator before awaiting network delivery, so
the periodic driver, POST /flush, and the shutdown drain could overlap
and let one report success on empty queues while another's send was
still in flight. All three now run the barrier-plus-flush sequence under
a shared lock, which also makes the final drain wait for an in-flight
periodic flush.

🤖
A closed barrier channel or a dropped acknowledgement means the forwarder
task is gone and the payloads queued ahead of it were lost, not drained.
The barrier now returns an error instead of treating that as success, and
POST /flush reports 502 rather than 204.

🤖
The endpoint table listed only the five core paths, but the binary also
serves every proxy route the trace agent's router registers.

🤖
The handler now takes the flush work as an injectable operation, so the
204, 502, 500 and 504 branches can be driven directly. Without this a
regression in the status the harness reads would pass CI.

🤖
The ingest barrier held only its own acknowledgement channels, while every
payload sender for the stats forwarder lived inside the trace agent. On the
normal test-mode shutdown path the agent is dropped before the final drain
runs, so the stats forwarder had already exited (after draining its queue)
and the barrier reported it as dead, logging that accepted payloads were
lost on every clean shutdown even though nothing was lost.

The barrier now keeps a payload sender for each forwarder, so a live barrier
keeps both forwarders running and a barrier error again means a forwarder
genuinely died. The stats route is also no longer passed in to the router
builder, so it cannot be wired to a channel outside the barrier's coverage.

🤖
Only SIGINT was handled, so a container runtime or harness stopping the
binary with SIGTERM killed it outright and the final drain never ran,
losing the last accepted payloads. The signal driver is now pulled in by
the test-mode feature rather than unconditionally, so the shipped
extension no longer carries it.

Also corrects the periodic flush comment: the `end` strategy yields the
placeholder interval that means "never race a flush", so it is periodic
only in name.

🤖
The path the config crate appends to the trace intake URL was declared
twice, once for data streams and once for the test-mode binary. Both now
use a single definition.

🤖
@lucaspimentel
lucaspimentel force-pushed the lpimentel/bottlecap-testmode-binary branch from bf758e6 to ec8ff3b Compare September 11, 2026 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants