Skip to content

test(bottlecap): fix clock race in increment metric tests - #1368

Draft
lucaspimentel wants to merge 1 commit into
mainfrom
lpimentel/fix-flaky-enhanced-metrics-tests
Draft

test(bottlecap): fix clock race in increment metric tests#1368
lucaspimentel wants to merge 1 commit into
mainfrom
lpimentel/fix-flaky-enhanced-metrics-tests

Conversation

@lucaspimentel

Copy link
Copy Markdown
Member

Overview

Fixes a flaky race condition in two enhanced-metrics tests in bottlecap/src/metrics/enhanced/lambda.rs:

  • test_increment_invocation_metric
  • test_increment_errors_metric

Each test read the wall clock twice: once for the timestamp passed to the increment method, and again (shadowing the first value) for the timestamp passed to assert_sketch. The dogstatsd dependency rounds timestamps down to ten-second buckets ((timestamp / 10) * 10), and that bucket participates in the aggregator metric ID. When the two clock reads straddled a ten-second boundary, the metric was inserted under one ID and looked up under another, causing failures such as aws.lambda.enhanced.errors not found.

This was observed in job 102934913271, which failed at 0.076s immediately after the clock crossed a ten-second boundary and passed on rerun with no code changes.

The fix reuses the test's first now value for both the increment method and assert_sketch, guaranteeing insertion and lookup use the same timestamp bucket. Test-only change: 12 deletions, no production code touched.

Generator-level timestamp-injection clock races are intentionally out of scope and reserved for a follow-up PR.

Testing

  • cargo nextest run -p bottlecap metrics::enhanced::lambda::tests: 17 passed
  • cargo nextest run --workspace: 684 passed, 0 failed, 0 skipped
  • cargo fmt --all -- --check: pass
  • RUSTFLAGS="-D warnings" cargo clippy --workspace --all-targets --features default: pass
  • RUSTFLAGS="-D warnings" cargo clippy --workspace --all-targets --no-default-features --features fips: pass

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused test-only change correctly eliminates the documented timestamp race.

Pull request overview

Fixes flaky enhanced-metrics tests by reusing one timestamp for metric insertion and lookup.

Changes:

  • Removes duplicate wall-clock reads from two tests.
  • Prevents crossing ten-second aggregation bucket boundaries.
File summaries
File Description
bottlecap/src/metrics/enhanced/lambda.rs Reuses the original timestamp in invocation and error metric assertions.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@datadog-official

datadog-official Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Pipelines

Unblock PR with BitsAI

⚠️ Warnings

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 1 Pipeline job failed

DataDog/datadog-lambda-extension | e2e-test-status (amd64, fips) — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 531cdd3 | Docs | View more details | Give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants