Skip to content

Decrease in trace metrics after updating to >v68 #688

Description

@HiDAl

Hello,

some days ago my team tried to update the currently used layer to the latest version from v65. First, we tried with v68 and noticed a huge decrease in the hits metrics generated through the tracing tool: for example when using tracer.trace('name', ...), we saw a decrease in the metrics trace.name.hits.
Checking documentation and such, we noticed that the env var DD_TRACE_SAMPLE_RATE was deprecated so we migrated to use the new one DD_TRACE_SAMPLING_RULES (moving from DD_TRACE_SAMPLE_RATE=0.4 to DD_TRACE_SAMPLING_RULES='[{"sample_rate": 0.4}]'. Sadly, the issue was still there. We decided to give a try to v78, but the problem persisted.
When enabling the debugging option, I see the following output (shortened for simplicity):

// v65 using DD_TRACE_SAMPLE_RATE
{
  "os_name": "Linux",
  "os_version": "5.10.236-249.934.amzn2.aarch64",
  "architecture": "arm64",
  "version": "4.44.0",
  "lang": "nodejs",
  "lang_version": "20.19.1",
  "sample_rate": 0.4,
  "sampling_rules": [
    {
      "matchers": [],
      "_sampler": {
        "_rate": 0.4
      },
      "_limiter": {
        "_rateLimit": 100,
        "_limiter": {
          "tokenBucket": {
            "bucketSize": 100,
            "tokensPerInterval": 100,
            "interval": 1000,
            "parentBucket": null,
            "content": 99,
            "lastDrip": 1748592474968
          },
          "curIntervalStart": 1558141,
          "tokensThisInterval": 2
        },
        "_tokensRequested": 2,
        "_prevIntervalTokens": 0,
        "_prevTokensRequested": 0
      }
    }
  ],
  ...
}

// v78 using DD_TRACE_SAMPLING_RULES
{
  "os_name": "Linux",
  "os_version": "5.10.236-249.934.amzn2.aarch64",
  "architecture": "arm64",
  "version": "4.44.0",
  "lang": "nodejs",
  "lang_version": "20.19.1",
  "sampling_rules": [
    {
      "matchers": [],
      "_sampler": {
        "_rate": 0.4
      }
    }
  ],
  ...
}

One thing that I noticed, but not sure at all, is that even though the sample rate was configured, I saw a 100% of the traces —no sampling?—, so not sure whether it's normal after the update.

thank you

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingenhancementNew feature or requestversion/nextIssues related to DD_EXTENSION_VERSION: next

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions