Skip to content

Lambda Extension ignores additional endpoint Secrets Manager variables when WIF is enabled #1370

Description

@dclarkNV

Dual shipping from Datadog Lambda Extension is documented as supported: https://docs.datadoghq.com/serverless/aws_lambda/configuration/#dual-shipping

But I don't believe that appears to be the case. The Datadog Lambda Extension does not resolve these documented environment variables:

  • DD_ADDITIONAL_ENDPOINTS_SECRET_ARN
  • DD_APM_ADDITIONAL_ENDPOINTS_SECRET_ARN
  • DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS_SECRET_ARN

The primary organization uses AWS Workload Identity Federation through DD_ORG_UUID. The additional
endpoints contain a conventional API key for a secondary organization.

Primary delivery succeeds, but the extension does not retrieve the additional-endpoint secrets or
attempt delivery to the secondary organization.

Environment

  • AWS Lambda
  • Datadog Lambda Extension: v99
  • Primary authentication: AWS Workload Identity Federation
  • Additional endpoint credentials: AWS Secrets Manager
  • Telemetry types tested: metrics, traces, and logs

Expected behavior

The extension should:

  1. Retrieve each configured additional-endpoint secret from AWS Secrets Manager.
  2. Parse its value as the corresponding plaintext configuration:
    • DD_ADDITIONAL_ENDPOINTS
    • DD_APM_ADDITIONAL_ENDPOINTS
    • DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS
  3. Send the same telemetry to both the primary and additional endpoints.

This matches the documented statement that the Lambda Extension automatically retrieves Secrets
Manager values for the corresponding *_SECRET_ARN variables.

Actual behavior

  • Primary WIF authentication succeeds.
  • Telemetry reaches the primary organization.
  • The extension does not request the three additional-endpoint secrets from Secrets Manager.
  • No delivery attempt is made to the additional metrics, traces, or logs endpoints.
  • The secondary organization receives none of this telemetry.
  • No configuration error is reported for the ignored variables.

The function role has secretsmanager:GetSecretValue permission for all three secrets, and their
contents have been validated independently.

Reproduction

Configure a Lambda function with:

  DD_ORG_UUID=<primary-org-uuid>
  DD_ADDITIONAL_ENDPOINTS_SECRET_ARN=<metrics-secret-arn>
  DD_APM_ADDITIONAL_ENDPOINTS_SECRET_ARN=<traces-secret-arn>
  DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS_SECRET_ARN=<logs-secret-arn>
  DD_LOGS_CONFIG_FORCE_USE_HTTP=true

Grant the function role secretsmanager:GetSecretValue for all three secrets, then invoke the
function.

Observe that primary WIF delivery succeeds, but Secrets Manager receives no requests for the
additional-endpoint secrets and no telemetry reaches the additional endpoints.

Replacing the secret-backed variables with their plaintext equivalents causes the extension to
construct the additional endpoint flushers, indicating that endpoint delivery itself is supported.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions