Dual shipping from Datadog Lambda Extension is documented as supported: https://docs.datadoghq.com/serverless/aws_lambda/configuration/#dual-shipping
But I don't believe that appears to be the case. The Datadog Lambda Extension does not resolve these documented environment variables:
DD_ADDITIONAL_ENDPOINTS_SECRET_ARN
DD_APM_ADDITIONAL_ENDPOINTS_SECRET_ARN
DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS_SECRET_ARN
The primary organization uses AWS Workload Identity Federation through DD_ORG_UUID. The additional
endpoints contain a conventional API key for a secondary organization.
Primary delivery succeeds, but the extension does not retrieve the additional-endpoint secrets or
attempt delivery to the secondary organization.
Environment
- AWS Lambda
- Datadog Lambda Extension: v99
- Primary authentication: AWS Workload Identity Federation
- Additional endpoint credentials: AWS Secrets Manager
- Telemetry types tested: metrics, traces, and logs
Expected behavior
The extension should:
- Retrieve each configured additional-endpoint secret from AWS Secrets Manager.
- Parse its value as the corresponding plaintext configuration:
DD_ADDITIONAL_ENDPOINTS
DD_APM_ADDITIONAL_ENDPOINTS
DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS
- Send the same telemetry to both the primary and additional endpoints.
This matches the documented statement that the Lambda Extension automatically retrieves Secrets
Manager values for the corresponding *_SECRET_ARN variables.
Actual behavior
- Primary WIF authentication succeeds.
- Telemetry reaches the primary organization.
- The extension does not request the three additional-endpoint secrets from Secrets Manager.
- No delivery attempt is made to the additional metrics, traces, or logs endpoints.
- The secondary organization receives none of this telemetry.
- No configuration error is reported for the ignored variables.
The function role has secretsmanager:GetSecretValue permission for all three secrets, and their
contents have been validated independently.
Reproduction
Configure a Lambda function with:
DD_ORG_UUID=<primary-org-uuid>
DD_ADDITIONAL_ENDPOINTS_SECRET_ARN=<metrics-secret-arn>
DD_APM_ADDITIONAL_ENDPOINTS_SECRET_ARN=<traces-secret-arn>
DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS_SECRET_ARN=<logs-secret-arn>
DD_LOGS_CONFIG_FORCE_USE_HTTP=true
Grant the function role secretsmanager:GetSecretValue for all three secrets, then invoke the
function.
Observe that primary WIF delivery succeeds, but Secrets Manager receives no requests for the
additional-endpoint secrets and no telemetry reaches the additional endpoints.
Replacing the secret-backed variables with their plaintext equivalents causes the extension to
construct the additional endpoint flushers, indicating that endpoint delivery itself is supported.
Dual shipping from Datadog Lambda Extension is documented as supported: https://docs.datadoghq.com/serverless/aws_lambda/configuration/#dual-shipping
But I don't believe that appears to be the case. The Datadog Lambda Extension does not resolve these documented environment variables:
DD_ADDITIONAL_ENDPOINTS_SECRET_ARNDD_APM_ADDITIONAL_ENDPOINTS_SECRET_ARNDD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS_SECRET_ARNThe primary organization uses AWS Workload Identity Federation through
DD_ORG_UUID. The additionalendpoints contain a conventional API key for a secondary organization.
Primary delivery succeeds, but the extension does not retrieve the additional-endpoint secrets or
attempt delivery to the secondary organization.
Environment
Expected behavior
The extension should:
DD_ADDITIONAL_ENDPOINTSDD_APM_ADDITIONAL_ENDPOINTSDD_LOGS_CONFIG_ADDITIONAL_ENDPOINTSThis matches the documented statement that the Lambda Extension automatically retrieves Secrets
Manager values for the corresponding
*_SECRET_ARNvariables.Actual behavior
The function role has
secretsmanager:GetSecretValuepermission for all three secrets, and theircontents have been validated independently.
Reproduction
Configure a Lambda function with:
Grant the function role secretsmanager:GetSecretValue for all three secrets, then invoke the
function.
Observe that primary WIF delivery succeeds, but Secrets Manager receives no requests for the
additional-endpoint secrets and no telemetry reaches the additional endpoints.
Replacing the secret-backed variables with their plaintext equivalents causes the extension to
construct the additional endpoint flushers, indicating that endpoint delivery itself is supported.