Extension Error Reading A Body From Connection

[ggoranov-smar]

Hi Team!

We are currently using this lambda extension for tracing and we are experiencing failed lambda invocations with error log from the extension itself:

DD_EXTENSION | ERROR | TRACE_AGENT | handle_proxy | Error extracting request body: Failed to buffer the request body: error reading a body from connection

As far as I understand this is caused by the AWS Lambda runtime freezing or terminating the lambda instance without gracefully waiting for all traces to be processed. The business logic in the lambda function executes without issues so this does not cause any serious issues, just noise in the logs and maybe missing traces.

Is this understanding correct? Do you guys have a suggestion for how to setup the Lambda to avoid these error logs?

[duncanista]

Hey @ggoranov-smar,

Could you tell me which Extension version are you using?
Which runtime (node, python, .net, java, ruby, go, other) are you experiencing the error with?

Which products do you have enabled?

Based on this I might try to understand your problem properly

[ggoranov-smar]

Hey @duncanista,

Here are the requested details:

• Datadog Lambda Extension Version: 92
• Lambda Runtime: Go
• Go Packages:
  • github.com/DataDog/dd-trace-go/contrib/net/http/v2 v2.5.0
  • github.com/DataDog/dd-trace-go/v2 v2.5.0
  • gopkg.in/DataDog/dd-trace-go.v1 v1.74.0 -> This one I just noticed. I should replace it with the one above and see if the error persists.

Can you guide me how to get the products enabled? I know we use tracing, metrics and monitors for sure, no log shipping.

[jchrostek-dd]

@ggoranov-smar can you create a support ticket? You can mention serverless-aws team asked you to do so.

[litianningdatadog]

@ggoranov-smar Your understanding is correct. The error is harmless and can be safely ignored. In the next release, we’ll lower the log level to prevent confusion.

[erazemk]

Hey @litianningdatadog, are the errors for sure harmless? We've been seeing them pretty consistently in the past 2 days and never before that. We're using the latest version of the lambda through docker: COPY --from=public.ecr.aws/datadog/lambda-extension:latest /opt/. /opt/.

We're also only seeing them on the latest version, not on some other lambdas with version 65.

[litianningdatadog]

@erazemk have you seen this reported issue recently? The change has been included in v94 release.

To answer your questions:

• Why it is safe to ignore?
  When AWS Lambda freezes or terminates a function instance, the OS immediately closes all TCP sockets. The extension's proxy HTTP handler (handle_proxy) was mid-read on the request body — this read fails with error reading a body from connection and was logged at ERROR level, generating noise with no actionable signal. This is expected Lambda lifecycle behavior: the Lambda function completed successfully and nothing in the extension is broken. The ERROR level implies an extension fault, which misleads operators.

• Why it was never observed in v65?
  We fully migrated away from the Go-based agent in v88. Since the implementations differ between the old and new versions, this could explain the change in behavior.

[erazemk]

We have not seen it recently, since we switched all our :latest tags to :65. If the issue is harmless, then why must it show up as an error? Errors immediately prompt us to investigate, taking away time from actual issues. Also, if there's nothing we can do, then they really should not even show up.

Are you confirming that the errors won't show up in versions 94+?

[Nitjsefni7]

I can confirm that logs are still visible in v94.

[erazemk]

+1, we're still seeing them with the :latest tag.

[erazemk]

@litianningdatadog Looks like there is also a potential nil dereference in the latest version:

I'm not 100% sure, but it looks like they are coming from the DD extension, based on the /tmp/dd path.

CC: @GasperSenk @erazem-kos