Skip to content

Node layer never flushes LoggerProvider on invocation completion — logs silently dropped without manual forceFlush() #2537

Description

@misl-smlz

Describe the bug
When using the Node layer with AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler and OTLP logs export, logs emitted via the Logs API are unreliably delivered to the backend. They are only reliably delivered if the application explicitly calls forceFlush() on the logger provider before the handler returns. wrapper.ts constructs the LoggerProvider with a BatchLogRecordProcessor but no code path calls forceFlush()/shutdown() on it after the wrapped handler resolves, nor on a Lambda Extensions API SHUTDOWN event. AWS may freeze the execution environment shortly after the handler returns, before the batch processor's timer fires, dropping any log not yet exported.

Steps to reproduce
Deploy a Lambda with the layer below and AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler, OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=<your OTLP endpoint>, using this handler:

import { logs, SeverityNumber } from '@opentelemetry/api-logs'
import { randomUUID } from 'node:crypto'

export async function handler() {
    const experimentId = randomUUID()

    logs.getLogger('otel-logs').emit({
        body: 'Lambda test log',
        severityNumber: SeverityNumber.INFO,
        severityText: 'INFO',
        attributes: {
            experiment_id: experimentId,
        },
    })

    // works only with this line uncommented
    // await (logs.getLoggerProvider() as any).forceFlush?.()

    return {
        experimentId,
    }
}

Invoke repeatedly with the forceFlush() line commented out: log records are missing in the backend intermittently. Uncommenting the forceFlush() line (or alternatively sleeping ~5s before returning) makes delivery reliable.

What did you expect to see?
Logs emitted during a Lambda invocation are flushed/delivered automatically by the layer before the execution environment is frozen, the same "zero code change" guarantee the layer provides for traces.

What did you see instead?
Log records queued in the (default) BatchLogRecordProcessor are lost intermittently, with no application-level indication of the drop.

What version of collector/language SDK version did you use?
Version: layer opentelemetry-nodejs-0_23_0 (arn:aws:lambda:eu-central-1:184161586896:layer:opentelemetry-nodejs-0_23_0:1)

What language layer did you use?
Config: Node.js

Additional context
Relevant code: wrapper.ts constructs the LoggerProvider with a BatchLogRecordProcessor (no schedule-delay override), but no flush/shutdown call exists in the wrapper for the logger (or tracer/meter) provider.

Suggested fix: flush all registered providers (logger, tracer, meter) automatically once the wrapped handler settles, before returning control to the Lambda runtime, and/or register a Lambda Extensions API SHUTDOWN handler to flush on environment termination so in-flight logs aren't lost if the process is killed abnormally.

Workaround: manually call await (logs.getLoggerProvider() as any).forceFlush?.() at the end of every handler.

Possibly related (metrics, not logs, but similar auto-flush gap): #1949

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions