[Discussion] Configurable traffic-log sinks; write traffic logs to a file or push them to a log platform

[DinithHerath]

Problem

The traffic log is the only thing the gateway emits that deliberately carries request and
response bodies, and today it has exactly one destination, hard-coded:

// gateway-runtime/policy-engine/internal/analytics/publishers/log.go
out *os.File
...
out: os.Stdout,

Everything unwelcome follows from that single field:

• Bodies land in the container log, so the kubelet writes them to /var/log/pods/… on the node.
• Anyone with kubectl logs on the namespace can read unredacted PII.
• Any node-level log collector already running — a DaemonSet agent, a Splunk Universal
  Forwarder on the host — picks them up and ships them onward automatically, with no
  configuration. On a cluster that already forwards container logs to a SIEM, enabling
  [traffic_logging] silently starts sending request bodies to a long-retention,
  org-wide-searchable index.
• Reading the log requires something co-located (a sidecar, or a DaemonSet with hostPath).
  Several customers are not permitted to run either.
• The line is wrapped by the entrypoint's [pol] stdout prefix, which forces a fragile
  strip-the-prefix parser downstream. A JSON parse failure there previously let an
  unredacted line through to a log store.

This is structural to the stdout path — no downstream configuration fixes it, because the
unredacted copy already exists on disk by the time anything else sees it.

Architecure

Proposed solution

Introduce a sink abstraction behind the existing Log publisher and add two sinks. Both are
config-level, and neither changes default behaviour — absent new config the gateway writes
to stdout exactly as before.

Sink	What it does	What it solves
file	Appends each line to a file on a mounted volume, rotating at a size ceiling	Removes bodies from kubectl logs and from every node-level collector
http	Batches lines as NDJSON and POSTs them to an operator-named endpoint	Removes the need for any co-located collector; nothing is written to disk

They compose: outputs = ["file", "http"] writes both.

Configuration

[traffic_logging]
enabled = true
outputs = ["file", "http"]     # any of stdout|file|http. Default: ["stdout"]
shutdown_timeout = "5s"

[traffic_logging.file]
path = "/var/log/wso2/traffic/traffic.log"
max_size_mb = 100              # one backup kept -> worst case 2 x this. 0 disables rotation.

[traffic_logging.http]
endpoint = "https://splunk.example.com:8088/services/collector/raw?sourcetype=wso2:gateway:traffic"
content_type = "application/x-ndjson"
batch_max_events = 100
batch_max_bytes = 1048576
flush_interval = "5s"
queue_capacity = 10000         # bounded; overflow drops + counts, never blocks
on_queue_full = "drop_new"     # or "drop_oldest"
request_timeout = "10s"
max_retries = 3                # transport errors, 5xx and 429 only
retry_backoff = "1s"           # exponential, jittered

[traffic_logging.http.auth]
type = "header"                # none | bearer | basic | header
name = "Authorization"
value = 'Splunk {{ file "/secrets/gateway-runtime/hec-token" }}'

[traffic_logging.http.tls]
ca_file = ""
cert_file = ""
key_file = ""
insecure_skip_verify = false

The header auth type is not cosmetic: Splunk HEC expects Authorization: Splunk <token>,
which bearer cannot express.

The NDJSON body is accepted as-is by Splunk HEC /raw, Fluent Bit's http input, the
OpenTelemetry Collector, Datadog's log intake, and (via a collector) Elasticsearch/OpenSearch.

Design guarantees

These are the properties the tests exist to defend:

1. Default behaviour is unchanged. No outputs key ⇒ ["stdout"] ⇒ byte-identical output.
2. A sink that cannot initialise refuses process startup. It never silently degrades to
   stdout, which would put bodies back into the container log — the exact disclosure a
   file/http sink is configured to prevent.
3. A runtime sink failure never falls back to stdout. It drops the line and counts it.
4. A sink failure never affects request handling. No error path reaches a client.
5. Write never blocks. It runs on the ALS ingest path; blocking backpressures Envoy.
6. The traffic-log file is 0600 inside a 0700 directory, both established at creation.
7. Auth material never appears in any log line, including on transport errors.
8. Shutdown flushes buffering sinks, bounded by shutdown_timeout.

Note (3): unlike Fluent Bit or the OTel Collector, the HTTP sink's queue is memory-only, so a
receiver outage costs events rather than delaying them. That is the deliberate trade for never
touching disk, and it is why dropped_total exists.

Metrics

Metric	Type	Labels
policy_engine_traffic_log_written_total	Counter	sink
policy_engine_traffic_log_dropped_total	Counter	sink, reason
policy_engine_traffic_log_queue_depth	Gauge	sink
policy_engine_traffic_log_flush_duration_seconds	Histogram	sink
policy_engine_traffic_log_write_errors_total	Counter	sink, code

dropped_total is the series to alert on; the rest exist to diagnose it.

Related change: publisher lifecycle

Moesif.Close() existed but was never called from anywhere — Analytics was constructed
inside newAccessLogServiceServer and never surfaced to main.go, so shutdown stopped the
gRPC server and exited without flushing any publisher. Harmless for unbuffered stdout; it would
lose the HTTP sink's in-flight batch on every restart, rolling update and scale-down.

Added a Closer interface, Analytics.Close(ctx), and wired it into main.go after
alsServer.GracefulStop() so the flush cannot race arriving events.

Scope of change

New

gateway-runtime/policy-engine/internal/analytics/publishers/sink.go
gateway-runtime/policy-engine/internal/analytics/publishers/sink_file.go
gateway-runtime/policy-engine/internal/analytics/publishers/sink_http.go
gateway-runtime/policy-engine/internal/analytics/publishers/sink_factory.go
  + matching _test.go files
gateway-runtime/policy-engine/internal/config/traffic_log_sinks_test.go

Modified

gateway-runtime/policy-engine/internal/analytics/publishers/log.go        # out *os.File -> sinks []Sink
gateway-runtime/policy-engine/internal/analytics/publishers/publisher.go  # + Closer
gateway-runtime/policy-engine/internal/analytics/publishers/moesif.go     # Close() -> Close(ctx) error
gateway-runtime/policy-engine/internal/analytics/analytics.go             # + Analytics.Close
gateway-runtime/policy-engine/internal/utils/access_logger_server.go      # surface *Analytics
gateway-runtime/policy-engine/cmd/policy-engine/main.go                   # flush on shutdown
gateway-runtime/policy-engine/internal/config/config.go                   # config + fail-closed validation
gateway-runtime/policy-engine/internal/metrics/metrics.go                 # 5 metrics
kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-config.yaml
kubernetes/helm/gateway-helm-chart/templates/gateway/gateway-runtime/deployment.yaml
kubernetes/helm/gateway-helm-chart/values.yaml

No new Go dependencies. Rotation is ~40 lines of stdlib rather than a logging library.

Helm

• traffic_logging.file / .http sub-tables rendered into config.toml.
• A writable emptyDir for the file sink, auto-enabled when outputs contains file so a
  path cannot be configured with nowhere to write it. Mounted at the parent of the log
  directory, because the kubelet creates a mount point 0777 and a volume mount cannot set a
  directory mode — mounting one level up lets the sink create the leaf at 0700.
• gateway.config_toml now renders before the first [section] instead of being appended.
  TOML cannot return to the root document once a table is open, so a root-level key written
  after the generated sections silently became a member of [mcp]. This also makes policy
  systemParameters (${config.<key>}) settable for the first time.

Known gaps / follow-ups

• ErrorUnused on the config decoder. Unknown keys are silently discarded today, so a
  typo'd pth = … is a no-op. The sink validation catches the cases that matter here
  (unknown sink name, unusable sink), but this remains a trap for every other config key.
  Deserves its own PR — turning it on will break any deployment carrying a stale key.
• spill_to_file when the HTTP receiver is down. The right answer to the durability
  trade, deferred because it reintroduces PII at rest and deserves its own decision.
• Syslog/TCP and gRPC/OTLP sinks. Additive behind the Sink interface; HTTP covers every
  receiver we have needed so far.
• [collector].ignore_path_prefixes does not match the real health-probe path
  (/_gateway-health/ready), so probes dominate traffic-log volume. Small, unrelated fix.

Out of scope

• Redaction itself. These sinks change where the log goes, not what is in it; an in-process
  redaction step composes cleanly with both.
• Application logs ([rtr], [pol], [pye]) — still stdout. At logging.level = "debug"
  those can also contain payload fragments; moving the traffic log does not move those.
• Envoy's own access log, and other analytics publishers.

Related Issues

#3201