otel-demo: upgrade docker-compose integration to 3.0.0

[kylehounslow]

Rebased onto current main. The only upstream change since the branch point was opensearch-project#432 (quickstart docs), which touches none of the eight files here.

What

Upgrades the docker-compose otel-demo integration from 2.2.0 to 3.0.0. Required to stay on the otel-demo adopter list (open-telemetry/opentelemetry-demo#3720). This is a breaking upgrade, not a version bump: 3.0.0 renamed docker-compose.yml to compose.yaml and split the core demo from a full profile.

Anchored to the upstream open-telemetry/opentelemetry-demo 3.0.0 tag, diffed against 2.2.0. Draft, and scoped to docker-compose only. The diff touches only .env, docker-compose.otel-demo.yml, and files under docker-compose/. Nothing under charts/ (Helm), k8s, or AWS CLI/CDK. Those are follow-ups.

Service changes (24 to 20)

• Removed from core: accounting, fraud-detection, kafka, llm, product-reviews. In 3.0 the first three moved to compose.full.yaml; llm and product-reviews are gone.
• Renamed postgresql to astronomy-db with the 3.0 astronomy_user / monitoring_user scheme. product-catalog now reads catalog.products via postgres://astronomy_user:...@astronomy-db/astronomy_db.
• Added telemetry-docs.
• Load generator switched from Locust to k6 (LOAD_GENERATOR_VUS, K6_TARGET_URL replace the LOCUST_* vars).
• Image pins bumped for the services the stack actually runs: flagd v0.16.0, valkey 9.0.4, astronomy-db postgres 18.4. Unused backend/build pins in .env (GRAFANA_IMAGE, JAEGERTRACING_IMAGE, OPENSEARCH_IMAGE, TRACETEST_IMAGE*, COLLECTOR_CONTRIB_IMAGE, java/cpp agent versions) were left untouched. The stack provides its own backends and does not reference them; prune separately if desired.

Stack customizations preserved on every ported service: observability-stack-network, shared logging, container names, memory limits, telemetry pointed at the stack collector (OTEL_EXPORTER_OTLP_ENDPOINT / depends_on: otel-collector), and the stack-specific otel-demo-alerting-rules-monitors-init service.

Files

• .env, docker-compose.otel-demo.yml
• Vendored config refreshed to 3.0.0: docker-compose/opentelemetry-demo/src/flagd/demo.flagd.json, .../postgresql/init.sql, new .../product-catalog/otel-config.yml
• Docker-compose dashboards + alerts for the new service set: docker-compose/opensearch-dashboards/init/dashboard-astronomy-shop.ndjson, .../dashboard-astronomy-service-telemetry.ndjson, docker-compose/prometheus/rules-otel-demo/otel-demo-alerts.yml

The astronomy dashboards and otel-demo alert rules are mirrored into charts/observability-stack/files/, which feed the Helm init-dashboards and cortex-rules configmaps. Those copies are deliberately left unchanged here: the opentelemetry-demo subchart is still pinned at 0.40.8, so shipping 3.0-service-set dashboards and alerts into the chart ahead of the subchart bump would leave the Helm path half-updated (astronomy-db and dropped-service references against a pre-3.0 deployment). The chart-side dashboard and alert updates land with the helm 3.0 follow-up that also bumps the subchart.

Validation

Static only, confirmed this pass:

• INCLUDE_COMPOSE_OTEL_DEMO=docker-compose.otel-demo.yml docker compose --env-file .env config -q exits 0, no warnings.
• Rendered config resolves all demo images to ghcr.io/open-telemetry/demo:3.0.0-*; DB_CONNECTION_STRING and K6_TARGET_URL interpolate correctly.
• Dashboard ndjson is valid JSON on every line; edits are limited to dropping product-reviews saved objects and repointing the product-catalog throughput panel.

Not yet validated (needs a live bring-up; gates promotion off the fork):

• Full docker compose up of the 3.0 stack end to end with telemetry flowing. A live 2.2.0 stack runs on the dev host and was deliberately not disturbed; the e2e run needs an isolated compose project.
• The repointed product-catalog throughput panel: the old panel used npgsql byte counters (2.2.0 product-catalog was .NET). 3.0 product-catalog is Go, so the panel now uses db_client_operation_duration_seconds_count. The exact metric name and labels emitted by the 3.0 Go service need live confirmation.
• astronomy-db init seeding and product-catalog reads under the new scheme.
• Dashboard import into OpenSearch Dashboards and alert-rule firing under load.

Follow-ups (out of scope here)

• k8s/helm: bump the opentelemetry-demo subchart to a 3.0-compatible version, apply the astronomy-db rename, and update the charts/observability-stack/files/ dashboard and alert copies to match this PR's docker-compose versions.
• AWS CLI/CDK deployment paths.
• Stack collector config may need new 3.0 scrape targets (ad Prometheus endpoint on AD_PROMETHEUS_PORT, astronomy-db monitoring_user); that lives in the main compose/collector config, not this file.