diff --git a/docs/o11y.md b/docs/o11y.md
new file mode 100644
index 00000000..c33123a1
--- /dev/null
+++ b/docs/o11y.md
@@ -0,0 +1,792 @@
+# First context, made at stats-lake:
+
+@decocms/start Observability — Investigation & Implementation Brief
+
+Goal
+
+Make @decocms/start’s instrumentWorker() produce rich, well-shaped OTel telemetry that lands in our existing ClickStack instance (hyperdx.clickhouse.cloud) backed by stats-lake ClickHouse. The transport is already solved (CF Destinations → deco-otel-ingest Worker → otel_traces / otel_logs). This brief is about what the framework should emit.
+
+Context — what’s already built
+
+casaevideo-tanstack (CF Worker)
+  observability.destinations in wrangler.jsonc
+        │  OTLP/HTTP JSON
+        ▼
+deco-otel-ingest  (https://deco-otel-ingest.deco-cx.workers.dev)
+  maps OTLP → ClickHouse clickhouseexporter schema
+  redacts PII (cookie, authorization, x-vtex-*)
+        ▼
+stats-lake ClickHouse
+  default.otel_traces / default.otel_logs (30-day TTL)
+        ▼
+hyperdx.clickhouse.cloud (ClickStack UI)
+End-to-end is verified. Synthetic spans + logs ingest correctly, render in the ClickStack UI under service:deco-otel-ingest-smoke.
+
+Repos involved:
+
+/Users/fernandofrizzatti/development/workspace/decocms/deco-start/ — this brief is about this repo
+/Users/fernandofrizzatti/conductor/workspaces/stats-lake/calgary/ — ingest Worker + DDL (Phase 1, PR https://github.com/decocms/stats-lake/pull/1)
+/Users/fernandofrizzatti/development/workspace/decocms/casaevideo-storefront/ — first site to wire (Phase 3)
+/Users/fernandofrizzatti/development/workspace/decocms/apps/ — @decocms/apps, has createInstrumentedFetch used by sites
+What works today in @decocms/start v5.0
+
+instrumentWorker(handler, { serviceName }) — wraps a Worker; configures structured JSON logger + per-span attribute floor via configureTracer() bridge onto @opentelemetry/api global tracer
+withTracing("name", fn, attrs) — public API, called from workerEntry.ts:981 around every request as deco.http.request
+defaultLoggerAdapter — writes { level, msg, timestamp, ...attrs } to console.*, which CF Destinations captures as OTLP logs
+AE meter wiring via DECO_METRICS binding (existing, unrelated to the OTLP path)
+Single dependency: @opentelemetry/api ^1.9.1. No SDK, no exporters. The transport is entirely CF Destinations.
+
+Confirmed gaps (ranked by impact)
+
+1. service.version is never set from CF_VERSION_METADATA
+instrumentWorker() reads decoRuntimeVersion option (defaults to hardcoded "5.0.0") and stamps it as deco.runtime.version
+It does NOT read env.CF_VERSION_METADATA?.id to set the OTel-standard service.version
+Why this matters: dashboards correlate errors / latency with specific deployments via service.version. Without it, “did the regression land in deployment X?” is unanswerable
+Fix: in bootObservability(), read env.CF_VERSION_METADATA?.id and stamp it as service.version in spanAttributeFloor AND in the logger floor. Files: src/sdk/otel.ts:168-214
+2. Per-span attribute floor only covers framework-created spans, not CF auto-instrumented ones
+spanAttributeFloor (deco.runtime.version, deployment.environment) is applied inside the configureTracer bridge in otel.ts:137-149 — only spans created through withTracing() get it
+CF runtime auto-instrumented spans (fetch/KV/R2/DO) come out without these attributes
+Why this matters: ~90% of spans in a typical request come from CF auto-instrumentation. Without the floor, dashboards can’t filter those by environment or runtime version
+Fix options:
+(a) Move stamping into ingest Worker: if a resource attribute is missing, fill from the request’s known service. Requires the ingest Worker to know per-service defaults — leaks coupling.
+(b) Accept the gap. Filter dashboards by service.name only (resource attribute, always present). deco.runtime.version becomes “best-effort for framework spans”
+(c) Use CF Tail Workers to enrich spans post-hoc (heavy)
+Recommendation: (b) for now, document explicitly. Revisit if a real dashboard need surfaces.
+3. Sampling: head-only via CF Destinations, augmented by a tail worker (RESOLVED)
+CF Destinations only supports head_sampling_rate — decided at trace start, before status is known. The original goal of "100% errors + 1% baseline" is now met by a three-channel model, see docs/observability.md → "Error capture — three-channel model". Summary:
+  - CF Destinations carries 1% sampled traces + 1% sampled info/warn logs (cheap bulk volume).
+  - Direct-POST in-worker carries 100% of framework logger.error(...) and 100% of OTel metrics (CF Destinations can't carry either).
+  - Cloudflare Tail Worker (deco-otel-tail, Strategy B) carries 100% of uncaught throws, exceededCpu/Memory outcomes, and raw console.error from third-party SDKs.
+The tail worker lives in decocms/stats-lake/ingestion/otel-tail. Producer-side wiring is one line in wrangler.jsonc: `tail_consumers: [{ service: "deco-otel-tail" }]`. See MIGRATION_TOOLING_PLAN.md D-8 for the decision record.
+4. Logger output is JSON-in-stringValue, not native OTel log attributes
+defaultLoggerAdapter does console.log(JSON.stringify({ level, msg, timestamp, ...attrs }))
+CF Destinations wraps that into an OTLP LogRecord with body.stringValue = "<the JSON>". The attrs object isn’t exposed as native log attributes
+Our ingest Worker maps body.stringValue → otel_logs.Body verbatim. Queries need JSONExtract(Body, 'level', 'String') to filter by level
+Why this matters: ClickStack UI auto-detects severity from body.stringValue substring matching, so “INFO” / “ERROR” show up correctly (confirmed empirically). But filtering by custom attributes (requestId, userId) requires JSON extraction at query time
+Fix options:
+(a) Have the logger emit severityText and severityNumber to a separate sidecar that’s understood by CF Destinations. Looks hard — CF only sees console output
+(b) Accept the JSON-in-body shape, document the JSONExtract pattern for queries
+(c) Change the logger to log key-value pairs as separate console.log calls, but that breaks correlation
+Recommendation: (b). Document with example queries in @decocms/start observability README.
+Open investigation points (verify empirically with casaevideo)
+
+What does CF Destinations actually emit for resource attributes? I assumed service.name = Worker name from wrangler.jsonc. Confirm by reading from otel_traces.ResourceAttributes after the first casaevideo request lands. Also check: cloud.provider, cloud.platform, faas.name, cloudflare.script_version.id.
+
+What does withTracing() in workerEntry.ts:981 actually emit? Confirm deco.http.request spans show up with http.method + url.path and inherit the spanAttributeFloor. Cross-check that the framework span and the CF root span are properly parent-child related (or whether they’re siblings — depends on how CF auto-instrumentation handles user spans).
+
+createInstrumentedFetch("vtex") from @decocms/apps — does it use withTracing from @decocms/start, or its own tracer? Either way, do VTEX calls show up as child spans of the request root in ClickStack? Trace propagation (W3C traceparent header) to VTEX would be a nice future win.
+
+TanStack-specific spans — server entry render, data fetch, route resolution. Are there withTracing() calls in the framework / @tanstack/react-start/server-entry that produce meaningful spans? If not, add them at strategic points.
+
+Error reporting — when a request throws, does withTracing() set setError on the span (workerEntry.ts:996 area)? Verify the span gets StatusCode = STATUS_CODE_ERROR and a StatusMessage. If not, fix in src/middleware/observability.ts:77-95.
+
+CF_VERSION_METADATA binding — is it declared in casaevideo’s wrangler.jsonc? (Yes, confirmed earlier.) Does instrumentWorker() receive it via env? Should: yes, since the binding is on env.
+
+Concrete deliverables for the @decocms/start PR
+
+In priority order:
+
+src/sdk/otel.ts — read env.CF_VERSION_METADATA?.id in bootObservability(). Stamp as service.version in both spanAttributeFloor and the logger floor. Boot-time log should include version. ~10 LOC.
+
+src/sdk/otel.ts — also stamp service.name in both floors (so framework-created spans carry it, even though the resource attribute already has it from CF). Defensive, lets ClickStack dashboards filter by either path. ~3 LOC.
+
+src/middleware/observability.ts:77-95 — confirm withTracing() sets span status to STATUS_CODE_ERROR on thrown exception via the bridge’s setError. Add a test.
+
+README.md / docs/observability.md — document the architecture: instrumentWorker() + CF Destinations + ingest Worker. Show the recommended wrangler.jsonc shape. Show the ClickStack query patterns (JSONExtract for log filtering).
+
+Optional: add withTracing() calls at strategic framework points — render entry, loader resolution, data fetching. Only if the empirical investigation (point 2 above) shows current coverage is too thin.
+
+Out of scope for this PR
+
+Reviving the OTLP SDK exporters that lived at v4.6.0 — not needed; CF Destinations replaces that path
+Tail-on-error sampling at the framework layer — now lives in the deco-otel-tail Cloudflare Tail Worker (decocms/stats-lake), not in @decocms/start. See docs/observability.md and D-8.
+@microlabs/otel-cf-workers integration — rejected; CF Destinations + thin framework attributes is sufficient
+PII redaction — handled at ingest Worker, no per-site code needed
+How to validate the framework changes
+
+Branch off @decocms/start main, apply the 4 changes above
+pnpm link it into casaevideo-storefront
+Add destinations block to casaevideo/wrangler.jsonc pointing at https://deco-otel-ingest.deco-cx.workers.dev
+Wrap casaevideo/src/worker-entry.ts final export with instrumentWorker(handler, { serviceName: "casaevideo-tanstack" })
+wrangler deploy (preview URL is fine; routes are still commented out in production)
+Hit the preview URL a few times
+ClickHouse: verify service.version lands in otel_traces.ResourceAttributes['service.version'] AND otel_traces.SpanAttributes['service.version']
+ClickStack UI: filter service:casaevideo-tanstack, confirm spans show with sensible names, durations, and errors-as-errors
+Reference files
+
+src/sdk/otel.ts — instrumentWorker(), bootObservability(), attribute floor wiring
+src/sdk/logger.ts — defaultLoggerAdapter, severity routing
+src/sdk/workerEntry.ts:968-996 — createDecoWorkerEntry, RequestContext.run, root withTracing("deco.http.request", ...)
+src/middleware/observability.ts:77-95 — withTracing() implementation, error path
+package.json — only @opentelemetry/api ^1.9.1, no SDK
+src/sdk/otelAdapters/clickhouseCollector.ts — documented stub, unchanged; leave for future use
+
+
+# Second Context: made at deco-start
+
+Let's plan for all of those, straight away. we need to be able to track everything we need; cache, performance, errors, etc; 
+Separate what goes into this repo as a PLAN, and separate what goes into decocms/apps as Plan prompt for investigation and stuff; please. 
+Let's implement what goes into this repo, and I will send the prompt to apps in another thread.
+Observability completeness — full coverage for cache, performance, errors
+
+Context
+
+The brief at /tmp/attachments/pasted_text_2026-05-15_15-42-09.txt covers the next iteration of @decocms/start observability. Transport is solved end-to-end:
+
+site Worker → CF Destinations (OTLP/HTTP JSON)
+  → deco-otel-ingest (Worker)
+  → stats-lake ClickHouse (otel_traces / otel_logs)
+  → hyperdx.clickhouse.cloud (ClickStack UI)
+The framework already has scaffolding in place — instrumentWorker, withTracing, AE meter, structured logger, log severity floor — but the signal it emits is incomplete:
+
+Identity fields missing: service.version, service.name aren’t on the per-span/per-log floor.
+Error spans don’t carry STATUS_CODE_ERROR — they look successful with an exception attached.
+Two CMS hotspots are already instrumented (resolveDecoPage → deco.cms.resolvePage, runSingleSectionLoader → deco.section.loader). The rest of the framework is dark to ClickStack.
+No cache decision spans / metrics at the application layer (CF Cache API spans show storage ops but not HIT/MISS/profile).
+Admin protocol handlers (handleMeta, handleDecofile, handleRender, handleInvoke) emit no spans, so admin traffic mixes with customer traffic in the trace tree.
+Deferred section loads (loadDeferredSection server fn) have no span — on-scroll latency is invisible.
+logRequest doesn’t emit traceId, so logs and traces can’t be cross-referenced in ClickStack.
+No W3C trace-context propagation outbound — upstream services (VTEX, Shopify) can’t link their spans to ours.
+No docs/observability.md.
+Goal: ship a single coherent PR that makes the framework emit rich, queryable telemetry covering cache decisions, request shape, admin protocol, errors, and trace correlation — without changing the public API. The companion work in @decocms/apps (createInstrumentedFetch, VTEX-specific spans) is described as a standalone brief at the bottom of this document for the user to send to a separate agent thread.
+
+Plan A — Changes in @decocms/start (this repo)
+
+A1. Identity on the floor (src/core/sdk/otel.ts:168-214)
+bootObservability():
+
+Read env.CF_VERSION_METADATA?.id; if present, add service.version to floor.
+Add service.name (already computed at line 171) to floor.
+Update the boot breadcrumb (otel.ts:208-213) to log version when known.
+Resulting shape:
+
+const floor: Record<string, string> = {
+  "service.name": serviceName,
+  "deco.runtime.version": decoRuntimeVersion,
+  "deployment.environment": deploymentEnvironment,
+};
+const versionId = (env.CF_VERSION_METADATA as { id?: string } | undefined)?.id;
+if (versionId) floor["service.version"] = versionId;
+if (opts.decoAppsVersion) floor["deco.apps.version"] = opts.decoAppsVersion;
+Both spanAttributeFloor = floor and setLoggerAttributeFloor(floor) already exist (lines 185, 191) — no plumbing needed downstream.
+
+A2. Span-status on errors (src/core/sdk/otel.ts:137-149)
+In the configureTracer bridge, extend setError to also call span.setStatus({ code: SpanStatusCode.ERROR, message }). Import SpanStatusCode from @opentelemetry/api.
+
+setError: (error) => {
+  const message = error instanceof Error ? error.message : String(error);
+  if (error instanceof Error) span.recordException(error);
+  span.setStatus({ code: SpanStatusCode.ERROR, message });
+},
+withTracing() at src/core/sdk/observability.ts:101-105 already calls span.setError?.(error) in its catch — no change needed there.
+
+A3. Extend Span interface to expose trace context (src/core/sdk/observability.ts:43-47)
+Required by A7 (logger traceId) and A8 (traceparent propagation).
+
+export interface Span {
+  end(): void;
+  setError?(error: unknown): void;
+  setAttribute?(key: string, value: string | number | boolean): void;
+  spanContext?(): { traceId: string; spanId: string; traceFlags: number };
+}
+In the bridge (src/core/sdk/otel.ts:140-148), expose spanContext by delegating to the OTel API span:
+
+spanContext: () => {
+  const ctx = span.spanContext();
+  return { traceId: ctx.traceId, spanId: ctx.spanId, traceFlags: ctx.traceFlags };
+},
+This is opt-in via the optional method — adapters that don’t implement it still work; helpers that need it (logger, traceparent) just no-op gracefully.
+
+A4. Cache decision spans + metrics (src/tanstack/sdk/workerEntry.ts)
+Two cache stores, two lookup paths. Wrap each cache.match call in withTracing("deco.cache.lookup", ...) and emit recordCacheMetric(...) at the decision points already in the code.
+
+Server-function cache (POST _serverFn): workerEntry.ts:1246-1303
+
+Wrap line 1247-1251 lookup: span deco.cache.lookup, attrs { "cache.profile": "listing", "cache.kind": "serverFn" }.
+At each decision point (HIT line 1254-1264, STALE-HIT line 1267-1292, MISS line 1303+), call recordCacheMetric(hit, profile) and set span attribute cache.decision.
+Wrap cache.put at line 1334 with deco.cache.store, attrs { "cache.profile": "listing", "cache.kind": "serverFn" }.
+HTML cache (GET): workerEntry.ts:1484-1595
+
+Wrap line 1486-1489 lookup: span deco.cache.lookup, attrs { "cache.profile": profile, "cache.kind": "html" }.
+At each decision point (HIT line 1492-1501, STALE-HIT line 1502-1511, MISS line 1514+), call recordCacheMetric(...) with the right profile, set cache.decision.
+Wrap cache.put (helper storeInCache line 1446-1458) with deco.cache.store, attrs { "cache.profile": profile, "cache.kind": "html" }.
+Attributes on every cache span: cache.profile, cache.kind, cache.decision (HIT/STALE-HIT/MISS), cache.age_seconds (when known).
+
+A5. Section loader batch wrapper (src/core/cms/sectionLoaders.ts:253)
+Wrap runSectionLoaders body in withTracing("deco.section.loaders.batch", ..., { "section.count": sections.length }) so the existing per-section spans (deco.section.loader at sectionLoaders.ts:332-334) nest cleanly under one batch parent. Makes traces readable when a page has 15+ sections.
+
+A6. Admin protocol handler spans
+The map agent found 5 admin handlers. Wrap each at the call site in src/tanstack/sdk/workerEntry.ts (inside tryAdminRoute) — not inside the core handler — so the handlers stay framework-agnostic and reusable from src/next/. Names + attrs:
+
+Span name	Call site	Attributes
+deco.admin.meta	wrapper around handleMeta	(none — handler is sync, fast)
+deco.admin.decofile.read	wrapper around handleDecofileRead	(none)
+deco.admin.decofile.reload	wrapper around handleDecofileReload	decofile.block_count (set after reload)
+deco.admin.render	wrapper around handleRender	cms.component (from pathComponent), cms.resolve_chain (bool)
+deco.admin.invoke	wrapper around handleInvoke	invoke.key, invoke.kind (loader
+For the sync handlers (handleMeta, handleDecofileRead), use a thin async wrapper inside the withTracing callback. Negligible cost; uniform span shape.
+
+A7. Logger trace correlation (src/core/sdk/observability.ts:189-216)
+Inside logRequest, before the JSON-serialize step, pull the active span’s spanContext() (via the interface from A3) and add traceId / spanId to the payload. No-op when no active span. Field names match OTel conventions so ClickStack’s auto-correlation panel works (trace_id, span_id).
+
+Same enrichment for the logger floor in setLoggerAttributeFloor — actually skip that: the floor is per-boot, not per-request. Instead, plumb traceId inside the emit() function in src/core/sdk/logger.ts:228-250 so every log line (not just logRequest) carries trace correlation. Cheap: one getActiveSpan()?.spanContext?.() call per line, no allocation when no active span.
+
+A8. W3C trace-context propagation helper (src/core/sdk/http.ts or new src/core/sdk/traceContext.ts)
+Add a small helper:
+
+export function injectTraceContext(headers: Headers): void {
+  const ctx = getActiveSpan()?.spanContext?.();
+  if (!ctx) return;
+  const flags = ctx.traceFlags.toString(16).padStart(2, "0");
+  headers.set("traceparent", `00-${ctx.traceId}-${ctx.spanId}-${flags}`);
+}
+Frame as a public helper exported from @decocms/start/sdk/observability so apps repo (@decocms/apps) can call it from createInstrumentedFetch. No new dependency — we already have @opentelemetry/api.
+
+Document on the export: “Call before sending an outbound fetch from within a withTracing scope to enable upstream trace correlation.”
+
+A9. Deferred section span (src/tanstack/routes/cmsRoute.ts:167-214)
+Wrap the internal resolveDeferredSectionPure call at line 197:
+
+const result = await withTracing(
+  "deco.section.deferred.load",
+  () => resolveDeferredSectionPure(pagePath, component, matcherCtx, opts),
+  { "section.name": component, "section.index": index, "page.path": pagePath },
+);
+Lives inside the TanStack server fn handler — the span naturally parents under whatever withTracing context CF threads through (or stands alone if not).
+
+A10. Tests
+src/core/sdk/otel.test.ts:
+
+service.version lands on the logger floor when env.CF_VERSION_METADATA = { id: "abc" }.
+service.name lands on the logger floor (from serviceName option and DECO_SITE_NAME fallback).
+The bridge’s setError sets SpanStatusCode.ERROR — assert via a vi.spyOn(trace, "getTracer") stub that returns a recording span.
+spanContext() round-trips traceId/spanId from a stub OTel span.
+src/core/sdk/observability.test.ts (new file):
+
+injectTraceContext writes a well-formed traceparent when an active span exists, no-ops otherwise.
+withTracing nests getActiveSpan correctly across await.
+src/core/cms/sectionLoaders.test.ts:
+
+New runSectionLoaders batch test: with a stub tracer, verify one deco.section.loaders.batch parent appears with section.count set.
+src/tanstack/sdk/workerEntry.test.ts (new file or extend existing):
+
+Stub caches.open + a fake cache that returns a fresh Response — verify span deco.cache.lookup is emitted with cache.decision = "HIT" and recordCacheMetric(true, profile) is called.
+Same for MISS.
+Admin route hits emit the right span names.
+src/core/sdk/logger.test.ts:
+
+A log emitted inside withTracing carries trace_id / span_id fields.
+A log emitted outside any span doesn’t carry them.
+A11. Documentation — docs/observability.md (new) + README link
+Single page covering:
+
+Architecture — site Worker → CF Destinations → deco-otel-ingest → ClickHouse → ClickStack. Diagram in plain text.
+What’s instrumented — table of every span name and its attributes:
+Span	Source	Key attributes
+deco.http.request	workerEntry root	http.method, url.path
+deco.cms.resolvePage	resolveDecoPage	deco.route
+deco.section.loaders.batch	runSectionLoaders	section.count
+deco.section.loader	runSingleSectionLoader	deco.section
+deco.section.deferred.load	loadDeferredSection	section.name, section.index
+deco.cache.lookup	edge cache match	cache.profile, cache.kind, cache.decision, cache.age_seconds
+deco.cache.store	edge cache put	cache.profile, cache.kind
+deco.admin.meta / decofile.* / render / invoke	admin route wrappers	see table in A6
+What’s measured — table of metric names:
+Metric	Source	Labels
+http_requests_total (counter)	workerEntry	method, path, status
+http_request_duration_ms (histogram)	workerEntry	method, path, status
+http_request_errors_total (counter, 5xx)	workerEntry	method, path, status
+cache_hit_total / cache_miss_total (counter)	workerEntry	profile
+resolve_duration_ms (histogram)	resolveDecoPage	(none)
+Identity stamped on every span and log: service.name, service.version (when CF_VERSION_METADATA bound), deco.runtime.version, deployment.environment, deco.apps.version (optional). Each log line additionally carries trace_id / span_id when emitted inside an active span.
+
+Required wrangler.jsonc — lift the JSDoc example from src/core/sdk/otel.ts:19-30, plus the observability.logs.destinations / observability.traces.destinations blocks pointing at https://deco-otel-ingest.deco-cx.workers.dev. Include version_metadata binding and analytics_engine_datasets.
+
+Logger output shape gap — JSON-in-stringValue for log records. Show one ClickHouse query:
+
+SELECT JSONExtract(Body, 'level', 'String') as level, count()
+FROM otel_logs
+WHERE ServiceName = 'my-store' AND Timestamp > now() - INTERVAL 1 HOUR
+GROUP BY level
+Cross-referencing logs and traces in ClickStack — filter logs by trace_id to jump from a log line to its trace.
+
+Outbound trace propagation — short note on injectTraceContext(headers) for sites/apps that issue outbound fetches.
+
+Not in scope — tail-on-error sampling at the framework layer (now implemented as the deco-otel-tail Cloudflare Tail Worker in decocms/stats-lake; see D-8), VTEX/Shopify-specific spans (lives in @decocms/apps).
+
+Add a link line under the “Documentation” section of README.md.
+
+Critical files
+src/core/sdk/otel.ts (A1, A2, A3)
+src/core/sdk/observability.ts (A3, A7, A8)
+src/core/sdk/logger.ts (A7)
+src/core/sdk/http.ts or new src/core/sdk/traceContext.ts (A8)
+src/core/cms/sectionLoaders.ts (A5)
+src/tanstack/sdk/workerEntry.ts (A4, A6)
+src/tanstack/routes/cmsRoute.ts (A9)
+src/core/sdk/otel.test.ts, src/core/sdk/observability.test.ts (new), src/core/sdk/logger.test.ts, src/core/cms/sectionLoaders.test.ts, src/tanstack/sdk/workerEntry.test.ts (new) (A10)
+docs/observability.md (new), README.md (link) (A11)
+Existing utilities to reuse
+withTracing (src/core/sdk/observability.ts:88-106) — span lifecycle.
+recordCacheMetric / recordRequestMetric (src/core/sdk/observability.ts:145-167) — existing meter helpers; just wire them at the cache decision points.
+MetricNames (src/core/sdk/observability.ts:131-139) — already declares RESOLVE_DURATION_MS, FETCH_DURATION_MS, CACHE_HIT, CACHE_MISS. No new metric names needed.
+getActiveSpan (src/core/sdk/observability.ts:79-81) — used in A7 (logger) and A8 (traceparent helper).
+setLoggerAttributeFloor (src/core/sdk/logger.ts:118-120) — already wired from bootObservability.
+defaultLoggerAdapter (src/core/sdk/logger.ts:49-82) — wraps console.*, picked up by CF Destinations.
+Verification
+bun run typecheck — clean.
+bun test — all new + existing tests green.
+bun run build — package builds clean; dist/ exports unchanged.
+Manual smoke on a preview deployment (after the user merges):
+Deploy site Worker with version_metadata binding + observability.{logs,traces}.destinations pointing at deco-otel-ingest.
+Hit a PDP, a PLP, a cart route (so we exercise multiple cache profiles).
+ClickHouse query confirms ResourceAttributes['service.version'], SpanAttributes['service.version'], and SpanAttributes['cache.profile'] are populated.
+Force a 5xx (invalid CMS resolve), confirm the framework span shows STATUS_CODE_ERROR and the exception message in ClickStack.
+Filter ClickStack logs by trace_id from a known request, confirm logs and traces correlate.
+Release channel
+All changes are additive — no public API removed or renamed. The only behavioral change for sites that don’t have CF_VERSION_METADATA bound is that they get richer spans and per-line trace_id on logs; no breaking surface. Target main → @decocms/start@latest. Confirm with user before opening the PR.
+
+Plan B — Brief for @decocms/apps (paste into a separate agent thread)
+
+The user will copy the section below verbatim into a new agent conversation rooted at /Users/fernandofrizzatti/development/workspace/decocms/apps/. Self-contained, no shared state with this thread.
+
+Brief: instrument @decocms/apps for end-to-end observability
+Context
+
+The deco platform pipes telemetry from site Workers → CF Destinations → deco-otel-ingest (https://deco-otel-ingest.deco-cx.workers.dev) → stats-lake ClickHouse → ClickStack UI (hyperdx.clickhouse.cloud). The framework (@decocms/start) emits service identity (service.name, service.version from CF_VERSION_METADATA), a root deco.http.request span, per-section spans, cache decision spans, admin protocol spans, and W3C traceparent-injection helper injectTraceContext(headers) exported from @decocms/start/sdk/observability.
+
+What’s missing is per-app commerce instrumentation. VTEX, Shopify, Nuvemshop, etc. all issue many outbound fetches per request — search, catalog, OMS, intelligent-search, checkout. Today these are CF-auto-instrumented as flat fetch spans with no semantic grouping. Operators can’t answer “is intelligent-search slow today?” without grep-style URL filtering.
+
+Investigation tasks
+
+Audit createInstrumentedFetch — find its current implementation in this repo (likely under vtex/utils/ or utils/). For each:
+Does it use withTracing from @decocms/start? If yes, what’s the span name shape?
+Are spans named per-endpoint (e.g. vtex.intelligent-search.product_search) or just per-client (vtex.fetch)?
+Are HTTP-level attributes set (http.method, http.url, http.status_code, http.response.body_size)?
+Does it set STATUS_CODE_ERROR on 5xx responses or on thrown errors?
+Is response timing recorded as a histogram metric (e.g. vtex_request_duration_ms with endpoint label)?
+W3C trace propagation — does createInstrumentedFetch inject traceparent headers? If not, import injectTraceContext from @decocms/start/sdk/observability and call it before each outbound fetch. Confirms downstream correlation if upstream is OTel-aware.
+Per-app coverage matrix — list every commerce app (VTEX, Shopify, Wake, Nuvemshop, …) and whether each uses createInstrumentedFetch consistently across loaders, actions, and hooks. Identify any direct fetch() calls that bypass instrumentation.
+Cross-app pattern — should there be a shared createInstrumentedFetch(provider: string) factory that emits spans {provider}.{endpoint}, with provider as a label on the per-app histogram? Today each app likely has its own.
+GraphQL — VTEX/Shopify use GraphQL. Should the span attributes include graphql.operation.name so dashboards can group by GraphQL operation rather than POST endpoint?
+Deliverables (after investigation)
+
+A short report (in the conversation, no file) summarizing the current state of fetch instrumentation across apps.
+A unified PR (or per-app PRs) that:
+Standardizes createInstrumentedFetch to emit a span per outbound call with {provider}.{operation_or_endpoint} shape, full HTTP attrs, error status on failures, and traceparent propagation.
+Adds a per-provider duration histogram (vtex_request_duration_ms, shopify_request_duration_ms) with endpoint / operation labels.
+Wraps GraphQL clients to expose graphql.operation.name on the span.
+Audits direct fetch() calls and converts them to createInstrumentedFetch.
+Local smoke verification (run a site against a real VTEX account, confirm spans appear in ClickStack with the expected shape).
+Constraints
+
+Don’t add new dependencies. @opentelemetry/api is already pulled in transitively via @decocms/start.
+Don’t change public function signatures — wrap implementations, don’t break consumers.
+PII redaction is handled at the ingest Worker; no per-app code needed for that.
+Don’t introduce a custom OTLP exporter — telemetry rides the CF Destinations pipe.
+Reference files (in @decocms/start, after the upcoming PR lands)
+
+src/core/sdk/observability.ts — withTracing, injectTraceContext, getActiveSpan.
+src/core/sdk/logger.ts — logger.info|warn|error (logs carry trace_id automatically).
+docs/observability.md — full pipeline + query patterns.
+Approved
+All 11 plan items complete. Verification clean: bun run typecheck, all 553 tests pass, bun run build succeeds, biome lint clean on touched files, tier boundaries clean.
+
+Summary of changes (this repo)
+
+Identity & errors — service.name and service.version (from env.CF_VERSION_METADATA.id) now stamp on every framework span and every log line via the attribute floor. Framework spans correctly carry SpanStatusCode.ERROR with the exception message when withTracing() rethrows. Span interface gained an optional spanContext() so logger/propagation helpers can read trace IDs.
+
+New spans — deco.section.loaders.batch, deco.section.deferred.load, deco.cache.lookup (×2 paths: SFn + HTML), deco.cache.store (×2), deco.admin.{meta,decofile.read,decofile.reload,render,invoke}. Wired at workerEntry callsites and TanStack admin routes.
+
+Cache metrics — extended recordCacheMetric(hit, profile, decision) with a CacheDecision union (“HIT” | “STALE-HIT” | “STALE-ERROR” | “MISS” | “BYPASS”). Wired at every HIT / STALE-HIT / MISS / STALE-ERROR decision point in both cache paths.
+
+Trace correlation — every log emitted inside a withTracing() scope now carries trace_id / span_id for cross-referencing logs and traces in ClickStack. logRequest does the same.
+
+injectTraceContext(headers) — new public helper that writes a W3C traceparent header from the active span. Exported from @decocms/start/sdk/observability (and /middleware) so @decocms/apps’s createInstrumentedFetch can propagate context to upstream services.
+
+Tests — 11 new test cases covering: identity floor, error status, spanContext round-trip, batch-loader span, deferred span, trace-id in logs, injectTraceContext happy/no-op paths, getActiveSpan nesting.
+
+Docs — new docs/observability.md covering architecture, instrumented spans, metrics, identity floor, wrangler.jsonc shape, ClickHouse JSONExtract query patterns, log-to-trace correlation, injectTraceContext usage, sampling notes, and what’s out of scope. README links it under “Documentation”.
+
+Apps brief
+
+The standalone apps brief is in Plan B of /Users/fernandofrizzatti/.claude/plans/system-instruction-you-are-working-shimmying-badger.md — copy that section into a new agent thread rooted at /Users/fernandofrizzatti/development/workspace/decocms/apps/.
+
+PR instructions.md
+The user likes the current state of the code. Ther...
+Create a PR
+PR created: https://github.com/decocms/deco-start/pull/179
+
+The branch is pushed, the commit follows the repo’s conventional-commit style, and the PR body lays out what shipped plus the post-merge smoke steps from the plan.
+
+
+# Third Context: decocms/apps
+https://github.com/decocms/deco-start/pull/179 check that PR, check if that solves some problems, if any
+Exit code 1 accepts at most 1 arg(s), received 3
+PR 179 in decocms/deco-start directly addresses the framework gaps I found. Summary:
+
+injectTraceContext(headers: Headers) is added and exported from @decocms/start/sdk/observability — exactly the symbol the brief assumed exists. No-op when no active span. ✅
+Log↔trace correlation is automatic: any log emitted inside a withTracing(...) scope gets trace_id/span_id stamped via the logger attribute floor. Free win for @decocms/apps once we wrap call sites.
+recordCacheMetric(hit, profile, decision?) extended with decision label (HIT|STALE-HIT|STALE-ERROR|MISS|BYPASS).
+New framework spans added (cache lookup/store, loaders batch, deferred section load, admin handlers). Doesn’t affect us, but means once our outbound fetch spans nest under deco.http.request, the whole tree is contiguous.
+spanContext?() added to Span — adapters that implement it light up trace propagation; adapters that don’t degrade gracefully.
+What PR 179 does not solve (still on the @decocms/apps plate):
+
+createInstrumentedFetch is unchanged — still emits only ${name}.fetch spans, still does not call injectTraceContext (the PR’s doc claims it does, but the diff doesn’t touch src/core/sdk/instrumentedFetch.ts). So either (a) we add injectTraceContext inside our per-app fetch wrappers in this repo, or (b) we open a follow-up upstream PR to wire it into createInstrumentedFetch itself.
+Per-operation span granularity — upstream still produces vtex.fetch. To get vtex.intelligent-search.product_search, we wrap call sites in withTracing(...) inside @decocms/apps.
+GraphQL operationName on spans — needs to happen in our Shopify GraphQL client.
+~27 raw fetch( sites that bypass _fetch need to be routed through it.
+Per-provider duration histograms — need to call getMeter()?.histogramRecord(...) ourselves at the per-app fetch helper.
+Let me write the plan, then I’ll ask a couple of decisions via ExitPlanMode.
+Instrument @decocms/apps for end-to-end observability
+
+Context
+
+The deco platform pipes telemetry from site Workers → CF Destinations → deco-otel-ingest → ClickHouse stats-lake → ClickStack UI. The framework half (@decocms/start) emits a root deco.http.request span, per-section spans, cache decision spans, admin protocol spans, and exposes tracer/meter primitives. The apps half (@decocms/apps — this repo) is the integration layer where outbound commerce fetches happen (VTEX REST, Shopify GraphQL, Resend, Google Fonts, etc.). Today those fetches are essentially invisible to operators — the only span shape produced is vtex.fetch / shopify.fetch via createInstrumentedFetch(name), and most fetches bypass even that.
+
+The goal: every outbound fetch in every app produces a span named {provider}.{operation}, carries full HTTP attributes, sets error status on failures, records a per-provider duration histogram with operation label, and propagates W3C traceparent so any OTel-aware upstream joins the trace.
+
+Upstream dependency: this plan depends on decocms/deco-start#179 being released. That PR adds injectTraceContext, CacheDecision, spanContext?() on the Span interface, and automatic log/trace correlation via the logger attribute floor. Without it, we’d need to vendor a W3C propagator. Once it ships, we just import.
+
+Current state (from exploration)
+
+createInstrumentedFetch is defined in @decocms/start/sdk/instrumentedFetch, not in this repo. Span shape is fixed at ${name}.fetch. No HTTP attributes, no traceparent injection, no per-operation granularity.
+Wiring lives in this repo via setter pattern:
+vtex/client.ts:117 — setVtexFetch(fetchFn) stores into module-level _fetch. Called at startup in setup.ts.
+shopify/client.ts:24 — setShopifyFetch(fetchFn) does the same for Shopify.
+Only 4 paths actually use _fetch:
+vtex/client.ts — vtexFetchResponse:214, vtexCachedFetch:260, intelligentSearch:360
+shopify/utils/graphql.ts:32 — createGraphqlClient.query (when fetchFn arg passed)
+~27 raw fetch( sites bypass instrumentation entirely:
+VTEX actions (6): actions/analytics/sendEvent.ts:54, actions/profile.ts, actions/masterData.ts, …
+VTEX hooks (9): hooks/useCart.ts:77,96,111,131, hooks/useWishlist.ts:34,48,58, hooks/useAutocomplete.ts:49, hooks/useUser.ts:34
+VTEX utils (5): utils/authHelpers.ts:52, utils/proxy.ts:176,383, utils/fetch.ts:91
+Shopify: 1 fallback in utils/graphql.ts when fetchFn is undefined
+Resend: resend/actions/send.ts:35
+Website: website/loaders/fonts/googleFonts.ts:97,103
+Shopify GraphQL client (shopify/utils/graphql.ts) does not surface operationName anywhere — span attribute opportunity.
+No file in this repo currently imports from @decocms/start/sdk/observability.
+Approach
+
+Two layers of change in this repo:
+
+A. A small shared “traced fetch” helper
+Add a single helper that all per-app fetch utilities funnel through. It composes the three pieces we need on every outbound call:
+
+// utils/tracedFetch.ts (new file)
+import { withTracing, injectTraceContext, getMeter, setSpanAttribute } from "@decocms/start/sdk/observability";
+
+export interface TracedFetchOptions {
+  provider: string;            // e.g. "vtex"
+  operation: string;           // e.g. "intelligent-search.product_search"
+  attributes?: Record<string, string | number | boolean>;
+}
+
+export async function tracedFetch(
+  input: Request | URL | string,
+  init: RequestInit | undefined,
+  opts: TracedFetchOptions,
+  underlyingFetch: typeof fetch = fetch,
+): Promise<Response> {
+  const headers = new Headers(init?.headers);
+  injectTraceContext(headers);
+  const url = typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
+  const method = init?.method ?? "GET";
+  const start = performance.now();
+
+  return withTracing(
+    `${opts.provider}.${opts.operation}`,
+    async () => {
+      setSpanAttribute("http.method", method);
+      setSpanAttribute("http.url", redactUrl(url));   // strip query secrets like authToken
+      for (const [k, v] of Object.entries(opts.attributes ?? {})) setSpanAttribute(k, v);
+
+      let res: Response;
+      try {
+        res = await underlyingFetch(input, { ...init, headers });
+      } catch (err) {
+        recordDuration(opts, performance.now() - start, "error");
+        throw err;
+      }
+
+      setSpanAttribute("http.status_code", res.status);
+      const len = res.headers.get("content-length");
+      if (len) setSpanAttribute("http.response.body_size", Number(len));
+      recordDuration(opts, performance.now() - start, String(res.status));
+      return res;
+    },
+  );
+}
+
+function recordDuration(opts: TracedFetchOptions, ms: number, status: string) {
+  getMeter()?.histogramRecord?.("commerce_request_duration_ms", ms, {
+    provider: opts.provider,
+    operation: opts.operation,
+    status,
+  });
+}
+Notes:
+
+Error status on 5xx and on thrown errors is handled by withTracing itself (PR 179 sets SpanStatusCode.ERROR when the wrapped function rethrows). For 5xx responses we don’t throw — we just rely on the http.status_code attribute for filtering, which is the OTel-recommended approach (status codes ≠ span errors unless the client treats them as errors).
+We rely on getMeter()?.histogramRecord?.(...) from PR 179. Single generic histogram commerce_request_duration_ms with provider + operation labels is more useful for cross-provider dashboards than per-provider names; matches how http_request_duration_ms is shaped upstream.
+redactUrl strips known-sensitive query params (authToken, cookie, vtex_session, etc.) consistent with the existing sanitization in vtex/utils/fetch.ts.
+B. Funnel every raw fetch through it
+VTEX (vtex/):
+
+In vtex/client.ts, change vtexFetchResponse, vtexCachedFetch, intelligentSearch to call tracedFetch(url, init, { provider: "vtex", operation }, _fetch) instead of _fetch(url, init). operation is hardcoded per call path:
+vtex.api (catch-all in vtexFetchResponse)
+vtex.intelligent-search.{endpoint} — parsed once from the URL path inside intelligentSearch
+vtex.catalog.*, vtex.checkout.orderForm, etc. as appropriate
+Convert each raw fetch(...) in vtex/actions/, vtex/hooks/, vtex/utils/authHelpers.ts, vtex/utils/proxy.ts, vtex/utils/fetch.ts to either:
+Go through _fetch (preferred — they’re VTEX-account-aware), then through tracedFetch, with an operation derived from the call site (e.g. vtex.checkout.addItems, vtex.events.send, vtex.wishlist.list, …).
+For the proxy paths (utils/proxy.ts) where we forward arbitrary requests, use operation vtex.proxy and set proxy.path as an attribute.
+The hook fetches (hooks/useCart.ts, etc.) currently run on the browser via the invoke proxy → server function path. They still need instrumentation on the server side; if the hooks themselves run on the client they won’t have a span context, and injectTraceContext will no-op. That’s fine. The server-side _fetch call is where instrumentation lives.
+Shopify (shopify/):
+
+shopify/utils/graphql.ts — wrap the POST in tracedFetch with operation: graphqlOperationName(query) ?? "graphql" and set graphql.operation.name as a span attribute. Parse the operation name with a small regex on the GraphQL document (/^\s*(?:query|mutation|subscription)\s+(\w+)/) — no external dep needed.
+Remove the globalThis.fetch fallback path: require that callers route through setShopifyFetch at startup. (Or wrap the fallback in tracedFetch too — pick whichever is least disruptive to existing tests.)
+Resend, Website:
+
+resend/actions/send.ts:35 — wrap in tracedFetch with provider: "resend", operation: "emails.send".
+website/loaders/fonts/googleFonts.ts:97,103 — wrap with provider: "google-fonts", operation: "css". Low priority but cheap.
+Setter signature unchanged: setVtexFetch / setShopifyFetch still accept the result of createInstrumentedFetch("vtex") / ("shopify") from @decocms/start. The new tracedFetch helper sits between the call site and _fetch, so the upstream ${provider}.fetch span becomes the parent of our per-operation span. Good — gives operators both views.
+
+Files to modify
+
+utils/tracedFetch.ts (new)
+utils/tracedFetch.test.ts (new — exercise span name, attributes, traceparent header, histogram record, error status)
+vtex/client.ts — route vtexFetchResponse, vtexCachedFetch, intelligentSearch through tracedFetch
+vtex/utils/fetch.ts — route the generic wrapper through tracedFetch
+vtex/utils/proxy.ts — wrap the two fetch calls
+vtex/utils/authHelpers.ts — wrap the logout call
+vtex/actions/analytics/sendEvent.ts, vtex/actions/profile.ts, vtex/actions/masterData.ts, and the rest of the VTEX action files identified during audit
+vtex/hooks/useCart.ts, vtex/hooks/useWishlist.ts, vtex/hooks/useAutocomplete.ts, vtex/hooks/useUser.ts — convert direct fetch to the per-app wrapper
+shopify/utils/graphql.ts — wrap query in tracedFetch, surface graphql.operation.name
+resend/actions/send.ts — wrap in tracedFetch
+website/loaders/fonts/googleFonts.ts — wrap in tracedFetch
+package.json — bump @decocms/start to the version that includes PR 179 (likely a next channel release first)
+Reused helpers
+
+withTracing, injectTraceContext, setSpanAttribute, getMeter, MetricNames — all from @decocms/start/sdk/observability (post-PR-179).
+createInstrumentedFetch — still the boundary at startup, unchanged.
+RequestContext from @decocms/start/sdk/requestContext — already imported in this repo; useful inside tracedFetch if we want to stamp cookie.account or similar on spans.
+Release path
+
+Per CLAUDE.md, this is behavior-changing (new spans, new metric names, log volume). Route through @next:
+
+Open PR to next branch (not main). Title/body must avoid the canonical CI-skip token.
+Validate on a real VTEX storefront via npm install @decocms/apps@next.
+Confirm spans appear in ClickStack with expected shape (vtex.intelligent-search.product_search, full HTTP attrs, traceparent received downstream).
+Promote next → main via PR.
+Verification
+
+npm run check clean (typecheck + biome + knip).
+npm test — new unit tests on tracedFetch pass:
+Asserts span name is ${provider}.${operation}.
+Asserts http.method, http.url (redacted), http.status_code attributes set.
+Asserts traceparent header is injected when called inside withTracing (use the fake tracer pattern from PR 179’s observability.test.ts).
+Asserts histogram histogramRecord is called with commerce_request_duration_ms and the right labels.
+Asserts span has STATUS_CODE_ERROR when the underlying fetch throws.
+Local smoke against a real VTEX storefront:
+Bind to a dev site, hit PLP/PDP/cart pages.
+Tail ClickHouse / ClickStack for service.name = <site> and confirm vtex.intelligent-search.*, vtex.checkout.*, vtex.api spans appear nested under deco.http.request.
+Confirm logs from inside those fetches show trace_id/span_id matching the parent trace.
+Confirm at least one VTEX endpoint that has its own OTel collector receives our traceparent header (curl or check VTEX-side logs if available).
+Open decisions
+
+Operation-name source — explicit per-caller string vs URL-derived inside tracedFetch. The plan above assumes explicit per-caller (most accurate, no regex maintenance), but a URL→operation router inside tracedFetch would mean fewer per-call-site edits. Worth deciding before implementation starts.
+Should injectTraceContext live in upstream createInstrumentedFetch instead? PR 179’s doc claims it does, but the diff doesn’t touch instrumentedFetch.ts. If yes, file a separate upstream PR; if no, our tracedFetch is the right place.
+Per-provider vs generic histogram name — plan uses generic commerce_request_duration_ms with labels. Brief asked for vtex_request_duration_ms. Generic is more flexible; brief-literal is matchable to existing per-provider dashboards if any exist. Decide.
+
+
+# Fourth Context: Observability Architectural Refinement (PRs A/B/C/D — May 22 2026)
+
+After PR 179 landed and rolled out to a few storefronts, three structural gaps surfaced in real traffic:
+
+1. **Fragmented observability.** Three telemetry channels (CF Destinations, direct-POST in-worker, tail worker) produced overlapping but uncorrelated views. Same request appeared in three places without a stable join key, so operators couldn't follow a request end-to-end.
+2. **Context gaps.** Logs and tail records lacked stable `request.id` / `trace.id`, so logs and traces couldn't be joined. Tail rows had no method, URL, colo, or runtime cost attributes. The "false-positive ERROR flood" (`outcome=canceled` rows that aren't actually errors) was burying real errors at ~3K/hour.
+3. **Cost-effectiveness.** 100% direct-POST traces from the framework would scale linearly with traffic. We need sampled bulk + 100% errors + 100% metrics — and the sampling decision needs to align with what we already do at CF Destinations.
+
+The refinement plan (`~/.cursor/plans/observability_refinement_plan_4fa41548.plan.md`) decomposed this into 7 phases shipped as four PRs across two repos:
+
+## The 4-PR split
+
+| PR | Repo | Branch | Scope |
+|---|---|---|---|
+| **A** | `decocms/stats-lake` | `feat/otel-ingest-and-tail-enrichment` | Phase 1 tail enrichment (severity reclassification, `request/url/colo/cpu/wall` attrs, `request.id`/`trace.id` extraction from response headers); Phase 4 server-side JSON log normalization in `deco-otel-ingest`; Phase 7 `COLLECTOR_TARGET.md` documenting the optional collector swap. |
+| **B** | `decocms/stats-lake` | `feat/dashboard-provisioning` | Auto-provisioned per-site Grafana dashboards from `dim_sites`. Templates include `fleet-cost.json` (projected monthly spend, per-site ingestion volume, direct-POST vs Destinations ratio) + per-site SRE views. Alerts gated behind `--with-alerts` (opt-in for v1). |
+| **C** | `decocms/deco-start` | `feat/observability-identity-outcomes-spans` | Phases 1–3 framework-side: identity propagation (`X-Request-Id`, `X-Trace-Id`, `service.version`, `deployment.environment` in response headers + logger floor + span attributes), outcome metric layer (`http_requests_total{route_pattern, status_class}`, `cache_hit_total{cache_layer, cache_decision}`, `commerce_request_duration_ms{provider, operation}`), and the direct-POST OTLP trace exporter for `deco.*` spans with head sampling. D-records D-9 through D-15 added to `MIGRATION_TOOLING_PLAN.md`. Released as `@decocms/start@6.1.0`. |
+| **D** | `decocms/deco-start` | `feat/observability-governance-audit-runbooks` | Phase 6 governance: `deco-audit-observability` script (covers full fleet bindings, not just the observability block; `--mode warn\|block`; `--github` annotation output), five runbooks (HTTP error spike, HTTP latency spike, cache hit drop, commerce upstream slow, tail exception spike), RUM plan document with three scope tiers. D-records D-16 and D-17 added. Released as `@decocms/start@6.2.0`. |
+
+All four PRs merged. One post-merge bug found and fixed — see "Phase 4 JSON-body lift bug" below.
+
+## Decision records added by this refinement (D-9 → D-17, see `MIGRATION_TOOLING_PLAN.md` for full text)
+
+- **D-9** — `request.id` is the stable join key across all three channels. CF auto-injects; framework reads from `cf-ray` or generates if absent; logger floor stamps it on every log line; trace exporter writes it as a resource attribute; response headers carry `X-Request-Id` so the tail worker reads it back. One ID, four surfaces.
+- **D-10** — Server-side log normalization happens at the ingest worker, not in the framework. Framework keeps writing `console.log(JSON.stringify({...}))`; `deco-otel-ingest` parses any JSON-shaped `body.stringValue` and lifts fields into `LogAttributes`. Cost-neutral, query-side wins.
+- **D-11** — Outcome metrics (`outcome=ok|exception|canceled|exceededMemory|exceededCPU`) become the truth source for "did we serve users today?". Tail worker no longer classifies `canceled` and `responseStreamDisconnected` as `ERROR` — they're `INFO` outcomes.
+- **D-12** — Direct-POST OTLP trace exporter for framework `deco.*` spans only. CF auto-instrumented `fetch`/KV/R2 spans continue to ride CF Destinations sampling. Two-channel for traces matches the two-channel for logs.
+- **D-13** — Per-site Grafana dashboards are auto-provisioned from `dim_sites`, not hand-maintained. Adding a site to the dim table = dashboard appears on next provision run.
+- **D-14** — `deco-audit-observability` covers full fleet bindings (`tail_consumers`, `version_metadata`, `analytics_engine_datasets`, `DECO_OTEL_*` vars, observability block, `head_sampling_rate`), not just the observability block.
+- **D-15** — The OTel Collector swap (replacing `deco-otel-ingest` Worker with a canonical collector) is a documented target state, not a committed milestone. Revisit when (a) Worker hits cost ceiling, (b) we need processors the Worker doesn't have, or (c) we need to ingest from non-CF sources.
+- **D-16** — `deco-audit-observability` is warn-by-default; promote to `--mode block` per-repo as each section reaches clean. Avoids one breaking change for the whole fleet on day one.
+- **D-17** — Alerting venue for v1: none. Ship dashboards-only. Pick venue (PagerDuty, Slack, Linear) after 30 days of dashboard data informs what should alert.
+
+## Phase 4 JSON-body lift bug (post-merge fix, May 22)
+
+After PR A merged and `deco-otel-ingest` redeployed, the JSON-body lift in `logsToRows` was suppressed for 100% of production traffic. Original guard:
+
+```ts
+if (Object.keys(baseAttrs).length === 0) {
+  // ...attempt JSON.parse and lift fields into LogAttributes
+}
+```
+
+Intent: "don't overwrite attributes from direct-POST logs that already have structured LogAttributes". But Cloudflare Workers Observability auto-injects two attributes (`name`, `cloudflare.invocation.sequence.number`) into *every* LogRecord — even from a bare `console.log("hello")`. So `baseAttrs.length === 0` was never true in production, and Phase 4 became a no-op.
+
+Fixed in `fix/otel-ingest-phase4-merge-attrs` (PR #7 in stats-lake):
+- Removed the `length === 0` guard.
+- Attempt JSON parse on any `body.stringValue` shaped like JSON.
+- Merge with `{ ...redact(promoted.attrs), ...baseAttrs }` — existing attributes win on key collision, so direct-POST structured logs are still preserved.
+
+Lesson worth carrying forward: a "skip if non-empty" guard against a platform-injected map is the kind of bug only real production traffic surfaces. Unit tests for `logsToRows` would have caught it — see P3 below.
+
+
+# Fifth Context: Deploy + validate on lebiscuit-tanstack (May 25–27 2026)
+
+Sequence after the 4 PRs merged:
+
+| Step | Status | Notes |
+|---|---|---|
+| Deploy `deco-otel-ingest` (with Phase 4 fix) to production | ✅ Done May 22 | `wrangler deploy` from `stats-lake/ingestion/otel-ingest`. |
+| Deploy `deco-otel-tail` to production | ✅ Done May 22 | `wrangler deploy` from `stats-lake/ingestion/otel-tail`. Severity reclassification confirmed: `outcome=canceled` ERROR flood dropped from 294 in 5 min pre-deploy → 0 post-deploy. Attribute enrichment (`colo`, `cpu_time_ms`, `wall_time_ms`, `method`, `url.path`, `service.version`, `deployment.environment`) verified. |
+| Pick test storefront | ✅ Done May 25 | **lebiscuit-tanstack** chosen — tail consumer + observability block already wired in `wrangler.jsonc`, enough traffic to produce signal quickly, lower blast radius than casaevideo. |
+| Bump lebiscuit `@decocms/start` to 6.2.0 | ✅ PR open May 25 | [deco-sites/lebiscuit-tanstack#52](https://github.com/deco-sites/lebiscuit-tanstack/pull/52). Fresh branch off `origin/main`; left the pre-existing `chore/bump-decocms-start-6-apps-2` WIP branch alone. |
+| CF Workers Builds preview compiles cleanly | ✅ Green | 27 typecheck errors exist on both `main@6.0.1` and `bump@6.2.0` — pre-existing, not regressions. |
+| Merge → CF Workers Builds auto-deploys prod (D-6.3) | ⏸️ Pending user merge | Per migration-tooling-policy: agent doesn't self-merge customer storefronts. |
+| Run validation queries against ClickHouse | ⏸️ Blocked on deploy | Phases 1, 2, 3 against `service.name='lebiscuit-tanstack'`, 5-min windows pre/post deploy timestamp. |
+
+## Validation queries to run once deployed
+
+| Phase | Source | Expected change |
+|---|---|---|
+| 1 — identity in logs | `otel_logs.LogAttributes['request.id']`, `['trace.id']` | 0 → growing non-zero count for `service.name='lebiscuit-tanstack'` |
+| 1 — cross-channel join | JOIN `otel_logs` on `LogAttributes['request.id']` against `deco-otel-tail` rows | Tail rows now carry the same `request.id` (was 0 join coverage before) |
+| 2 — outcome metrics | `otel_metrics_sum` — `http_requests_total` with labels `route_pattern`, `status_class` | New label keys present (absent in 6.0.1) |
+| 2 — cache metrics | `cache_hit_total{cache_layer, cache_decision}`, `cache_miss_total{...}` | First entries for this service |
+| 3 — framework spans | `otel_traces` — span names `deco.cms.resolve`, `deco.section.render`, `deco.cache.lookup`, `deco.commerce.fetch` | First spans for this service appearing under the parent `deco.http.request` |
+| 4 — log normalization | `otel_logs.LogAttributes` populated from JSON body | Continues to work for lebiscuit's structured logs (was already validated on the Phase 4 fix smoke) |
+
+If any phase shows zero post-deploy, that's a real bug the validation has surfaced — exactly what the lebiscuit test is for. Do not roll out to other storefronts before that's root-caused.
+
+
+# Sixth Context: Future work (open items, ordered by priority)
+
+Framing each with both lenses where there's a choice to make. P1 is "shippable now"; P2–P7 each carry a tradeoff between shipping fast and the architecturally refined target.
+
+## P1 — Close the lebiscuit validation loop (next session)
+
+**Shippable now:** user merges PR #52 → CF Workers Builds auto-deploys prod → I run the six validation queries → record results in this file under a "Seventh Context".
+
+No architectural refinement at this layer — it's just the gate that lets P2 begin.
+
+## P2 — Fleet rollout (gated on P1 green)
+
+Risk-gradient order, smallest blast radius first:
+
+1. `als-tanstack` — internal-ish, lower traffic.
+2. `baggagio-tanstack` — medium traffic.
+3. `miess-01-tanstack` — medium traffic.
+4. `casaevideo-storefront` — highest traffic, biggest customer. Goes last.
+
+**Shippable now:** open identical-shape PRs per storefront — one-line `@decocms/start` bump, no code changes, CF Workers Builds handles deploy. ~10 minutes per storefront if `wrangler.jsonc` is already wired.
+
+**Architecturally refined:** sites missing the `tail_consumers` / `observability` block need that wired first. `deco-audit-observability` already flags them. The refined version is a single batched audit-and-fix PR per storefront, not one PR per finding.
+
+**Risk:** the framework changes are additive, no API removed, but rolling out a behavior change to every storefront in a 24-hour window risks one shared regression across the fleet. **Mitigation:** soak each storefront 24h before promoting the next.
+
+**Evolution path:** once 4/4 storefronts have soaked clean, the audit script flips from warn to block (P4) — making any future storefront missing the bindings break CI rather than silently miss telemetry.
+
+## P3 — Vitest coverage on stats-lake/ingestion/otel-ingest
+
+Flagged at fix-time of the Phase 4 bug. The merge-attrs logic in `logsToRows` is exactly the kind of code where a hidden guard silently no-ops.
+
+**Shippable now:** five test cases covering the main behaviors:
+- JSON-shaped body with no pre-existing LogAttributes → fields lifted.
+- JSON-shaped body with CF-injected attributes only → fields lifted *and* CF attrs preserved.
+- JSON-shaped body with key collision against pre-existing attribute → pre-existing wins.
+- Non-JSON body → body left as-is, no lift attempted.
+- Malformed JSON body → body left as-is, no throw.
+
+~80 LOC, contained to one test file. No tradeoff.
+
+**Architecturally refined:** snapshot-test the full OTLP→ClickHouse-row transform with a golden fixture of recorded real OTLP payloads. Catches regressions across the whole `logsToRows`/`tracesToRows`/`metricsToRows` surface, not just the merge-attrs bug. ~2x the work.
+
+**Evolution path:** ship the five-case version first to close the immediate gap; pull a real OTLP payload into a golden fixture next time a similar bug bites.
+
+## P4 — Promote `deco-audit-observability` from warn → block (per D-16)
+
+**Shippable now:** flip every clean repo to `--mode block` in one batched PR. Fast but breaks CI in any repo that hasn't been fixed yet.
+
+**Architecturally refined:** per-repo, per-section gradual promotion:
+1. Land the audit script in CI as warn-only on every `decocms/*` and `deco-sites/*` repo.
+2. Wait one cycle (a week) to see which findings hit which repos.
+3. For each clean repo, flip to `--mode block` via a separate PR.
+4. For dirty repos, fix findings before flipping.
+
+**Tradeoff:** refined version costs ~3 weeks of calendar; shippable-now version costs ~1 PR but every dirty repo's CI breaks Monday morning.
+
+**Recommendation:** refined version. The "Strict over flexible" principle (constitutional principle 2) earns its strictness incrementally rather than by breaking everyone at once.
+
+## P5 — Dashboard alert wiring (deferred per D-17)
+
+30 days after the dashboards are live, review:
+
+- Which dashboards have we actually opened during the 30 days?
+- Which panels would have caught a real incident if they'd been alerts?
+- What's the right venue — PagerDuty for paging, Slack for visibility, Linear for tracking?
+
+**Shippable now:** pick PagerDuty for the top 3 panels (HTTP error rate, P95 latency, cache hit ratio) — these are universally useful and we know we want pages on them.
+
+**Architecturally refined:** anomaly-band alerts (mean ± 3σ over a rolling 7-day baseline) instead of fixed thresholds. Reduces false positives on legitimate traffic shifts (Black Friday, promo campaigns). Requires a stable historical baseline — needs the 30-day soak before it makes sense.
+
+**Evolution path:** ship fixed-threshold PagerDuty pages for the top 3 panels first. Migrate to anomaly bands once we have 60 days of baseline.
+
+## P6 — Phase 7 OTel Collector swap (gated per D-15)
+
+Documented target state in `stats-lake/ingestion/otel-ingest/COLLECTOR_TARGET.md`. Revisit when **one of**:
+
+- `deco-otel-ingest` Worker hits the cost ceiling (currently far under).
+- We need processors the Worker doesn't have (filter by attribute, tail-based sampling, multi-tenant routing).
+- We need to ingest from non-CF sources (on-prem services, mobile RUM SDK).
+
+**Shippable now:** keep the Worker. Zero work. The Worker has been deployed, tested, and is producing correct data.
+
+**Architecturally refined:** swap to canonical OTel Collector behind the same OTLP endpoint shape and same ClickHouse schema. Migration path is reversible — A/B test the Collector behind a fraction of traffic before flipping cutover.
+
+**Tradeoff:** the Worker is doing the job. The Collector is doing-the-job-with-someone-else's-roadmap-of-features. Pulling Collector in now is over-engineering until one of the triggers above fires.
+
+## P7 — RUM (Real User Monitoring) — separate scope per user decision
+
+Document: `decocms/deco-start/docs/rum-plan.md`. Three tiers were drafted; user has not yet picked one:
+
+| Tier | Scope | Footprint | Cost estimate |
+|---|---|---|---|
+| **Core Web Vitals only** | LCP, CLS, INP via `web-vitals` library; ship to `deco-otel-ingest` via `sendBeacon` | ~3 KB gzipped, no UX impact | ~$0.50 / 1M PVs |
+| **Custom Events** | CWV + page navigation timings + custom `track(eventName, attrs)` API for app-level events (add-to-cart, view-product, etc.) | ~8 KB gzipped | ~$2 / 1M PVs |
+| **Session Replay** | All of the above + DOM-mutation-based replay (PostHog/FullStory shape) | ~50 KB gzipped, real privacy + perf concerns | ~$15 / 1M PVs + S3 storage |
+
+**Blocked on:** business question of which questions we actually need RUM to answer. Until that's specified, the plan sits. Pushing back on shipping any tier prematurely — Session Replay in particular has a privacy review I don't want to absorb without a clear product reason.
+
+
+# Source-of-truth links
+
+- Architectural refinement plan (read-only): `~/.cursor/plans/observability_refinement_plan_4fa41548.plan.md`
+- Decision records D-9 → D-17 (full text): `decocms/deco-start/MIGRATION_TOOLING_PLAN.md`
+- Runbooks: `decocms/deco-start/docs/runbooks/{README,http-error-spike,http-latency-spike,cache-hit-drop,commerce-upstream-slow,tail-exception-spike}.md`
+- RUM scope tiers: `decocms/deco-start/docs/rum-plan.md`
+- Phase 7 collector target state: `decocms/stats-lake/ingestion/otel-ingest/COLLECTOR_TARGET.md`
+- Audit script: `decocms/deco-start/scripts/audit-observability-config.ts` (+ `.test.ts`)
+- Dashboard templates: `decocms/stats-lake/observability/dashboards/templates/*.json`
+- Dashboard provisioning script: `decocms/stats-lake/observability/scripts/provision-dashboards.ts`
+- ClickStack query patterns: `decocms/deco-start/docs/observability.md`
\ No newline at end of file