Unified Timeline Observability: establish feasibility and target design

[Coekjan]

Outcome and baseline

Enable one queryable causal timeline across the daemon, serving Instances,
AtnAgents, FfnAgents, CPUs, accelerators, and eventually multiple hosts.
CrossPool already has typed Transport, Fabric, Graph, and Routing Observer
evidence and propagated protocol identities. These are useful inputs, but
process-local snapshots do not establish a unified event or clock model.

This issue covers the discovery, minimal prototype, and target-design stages
below. Production collection and qualification are suggested follow-on work.

Suggested implementation route

1. Define the questions the timeline must answer and the existing identities
   needed to correlate events. Compare Perfetto, Chrome Trace, OpenTelemetry,
   and NVTX/CUPTI for collection, export, causality, queryability, capacity,
   overhead, and non-CUDA portability. Recommend an interface combination
   against these requirements rather than selecting a format in advance.
2. Build a throwaway export prototype over existing Observer evidence. Show a
   causal path across participants and distinguish event time, collection
   time, and ordering. Measure process/Host/accelerator clock alignment,
   uncertainty, and drift. Specify the two-host experiment needed before any
   cross-host alignment claim; execute it when that claim is in scope.
3. Propose an event identity and clock-domain contract, bounded collection
   behavior, overflow/loss reporting, export ownership, and a scoped target
   plan. Record which questions the measured fidelity can actually answer.
4. After accepting that plan, implement the chosen collection/export path at
   existing Hook Points and expose a usable query or visualization workflow.
5. Qualify causal correlation, clock error, collection overhead, data volume,
   and loss behavior for the selected deployment. Expand platform and host
   coverage only with corresponding evidence.

Discovery completion evidence

• An interface comparison and requirements report with an evidence-backed
  recommendation.
• A minimal prototype, representative raw trace, and reproducible clock and
  overhead measurements for the claimed scope.
• A reviewed target-design recommendation or a documented, evidence-backed
  no-go. Outstanding cross-host measurements remain explicit prerequisites.

Closing this issue records discovery completion, not production support.
Unavailable measurement hardware does not satisfy a required experiment.
Continuous streaming, alerting, and a production monitoring service are outside
this scope. Existing typed Observer snapshots and Hook Points are inputs, not
replacement production protocols.

References

• Roadmap
• Devkit
• Qualification

[Coekjan]

cc @saitewasreset