Eclipse asks a harder question than “which driving events look unusual?”:
When a vehicle can upload only a small fraction of what it observes, which events should it send, and how can the fleet estimate what happened in the data it did not collect?
The project is a reproducible fleet-event replay laboratory. It compares four acquisition policies under fixed upload budgets, logs every decision probability, preserves randomized sentinel exploration, and uses inverse-propensity and doubly robust estimators to audit the selection bias introduced by targeted collection.
The committed benchmark is byte-stable across runs: its random seed, policy versions, event ordering, and benchmark date are fixed, so reviewers can reproduce the published evidence exactly.
Targeted fleet collection creates missing-not-at-random data. A trigger that uploads difficult scenarios will make the uploaded sample look less safe even if fleet behavior has not changed. It can also create a feedback loop in which a model controls what future training data it sees and therefore never discovers its own blind spots.
Eclipse treats the acquisition policy as a versioned part of the learning system.
- Four policies: random, rule triggers, uncertainty-only, and the Eclipse multi-objective policy.
- Upload-budget calibration at 1%, 3%, 5%, 10%, and 15%.
- A randomized sentinel floor that gives every event a non-zero upload probability.
- A complete decision ledger containing policy version, propensity, random draw, decision lane, score, and contributing signals.
- Naive, inverse-propensity, and doubly robust fleet-rate estimators.
- Coverage, critical-event recall, drift recall, bandwidth, and information-density metrics.
- A normalized SQLite reference warehouse with event, policy-run, and upload-decision models.
- A versioned JSON Schema data contract and a PostgreSQL-style analytical model.
- An interactive React control room, bias audit, replay map, and searchable decision ledger.
- Automated unit, data-integrity, production-build, interaction, and responsive-layout checks.
The committed benchmark contains 6,000 deterministic fleet-event summaries and evaluates 20 policy/budget combinations, producing 120,000 auditable upload decisions. At a 5% target budget:
| Policy | Actual upload | Critical recall | Scenario coverage | Value / GB | Naive bias | DR error |
|---|---|---|---|---|---|---|
| Random | 5.6% | 5.7% | 31.1% | 21.0 | +0.4 pp | 2.0 pp |
| Rules | 5.0% | 15.3% | 33.1% | 29.0 | +41.8 pp | 4.4 pp |
| Uncertainty | 5.4% | 12.9% | 37.6% | 27.1 | +28.2 pp | 3.6 pp |
| Eclipse | 5.5% | 13.0% | 43.6% | 29.4 | +27.2 pp | 2.6 pp |
The result is deliberately not presented as “Eclipse wins everything.” Rule triggers recover more obvious critical events at this budget. Eclipse obtains broader scenario coverage and higher information density while retaining statistical support for fleet-level estimation. That tradeoff is the point of the experiment.
flowchart LR
A[Vehicle event summary] --> B[Policy scorer]
B --> C{Dual-stream decision}
C -->|Targeted| D[Policy upload]
C -->|Randomized| E[Sentinel upload]
C -->|Rejected| F[Metadata-only ledger]
D --> G[(Versioned event warehouse)]
E --> G
F --> H[(Decision warehouse)]
G --> I[Coverage and safety metrics]
H --> J[IPS and doubly robust audit]
I --> K[Policy comparison UI]
J --> K
The browser receives a compact replay window. All metrics are calculated against the complete 6,000-event benchmark, and all 120,000 decisions are validated in the local warehouse build.
See the architecture note for policy equations, data lineage, and production extensions.
- Policy and data engine: Python 3, deterministic hashing, SQLite, SQL
- Statistical evaluation: inverse propensity weighting, doubly robust estimation
- Application: React, Vite, D3, Lucide
- Quality:
unittest, schema checks, Playwright-driven Chrome QA - Deployment: static production build suitable for Vercel
npm install
npm run data
npm test
npm run build
npm run devThe application will be available at the local URL printed by Vite.
npm test # 9 policy, reproducibility, estimator, and warehouse tests
npm run build # production bundle
npm run qa:browserBrowser QA exercises the budget and policy controls, replay action, bias audit, searchable ledger, and method view at 1440x1000 and 390x844. It also fails on browser errors or horizontal overflow.
The default benchmark is a deterministic event replay modeled on public autonomous-driving scenario taxonomies. It does not contain Tesla data, personal data, camera footage, or proprietary telemetry. The event fields reflect concepts used in motion-forecasting and scenario-mining work, but the benchmark outcomes must not be interpreted as real-world vehicle safety results.
The architecture is prepared for an Argoverse 2 motion-forecasting adapter. The public Argoverse 2 dataset contains 250,000 motion-forecasting scenarios recorded at 10 Hz across six cities. A full-data adapter is intentionally left outside the default deployment because the source dataset is large and the browser demonstration does not require redistributing it.
Methodological references:
- Wang, Agarwal, and Dudik, Optimal and Adaptive Off-policy Evaluation in Contextual Bandits
- Dudik et al., Doubly Robust Policy Evaluation and Optimization
- Lu et al., ActiveAD: Planning-Oriented Active Learning for End-to-End Autonomous Driving
pipeline/policy_engine.py acquisition policies and estimators
pipeline/warehouse.py normalized warehouse build
pipeline/generate_benchmark.py reproducible benchmark generator
schemas/ versioned event contract
sql/ propensity-aware analytical model
src/ interactive control room
tests/ behavioral and data-integrity tests
docs/ architecture and evidence ledger
- Add a licensed Argoverse 2 or Waymo Open Dataset ingestion adapter.
- Evaluate learned acquisition policies against a real motion-forecasting model.
- Move the reference warehouse to ClickHouse and the replay transport to Kafka/Redpanda.
- Add confidence intervals and effective-sample-size diagnostics to the policy audit.
- Run candidate policy versions in true shadow mode before promotion.
MIT. See LICENSE.