Sharing upstream outputs across universes — is there a better way?

[cailmdaley]

Two universes that differ in just one setting usually share most of their upstream — the same maps, redshifts, and spectra. But Lightcone seems to treat them as entirely separate runs: it won't build them at the same time, and it rebuilds the shared upstream once per universe.

            shared upstream
         maps · n(z) · spectra
           (same for both)
                  │
        ┌─────────┴─────────┐
        ▼                   ▼
    universe A          universe B
   estimator = gmvbh   estimator = gmv
        │                   │
   rebuilds the        rebuilds the      ◄─ identical upstream
   whole upstream      whole upstream       built twice, and the
   + its own outputs   + its own outputs    two runs can't overlap

We've found a workaround — give each run its own LIGHTCONE_SCRATCH directory so they don't share a lock — but we're not sure it's the right approach, and it still rebuilds the shared upstream for each universe.

Could this be fixed so that universes recognize a shared upstream and build it just once? Or is there a recommended pattern we're missing?

— Claude, on behalf of Cail

[cailmdaley]

A quantified case from the UNIONS Paper II reproduction: we're materializing two universes that differ in exactly one decision (pure_eb_integration_method — the quadrature inside the pure-E/B transform). Everything upstream of the transform is decision-independent: 1050 per-mock TreeCorr measurements (3 scans × 350 GLASS realisations), the per-version ξ± sweeps, and the CosmoCov covariances. lc plans all of it once per universe — O(1000) core-hours of byte-identical recompute for the second universe, dwarfing the forked part (the transform + PTE layers) by an order of magnitude.

Since a task's identity is already (recipe string, resolved decisions, input versions), two universes whose tasks have identical identity could share the artifact — the manifest already records everything needed to recognize the match.

— Claude, on behalf of Cail