Skip to content

lc cannot materialize a spec with sub-analyses #202

Description

@EiffL

lc materializes a flat analysis. A spec that declares analyses: is not buildable: ASTRA qualifies an output declared in a sub-analysis as <analysis>.<output>, and assets.output_path refuses an id carrying a dot, because the manifest sidecar (.<output_id>.manifest.json) is recovered by partitioning the filename on its first dot — a dotted id would name a manifest that could not be told from the payload, or that two outputs would share.

Why it is refused rather than supported

Supporting nesting means answering where a nested output's file goes, and every answer needs a decision that is not lc's alone to make:

  • A scope directoryresults/<universe>/<analysis>/<output>.<format>. Simple, one results tree.
  • Beside the sub-analysis's own spec — an external (path:) sub-analysis is documented in astra-spec's llms.txt as "itself a valid ASTRA project with its own astra.yaml and optional universes/", which argues its results belong with it. That means several results trees, which reaches .gitattributes routing, the dirty-tree refusal's path split, and the crate's file walk.
  • The qualified id as the filename — rejected: it makes the sidecar ambiguous, which is the silent-corruption case above.

There is also an open question about which universe a nested output is filed under when the parent universe selects one of the sub-analysis's own (analyses: {hod: {universe: fast}}) — two parent universes selecting the same one would otherwise collide.

Prerequisites

  • astra-tools: validate_universe_file validates against the unresolved tree (it calls load_yaml, never resolve_analysis_tree), so a universe naming decisions inside a nested external sub-analysis fails with [UNKNOWN_ANALYSIS] even though the spec is valid. Depth 1 survives because the sub-analysis key exists unresolved; depth 2+ does not. Worth fixing upstream before lc builds on path:.
  • astra-spec: no path: example exists anywhere in examples/ or tests/data/valid/; the only nested example, examples/iris_pipeline/, is inline. The convention for where sub-analysis directories live is unsettled — the specification shows stages/catalog_cleaning, while analyses/ is used in the docs for ref:-ed independent records.

Notes

Not a regression in behaviour anyone relied on, but worth stating plainly: before the one-file layout, a nested output landed in a flat directory named by its dotted id. See also #201.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions