Skip to content

Add how-data-products-are-resolved.md #72

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
beojan wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add how-data-products-are-resolved.md #72

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cdf31f1
Add how-data-products-are-resolved.md
beojan Jul 20, 2026
81187c1
Fix data product resolution Markdown formatting
beojan Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions design_wiki/seeds/how-data-products-are-resolved.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# How Data Products are Resolved

## Product Selectors

- Data products are selected using a `product_selector`
- A `product_selector` *must* specify a data product concept
- This concept may be automatically determined from the input type required by the CHOF being registered
- A `product_selector` *may* specify zero or more of the following
- Creator name -- This can match either the plugin name or the algorithm name (or both) of the creating algorithm
- Suffix
- Stage name -- This can be "CURRENT" to match whatever the current stage name is
- Layer name

## Layers

- Each data cell is associated with a layer path, which determines its home in the hierarchy
- The "layer name" is the name of the lowest ("leaf") layer in the path
- Each instance of a data product is placed in a data cell
- Each instance of a data product is therefore transitively associated with a layer path, and layer name
- These do not have to be constant across instances of a data product

@knoepfel knoepfel Jul 22, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would word this differently. Perhaps: "A product_selector instance may select data products that do not all have the same data layer."

Once assigned to a data product, the data layer is constant.

@beojan beojan Jul 30, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not quite what I was meaning to say. I meant that one particular data product produced by a particular algorithm might be placed in "spill" when the input to the algorithm was in "spill", then in "mc_event" when the input was in "mc_event". That's independent of what product_selectors might select this data product.

Maybe what I should say is "The layer path and layer name do not have to be the same for all data products in a data product family"?

@beojan beojan Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even that isn't right. If the layer can change then the index set also changes, which means we don't have an indexed family.

@knoepfel knoepfel Aug 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even that isn't right. If the layer can change then the index set also changes, which means we don't have an indexed family.

I believe we actually do have an indexed family. A product_selector will be "evaluated" at graph-initialization time. At that time, a product_selector may match one or more product specifications corresponding to $n$ different layers. In such a case, the index set would be

$$ \mathcal{I}=\mathcal{I}_{layer\ 1} \cup \mathcal{I}_{layer\ 2} \cup \cdots \cup \mathcal{I}_{layer\ n} $$

It is not required that each index set used in the above union must have identically-shaped indices (e.g., one set's indices may look like [Run: 1, Spill: 6] while another set's indices may look like [Run: 1, Subrun: 9, Spill: 8]).

@beojan beojan Aug 5, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is, these do not have to be constant across invocations of the algorithm producing the data product.


## Resolution

- All fields except layer name are used to resolve the data product at graph creation time
- This is because the layer name may not be constant across instances of the product
- Edges are created only from input products which match all specified fields of the selector
- Where the product is produced by a HOF other than a provider the relevant fields are: Concept, Creator name, and suffix
- Where the product is produced by a provider, the relevant fields are: Concept, creator name, suffix, and layer name
- Where the product is produced by an *explicit* provider, the layer name is also used at graph creation time
- Explicit providers statically specify the layer path associated with their output
Comment thread
coderabbitai[bot] marked this conversation as resolved.

## Layers Checking / Resolution

- It is desirable that a CHOF be able to take a given input product from different layers on different iterations
- On each iteration the output layer of the CHOF is determined by the layers of its inputs
- Therefore layer name checking must be delayed til runtime, when data flows along the created edges

@knoepfel knoepfel Jul 22, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the idea of layer-checking needs to be fleshed out a little. For example, at what points in the execution of a single graph iteration does layer-checking need to happen (and why)? Right now I can only think of two scenarios:

  1. Routing—the initial sending of the data-cell index to the correct providers, which requires inspecting the layer of the data-cell index (currently achieved by the index_router).
  2. Layer filtering—when a CHOF with at least one input product selector having a specified layer field is downstream of a CHOF that does not have a specified layer field in its input product selectors.

Am I missing other scenarios?


### Contentious Part

- A selector may match multiple products when its layer name is ignored, while only matching a single product in any given data cell
- This requires allowing the creation of multiple edges for the same input port
- We must nevertheless prevent duplicate matches within a single data cell

1. Could require all other fields to match
- Perhaps places unreasonable burden on user (all other fields must be specified in the query)
- Reduces utility of multiple match feature, perhaps completely
2. Could (somehow) statically enforce behaviour as products are placed into cells
3. Could adopt a "cycle" concept, where multiple matches within a data cell have a priority order
- Likely to cause surprising behaviour, at least sometimes.