Skip to content
Open
16 changes: 14 additions & 2 deletions converters/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ The top-level container. Maps to the root object in each vendor's format.
| `ai_context` | Instructions, synonyms for AI tools | Map if vendor supports AI/LLM annotations |
| `datasets` | Logical datasets (fact/dimension tables) | See dataset mapping below |
| `relationships` | Foreign key connections | See relationship mapping below |
| `metrics` | Aggregate measures | See metric mapping below |
| `metrics` | Model-scoped aggregate measures | See metric mapping below |
| `custom_extensions` | Vendor-specific metadata | Extract extensions matching the target vendor |

### Datasets
Expand All @@ -108,6 +108,7 @@ Datasets represent logical tables (fact or dimension tables). They contain field
| `primary_key` | Single or composite primary key | Map to vendor's PK syntax; note composite keys use arrays like `[order_id, line_number]` |
| `unique_keys` | Alternative unique identifiers | Map if vendor supports unique constraints |
| `fields` | Row-level attributes (columns) | See field mapping below |
| `metrics` | Dataset-scoped aggregate measures whose expressions resolve entirely within this dataset | See metric mapping below. A converter that reads only `semantic_model.metrics` will silently drop these |
| `ai_context` | Synonyms and context for AI | Map if vendor supports semantic annotations |
| `custom_extensions` | Vendor-specific metadata | Extract extensions matching the target vendor |

Expand Down Expand Up @@ -166,7 +167,18 @@ means `from.product_id = to.id AND from.variant_id = to.variant_id`. The convert

### Metrics

Metrics are aggregate measures defined at the semantic model level. They can span multiple datasets via relationships.
Metrics are aggregate measures. They appear in two places, and **a converter must read both**:

- `semantic_model.metrics`: model-scoped. May span multiple datasets via relationships. Expressions reference fields by qualified name (`SUM(orders.amount)`).
- `datasets[].metrics`: dataset-scoped. The expression may reference this dataset's declared fields, written `dataset.field` (`SUM(orders.net_amount)`), and the columns of its `source`, written unqualified (`SUM(tax)`). A qualified reference must name a declared field.

Both placements use the identical metric structure, so the field mapping below applies to each.

A converter that reads only `semantic_model.metrics` produces an incomplete model. That is a lossy conversion, not a valid one: it SHOULD warn, naming the metrics it dropped, and MUST NOT present the output as a faithful representation of the source.

When the target format has only one metric namespace, hoist dataset-scoped metrics to the model level. This requires two changes: qualify the expression's unqualified column references with the dataset name, since references to declared fields are already qualified, and qualify the metric's own name as `dataset_name.metric_name`, since two datasets may each declare a metric with the same local name. If the target namespace disallows dots, use an equivalent encoding.

See [Metric Scoping](../core-spec/spec.md#metric-scoping) for the full rules.

| Ossie Field | Description | Converter Consideration |
|-----------|-------------|------------------------|
Expand Down
9 changes: 8 additions & 1 deletion core-spec/ossie-schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,13 @@
"$ref": "#/$defs/Field"
}
},
"metrics": {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OssieDataset in python/src/ossie/models.py (the model shipped as the apache-ossie package) has no metrics field. Pydantic's default extra='ignore' means dataset-scoped metrics are discarded on load with no error:

doc = OssieDocument.model_validate(yaml.safe_load(open('examples/tpcds_semantic_model.yaml')))
# store_sales has no `metrics` key; only customer_lifetime_value and
# store_productivity survive to_ossie_yaml(). Three metrics vanish silently.

Adding metrics: list[OssieMetric] | None = None to OssieDataset fixes it.

Worth noting the reason nothing caught this: python/tests/test_models.py:75 only cross-checks the DataType enum against the JSON Schema, so structural drift between the schema and the pydantic model is invisible. A test that walks $defs properties and asserts each one exists on the corresponding model would have failed here.

"type": "array",
"items": {
"$ref": "#/$defs/Metric"
},
"description": "Metrics that aggregate data held by this dataset. The expression may reference this dataset's declared fields, written dataset_name.field_name, and the columns of its source, written unqualified; a column used only inside a metric does not need to be declared as a field. A qualified reference must name a declared field of this dataset. An expression referencing another dataset is model-scoped and belongs in semantic_model.metrics. This constrains the expression only, not the query: the metric is joined and grouped like any other, using the model's relationships. Names must be unique within the dataset and must not collide with a field name of this dataset. Referenced from outside the dataset as dataset_name.metric_name. See core-spec/spec.md#metric-scoping."
},
"custom_extensions": {
"type": "array",
"items": {
Expand Down Expand Up @@ -336,7 +343,7 @@
"items": {
"$ref": "#/$defs/Metric"
},
"description": "Quantifiable measures spanning datasets"
"description": "Model-scoped metrics. Expressions reference fields by qualified name (dataset.field). Names must be unique across the semantic model; reusing the name of a dataset-scoped metric should produce a warning rather than an error. An unqualified metric reference resolves to a model-scoped metric. See core-spec/spec.md#metric-scoping."
},
"custom_extensions": {
"type": "array",
Expand Down
189 changes: 168 additions & 21 deletions core-spec/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@
4. [Relationships](#relationships)
5. [Fields](#fields)
6. [Metrics](#metrics)
7. [Examples](#examples)
7. [Metric Scoping](#metric-scoping)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: every other numbered entry (1–6, 8) maps to a ## heading, but Metric Scoping is authored as ### Metric Scoping at line 427, nested under ## Metrics. Rendered docs get a top-level entry that jumps into the middle of the Metrics section.

Either promote it to ## Metric Scoping, or drop the TOC entry and keep the in-section link at line 370.

8. [Examples](#examples)

---

Expand Down Expand Up @@ -92,7 +93,7 @@ The top-level container that represents a complete semantic model, including dat
| `ai_context` | string/object | No | Additional context for AI tools (e.g., custom instructions) |
| `datasets` | array | Yes | Collection of logical datasets (fact and dimension tables) |
| `relationships` | array | No | Defines how logical datasets are connected |
| `metrics` | array | No | Quantifiable measures defined as aggregate expressions on fields from logical datasets |
| `metrics` | array | No | Model-scoped metrics: for expressions that must combine fields from more than one dataset. See [Metric Scoping](#metric-scoping). |
| `custom_extensions` | array | No | Vendor-specific attributes for extensibility |

### Example
Expand Down Expand Up @@ -130,6 +131,7 @@ Logical datasets represent business entities or concepts (fact and dimension tab
| `description` | string | No | Human-readable description |
| `ai_context` | string/object | No | Additional context for AI tools (e.g., synonyms, common terms) |
| `fields` | array | No | Row-level attributes for grouping, filtering, and metric expressions |
| `metrics` | array | No | Dataset-scoped metrics whose expressions resolve entirely within this dataset. See [Metric Scoping](#metric-scoping). |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • docs/index.md:251 (FAQ) — "Metrics are defined at the semantic model level (not within a dataset)"
  • docs/index.md:58 and docs/index.md:321 (glossary) — same claim
  • converters/README.md:169 — "Metrics are aggregate measures defined at the semantic model level"
  • converters/README.md:97 — the dataset property table omits metrics

The converter guide one matters most: an author following it writes a converter that drops dataset-scoped metrics, which is exactly the failure mode already present in all the existing converters.

| `custom_extensions` | array | No | Vendor-specific attributes |

### Primary Key Examples
Expand Down Expand Up @@ -358,7 +360,14 @@ Common combinations:

## Metrics

Quantitative measures defined on business data, representing key calculations like sums, averages, ratios, etc. Metrics are defined at the semantic model level and can span multiple datasets.
Quantitative measures defined on business data, representing key calculations like sums, averages, ratios, etc.

Metrics may be defined in two placements, using the same structure in both:

- **Model-scoped** (`semantic_model.metrics`): for expressions that must combine fields from more than one dataset.
- **Dataset-scoped** (`datasets[].metrics`): aggregates data held by one dataset. Still joined and grouped through the model's relationships like any other metric.

See [Metric Scoping](#metric-scoping) for the rules governing each.

### Schema

Expand Down Expand Up @@ -417,6 +426,133 @@ expression:

---

## Metric Scoping

A metric may be defined at the semantic model level or on an individual dataset. Both placements use the identical metric structure; only the resolution rules differ.

| | Model-scoped (`semantic_model.metrics`) | Dataset-scoped (`datasets[].metrics`) |
|---|---|---|
| Expression references | Fields of any dataset in the model, qualified: `dataset.field` | Fields of its own dataset, qualified: `dataset.field`; columns of its `source`, unqualified: `column` |
| Name uniqueness | Unique across the semantic model | Unique within its dataset, and distinct from that dataset's field names |
| Referenced as | `metric_name` | `dataset_name.metric_name` |

**Rules**

1. A dataset-scoped metric's expression MAY reference the declared fields of its dataset and the columns of its `source`. A declared field is written `dataset_name.field_name`; a column of the `source` is written unqualified: `SUM(orders.net_amount) / COUNT(DISTINCT tax_id)`. A qualified reference MUST name a declared field of the declaring dataset. An expression that references another dataset MUST be declared in `semantic_model.metrics`.
2. Dataset-scoped metric names MUST be unique within their dataset. Two datasets MAY each declare a metric with the same name.
3. A dataset-scoped metric name MUST NOT collide with the name of a field of the same dataset, which under rule 5 would leave `orders.amount` ambiguous.
4. A model-scoped metric SHOULD NOT reuse the name of a dataset-scoped metric in the same semantic model. Validators SHOULD warn, and SHOULD attribute the warning to the model rather than to the dataset.
5. An unqualified metric reference resolves to a model-scoped metric. A dataset-scoped metric is referenced as `dataset_name.metric_name`.

A column used only inside a metric does not need to be declared as a field. Declaring one is how a metric reuses a field's expression instead of repeating it.

Because the two are spelled differently, a field and a source column MAY share a name without ambiguity. In a metric of `orders`, `orders.foo` is the declared field and `foo` is the source column, so declaring a field named after an existing column does not change the meaning of an expression already using the bare name.

Whether a bare name is a real column of the `source` is not checked, because that requires catalog metadata the model does not carry. A qualified reference is checked, because the field list is in the model.

Rule 1 places no limit on the complexity of the aggregation, and constrains the expression rather than the query. A dataset-scoped metric is joined and grouped like any other using the relationships declared in the model, so a metric declared on `store_sales` as `SUM(ss_ext_sales_price)` may still be grouped by `item.i_brand` or `store.s_state`.

**Which names may repeat**

Names are addressed in three ways, so a name may repeat across them without ambiguity:

| Kind | Addressed as |
|---|---|
| Model-scoped metric | Bare: `revenue` |
| Dataset-scoped metric | Qualified: `orders.revenue` |
| Field | Qualified: `orders.revenue` |
| Dataset | Only ever as a qualifier |

A model-scoped metric MAY therefore share a name with a field, or with a dataset. Rule 3 is an error because a field and a metric of one dataset share the same qualified namespace; rule 4 is a warning because the two names remain separately addressable.

**Example: dataset-scoped metrics**

```yaml
datasets:
- name: orders
source: sales.public.orders
primary_key: [order_id]
fields:
- name: order_id
expression:
dialects:
- dialect: ANSI_SQL
expression: order_id
description: Order identifier
- name: net_amount
expression:
dialects:
- dialect: ANSI_SQL
expression: amount - discount
description: Order amount after discount

metrics:
# Qualified, so this references the declared field net_amount
- name: total_net_amount
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(orders.net_amount)
description: Total order amount after discount
datatype: Decimal

# Unqualified, so this references tax, a column of the source that is
# not declared as a field
- name: total_tax
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(tax)
description: Total tax collected
datatype: Decimal

- name: order_count
expression:
dialects:
- dialect: ANSI_SQL
expression: COUNT(order_id)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The orders dataset above declares exactly one field, amount; order_id appears only in primary_key. So the metric introduced as "unqualified reference to a field of the declaring dataset" references something that is not among the declaring dataset's fields, the example undercuts the point it is making.

Either add order_id to fields, or use COUNT(amount).

description: Number of orders
datatype: Integer
```

Referenced from a consumer as `orders.total_net_amount`, `orders.total_tax` and `orders.order_count`.

**Example: invalid dataset-scoped metrics**

```yaml
datasets:
- name: orders
source: sales.public.orders
fields:
- name: net_amount
expression:
dialects:
- dialect: ANSI_SQL
expression: amount - discount
metrics:
# INVALID (rule 1): references the customers dataset, so this metric is
# model-scoped and belongs in semantic_model.metrics
- name: revenue_per_customer
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(net_amount) / COUNT(DISTINCT customers.id)

# INVALID (rule 1): a qualified reference must name a declared field, and
# tax is only a column of the source. Write it unqualified as SUM(tax)
- name: total_tax
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(orders.tax)
```

**Consumer guidance: flattening to a single metric namespace**

Hoisting a dataset-scoped metric to the model level requires qualifying its field references with the dataset name, and qualifying the metric's own name as `dataset_name.metric_name` since two datasets may reuse a local name. Use an equivalent encoding if the target namespace disallows dots.

A consumer that reads only `semantic_model.metrics` produces an incomplete representation of the model. This is a lossy conversion rather than a valid one: such a consumer SHOULD warn, naming the metrics it dropped, and MUST NOT present the result as a faithful representation of the source. Producers requiring maximum compatibility may continue declaring all metrics at the model level.

## Custom Extensions

Custom extensions allow vendors to add platform-specific metadata without breaking core compatibility. Each extension includes a vendor name and arbitrary JSON data.
Expand Down Expand Up @@ -546,6 +682,16 @@ semantic_model:
expression: amount
description: Order amount

# Dataset-scoped: resolves entirely within the orders dataset
metrics:
- name: total_amount
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(amount)
description: Total order amount
datatype: Decimal

- name: customers
source: sales.public.customers
primary_key: [id]
Expand All @@ -565,6 +711,20 @@ semantic_model:
expression: email
description: Customer email

# Dataset-scoped: resolves entirely within the customers dataset
metrics:
- name: customer_count
expression:
dialects:
- dialect: ANSI_SQL
expression: COUNT(DISTINCT id)
description: Total number of customers
datatype: Integer
ai_context:
synonyms:
- "total customers"
- "customer base"

relationships:
- name: orders_to_customers
from: orders
Expand All @@ -573,27 +733,14 @@ semantic_model:
to_columns: [id]

metrics:
- name: total_revenue
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(orders.amount)
description: Total revenue from all orders
ai_context:
synonyms:
- "total sales"
- "revenue"

- name: customer_count
# Model-scoped: spans orders and customers via the relationship

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says "spans orders and customers via the relationship", but it applies to a metrics: list whose second entry (customer_count, line 722) is COUNT(DISTINCT customers.id), single-dataset. By the guidance at line 453 ("prefer dataset-scoped for simple aggregations that belong conceptually to one entity") it belongs on customers.metrics.

Commit 4 moved all three single-dataset metrics in the TPC-DS example for exactly this reason, so the two examples in the PR now teach opposite things. Either move customer_count onto customers here too, or scope the comment to revenue_per_customer alone.

- name: revenue_per_customer
expression:
dialects:
- dialect: ANSI_SQL
expression: COUNT(DISTINCT customers.id)
description: Total number of customers
ai_context:
synonyms:
- "total customers"
- "customer base"
expression: SUM(orders.amount) / COUNT(DISTINCT customers.id)
description: Average revenue per customer
datatype: Decimal

custom_extensions:
- vendor_name: SNOWFLAKE
Expand Down
39 changes: 37 additions & 2 deletions core-spec/spec.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,16 @@ semantic_model:
# See Relationships section below for detailed structure
relationships: []

# Optional:
# These metrics can span one or more logical datasets and use relationships
# Optional: Model-scoped metrics
# Expressions reference fields by qualified name (e.g., SUM(orders.amount)),
# unlike dataset-scoped metrics, which use unqualified names
# Names MUST be unique across the semantic model
# A model-scoped metric SHOULD NOT reuse the name of a dataset-scoped metric;
# validators warn rather than fail, and attribute the warning to the model
# An unqualified metric reference resolves to a model-scoped metric
#
# Whether a metric expression may reference another metric is not yet defined
# by this specification and is being handled as a separate proposal
# See Metrics section below for detailed structure
metrics: []

Expand Down Expand Up @@ -134,6 +142,29 @@ datasets:
# See Fields section below for detailed structure
fields: []

# Optional: Dataset-scoped metrics
# Metrics that aggregate data held by this dataset. Same structure as
# model-scoped metrics; see the Metrics section below.
#
# Rules (see core-spec/spec.md#metric-scoping):
# - The expression MAY reference this dataset's declared fields and the
# columns of its source. A declared field is written
# dataset_name.field_name; a source column is written unqualified:
# SUM(orders.net_amount) / COUNT(DISTINCT tax_id). A column used only
# inside a metric does not need to be declared as a field.
# - A qualified reference MUST name a declared field of this dataset.
# - An expression referencing another dataset belongs in
# semantic_model.metrics.
# - Names MUST be unique within this dataset and MUST NOT collide with a
# field name of this dataset.
# - Referenced from outside the dataset as dataset_name.metric_name. An
# unqualified metric name refers to a model-scoped metric.
#
# These rules constrain the EXPRESSION only, not the query. The metric is
# joined and grouped like any other, using the model's relationships, so it
# can be sliced by dimensions of any dataset the model connects.
metrics: []

# Optional: Vendor-specific attributes for extensibility
custom_extensions:
- vendor_name: string # Free-form string identifying the vendor
Expand Down Expand Up @@ -235,6 +266,10 @@ fields:
# Metrics Schema
# Quantitative measures defined on business data
# Represents key calculations like sums, averages, ratios, etc.
#
# The same structure is used in two placements:
# - semantic_model.metrics (model-scoped): may span datasets via relationships

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add here that semantic_model.metrics can be used to combine multiple dataset scoped metrics ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To Josh's PR review questions, this fits with the question on if / how to build metrics from other metrics. That's probably worth a followup PR / discussion since it touches on how to extend the expression language

# - datasets[].metrics (dataset-scoped): must resolve within one dataset
metrics:
# Required: Unique identifier for the metric
- name: string
Expand Down
Loading