Feat: forecast input bounds

[BelhsanHmida]

Description

Stacked on #2542 — base this PR on feat/forecast-target-source-filter, not main. It depends on that PR's change making the forecast target accept a sensor reference.

lower, upper and snap currently shape a forecast on its way out, right before it is stored. This PR lets the same three fields be set on an individual regressor or on the target, where they clean that sensor's readings on the way in, before the model trains on them. It is for a sensor whose recorded data is not trustworthy as it stands — an occasional implausible spike, or an error sentinel such as -9999 — that you would otherwise have to correct upstream or copy to a second sensor.

• Regressor and target references accept lower, upper and snap alongside the source filters they already take. A bare sensor ID keeps working unchanged.

• The bounds live on the shared SensorReferenceSchema, so any sensor reference can carry them. Only forecasters apply them in this PR: flex-model and flex-context references refuse them for now, rather than accepting bounds they would ignore. Scheduling applies them in the follow-up, Let the scheduler clean its inputs with bounds on a sensor reference #2561.

• Bounds that cannot be applied to the referenced sensor (an incompatible unit, a snap target outside its interval, a lower bound above the upper bound) are refused when the reference is loaded, not inside the queued job.

• Each sensor's bounds are read in that sensor's own unit, not the unit of the sensor being forecast.

• Bounding runs after gap filling, so a value interpolated across a gap is bounded too.

• Snapping and clipping reuse the output path's helpers, now in flexmeasures/utils/bound_utils.py, so the snap-then-clip order and the [first, second) interval rule are identical on both sides by construction. A reference parses its bounds once, not once per read.

• Bounds survive the queued-job payload, and are omitted when unset so a reference without cleaning serialises exactly as before.

• Also fixes a pre-existing bug found on the way: detect_and_fill_missing_values handed the model N copies of all N regressor columns. That fix is split out into Hand the forecasting model each regressor once, not once per regressor #2560 (with its changelog entry), so it can land and be backported on its own; the same commit stays here, since this PR builds on it.

• Changelog: one entry under New features.

• Added changelog item in documentation/changelog.rst

Look & Feel

Regressor bounds go in the forecaster config, next to the regressors:

{
  "past-regressors": [
    2094,
    {"sensor": 2095, "lower": "0 kW", "snap": {"0 kW": ["0 kW", "0.5 kW"]}}
  ]
}

The target is named in the forecast parameters rather than the config, so its bounds go there:

{
  "sensor": {"sensor": 2092, "upper": "20 kW"}
}

Both are passed with flexmeasures add forecasts --config / --parameters, or as the API config / parameters payload. As with the existing output bounds, there are no dedicated CLI flags.

Given a regressor reading [-5, 0.3, 99, <gap>, 4] with lower: 0 kW, upper: 20 kW and snap: {"0 kW": ["0 kW", "0.5 kW"]}:

	values reaching the model
without bounds	[-5.0, 0.3, 99.0, 51.5, 4.0]
with bounds	[0.0, 0.0, 20.0, 20.0, 4.0]

-5 clips to 0, 0.3 snaps to 0, 99 clips to 20, and the gap — which interpolates to 51.5 between 99 and 4 — is clipped to 20 as well, because bounding runs after filling.

How to test

flexmeasures/data/tests/test_forecasting_pipeline.py:

• test_input_bounds_clean_a_regressor_after_its_gaps_are_filled
• test_input_bounds_leave_an_unbounded_regressor_alone
• test_input_bounds_are_read_in_the_regressors_own_unit
• test_input_bounds_reject_a_unit_the_regressor_cannot_take
• test_filling_gives_each_regressor_exactly_one_component

flexmeasures/data/schemas/tests/test_forecasting.py:

• test_forecaster_config_schema_loads_regressor_cleaning_bounds
• test_forecaster_config_schema_keeps_an_unbounded_regressor_a_plain_sensor
• test_forecaster_parameters_schema_loads_target_cleaning_bounds
• test_forecaster_config_schema_rejects_an_unparseable_regressor_bound
• test_cleaning_bounds_live_on_the_shared_sensor_reference
• test_a_reference_without_bounds_serialises_as_it_did_before
• test_sensor_reference_refuses_bounds_it_cannot_apply_when_loaded
• test_forecaster_config_schema_refuses_a_regressor_bound_in_an_incompatible_unit

flexmeasures/data/schemas/tests/test_sensor.py:

• test_scheduling_references_refuse_bounds_they_would_ignore

What was broken to prove these are not vacuous:

• Making input bounding a no-op reddened the three bounding tests.
• Restoring the whole-frame copy reddened test_filling_gives_each_regressor_exactly_one_component (four components instead of two).
• Bounding against the target's unit instead of the regressor's reddened the unit test.
• Skipping the unit check when a reference is loaded reddened all cases of test_sensor_reference_refuses_bounds_it_cannot_apply_when_loaded and test_forecaster_config_schema_refuses_a_regressor_bound_in_an_incompatible_unit.
• Removing the refusal from flex-model and flex-context references reddened test_scheduling_references_refuse_bounds_they_would_ignore.

test_input_bounds_leave_an_unbounded_regressor_alone survives all of those breaks. It is a control, not a binding test — it pins the unbounded baseline (51.5) that gives the "after filling" claim in the test above it its meaning, and it would catch a regression that made bounding unconditional. Read it as documentation of the baseline rather than as coverage of the feature.

The existing output post-processing tests were left untouched and still pass, which is what establishes that pulling parse_bounds / apply_bounds_to_values out of apply_forecast_post_processing did not change output behaviour.

Further Improvements

• Clipping after filling has a sharp edge. An out-of-range reading is not neutralised until after it has been used to interpolate its neighbours. Given 10, -9999, <gap>, 14 with lower: 0, the gap interpolates from -9999 and is then clipped to 0, where filling between 10 and 14 would have given roughly 12. Where readings are wrong rather than merely out of range, the more honest operation may be to treat out-of-bounds values as missing and let interpolation fill across them, instead of clipping them to the bound and injecting a fabricated 0 into the training data. That interacts with the missing_threshold accounting, so it is left as a separate decision.
• Scheduling applies the bounds in Let the scheduler clean its inputs with bounds on a sensor reference #2561. It lifts the refusal on flex-model and flex-context references.
• Deleting a sensor does not prune references that carry a default or bounds from stored flex configs: Deleting a sensor leaves flex-config references to it behind when they carry a default (or bounds) #2558.
• test_forecasting_pipeline.py mixes the db and fresh_db fixtures (2 uses against 49), which .github/instructions/testing.instructions.md explicitly warns will hang CI. Pre-existing and untouched here, but worth its own issue.

Related Items

• Builds on Let a forecast target name the sources it trains on #2542 (forecast target source filter), which this PR is stacked on.
• Let the scheduler clean its inputs with bounds on a sensor reference #2561 applies the bounds in scheduling, and is stacked on this PR.
• Hand the forecasting model each regressor once, not once per regressor #2560 carries the duplicated-regressor fix on its own.
• Extends the forecast post-processing added in Feat: add configurable forecast post processing  #2273, which introduced lower, upper and snap for forecast output.

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #34619694 | 📁 Comparing 836b807 against latest (e3cf3ba)

  🔍 Preview build  

20 files changed · + 1 added · ± 19 modified

+ Added

• _autosummary/flexmeasures.utils.bound_utils.html

± Modified

• changelog.html
• genindex.html
• py-modindex.html
• _autosummary/flexmeasures.api.v3_0.assets.html
• _autosummary/flexmeasures.cli.utils.html
• _autosummary/flexmeasures.data.models.forecasting.pipelines.base.html
• _autosummary/flexmeasures.data.models.forecasting.pipelines.predict.html
• _autosummary/flexmeasures.data.models.forecasting.pipelines.train.html
• _autosummary/flexmeasures.data.models.forecasting.pipelines.train_predict.html
• _autosummary/flexmeasures.data.models.forecasting.utils.html
• and 9 more...

[Flix6x]

The bounds live on a forecaster-specific reference schema rather than the shared SensorReferenceSchema, which flex-model and flex-context also use and where these keys would mean nothing. That schema keeps refusing them.

Extending the shared SensorReferenceSchema would actually make this a much stronger feature. It is arguably an even more useful feature for data inputs to the scheduler.

Three mechanisms now bound values. ensure_positive still clips negatives to zero inside the model, next to the output bounds and now these input bounds. Worth deciding whether it should be deprecated in favour of the explicit bounds.

No rush on formally deprecating this, but worth a comment in the code that it is meant to be deprecated. If in the user documentation we advise using it (perhaps in an example) then let's replace that example with the new way.

Regressor bounds sit in the config while target bounds sit in the parameters, because that is where each sensor is named. Inherited rather than introduced here, but it makes the feature read as two places to configure one idea.

That's completely fine. I actually think all sensor references could use this.

[Copilot]

🟡 Changes recommended

Schema/OpenAPI metadata and error-handling/docstring convention issues need to be addressed before this can be safely approved.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR extends the forecasting pipeline to support input-side cleaning bounds (lower, upper, snap) on both target and regressor sensor references, so training data can be snapped/clipped before model fitting. It also fixes a regressor gap-filling bug where each per-sensor pass accidentally duplicated all columns into the Darts series, leading to duplicated model components.

Changes:

• Add a forecaster-specific sensor reference schema (ForecastInputReference*) that supports optional input bounds while keeping the shared SensorReferenceSchema strict.
• Apply input bounds after missing-value filling in BasePipeline.detect_and_fill_missing_values, and refactor output post-processing to reuse shared bound parsing/application helpers.
• Add/adjust tests and documentation, plus changelog entries for both the new feature and the regressor-component duplication bugfix.

File summaries

File	Description
flexmeasures/data/tests/test_forecasting_pipeline.py	Adds pipeline-level tests for input bounds and for the “one component per regressor” regression.
flexmeasures/data/schemas/tests/test_forecasting.py	Adds schema tests ensuring bounds load correctly for forecasters and remain rejected on shared sensor references.
flexmeasures/data/schemas/forecasting/references.py	Introduces ForecastInputReference, ForecastInputReferenceSchema, and ForecastInputField to support bounds on forecaster inputs.
flexmeasures/data/schemas/forecasting/pipeline.py	Switches forecaster regressor/target fields to ForecastInputField and updates schema behavior accordingly.
flexmeasures/data/models/forecasting/utils.py	Extracts reusable bound parsing/application (parse_bounds, apply_bounds_to_values) and reuses it for output post-processing.
flexmeasures/data/models/forecasting/pipelines/train_predict.py	Ensures queued-job payload round-trips forecasting-specific references (including bounds).
flexmeasures/data/models/forecasting/pipelines/base.py	Fixes regressor duplication during filling, and applies input bounds after filling via shared helpers.
documentation/features/forecasting.rst	Documents how to configure input-side bounds for regressors and the target.
documentation/changelog.rst	Adds changelog entries for the new feature and the regressor duplication bugfix.

Review details

• Files reviewed: 9/9 changed files
• Comments generated: 7
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[BelhsanHmida]

The bounds live on a forecaster-specific reference schema rather than the shared SensorReferenceSchema, which flex-model and flex-context also use and where these keys would mean nothing. That schema keeps refusing them.

Extending the shared SensorReferenceSchema would actually make this a much stronger feature. It is arguably an even more useful feature for data inputs to the scheduler.

Three mechanisms now bound values. ensure_positive still clips negatives to zero inside the model, next to the output bounds and now these input bounds. Worth deciding whether it should be deprecated in favour of the explicit bounds.

No rush on formally deprecating this, but worth a comment in the code that it is meant to be deprecated. If in the user documentation we advise using it (perhaps in an example) then let's replace that example with the new way.

Regressor bounds sit in the config while target bounds sit in the parameters, because that is where each sensor is named. Inherited rather than introduced here, but it makes the feature read as two places to configure one idea.

That's completely fine. I actually think all sensor references could use this.

All addressed, plus the seven Copilot comments

Shared schema: you're right, moved. lower/upper/snap now sit on SensorReferenceSchema, so any reference can carry them; the forecaster-specific schema is gone. That also fixes the OpenAPI Copilot flagged, by construction. Only forecaster inputs act on them so far — the fields say so, the way default documents its own gap.

Scheduling: follow-up PR, landing before v1.1. Small change, but it sits behind ~50 scheduling call sites, and a wrong bound there produces a plausible schedule rather than an error — not what I want in an rc. Changelog stays one entry, both PR links.

ensure_positive: flagged as deprecated in the description and at the line that applies it. Nothing in documentation/ recommends it, so no example to migrate.

[Copilot]

🟡 Changes recommended

There are a few correctness/documentation inconsistencies in the changed code paths (notably bounds refusal logic for scheduling references and ensure_positive default documentation) that should be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 14/14 changed files
• Comments generated: 4
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The current SensorIdOrReferenceField treats no-op bound keys as “cleaning requested” (changing backward-compatible deserialization/serialization), and boolean values are unintentionally accepted as numeric bounds.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

flexmeasures/utils/bound_utils.py:35

• _quantity_to_sensor_value accepts bool as a numeric bound because bool is a subclass of int (numbers.Real). That can silently turn true/false into 1.0/0.0 bounds.

It’s safer to explicitly exclude booleans from the numeric fast-path.
flexmeasures/utils/unit_utils.py:63

• is_parseable_quantity currently treats bool values as parseable quantities because bool is a subclass of int (numbers.Real). That means schema validation would accept true/false as bounds and later interpret them as 1.0/0.0, which is very likely unintended input.
  flexmeasures/utils/bound_utils.py:218
• apply_bounds_to_values says “Values that are not a number are left alone”, but the implementation coerces values to dtype=float, which will raise on non-numeric inputs (it only preserves NaNs). To avoid misleading readers, consider tightening the wording to describe NaN handling explicitly.

• Files reviewed: 14/14 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It changes core forecasting schema and pipeline behavior (input data cleaning and series construction), which merits final human review despite strong test coverage.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

flexmeasures/data/schemas/sensors.py:386

• The _sets_bounds docstring summary does not end with a period, which is inconsistent with the repo’s docstring convention and can make the docs/readability uneven.

• Files reviewed: 14/14 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Flix6x]

Nice work!