Read plugin-declared settings from the environment, and say whether a missing one has a default

[Flix6x]

Description

Plugin-declared settings could only be set in a config file. Setting one as an environment variable — the natural thing to do in a container — had no effect, and FlexMeasures still reported it as missing. On top of that, the "missing setting" message repeated whatever the plugin put in message_if_missing, so a plugin promising "'NL' will be used as a default." made a setting that has no fallback read as optional.

• Plugin settings that are still unset when the plugin is registered are now looked up in the environment. Plugins are registered after the config file has been read, so a value from the config file keeps precedence and the environment only fills the gaps — the same order read_env_vars() applies to FlexMeasures' own settings.
• Environment values are interpreted as the parse_as type the plugin declared: int/float as numbers, bool as True for 1/true/yes/on (case-insensitively), list/dict as JSON. A value we cannot convert is passed on unconverted, so the existing type check reports it. Without parse_as, the value stays a string.
• A __settings__ entry can now declare a default, which is applied when the setting is missing.
• The message for a missing setting now ends with a statement from FlexMeasures itself: either Falling back to the default declared by the plugin: 'NL'. or No default is declared for it, so it stays unset. A plugin's own message_if_missing can no longer be the last word on whether a setting is optional.
• Documented the keys of a __settings__ entry, and where those settings can be set.
• Added changelog item in documentation/changelog.rst

As before, nothing is read from the environment while testing or while building the documentation, which both run on defaults.

On reading everything from the environment

The issue discussion raises whether read_env_vars() should simply promote every environment variable. This PR does not do that: the app config would then absorb every variable in the environment (PATH, cloud credentials, whatever the deployment happens to export), and Flask config values are visible to every plugin and template. Promoting exactly the settings a plugin declared gets the reported use case working without that. If we do want the broader change, it seems worth a separate discussion.

Look & Feel

A plugin declaring:

__settings__ = {
    "MY_PLUGIN_TOKEN": {"description": "Token used by my plugin.", "level": "error"},
    "MY_PLUGIN_COUNTRY": {"level": "warning", "message_if_missing": "'NL' will be used as a default.", "default": "NL"},
    "MY_PLUGIN_TIMEOUT": {"level": "warning", "parse_as": int},
}

started with MY_PLUGIN_TOKEN=repro-token-value MY_PLUGIN_TIMEOUT=30:

INFO Importing plugin repro_plugin ...
WARNING Missing config setting 'MY_PLUGIN_COUNTRY'. 'NL' will be used as a default. Falling back to the default declared by the plugin: 'NL'.
INFO Loaded plugins: {'repro_plugin': '0.1'}

MY_PLUGIN_TOKEN   = 'repro-token-value'
MY_PLUGIN_COUNTRY = 'NL'
MY_PLUGIN_TIMEOUT = 30

Before this PR, MY_PLUGIN_TOKEN and MY_PLUGIN_TIMEOUT were both None and reported as missing, and the MY_PLUGIN_COUNTRY warning stopped after 'NL' will be used as a default. while MY_PLUGIN_COUNTRY stayed None.

How to test

pytest flexmeasures/utils/tests/test_plugin_utils.py covers env promotion, config-file precedence, the testing-mode exemption, each parse_as conversion, an unparsable value, and the missing-setting message with and without a default.

The issue's Docker repro is the end-to-end check: build the one-setting plugin image, set the setting as a container environment variable, and the server now boots without reporting it missing.

Further Improvements

• Whether read_env_vars() should read all settings from the environment (see above) is left open.

Related Items

Closes #2492

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

🤖 Generated with Claude Code

https://claude.ai/code/session_01X2KPuf29L3FLQhfcvdg85x

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #34465422 | 📁 Comparing b453918 against latest (b598863)

  🔍 Preview build  

5 files changed · ± 5 modified

± Modified

• changelog.html
• genindex.html
• _autosummary/flexmeasures.utils.plugin_utils.html
• api/v3_0.html
• plugin/customisation.html

[Copilot]

🟡 Changes recommended

It drops all existing register_plugins regression tests (not found elsewhere) and has a correctness gap where default values aren’t type-validated when parse_as is declared.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR improves plugin configuration handling by allowing plugin-declared __settings__ to be populated from environment variables (with config-file precedence), adds optional plugin-declared defaults, and makes the “missing setting” log message explicitly state whether a default is applied.

Changes:

• Read unset plugin-declared settings from environment variables, parsing values according to parse_as.
• Support a default key for plugin settings, and append an explicit default/unset statement to missing-setting logs.
• Update plugin documentation and add a main changelog entry describing the new behavior.

File summaries

File	Description
flexmeasures/utils/plugin_utils.py	Reads plugin settings from env, applies defaults, and clarifies missing-setting logging
flexmeasures/utils/tests/test_plugin_utils.py	Replaces prior plugin-loader tests with new tests for plugin setting env/default behavior
documentation/plugin/customisation.rst	Documents __settings__ keys and where plugin settings can be set (config vs env)
documentation/changelog.rst	Adds a changelog entry for plugin settings env/default support

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Flix6x]

Leaving this one. The ({description}). format predates this PR, so plugins that write a description ending in a period already get (... .). today; changing it would alter the log output of every plugin, which is beyond what this PR is about. Happy to do it as a follow-up if maintainers want it.

[Copilot]

🟢 Approval recommended

The behavior change is well-scoped, matches the PR/issue requirements, and is covered by targeted tests and updated documentation.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Flix6x]

@nhoening ready for review. Summary of where this landed, including your two design questions on the issue:

Should plugin settings be readable from env? Yes, but scoped: settings a plugin declares in __settings__ are looked up in the environment when the plugin registers, and only if still unset — so the config file keeps precedence, matching the order read_env_vars() uses for FlexMeasures' own settings. I did not go all the way to "read everything from env", since that would pull every environment variable (PATH, cloud credentials, …) into the Flask config where every plugin and template can see it. That broader change is left open in the PR description if you'd rather have it.

Should a plugin be able to claim a default it doesn't have? It still can, but it no longer gets the last word: FlexMeasures now appends its own statement, either Falling back to the default declared by the plugin: 'NL'. or No default is declared for it, so it stays unset. A plugin can now declare a real default, which is applied and type-checked like any other value.

Copilot reviewed twice; its first pass caught that I had overwritten this repo's plugin-loader tests, which are restored (the file's diff against main is additions only).

[nhoening]

Thanks!