Allow for processing of all available variables

[maxnutz]

Current status:
The code selects all the variables available, that are also part of the variable definition linked by the configuration "definitions_path". If an evaluation of a variable is available but the variable is not defined in the definitions in "definitions_path", that evaluation is not processed.

Goal:
Allow for the value "False" in the config "definitions_path". The value "False" in the config "definitions_path" should trigger the following behaviour:

• all variables available in the mapping file (default: mapping.default.yaml, can be set in configs "mapping_path") are evaluated. The output processing is exactly the same, but the units are not converted to the units defined in the definitions_path, but are kept as they are after the evaluation.

Acceptance:

• When definitions_path is set to a path, the current behaviour stays untouched.
• When definitions_path is set to "False", all variables are evaluated, units are kept unchanged and the evaluations are written to file – the evaluation and post processing is exactly the same as with a definitions_path, except the unit conversion.
• When defintions_path is empty, the code throws a comprehensive error.

[maxnutz]

Issue #98 — Allow definitions_path: False to process all available variables

First action on approval

Post the full contents of this plan as a comment on GitHub issue #98 via
gh issue comment 98 --body-file <this file>, before starting any code
changes.

Context

Today definitions_path (config.default.yaml) is a required path to an IAMC
variable-definitions folder. Network_Processor uses it for two things:

1. Variable selection — calculate_variables_values() iterates only
   over self.dsd.variable... (the definitions), so a variable that has a
   statistics function and a mapping entry but no definitions-folder entry
   is silently skipped.
2. Unit conversion — _convert_units_to_common_definitions() converts
   every output variable to the unit declared in self.common_dsd
   (also loaded from definitions_path).

Issue #98 asks for an escape hatch: setting definitions_path to False
should evaluate every variable in the mapping file (mapping_path,
default mapping.default.yaml) and skip unit conversion entirely, while
leaving all other post-processing (aggregation, Excel export) unchanged.
Setting it to an actual path must keep behaving exactly as today, and
leaving it empty/unset must raise a clear error explaining both valid
forms.

Design

All changes are in pypsa_validation_processing/class_definitions.py
(Network_Processor), plus tests in tests/test_network_processor.py and
tests/test_unit_conversion.py. No files in configs/ or definitions/
are touched, per CLAUDE.md.

1. New helper: detect the "disabled" sentinel

Add a static method near _is_valid_country_identifier:

@staticmethod
def _is_definitions_disabled(value: object) -> bool:
    """Check whether a definitions_path config value requests disabling definitions."""
    if isinstance(value, bool):
        return value is False
    if isinstance(value, str):
        return value.strip().lower() == "false"
    return False

This accepts both definitions_path: False (YAML bool) and
definitions_path: "False" / "false" (quoted string), per your answer.

2. __init__ — validate and branch (replaces lines ~167–181)

definitions_path = self.config.get("definitions_path", None)
is_empty = definitions_path is None or (
    isinstance(definitions_path, str) and definitions_path.strip() == ""
)
if is_empty:
    raise ValueError(
        f"'definitions_path' not set in config at {self.config_path}. "
        "Set it to a valid definitions folder path, or to False to "
        "evaluate all variables from the mapping file without unit "
        "conversion."
    )

self.use_definitions: bool = not self._is_definitions_disabled(definitions_path)

if self.use_definitions:
    self.definitions_path: Path | None = Path(definitions_path)
    if not self.definitions_path.exists():
        raise FileNotFoundError(
            f"Definition folder does not exist: {self.definitions_path}"
        )
else:
    self.definitions_path = None

if self.use_definitions and self.config.get("convert_units", True):
    self.common_dsd: nomenclature.DataStructureDefinition | None = (
        nomenclature.DataStructureDefinition(self.definitions_path)
    )
else:
    self.common_dsd = None

Note: the else: self.common_dsd = None branch is new and fixes a
pre-existing bug — currently, if convert_units: false is set (independent
of this issue), self.common_dsd is never assigned at all, so
_convert_units_to_common_definitions() raises AttributeError instead of
skipping conversion gracefully. Since this exact block is being rewritten
to add the use_definitions branch anyway, fixing it here is a one-line,
same-block change. Flagging it explicitly — this is the one piece of
"bonus" cleanup in the plan.

Later in __init__ (replaces line 209):

self.dsd: nomenclature.DataStructureDefinition | None = (
    self.read_definitions() if self.use_definitions else None
)

3. calculate_variables_values() — variable source (replaces line 802)

variables = (
    self.dsd.variable.to_pandas()["variable"]
    if self.use_definitions
    else list(self.functions_dict.keys())
)
for variable in variables:
    ...

self.functions_dict (from _read_mappings()) already holds every
IAMC variable name as a key (see mapping.default.yaml), so this reuses
existing state — no new I/O.

4. Unit conversion — no code change needed

_convert_units_to_common_definitions() already does
if self.common_dsd is None: return iam_df unchanged. With
self.common_dsd now correctly None whenever use_definitions is
False (regardless of convert_units), this path is skipped automatically.

5. __repr__ — cosmetic (replaces line 259)

f"  definitions_path: {self.definitions_path if self.use_definitions else 'False (all variables, no unit conversion)'}\n"

6. Docstrings

Update the class docstring (~line 113–123) and calculate_variables_values
docstring to describe the definitions_path: False behavior, and add the
new ValueError case to __init__'s Raises section.

7. README.md

Add one line under the definitions_path entry in the example config
block documenting the False option (mirrors the existing config comment
style, not a configs/ file so it's not covered by the CLAUDE.md restriction).

Tests

tests/test_network_processor.py (extend TestNetworkProcessorInit):

• definitions_path: false (YAML bool) → use_definitions is False,
  dsd is None, common_dsd is None, definitions_path is None.
• definitions_path: "False" (quoted string) → same assertions.
• definitions_path: "" → ValueError, message mentions both a path and False.
• existing test_init_missing_definitions_path_key_raises_value_error keeps
  passing unchanged (message still contains "definition_path", message text
  is only extended, not changed at the matched substring).
• existing test_init_nonexistent_definitions_path_raises unaffected (real
  path still validated with .exists()).

New test in TestAggregation/calculate_variables_values area:

• with use_definitions=False and a mocked functions_dict of 2 entries,
  assert _execute_function_for_variable is called once per mapping key
  (not once per self.dsd entry).

tests/test_unit_conversion.py:

• convert_units: false with a real definitions_path → common_dsd is None
  (currently would AttributeError; documents the bug fix from step 2).
• definitions_path: False with convert_units: true → common_dsd is None
  (definitions-disabled overrides convert_units).

Verification

1. pixi run test — full suite, including the new/updated tests above.
2. Manual check: point a scratch config at network_results_path with
   definitions_path: False, run
   Network_Processor(config_path=...).calculate_variables_values() +
   write_output_to_xlsx(), and confirm the output Excel contains one row
   per mapping-file variable with units unchanged from the raw statistics
   function output (e.g. via pixi run workflow_test against a scratch
   config copy — do not modify the checked-in configs/*.yaml).
3. git diff -- pypsa_validation_processing/class_definitions.py review
   before committing, confined to the sections above.