add base.context.dump to save experiment config as metadata

[vlap]

Adds toyaml (and to_yaml alias) Jinja filter to dump SE context as a yaml file (via yaml.dump).

[vlap]

hi @uwefladrich .

I am implementing "saveic" in ECE4. To make saved initial conditions fully self-documenting and traceable, I'd like saveic to also save a snapshot of the experiment config that defines the model setup (component list, resolutions/grids, forcing settings, and dates) that generated those initial conditions.

I did not find an easy way to dump from ScriptEngine in-memory context. With the new toyaml filter I can create
experiment-config.yml.j2 template

       {{ {'base.context': {'experiment': experiment, 'model_config': model_config}} | toyaml }}

and render it

            - base.template:
                src: "experiment-config.yml.j2"
                dst: "saveic/experiment-config.yml"

I hope this PR makes sense to you. Let me know if there is a better way to achieve the same.

[vlap]

when testing, I realized implementing a toyaml filter is complicated by ! tags, which are not rendered well.

So, I decided to add base.context.dump, a counterpart of base.context.load

[vlap]

I thought about copying the input file but it's fragile (cli args position uncertain, can be more than 1). So, base.context.dump is needed

   ```bash
   se my-user-settings.yml ../platforms/marenostrum5.yml my-experiment.yml scriptlib/main.yml

Also, some meaningful context, added via base.include can be missed (such as model tuning presets).

It can also filter out platform and job context to the model physics and experiment parameters.

base.context.dump is not meant to create a SE yml files with - base.context, just pure YAML, which can be loaded back into the context directly in downstream scripts using base.context.load.

[uwefladrich]

Hi @vlap, thanks for the PR!

Reasonable to have a base.context.dump task and also your implementation is good, imo! In fact, I thought to have it at the time I implemented base.context.load, but there were some issues that I couldn't make my mind up at the time so it never realized.

The main point is that base.context.dump is not exactly the counter part to base.context.load, which the user might expect:

• the se. namespace can't be copied, you also handle this in the code
• YAML contructor semantics might be lost, but I understand that you handle that as well
• include logic is lost
• not sure if some Jinja-parsing can be lost?
• more?

This is not a show stopper, rather the nature of base.context.dump, which dumps a certain state of the context and does not copy SE scripts. Maybe this could be further clarified in the documentation (I'll have a go).

So I am fine to go ahead with this new feature! 😃

One more technical question: You added duplicated constructors like

yaml.add_constructor(..., ..., Loader=yaml.SafeLoader)

what's the reason for that? I don't doubt it, I am just not familiar with this procedure.

[vlap]

Hi Uwe,

Thank you for the review. Agreed, it's not a true counter part to base.context.load. It is supposed to be a snapshot of the model runtime dictionary. Plus representers to handle tags. Clarifying this distinction in the documentation (that base.context.dump dumps model state context rather than re-creating SE scripts) is the right call.

Regarding the duplicated constructor (SafeLoader), it was added to allow calling base.context.load on the output of base.context.dump. (base.context.load (and SEContext.load_from_yaml) uses Loader=yaml.SafeLoader) But there is no real use-case for it that I can think of. There is already base.context. So, it is removed now. Thanks for pointing this out!

Happy to have assign base.context.dump any other name that you suggest if you have an idea.

[vlap]

I simplified this PR a bit. Hope it is more transparent now.

Tested to work. Let me know if there are other doubts.

[uwefladrich]

Great, I like it 👍 Also, nice to have the option to give a list of keys!

I would keep the name base.context.dump, I think it suggests the intent well.

Will merge later today!