fix: integrate select variables in dataloader

[icedoom888]

Description

Solves #899

What problem does this change solve?

What issue or task does this change relate to?

Additional notes

As a contributor to the Anemoi framework, please ensure that your changes include unit tests, updates to any affected dependencies and documentation, and have been tested in a parallel setting (i.e., with multiple GPUs). As a reviewer, you are also responsible for verifying these aspects and requesting changes if they are not adequately addressed. For guidelines about those please refer to https://anemoi.readthedocs.io/en/latest/

By opening this pull request, I affirm that all authors agree to the Contributor License Agreement.

[JPXKQX]

I think this needs to be discussed. These parameters are added just to the open_dataset call, should we add all anemoi dataset kwargs to this call (cutout, missing, join, ...)?

[dietervdb-meteo]

LGTM, lets run integration tests just to be sure:
https://github.com/ecmwf/anemoi-core/actions/runs/22057431865

[anaprietonem]

this is also related to the issue @VeraChristina opened last week - #891

[JPXKQX]

The solution to this is to pass all anemoi-datasets specifics arguments inside the dataset: argument, which is exactly the config passed to the open_dataset function.

[dietervdb-meteo]

@JPXKQX , so why did we have the separate drop and frequency arguments? They could be read from the open_dataset config as well?

[dietervdb-meteo]

Ok, see you already commented on that at #899 (comment)

[MicheleCattaneo]

Nice catch.
It feels like these bugs (there was another about beta's for the optimizers) are a bit sneaky due to the extra = "ignore" default value of pydantic's schemas.
What if extra was set to either "allow" or "forbid" everywhere and we let the object's constructor fail if the argument is not expected? Silently dropping arguments sounds too dangerous.
If "ignore" is really needed, maybe it's possible to add some kind of hook that at least throws a warning?

[icedoom888]

@JPXKQX this is a big change compared to how we used to write configs so far.
Has this been documented anywhere?

[mchantry]

@icedoom888 thanks for your contribution. It's true that we have an asymmetry between drop and select at present.
My suggestion is that we merge this lovely contribution (and the one from Vera that will help catch more of these).
But we also revisit in the near future moving all of these keywords to the lower level within datasets.