[Feature Request] Non-mutually exclusive config groups

[apsdehal]

🚀 Feature Request

Allow hydra to load multiple configurations in a single compositional block.

Motivation

For multitasking, one needs to load configurations for multiple datasets. Currently the way hydra is structure, I don't see a clear way of loading multiple configs in a single block.

Pitch

For e.g. let's say defaults look something like:

defaults:
  - dataset:
     - ???

and datasets folder is something like:

conf/
    dataset/
        vqa2.yaml
        textvqa.yaml

If I run, python main.py dataset=vqa2,textvqa, I should get a final config object something like

dataset:
  - vqa2: ...
  - textvqa: ...

We have something like this already implemented in Pythia's configuration system which is very close to hydra.

Are you willing to open a pull request? No

[omry]

Functionally speaking, I think #235 would address this.

In the short term, you can use lower level OmegaConf APIs to do the composition in your code.

Closing as dup, feel free to continue discussing here if you want further discussion.

[omry]

The fundamental issue is the options of each config group are mutually exclusive.
maybe I need to introduce a non-mutually-exclusive configuration group.

defaults:
  - dataset: a
  - dataset: b
  - dataset: c

Your suggestion for nesting there is syntactic sugar on this, and can actually be used to indicate this is one such special config group.

[omry]

Here are the docs.
This assumes knowledge of The Defaults List.