Allow configuring the search path via the config file

[omry]

Design doc outlining consideration and possible implementations:

https://docs.google.com/document/d/1Dx77SThg1ugnGHvZ8Dq1FQusXDvGn0tyyWBeJ3nXvro/edit?usp=sharing

[asherp]

Not sure if this is the feature I'm waiting for, but I want my users to install my app and easily point to a new config directory that overrides my defaults. My app is based on the hydra_app_example. Please advise.

[asherp]

Ok, I think I figured it out. First add a default search path to config.yaml:

search_path: config.yaml

Then merge it using OmegaConf

import hydra
from omegaconf import OmegaConf

@hydra.main(config_path='conf/config.yaml', strict = False)
def main(cfg):
    if cfg.search_path is not None:
        override_path = hydra.utils.to_absolute_path(cfg.search_path)
        override_conf = OmegaConf.load(override_path)
        cfg = OmegaConf.merge(cfg, override_conf)

Now the user can run your app from any directory containing a config.yaml, or specify their own:

myapp search_path=path/to/user/config.yaml

[omry]

That's a pretty good workaround. I would call this something like config_override and not search_path, as search path is an internal concept in Hydra that will eventually be exposed.

Asher, can you share a bit about your project? I am curious about how people are using Hydra and for what.

[asherp]

@omry It's for a project called Kamodo, which is a functional, publication-focused API for space weather models and data (though it can be used for much more!). Kamodo handles function composition, unit conversion, and automatic plot generation via function inspection, which should minimize the effort needed to create analysis pipelines and perform data-model comparisons. Kamodo is built with sympy and plotly.

Most of our science users are comfortable with command-line but are inexperienced with modern programing languages. The CLI for kamodo will allow them to configure their data sources and any equations they want to use for post processing and quickly generate plots. Our users will often test different models or data sources for science relevance and will want to easily reproduce those results, so Hydra's ability to compose several overlapping configurations and have timestamped output directories is perfect for our use case.

Github - https://github.com/nasa/Kamodo
Official site - https://ccmc.gsfc.nasa.gov/Kamodo/

I'm also considering Hydra for my other project Hourly, which uses git "clock in" and "clock out" messages to track hours worked, generate time sheets, and produce work charts. Since I started freelancing, I've logged at least 1000 sessions across 5 different projects using Hourly.

Currently, Hourly's cli is based on click, but I need to provide default functionality specific to a particular repo (to ignore errant clock-in/outs, etc). I'd also like to generate time sheets from related projects, so config composition would come in handy. Finally, I'm usually the only one who uses hourly, but eventually I'll need to iterate over all contributors to a project (assuming I get more users!).

Project page - https://asherp.github.io/hourly/index.html
Github - https://github.com/asherp/hourly

[omry]

Kamodo looks awesome, and I am really happy to start seeing serious projects that are making use of Hydra.

A few notes:

• Can you add Hydra as a dependency of Kamodo (in setup.py/cfg or requirements.txt). this will allow GitHub to detect it as a project using Hydra.
• Did you see the new compose API? it lets you compose configs with Hydra anywhere, not just in the main. This is specifically useful for Jupyter notebooks. here is an example (you can launch the binder to play with it online).
• Take a look at Pytorch, it might be very useful for Kamodo as an alternative to numpy. It allows highly optimized vector/matrix operations for both CPU and GPU, as well as automatic differentiation.
• Hourly looks like a nice and useful project for freelancers, it would be great if you use Hydra there as well, and I would love to know about your experience porting from Click to Hydra. Click is super popular but I think Hydra is much more powerful and it can probably be used as a replacement for Click for most use cases.
• I bet NASA got some serious compute clusters. While there are no official public plugins for it yet, we are using Hydra at FAIR to launch to our internal cluster. Hydra has a plugin mechanism that can support any cluster (including private clusters with non public APIs).
  I would be happy to guide you in writing a plugin that will allow you to use Hydra to launch to your internal cluster from the command line.

I am considering creating a "Who is using Hydra" section in the docs, would you agree to write a short testimonial saying why you find it useful?

[omry]

One more thing:
As a framework, Kamodo might want to provide some configuration to its users.
For example, some default configurations to instantiate commonly used models etc.
The best way to do it with Hydra is to add those configs to the search path.

You can do it programmatically and automatically via the search plugin API.
I don't yet have a super clean example for this in hydra/plugins/examples, but the colorlog plugin comes very close.
Essentially this is all its doing. check it out here.

[Stonesjtu]

I have the similar use cases as @asherp .

• The major interface of my program is in binary form, most users don't write any python codes.
• The default value is defined in structured config rather than a default config file
• Users write their own config file in their work dir, and runs through python -m xxx.main

@omry
I think the current version 1.0rc supports specify config_path, but the path is resolved relative to the main.py file, so I have to specify absolute path to make it work.

Can we resolve the config_path relative to the $pwd, which makes more sense in CLI cases

[omry]

You can probably achieve that now via a config searchpath plugin.
example.

[omry]

by the way, take a look at the Application Packaging page.

1. You CAN use config files with your application even if it's installed.
2. You can get a native command to execute it instead of the clunky python -m module.