feat: Add support for hydra style overrides

[hemildesai]

What does this PR do ?

Adds support for Hydra style overrides. See https://hydra.cc/docs/advanced/override_grammar/basic/#basic-override-syntax and https://hydra.cc/docs/advanced/override_grammar/extended/#extended-override-syntax

Issues

Closes #60

Usage

• You can potentially add a usage example below

python examples/run_grpo_math.py +new_key="sort([1,2,3,4])"

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• ...

[terrykong]

Are we currently mandating that the config be read-only after overrides are applied? If not, can that be added?

[hemildesai]

Are we currently mandating that the config be read-only after overrides are applied? If not, can that be added?

The config is converted to a dict after overrides are applied. The dict can be modified at any place in the code as of now. If we want to freeze the dict then we will need to convert it to a FrozenDict or similar.

[SahilJain314]

Are we currently mandating that the config be read-only after overrides are applied? If not, can that be added?

We currently have places where we modify the configs: https://github.com/NVIDIA/reinforcer/blob/main/examples/run_grpo_math.py#L255

how should we deal with this sort of thing? It's not actually obvious to me if we should hard prevent people from using the configs for global variable passing for minor things like this.

[terrykong]

how should we deal with this sort of thing? It's not actually obvious to me if we should hard prevent people from using the configs for global variable passing for minor things like this.

Could we do frozen dict as default and just expect the user to manually unfreeze to add? I agree forbidding users from adding things into the config is a little extreme. @hemildesai where did tron end up falling here?

[hemildesai]

Yeah I think we can enforce freezing and unfreezing of the config dict if/when needed. However, that would be outside the scope of this PR. Let's move the discussion to #106

@hemildesai where did tron end up falling here?
@terrykong Can you expand on what you mean by this?