support Python3.11

[skshetry]

Motivation

When trying to import hydra in Python 3.11, it throws following error:

  File "/home/user/projects/iterative/hydra/hydra/conf/__init__.py", line 75, in JobConf
    @dataclass
     ^^^^^^^^^
  File "/home/user/.pyenv/versions/3.11-dev/lib/python3.11/dataclasses.py", line 1221, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File "/home/user/.pyenv/versions/3.11-dev/lib/python3.11/dataclasses.py", line 1211, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/3.11-dev/lib/python3.11/dataclasses.py", line 959, in _process_class
    cls_fields.append(_get_field(cls, name, type, kw_only))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/3.11-dev/lib/python3.11/dataclasses.py", line 816, in _get_field
    raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory

This is due to a recent change in dataclasses in Python 3.11 that disallows mutable to be set as a default value. This is done via check for hashability.

See python/cpython#88840. Note that I am trying to add support for Python 3.11 to hydra. This seems like a major blocker.

For reproducibility, I used the following command to search for these:

rg ".*: .* = [A-Z].*\(.*\)"

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

If all tests pass, it should be good.

Related Issues and PRs

[Jasha10]

Hi @skshetry, thanks for working on Python3.11 support!

Using default_factory everywhere sounds like a good option.

The only other option I can think of is to use the decorator @dataclass(frozen=True) instead of @dataclass, which would make the dataclasses immutable. I think that option is sub-optimal because OmegaConf has special support for frozen dataclasses, using them to create read-only DictConfig instances.

We'll eventually need to make the same change for the OmegaConf codebase as for Hydra.

[Jasha10]

It might be a good idea to create some regression tests like the following (to make sure that OmegaConf and Hydra can still handle old dataclass pattern when python < 3.11):

if python_version < (3, 11):
    @dataclass
    class X:
        y: Y = Y()

[skshetry]

It might be a good idea to create some regression tests like the following (to make sure that OmegaConf and Hydra can still handle old dataclass pattern when python < 3.11):

if python_version < (3, 11):
    @dataclass
    class X:
        y: Y = Y()

I am hesitant to test that, as it is dataclasses behaviour that we’ll be testing. Instantiating via default_factory or default assigned value is its responsibility. And remember, this pattern is not going to change for <3.11.

[Jasha10]

When turning a dataclass into a DictConfig, the omegaconf._utils.get_dataclass_data method inspects the fields of the dataclass and takes a different code path depending on whether a field has a default value or a default_factory.

if python_version < (3, 11):
    @dataclass
    class X:
        y: Y = Y()
    
    # Will OmegaConf choke on dataclasses defined using
    # the old pattern with default values? Currently this works:
    cfg = OmegaConf.create(X)
    # We should test it to make sure it keeps working (for python 3.6-3.10).

I'm worried that future refactoring of omegaconf._utils.get_dataclass_data could make the above code fail.

[skshetry]

I was not sure where to add this test. Let me know where it fits.

[Jasha10]

Thanks @skshetry. I'm working on a python3.11 update for omegaconf. I'm planning to revisit this PR after the OmegaConf support is finished.

Edit: here's a link to the OmegaConf PR: omry/omegaconf#1032

[skshetry]

ping @Jasha10

[Jasha10]

Thanks @skshetry

[Jasha10]

FYI I've removed test_dataclass_with_assigned_mutable_default as it is redundant with the tests in omry/omegaconf#1032.