KeyError: Options in training stage for recipes

[joeoct93]

🐛 Bug

When I tried to run a recipe (SuDORMRFNET and SuDORMRFImprovedNet), I always end up running into this problem in stage 2 like the following.

Stage 2: Training
Traceback (most recent call last):
File "/mnt/c/Project/asteroid/egs/librimix/SuDORMRFNet/train.py", line 140, in
arg_dic, plain_args = parse_args_as_dict(parser, return_plain_args=True)
File "/home/cchoi1022/cd/envs/asteroid/lib/python3.10/site-packages/asteroid/utils/parser_utils.py", line 135, in parse_args_as_dict
args_dic["main_args"] = args_dic["optional arguments"]
KeyError: 'optional arguments'

To Reproduce

cd /asteroid/egs/librimix/SuDORMRFImprovedNet
. ./run.sh

or
python train.py

Expected behavior

The training commences

Environment

Package versions

Run asteroid-versions and paste the output here:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 584, in _build_master
ws.require(requires)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 901, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 792, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (PyYAML 5.3.1 (/usr/lib/python3/dist-packages), Requirement.parse('PyYAML>=5.4'), {'pytorch-lightning'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cchoi1022/.local/bin/asteroid-versions", line 33, in
sys.exit(load_entry_point('asteroid', 'console_scripts', 'asteroid-versions')())
File "/home/cchoi1022/.local/bin/asteroid-versions", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.8/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 848, in exec_module
File "", line 219, in _call_with_frames_removed
File "/mnt/c/Project/asteroid/asteroid/scripts/asteroid_versions.py", line 5, in
import pytorch_lightning as pl
File "/home/cchoi1022/.local/lib/python3.8/site-packages/pytorch_lightning/init.py", line 30, in
from pytorch_lightning.callbacks import Callback # noqa: E402
File "/home/cchoi1022/.local/lib/python3.8/site-packages/pytorch_lightning/callbacks/init.py", line 14, in
from pytorch_lightning.callbacks.base import Callback
File "/home/cchoi1022/.local/lib/python3.8/site-packages/pytorch_lightning/callbacks/base.py", line 25, in
from pytorch_lightning.utilities.types import STEP_OUTPUT
File "/home/cchoi1022/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/init.py", line 18, in
from pytorch_lightning.utilities.apply_func import move_data_to_device # noqa: F401
File "/home/cchoi1022/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 29, in
from pytorch_lightning.utilities.imports import _compare_version, _TORCHTEXT_LEGACY
File "/home/cchoi1022/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/imports.py", line 22, in
import pkg_resources
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 3254, in
def _initialize_master_working_set():
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 3237, in _call_aside
f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 3266, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 586, in _build_master
return cls._build_from_requirements(requires)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 599, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 792, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (PyYAML 5.3.1 (/usr/lib/python3/dist-packages), Requirement.parse('PyYAML>=5.4'), {'pytorch-lightning'})
(asteroid) cchoi1022@DESKTOP-Q470KUQ:/mnt/c/Project/asteroid/egs/librimix/SuDORMRFImprovedNet$ pyYAML
pyYAML: command not found
(asteroid) cchoi1022@DESKTOP-Q470KUQ:/mnt/c/Project/asteroid/egs/librimix/SuDORMRFImprovedNet$ pip install --upgrade pyyaml
Requirement already satisfied: pyyaml in /home/cchoi1022/cd/envs/asteroid/lib/python3.10/site-packages (5.4.1)
Collecting pyyaml
Using cached PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
Installing collected packages: pyyaml
Attempting uninstall: pyyaml
Found existing installation: PyYAML 5.4.1
Uninstalling PyYAML-5.4.1:
Successfully uninstalled PyYAML-5.4.1
Successfully installed pyyaml-6.0
(asteroid) cchoi1022@DESKTOP-Q470KUQ:/mnt/c/Project/asteroid/egs/librimix/SuDORMRFImprovedNet$ asteroid-versions
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 584, in _build_master
ws.require(requires)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 901, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 792, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (PyYAML 5.3.1 (/usr/lib/python3/dist-packages), Requirement.parse('PyYAML>=5.4'), {'pytorch-lightning'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cchoi1022/.local/bin/asteroid-versions", line 33, in
sys.exit(load_entry_point('asteroid', 'console_scripts', 'asteroid-versions')())
File "/home/cchoi1022/.local/bin/asteroid-versions", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.8/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 848, in exec_module
File "", line 219, in _call_with_frames_removed
File "/mnt/c/Project/asteroid/asteroid/scripts/asteroid_versions.py", line 5, in
import pytorch_lightning as pl
File "/home/cchoi1022/.local/lib/python3.8/site-packages/pytorch_lightning/init.py", line 30, in
from pytorch_lightning.callbacks import Callback # noqa: E402
File "/home/cchoi1022/.local/lib/python3.8/site-packages/pytorch_lightning/callbacks/init.py", line 14, in
from pytorch_lightning.callbacks.base import Callback
File "/home/cchoi1022/.local/lib/python3.8/site-packages/pytorch_lightning/callbacks/base.py", line 25, in
from pytorch_lightning.utilities.types import STEP_OUTPUT
File "/home/cchoi1022/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/init.py", line 18, in
from pytorch_lightning.utilities.apply_func import move_data_to_device # noqa: F401
File "/home/cchoi1022/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 29, in
from pytorch_lightning.utilities.imports import _compare_version, _TORCHTEXT_LEGACY
File "/home/cchoi1022/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/imports.py", line 22, in
import pkg_resources
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 3254, in
def _initialize_master_working_set():
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 3237, in _call_aside
f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 3266, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 586, in _build_master
return cls._build_from_requirements(requires)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 599, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 792, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (PyYAML 5.3.1 (/usr/lib/python3/dist-packages), Requirement.parse('PyYAML>=5.4'), {'pytorch-lightning'})

Alternatively, if you cannot install Asteroid or have an old version that doesn't have the asteroid-versions script,
please output the output of:

pip freeze | egrep -i 'pytorch|torch|asteroid'

Additional info

Additional info (environment, custom script, etc...)

[mpariente]

Did you try solving the issue?

[joeoct93]

I tried looking online for solutions, but could not find anything. Apparently this KeyError is telling me that "optional arguments" is not in the dictionary. But I simply installed asteroid, I don't understand why 'optional arguments' is not in the dictionary.

[miaris98]

same here any tips?

[mpariente]

Sorry, I don't have any bandwidth to debug this.
My tip : have a look at the other recipes and look at what's different in the argument parsing, you'll probably find our error

[miaris98]

i also tried SuDORMRFImprovedNet recipe and it gives me the same error any more tips?

[jbartolewska]

Key in argparse module for Python version above 3.10.10 changed from "optional arguments" into "options", hence two possible solutions:

1. change in asteroid/utils/parser_utils.py args_dic["optional arguments"] to args_dic["options"]
2. downgrade Python

[mpariente]

Thanks for the info, that was my idea !

We can also restrict argparse's version which is much less restrictive than the Python version.

[mpariente]

We can also restrict argparse's version which is much less restrictive than the Python version.

My bad, argparse is part of Python -_-

So, change to args_dic["options"] depending on python version here is probably the best. @jbartolewska would you like to do it ?