Error loading German model

[tannonk]

Hi,

I'd like to use benepar to parse German, however, when trying to add the German model to spacy's nlp_pipe, I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../lib/python3.10/site-packages/spacy/language.py", line 814, in add_pipe
    pipe_component = self.create_pipe(
  File ".../lib/python3.10/site-packages/spacy/language.py", line 702, in create_pipe
    resolved = registry.resolve(cfg, validate=validate)
  File ".../lib/python3.10/site-packages/confection/__init__.py", line 756, in resolve
    resolved, _ = cls._make(
  File ".../lib/python3.10/site-packages/confection/__init__.py", line 805, in _make
    filled, _, resolved = cls._fill(
  File ".../lib/python3.10/site-packages/confection/__init__.py", line 877, in _fill
    getter_result = getter(*args, **kwargs)
  File ".../lib/python3.10/site-packages/benepar/integrations/spacy_plugin.py", line 176, in create_benepar_component
    return BeneparComponent(
  File ".../lib/python3.10/site-packages/benepar/integrations/spacy_plugin.py", line 116, in __init__
    self._parser = load_trained_model(name)
  File ".../lib/python3.10/site-packages/benepar/integrations/downloader.py", line 34, in load_trained_model
    parser = ChartParser.from_trained(model_path)
  File ".../lib/python3.10/site-packages/benepar/parse_chart.py", line 186, in from_trained
    parser.load_state_dict(state_dict)
  File ".../lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ChartParser:
        Unexpected key(s) in state_dict: "pretrained_model.embeddings.position_ids".

To reproduce:

import spacy
import benepar

nlp = spacy.load('de_core_news_md')
nlp.add_pipe("benepar", config={"model": "benepar_de2"})

Libraries:

torch                    2.0.1
torch-struct             0.5
spacy                    3.6.1
benepar                  0.2.0

If I swap out the models for their English counterparts (en_core_web_md, benepar_en3), it runs fine. Any intuitions on why the German model fails to load?

[tannonk]

Update: I found a simple work around. Adding the strict=False argument on L186 in parse_chart.py seems to suffice.

[sujoung]

I think I got the same error recently, for me it worked when I downgraded my transformaers library to transformers==4.30.2

[theDebbister]

I had the same error and downgrading the transformers library and in addition downgrading protobuf to protobuf==3.20.0 worked.

[bisserai]

Thanks this fixed the same issue in French too !

[brown-callum]

Downgrading transformers and protobuf is now causing other dependency issues that I can't resolve

benepar_de2 and benepar_fr2 are both failing to load in spacy or nltk

tannonk's solution seems to work, but is difficult in a non-persistent environment

Update: I found a simple work around. Adding the strict=False argument on L186 in parse_chart.py seems to suffice.

It would be really helpful to be able to call the strict=False argument when we add the benepar pipe. Or set it to False by default as it seems to conflict with at least 2 models.

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

...

File /databricks/python/lib/python3.11/site-packages/google/protobuf/descriptor.py:796, in EnumValueDescriptor.__new__(cls, name, index, number, type, options, serialized_options, create_key)
    793 def __new__(cls, name, index, number,
    794             type=None,  # pylint: disable=redefined-builtin
    795             options=None, serialized_options=None, create_key=None):
--> 796   _message.Message._CheckCalledFromGeneratedFile()
    797   # There is no way we can build a complete EnumValueDescriptor with the
    798   # given parameters (the name of the Enum is not known, for example).
    799   # Fortunately generated files just pass it to the EnumDescriptor()
    800   # constructor, which will ignore it, so returning None is good enough.
    801   return None