Can't import gensim library - Python 3.8.5 + numpy 1.20.2

[norbertgieruc]

Hi,

I was trying to run script that I created some time ago on Python 3.6.5 and it seems to don't work anymore. I can't import the library. Upgrading numpy didn't help. Can I ask for a solution to this problem?

> from gensim import corpora
> Traceback (most recent call last):
> 
>   File "<ipython-input-5-0b009fd6379b>", line 1, in <module>
>     from gensim import corpora
> 
>   File "C:\Users\user\Anaconda3\lib\site-packages\gensim\__init__.py", line 11, in <module>
>     from gensim import parsing, corpora, matutils, interfaces, models, similarities, utils  # noqa:F401
> 
>   File "C:\Users\user\Anaconda3\lib\site-packages\gensim\corpora\__init__.py", line 6, in <module>
>     from .indexedcorpus import IndexedCorpus  # noqa:F401 must appear before the other classes
> 
>   File "C:\Users\user\Anaconda3\lib\site-packages\gensim\corpora\indexedcorpus.py", line 14, in <module>
>     from gensim import interfaces, utils
> 
>   File "C:\Users\user\Anaconda3\lib\site-packages\gensim\interfaces.py", line 19, in <module>
>     from gensim import utils, matutils
> 
>   File "C:\Users\user\Anaconda3\lib\site-packages\gensim\matutils.py", line 1024, in <module>
>     from gensim._matutils import logsumexp, mean_absolute_difference, dirichlet_expectation
> 
>   File "gensim\_matutils.pyx", line 1, in init gensim._matutils
> 
> ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
> 

Python 3.8.5
Numpy 1.20.2

Best regards,
Norbert

[piskvorky]

Thanks for reporting. That's weird – my understanding was that upgrading numpy should help.

Related to #3095 and numpy/numpy#16938.

@mpenkov could you have a look please? This Numpy's C API incompatibility fuckup seems critical, will hit a large percentage of Gensim users.

[mpenkov]

@norbertgieruc What version of numpy did you upgrade to?

Also, what O/S?

[norbertgieruc]

Current:
Windows 10
Numpy 1.20.2

If you are asking what Numpy version were working previously I don't remember. I created that code like 1 year ago and just wanted to rerun it now and failed.

[mpenkov]

@piskvorky Something really strange is happening with numpy.

In this ticket, the user is still unable to import gensim even after upgrading numpy to 1.20.2, when using Python 3.8.5 on Windows. For that configuration, Appveyor built the wheels using numpy 1.20.1 (see https://ci.appveyor.com/project/piskvorky/gensim-wheels-2x1bk/build/job/7bhvjbjhcr9mivx7#L1372). If numpy version 1.20.2 is incompatible with wheels built against 1.20.1, then we're in trouble.

[piskvorky]

Indeed. I'll open a ticket at numpy, but can you clarify one thing for me please? How come the wheels used numpy 1.20.1, when you switched to using oldest-supported-numpy? I don't understand how that can be.

Because if we're building against the latest version, that's against the official numpy recommendation, so I guess problems are to be expected.

BTW, Numpy have a nice signpost page for new issues: https://github.com/numpy/numpy/issues/new/choose
Let's see how they did it and do it for Gensim too :) Many don't read / respect our current issue template.

[mpenkov]

How come the wheels used numpy 1.20.1, when you switched to using oldest-supported-numpy? I don't understand how that can be.

It could be a bug in oldest-supported-numpy. Its choice of versions for Windows builds is quite strange:

• Py3.6: numpy 1.19.5
• Py3.7: numpy 1.20.1
• Py3.8: numpy 1.20.1

[mpenkov]

No, wait, the problem is that Windows builds happen using Appveyor, not Travis CI, and that uses a different build mechanism. This mechanism isn't picking up oldest-supported-numpy. I'll build new wheels and do a bugfix release. That should fix the wheels for Windows users (but, as I mentioned above, I'm not sure that will fix the ImportError that OP is having).

Have non-Windows users reported similar problems?

[piskvorky]

Oh crap, I just submitted the Numpy ticket :)

[mpenkov]

I think the Numpy ticket is still valid. We built the wheel using 1.20.1, and the user is unable to use it with numpy 1.20.2 installed. Is that expected? We built with an older version than what they have, so things should work, right?