rtree version 0.9 install error: OSError: libspatialindex_c.so: cannot open shared object file

[lior-k]

Hi,
running pip install "rtree>=0.8,<1" fails with this error:

Collecting rtree<1,>=0.8
  Downloading https://files.pythonhosted.org/packages/5e/64/a01a6675c39ddfba2467cc6b432ce9af2e71a3a67e9e8ace106ccda10df8/Rtree-0.9.0.tar.gz (52kB)
    100% |████████████████████████████████| 61kB 2.7MB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-r2lpov2a/rtree/setup.py", line 3, in <module>
        import rtree
      File "/tmp/pip-build-r2lpov2a/rtree/rtree/__init__.py", line 1, in <module>
        from .index import Rtree
      File "/tmp/pip-build-r2lpov2a/rtree/rtree/index.py", line 6, in <module>
        from . import core
      File "/tmp/pip-build-r2lpov2a/rtree/rtree/core.py", line 104, in <module>
        rt = ctypes.CDLL(lib_name)
      File "/usr/local/lib/python3.6/ctypes/__init__.py", line 348, in __init__
        self._handle = _dlopen(self._name, mode)
    OSError: libspatialindex_c.so: cannot open shared object file: No such file or directory

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-r2lpov2a/rtree/

I installed libspatialindex_c.so manually by running the following, but the above error still happens even though /usr/local/lib/libspatialindex_c.so exists now :

LIBSPATIALINDEX_VERSION=spatialindex-src-1.8.5
curl -sLO "http://download.osgeo.org/libspatialindex/$LIBSPATIALINDEX_VERSION.tar.gz" && \
    tar xzf "$LIBSPATIALINDEX_VERSION.tar.gz" && \
    rm -rf "$LIBSPATIALINDEX_VERSION.tar.gz" && \
    cd "$LIBSPATIALINDEX_VERSION" && \
    ./configure && \
    make -j $( getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1 ) && \
    make install

[lior-k]

worth mentioning that pip install "rtree>=0.8,<0.9"works.
this only happens on version 0.9

[DanielFEvans]

Seeing the same issue as of this morning on our CI server.

The system (running Scientific Linux 7) has libspatialindex v1.8.5 installed:

$ yum list installed | grep "spatialindex"
spatialindex.x86_64                   1.8.5-1.el7                 @epel

However, I have no shared library named libspatialindex_c.so, but instead libspatialindex_c.so.4:

$ locate spatialindex_c
/usr/lib64/libspatialindex_c.so.4
/usr/lib64/libspatialindex_c.so.4.1.0

The changes in this diff resulted in rtree being hardcoded to import spatialindex_c.so, rather than calling ctypes.util.find_library("spatialindex_c"), which was previously returning the correct name:

>>> import ctypes.util
>>> ctypes.util.find_library("spatialindex_c")
'libspatialindex_c.so.4'

[hobu]

Sorry for the churn. I was trying to clean up a messy area of the code that seemed to be getting patched by downstream packaging groups.

#56 implies that find_library doesn't find things in non-standard locations. Is that still true?

[hobu]

@DanielFEvans can you please try current master? If it fixes your issue, I will cut a 0.9.1 release immediately.

[DanielFEvans]

Appears that there was a change in Python 3.6 to cause find_library to search LD_LIBRARY_PATH: https://hg.python.org/cpython/rev/385181e809bc

I wonder if you could make more people happy with a fallback if find_library doesn't get a result:

lib_name = find_library('spatialindex_c')
if not lib_name:
    if 'linux' in sys.platform:
        lib_name = 'libspatialindex_c.so'
    elif 'darwin' in sys.platform:
        lib_name = 'libspatialindex_c.dylib'
    else:
        lib_name = 'libspatialindex_c'

Of course, that still doesn't work if you've got a libspatialindex_c.so.4 in a non-standard location (luckily, I don't!).

[hobu]

suggestion added in b85e2c8

[DanielFEvans]

@hobu - latest master installs without error, unlike 0.9.0. Thanks!

[hobu]

0.9.1 released to fix this issue. https://pypi.org/project/Rtree/

[davidedelvento]

Mmmm, this is still failing for me. It used to work with rtree v0.8.3 but now it fails with v0.9.1 (I am doing exactly the same thing, just with a different version of rtree since I install on servers with ansible)

[hobu]

0.9.2 was just released. Is that what is failing for you? If so, I need more information. The 0.9.2 code is an adaptation of what was there in 0.8.3.

[davidedelvento]

Thanks for the quick response. Yes, it is still failing with 0.9.2

My ansible script fails as follow:

pip3 install numpy scipy folium fiona shapely cesiumpy matplotlib pandas netCDF4 rtree geopandas plotly sklearn keras tensorflow pysal

and it fails with

fatal: [IP-NUMBER-EDITED]: FAILED! => {
    "changed": false, 
    "cmd": "/usr/local/bin/pip3 install numpy scipy folium fiona shapely cesiumpy matplotlib pandas netCDF4 rtree geopandas pl
otly sklearn keras tensorflow pysal"
}

MSG:

stdout: Collecting numpy
  Using cached https://files.pythonhosted.org/packages/ab/e9/2561dbfbc05146bffa02167e09b9902e273decb2dc4cd5c43314ede20312/nump
y-1.17.4-cp35-cp35m-manylinux1_x86_64.whl
Collecting scipy
  Using cached https://files.pythonhosted.org/packages/1e/9c/4d7a573d49c449c95d33df6648a62a522ec46403f13ffe2c8cd3f094f746/scip
y-1.3.3-cp35-cp35m-manylinux1_x86_64.whl
Collecting folium
  Using cached https://files.pythonhosted.org/packages/fd/a0/ccb3094026649cda4acd55bf2c3822bb8c277eb11446d13d384e5be35257/foli
um-0.10.1-py2.py3-none-any.whl
Collecting fiona
  Using cached https://files.pythonhosted.org/packages/03/68/d6a70cd10afc31c336d69e13c371ff0088e14390b310da1668a648afb19d/Fion
a-1.8.13-cp35-cp35m-manylinux1_x86_64.whl
Collecting shapely
  Using cached https://files.pythonhosted.org/packages/a2/6c/966fa320a88fc685c956af08135855fa84a1589631256abebf73721c26ed/Shap
ely-1.6.4.post2-cp35-cp35m-manylinux1_x86_64.whl
Collecting cesiumpy
  Using cached https://files.pythonhosted.org/packages/20/4d/079f09eed2ab0123c21a4728470f3c62b6bf5f6c7bf00d879f6dc41bcd3d/cesi
umpy-0.3.3.tar.gz
Collecting matplotlib
  Using cached https://files.pythonhosted.org/packages/89/61/465fb3bfba684b0f53b5c4829c3c89e86e6fe9fdcdfda93e38f1788090f0/matp
lotlib-3.0.3-cp35-cp35m-manylinux1_x86_64.whl
Collecting pandas
  Using cached https://files.pythonhosted.org/packages/74/24/0cdbf8907e1e3bc5a8da03345c23cbed7044330bb8f73bb12e711a640a00/pand
as-0.24.2-cp35-cp35m-manylinux1_x86_64.whl
Collecting netCDF4
  Using cached https://files.pythonhosted.org/packages/4c/d3/b42ffc1a0b89df26b9e7796e5b79155e7d89e8b50565699c06ce8d5907df/netC
DF4-1.5.3-cp35-cp35m-manylinux1_x86_64.whl
Collecting rtree
  Using cached https://files.pythonhosted.org/packages/6b/11/78e7cec7e502c2db845b931c4f26863dc6bb493c022883affea161ad6363/Rtre
e-0.9.2.tar.gz

:stderr:     ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3064k240/rtree/setup.
py'"'"'; __file__='"'"'/tmp/pip-install-3064k240/rtree/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=
f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /t
mp/pip-install-3064k240/rtree/pip-egg-info
         cwd: /tmp/pip-install-3064k240/rtree/
    Complete output (13 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-3064k240/rtree/setup.py", line 3, in <module>
        import rtree
      File "/tmp/pip-install-3064k240/rtree/rtree/__init__.py", line 1, in <module>
        from .index import Rtree
      File "/tmp/pip-install-3064k240/rtree/rtree/index.py", line 6, in <module>
        from . import core
      File "/tmp/pip-install-3064k240/rtree/rtree/core.py", line 139, in <module>
        rt = ctypes.CDLL(lib_name)
      File "/usr/lib/python3.5/ctypes/__init__.py", line 347, in __init__
        self._handle = _dlopen(self._name, mode)
    OSError: libspatialindex_c.so.4: cannot open shared object file: No such file or directory
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Attempting the same thing manually yields:

$ pip3 install rtree
Collecting rtree
  Downloading https://files.pythonhosted.org/packages/6b/11/78e7cec7e502c2db845b931c4f26863dc6bb493c022883affea161ad6363/Rtree
-0.9.2.tar.gz (515kB)
     |████████████████████████████████| 522kB 3.7MB/s
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-q6lf6jup/rtree/setup.
py'"'"'; __file__='"'"'/tmp/pip-install-q6lf6jup/rtree/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=
f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /t
mp/pip-install-q6lf6jup/rtree/pip-egg-info
         cwd: /tmp/pip-install-q6lf6jup/rtree/
    Complete output (13 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-q6lf6jup/rtree/setup.py", line 3, in <module>
        import rtree
      File "/tmp/pip-install-q6lf6jup/rtree/rtree/__init__.py", line 1, in <module>
        from .index import Rtree
      File "/tmp/pip-install-q6lf6jup/rtree/rtree/index.py", line 6, in <module>
        from . import core
      File "/tmp/pip-install-q6lf6jup/rtree/rtree/core.py", line 139, in <module>
        rt = ctypes.CDLL(lib_name)
      File "/usr/lib/python3.5/ctypes/__init__.py", line 347, in __init__
        self._handle = _dlopen(self._name, mode)
    OSError: libspatialindex_c.so.4: cannot open shared object file: No such file or directory
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

but note that the lib is there:

$ ls /usr/local/lib/libspatialindex* -l
-rw-r--r-- 1 root root 1702694 Dec  9 17:45 /usr/local/lib/libspatialindex.a
-rw-r--r-- 1 root root  414536 Dec  9 17:45 /usr/local/lib/libspatialindex_c.a
-rwxr-xr-x 1 root root    1064 Dec  9 17:45 /usr/local/lib/libspatialindex_c.la
lrwxrwxrwx 1 root root      26 Dec  9 17:45 /usr/local/lib/libspatialindex_c.so -> libspatialindex_c.so.4.0.1
lrwxrwxrwx 1 root root      26 Dec  9 17:45 /usr/local/lib/libspatialindex_c.so.4 -> libspatialindex_c.so.4.0.1
-rwxr-xr-x 1 root root  218944 Dec  9 17:45 /usr/local/lib/libspatialindex_c.so.4.0.1
-rwxr-xr-x 1 root root    1016 Dec  9 17:45 /usr/local/lib/libspatialindex.la
lrwxrwxrwx 1 root root      24 Dec  9 17:45 /usr/local/lib/libspatialindex.so -> libspatialindex.so.4.0.1
lrwxrwxrwx 1 root root      24 Dec  9 17:45 /usr/local/lib/libspatialindex.so.4 -> libspatialindex.so.4.0.1
-rwxr-xr-x 1 root root  918504 Dec  9 17:45 /usr/local/lib/libspatialindex.so.4.0.1

Thanks again and let me know if you need anything more than this.

[hobu]

@davidedelvento which platform and packages do you have installed? I just added ubuntu 16.04 and 18.04 pip tests that install and run correctly.

[hobu]

See https://github.com/Toblerity/rtree/pull/131/files for the libspatialindex package I am installing on ubuntu.