RuntimeError: NetCDF: Filter error: undefined filter encountered

[sebastic]

Some of the compression tests in 1.6.0 fail when building the Debian package (with NetCDF 4.9.0):

======================================================================
ERROR: runTest (tst_compression_bzip2.CompressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netcdf4-python-1.6.0/test/tst_compression_bzip2.py", line 25, in setUp
    write_netcdf(self.filename2,complevel=4) # with compression
  File "/build/netcdf4-python-1.6.0/test/tst_compression_bzip2.py", line 14, in write_netcdf
    foo = nc.createVariable('data',\
  File "src/netCDF4/_netCDF4.pyx", line 2838, in netCDF4._netCDF4.Dataset.createVariable
  File "src/netCDF4/_netCDF4.pyx", line 4038, in netCDF4._netCDF4.Variable.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: undefined filter encountered

======================================================================
ERROR: runTest (tst_compression_blosc.CompressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netcdf4-python-1.6.0/test/tst_compression_blosc.py", line 38, in setUp
    write_netcdf(self.filename,complevel=iblosc_complevel,blosc_shuffle=iblosc_shuffle)
  File "/build/netcdf4-python-1.6.0/test/tst_compression_blosc.py", line 17, in write_netcdf
    foo_lz = nc.createVariable('data_lz',\
  File "src/netCDF4/_netCDF4.pyx", line 2838, in netCDF4._netCDF4.Dataset.createVariable
  File "src/netCDF4/_netCDF4.pyx", line 4056, in netCDF4._netCDF4.Variable.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: undefined filter encountered

======================================================================
ERROR: runTest (tst_compression_zstd.CompressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netcdf4-python-1.6.0/test/tst_compression_zstd.py", line 25, in setUp
    write_netcdf(self.filename2,complevel=4) # with compression
  File "/build/netcdf4-python-1.6.0/test/tst_compression_zstd.py", line 14, in write_netcdf
    foo = nc.createVariable('data',\
  File "src/netCDF4/_netCDF4.pyx", line 2838, in netCDF4._netCDF4.Dataset.createVariable
  File "src/netCDF4/_netCDF4.pyx", line 4026, in netCDF4._netCDF4.Variable.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: undefined filter encountered

----------------------------------------------------------------------
Ran 100 tests in 50.665s

FAILED (errors=3)

blosc, bz2, zstd, and zlib compression were enabled in netcdf (1:4.9.0-1):

# NetCDF C Configuration Summary
==============================

# General
-------
NetCDF Version:         4.9.0
Dispatch Version:       5
Install Prefix:         /usr
Plugin Install Prefix:  N.A.

# Compiling Options
-----------------
C Compiler:             /usr/bin/cc
CFLAGS:                 -g -O2 -ffile-prefix-map=/build/netcdf-l9kbMS/netcdf-4.9.0=. -fstack-protector-strong -Wformat -Werror=format-security -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 
CPPFLAGS:                
LDFLAGS:                -Wl,-z,relro -Wl,-z,now 
AM_CFLAGS:
AM_CPPFLAGS:
AM_LDFLAGS:
Shared Library:         yes
Static Library:         no
Extra libraries:        -lhdf5_hl -lhdf5 -lcrypto -lcurl -lpthread -lsz -lz -ldl -lm -lblosc -lzstd -lbz2 -lxml2
XML Parser:             libxml2

# Features
--------
NetCDF-2 API:           yes
HDF4 Support:           no
HDF5 Support:           yes
NetCDF-4 API:           yes
NC-4 Parallel Support:  no
PnetCDF Support:        no
DAP2 Support:           yes
DAP4 Support:           yes
Byte-Range Support:     no
Diskless Support:       yes
MMap Support:           yes
JNA Support:            no
CDF5 Support:           yes
ERANGE Fill Support:    yes
Relaxed Boundary Check: yes
Parallel Filters:       yes
NCZarr Support:         yes
Multi-Filter Support:   yes
Quantization:           yes
Logging:                no
SZIP Write Support:     yes
Standard Filters:       deflate szip blosc zstd bz2
ZSTD Support:           yes
Benchmarks:             no

[jswhit]

You have to set the HDF5_PLUGIN_PATH env var to point to where the plugins were installed by netcdf-c

[sebastic]

The NetCDF build doesn't specify a HDF5 plugin path, how is one supposed to find out what it is?

[jswhit]

From the 'Developer Install' section of the docs:

if the env var NETCDF_PLUGIN_DIR is set to point to the location of the netcdf-c compression plugins built by netcdf >= 4.9.0, they will be installed inside the package. In this case HDF5_PLUGIN_PATH will be set to the package installation path on import, so the extra compression algorithms available in netcdf-c >= 4.9.0 will automatically be available. Otherwise, the user will have to set HDF5_PLUGIN_PATH explicitly to have access to the extra compression plugins.

[jswhit]

When building netcdf-c (using autotools) set --with-plugin-dir=$HDF5_PLUGIN_PATH.
With cmake, if HDF5_PLUGIN_PATH is set, and -DPLUGIN_INSTALL_DIR=YES then the plugins will be installed.

[sebastic]

NETCDF_PLUGIN_DIR isn't set either. The netcdf package is just built while having the compression libraries available and hence picked up by CMake at configure time:

-- Found Math library: /usr/lib/x86_64-linux-gnu/libm.so
-- Found Szip: headers at /usr/include, libraries at /usr/lib
--      library is /usr/lib/x86_64-linux-gnu/libsz.so
-- Found Bz2: headers at /usr/include, libraries at /usr/lib
--      library is /usr/lib/x86_64-linux-gnu/libbz2.so
-- Found Blosc: headers at /usr/include, libraries at /usr/lib
--      library is /usr/lib/x86_64-linux-gnu/libblosc.so
-- Found Zstd: headers at /usr/include, libraries at /usr/lib
--      library is /usr/lib/x86_64-linux-gnu/libzstd.so
>>> Standard Filter: szip
>>> Standard Filter: blosc
>>> Standard Filter: zstd
>>> Standard Filter: bz2

It does not result in any plugins being installed.

See also the settings:

Plugin Install Prefix:  N.A.

[sebastic]

With cmake, if HDF5_PLUGIN_PATH is set, and -DPLUGIN_INSTALL_DIR=YES then the plugins will be installed.

CMake is wrong:

# If user wants, then install selected plugins (default on)
SET(PLUGIN_INSTALL_DIR "NO" CACHE STRING "Whether and where we should install plugins; defaults to yes")

Default is not YES/ON.

[sebastic]

NetCDF build fails with -DPLUGIN_INSTALL_DIR:BOOL=ON:

Defaulting to -DPLUGIN_INSTALL_DIR=NOTFOUND

[jswhit]

setting HDF5_PLUGIN_PATH to where you want the plugins installed and then using -DPLUGIN_INSTALL_DIR=YES works for me. If you then set NETCDF_PLUGIN_DIR to HDF5_PLUGIN_PATH when building netcdf4-python, the plugins will be copied inside the netcdf4-python installation. Otherwise, you have to have HDF5_PLUGIN_PATH set for the python module to find them.

[jswhit]

@sebastic please consider creating a netcdf-c issue with the issues you are finding with plugin installation. There will soon be a 4.9.1 update, and it would be nice for these issues to be fixed in that release.

[jswhit]

thanks @sebastic. If you set HDF5_PLUGIN_PATH and set -DPLUGIN_INSTALL_DIR=YES do the netcfd4-python tests pass for you?

[sebastic]

This works:

--- a/debian/rules
+++ b/debian/rules
@@ -13,6 +13,10 @@ DEB_HOST_MULTIARCH ?= $(shell dpkg-architecture -qDEB_HOST_MULTIARCH)
 HDF5_INCLUDE_PATH=/usr/include/hdf5/serial
 HDF5_LIBRARY_PATH=/usr/lib/$(DEB_HOST_MULTIARCH)/hdf5/serial
 
+# Use H5_DEFAULT_PLUGINDIR from /usr/include/hdf5/serial/H5pubconf.h
+# https://github.com/Unidata/netcdf-c/issues/2428
+export HDF5_PLUGIN_PATH=$(HDF5_LIBRARY_PATH)/plugins
+
 include /usr/share/dpkg/pkg-info.mk
 
 UPSTREAM_VERSION = $(shell echo $(DEB_VERSION_UPSTREAM) | sed -e 's/\+.*//')
@@ -30,7 +34,8 @@ override_dh_auto_configure:
                -DENABLE_TESTS:BOOL=ON \
                -DENABLE_DOXYGEN:BOOL=ON \
                -DHDF5_INCLUDE_PATH=$(HDF5_INCLUDE_PATH) \
-               -DHDF5_LIBRARY_PATH=$(HDF5_LIBRARY_PATH)
+               -DHDF5_LIBRARY_PATH=$(HDF5_LIBRARY_PATH) \
+               -DPLUGIN_INSTALL_DIR=YES
 
 override_dh_auto_test:
        ARGS="--output-on-failure" dh_auto_test --max-parallel=1 || echo "Ignoring test failures"

But is not sufficient, the hdf5 plugins are not included in any package. The filenames look questionable:

-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nch5zstd.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nch5blosc.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nch5fletcher32.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nch5shuffle.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nch5deflate.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nczhdf5filters.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nczstdfilters.so
-- Installing: /build/netcdf-4.9.0/debian/tmp/usr/lib/x86_64-linux-gnu/hdf5/serial/plugins/lib__nch5szip.so

It's not clear what their relation is to libnetcdf or the executables, despite reading https://github.com/Unidata/netcdf-c/blob/main/docs/filters.md.

Based on the filenames the hdf5 plugins cannot be included in the libnetcdf19 binary package as there is no SOVERSION, so a new binary package (e.g. libnetcdf-hdf5-plugins) will be required for these files and have that be a Recommended dependency of libnetcdf19 as there doesn't seem to be a hard dependency. Or maybe something else is more appropriate.

Not including the plugins in the Debian package makes my life easier, I only maintain the NetCDF package because its a dependency of GDAL, I don't actually use them. So I think I'm going to update the netcdf package to explicitly not build the plugins and remove the related compression library dependencies.

[sebastic]

Disabling plugins with -DENABLE_PLUGINS:BOOL=OFF only helps a little:

======================================================================
ERROR: runTest (tst_compression_bzip2.CompressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netcdf4-python-1.6.0/test/tst_compression_bzip2.py", line 25, in setUp
    write_netcdf(self.filename2,complevel=4) # with compression
  File "/build/netcdf4-python-1.6.0/test/tst_compression_bzip2.py", line 14, in write_netcdf
    foo = nc.createVariable('data',\
  File "src/netCDF4/_netCDF4.pyx", line 2838, in netCDF4._netCDF4.Dataset.createVariable
  File "src/netCDF4/_netCDF4.pyx", line 4038, in netCDF4._netCDF4.Variable.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: undefined filter encountered

======================================================================
FAIL: runTest (tst_compression_blosc.CompressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netcdf4-python-1.6.0/test/tst_compression_blosc.py", line 53, in runTest
    assert f.variables['data_lz'].filters() == dtest
AssertionError

======================================================================
FAIL: runTest (tst_compression_zstd.CompressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/netcdf4-python-1.6.0/test/tst_compression_zstd.py", line 46, in runTest
    assert f.variables['data'].filters() ==\
AssertionError

----------------------------------------------------------------------
Ran 100 tests in 52.982s

FAILED (failures=2, errors=1)

[jswhit]

You can disable the netcdf4-python plugin tests by setting the env var NO_PLUGINS.

[jswhit]

Those are the correct filenames (lib__nc*so) for the netcdf-c plugins. They are dynamically loaded at runtime. You are probably right that they should be in a separate package.