'nccopy' much slower in 4.7.4 vs. 4.6.1

[strandwg]

I'm testing 'nccopy' to compress a ~ 3 GB netCDF-3 file.

On a local server with nccopy from netCDF library version 4.6.1, "time" reports about a minute to compress the file:

prompt> time nccopy -d1 -s 3test.nc ncc_out.nc
55.618u 4.231s 1:00.89 98.2% 0+0k 6115200+2831800io 30pf+0w

On the NCAR CISL DAV cluster, nccopy is from netCDF library version 4.7.4. The same command

prompt> time nccopy -d1 -s 3test.nc ncc_out.nc

Has now been running for more than 30 minutes with the expected output file 15% done.

I've seen reports of this behavior here and the solution seemed to be getting rid of any UNLIMITED dimensions. That doesn't seem like a good solution to me.

Any guidance/suggestions most welcome! Thanks!

[edwardhartnett]

Can you try it without the compression?

[DennisHeimbigner]

Also, can you post the output of

ncdump -h 3test.nc

So we can see what the metadata looks like?

[strandwg]

Can you try it without the compression?

time nccopy 3test.nc xtest.nc
2.378u 1.490s 0:04.10 94.1% 0+0k 0+0io 0pf+0w

The compression is the culprit on the timing in this instance.

[strandwg]

Also, can you post the output of

ncdump -h 3test.nc

So we can see what the metadata looks like?

See attached .txt file.

header_3test.txt

[edwardhartnett]

What version of HDF5 was each of these netCDF-4 builds built with?

[strandwg]

The user that reported this to me changed the version of netCDF she's using from 4.7.4 to 4.7.3 and the compression took under a minute.

I can verify that:

prompt> which nccopy
/glade/u/apps/dav/opt/netcdf/4.7.3/intel/19.0.5/bin/nccopy

time nccopy -d1 -s 3test.nc ncc_out_4.7.3.nc
47.047u 3.483s 0:52.32 96.5% 0+0k 0+0io 0pf+0w

[strandwg]

What version of HDF5 was each of these netCDF-4 builds built with?

How do I find that out, Ed?

PS - Hi!

[DennisHeimbigner]

My error I need the output of this:

ncdump -sh 3test.nc

since I need to see the compression parameters

[strandwg]

My error I need the output of this:

ncdump -sh 3test.nc

since I need to see the compression parameters

The file is also available via ftp://ftp.cgd.ucar.edu/pub/strandwg/3test.nc

Attached from 'ncdump -sh 3test.nc'

header_3test_sh.txt

[edwardhartnett]

Howdy @strandwg ! Look in the directory where netcdf-c is installed for a file called libnetcdf.settings.

Alternatively look in the config.log if you still have it.

[strandwg]

Howdy @strandwg ! Look in the directory where netcdf-c is installed for a file called libnetcdf.settings.

Alternatively look in the config.log if you still have it.

NetCDF C Configuration Summary

==============================

General

---

NetCDF Version: 4.7.4
Dispatch Version: 2
Configured On: Sun Nov 22 11:47:05 MST 2020
Host System: x86_64-pc-linux-gnu
Build Directory: /glade/scratch/vanderwb/hpci-stack/201122-1039/267091/netcdf-4.7.4
Install Prefix: /glade/u/apps/dav/opt/netcdf/4.7.4/intel/19.0.5

Compiling Options

---

C Compiler: /glade/u/apps/dav/opt/ncarcompilers/0.5.0/intel/19.0.5/icc
CFLAGS: -fPIC
CPPFLAGS: -fPIC -I/glade/u/apps/dav/opt/netcdf/4.7.4/intel/19.0.5//include
LDFLAGS: -L/glade/u/apps/dav/opt/netcdf/4.7.4/intel/19.0.5//lib -Wl,--disable-new-dtags
AM_CFLAGS:
AM_CPPFLAGS:
AM_LDFLAGS:
Shared Library: yes
Static Library: yes
Extra libraries: -lhdf5_hl -lhdf5 -lm -lz -lcurl

Features

---

NetCDF-2 API: yes
HDF4 Support: no
HDF5 Support: yes
NetCDF-4 API: yes
NC-4 Parallel Support: no
PnetCDF Support: no
DAP2 Support: yes
DAP4 Support: yes
Byte-Range Support: no
Diskless Support: yes
MMap Support: yes
JNA Support: no
CDF5 Support: yes
ERANGE Fill Support: no
Relaxed Boundary Check: yes
SZIP Support: yes
SZIP Write Support: yes
Parallel Filters: yes

[DennisHeimbigner]

It appears that the problem is a memory overflow in nccopy.
It appears to be fixed in the current master (and presumably in
the soon-to-be-released 4.8.0).
I do not have a separate fix for 4.7.4 at the moment.

[strandwg]

It appears that the problem is a memory overflow in nccopy.
It appears to be fixed in the current master (and presumably in
the soon-to-be-released 4.8.0).
I do not have a separate fix for 4.7.4 at the moment.

Thanks - we're doing OK with 4.7.3 as of now.