Recent 4.4.1-RC changes cause crashes reading HDF4 and HDF5 files

[czender]

Please provide as much of the following information as you can, as applicable to the issue being reported. Naturally, not all information is relevant to every issue, but the more information we have to start, the better!

Environment Information

Feel free to skip this if the issue is related to documentation, a feature request, or general discussion.

• What platform are you using? (please provide specific distribution/version in summary)
  • [ X] Linux
  • Windows
  • [ X] OSX
  • Other
  • NA
• 32 and/or 64 bit?
  • 32-bit
  • [ X] 64-bit
• What build system are you using?
  • [ X] autotools (configure)
  • cmake
• Can you provide a sample netCDF file or C code to recreate the issue?
  • Yes (please attach to this issue, thank you!)
  • [ X] No
  • Not at this time

Summary of Issue

Recent 4.4.1-RC crashes with HDF4, HDF5 files

Steps to reproduce the behavior

First let me say that things work more or less as expected/intended with NCO and RC2 on netCDF4 files. Second, I have not updated the underlying HDF4/5 libraries. That said:

NCO fails in nc_inq_varid() when reading HDF4 files. The failures seemed to be caused by changes to 4.4.1 development between 20160212 and 20160514 (i.e., things were working fine until I installed RC2).

It seems to happen on all HDF4 files, and results in

zender@frazil:~/nco/data$ ncks -m hdf.hdf
ERROR: nco_inq_varid() reports requested variable "date_int" is not defined in file
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_inq_varid()
nco_err_exit(): ERROR Error code is -49. Translation into English with nc_strerror(-49) is "NetCDF: Variable not found"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

ncdump does not seem to have any problems with HDF4 files (NCO ncks -m does lots of calls that ncdump does not do).

Also, the (lack of) an _NCProperties attribute in HDF5 files seems to cause unhappiness when opening such files. ncdump also has problems opening HDF5 files. here's what ncdump used to do (before 4.4.1-RC2) on small HDF5 test file:

zender@grele:~$ ncdump -h /home/zender/data/hdf/tdset.h5
netcdf tdset {
dimensions:
phony_dim_0 = 10 ;
phony_dim_1 = 20 ;
phony_dim_2 = 30 ;
variables:
int dset1(phony_dim_0, phony_dim_1) ;
double dset2(phony_dim_2, phony_dim_1) ;
}

Now it gives this:

zender@frazil:~/nco/data$ ncdump -h /home/zender/data/hdf/tdset.h5
netcdf tdset {
dimensions:
phony_dim_0 = 10 ;
phony_dim_1 = 20 ;
phony_dim_2 = 30 ;
variables:
NetCDF: Invalid dimension ID or name
Location: file ncdump.c; line 1697
int dset1(

ncks on the same file, with RC2, implicates _NCProperties code:

zender@frazil:~/nco/data$ ncks -m /home/zender/data/hdf/tdset.h5
HDF5-DIAG: Error detected in HDF5 (1.8.15-patch1) thread 0:
#000: ../../src/H5Adeprec.c line 243 in H5Aopen_name(): can't open attribute: '_NCProperties'
major: Attribute
minor: Can't open object
#1: ../../src/H5Aint.c line 451 in H5A_open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#2: ../../src/H5Oattribute.c line 537 in H5O_attr_open_by_name(): can't locate attribute: '_NCProperties'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.8.15-patch1) thread 0:
#000: ../../src/H5Adeprec.c line 243 in H5Aopen_name(): can't open attribute: '_NCProperties'
major: Attribute
minor: Can't open object
#1: ../../src/H5Aint.c line 451 in H5A_open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#2: ../../src/H5Oattribute.c line 537 in H5O_attr_open_by_name(): can't locate attribute: '_NCProperties'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.8.15-patch1) thread 0:
#000: ../../src/H5Adeprec.c line 243 in H5Aopen_name(): can't open attribute: '_NCProperties'
major: Attribute
minor: Can't open object
#1: ../../src/H5Aint.c line 451 in H5A_open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#2: ../../src/H5Oattribute.c line 537 in H5O_attr_open_by_name(): can't locate attribute: '_NCProperties'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.8.15-patch1) thread 0:
#000: ../../src/H5Adeprec.c line 243 in H5Aopen_name(): can't open attribute: '_NCProperties'
major: Attribute
minor: Can't open object
#1: ../../src/H5Aint.c line 451 in H5A_open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#2: ../../src/H5Oattribute.c line 537 in H5O_attr_open_by_name(): can't locate attribute: '_NCProperties'
major: Attribute
minor: Object not found
dset1: type NC_INT, 2 dimensions, 0 attributes, compressed? no, chunked? no, packed? no
dset1 size (RAM) = 10_20_sizeof(NC_INT) = 200*4 = 800 bytes
dset1 dimension 0: phony_dim_0, size = 10 (Non-coordinate dimension)
dset1 dimension 1: phony_dim_1, size = 20 (Non-coordinate dimension)

HDF5-DIAG: Error detected in HDF5 (1.8.15-patch1) thread 0:
#000: ../../src/H5Adeprec.c line 243 in H5Aopen_name(): can't open attribute: '_NCProperties'
major: Attribute
minor: Can't open object
#1: ../../src/H5Aint.c line 451 in H5A_open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#2: ../../src/H5Oattribute.c line 537 in H5O_attr_open_by_name(): can't locate attribute: '_NCProperties'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.8.15-patch1) thread 0:
#000: ../../src/H5Adeprec.c line 243 in H5Aopen_name(): can't open attribute: '_NCProperties'
major: Attribute
minor: Can't open object
#1: ../../src/H5Aint.c line 451 in H5A_open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#2: ../../src/H5Oattribute.c line 537 in H5O_attr_open_by_name(): can't locate attribute: '_NCProperties'
major: Attribute
minor: Object not found
dset2: type NC_DOUBLE, 2 dimensions, 0 attributes, compressed? no, chunked? no, packed? no
dset2 size (RAM) = 30_20_sizeof(NC_DOUBLE) = 600*8 = 4800 bytes
dset2 dimension 0: phony_dim_2, size = 30 (Non-coordinate dimension)
dset2 dimension 1: phony_dim_1, size = 20 (Non-coordinate dimension)

[WardF]

I believe this issue has been resolved, as per discussion in the other thread? Or am I confusing two separate things? May as well ask here while I go try to sort it out.

[DennisHeimbigner]

Yes, I think all of these problems have been addressed and solved.

[czender]

I think a serious bug remains with HDF4 files. The patches so far have been aimed at fixing the API (that issued is closed), the noisiness during loggin (issue is understood, but noisiness still plagues me), and another issue seen only in HDF4 files. It is the first issue described above: "NCO fails in nc_inq_varid() when reading HDF4 files. ...". Since I updated to RC2 NCO cannot read any HDF4 file. It always fails with nc_strerror(-49) = "NetCDF: Variable not found". These failures occur, I think, the first time nc_inq_varid() is called. The variable being inquired of is in the file. But libnetcdf returns an error. Best I can tell this behavior is due to libnetcdf changes between 20160212 and 20160514.

[WardF]

Thanks for weighing in Charlie; I'll try to duplicate the issue. On my end, hdf4 files are opening properly with ncdump, but I will try to duplicate with NCO+master and work my way backwards.

[WardF]

Using the hdf4 files fetched from a remote server during the netcdf tests, I see the following:

$ ./ncdump /working/MOD29.A2000055.0005.005.2006267200024.hdf 

Using ncks:

$ ./ncks /working/MOD29.A2000055.0005.005.2006267200024.hdf  
ERROR: nco_inq_varid() reports requested variable "Latitude" is not defined in file
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_inq_varid()
nco_err_exit(): ERROR Error code is -49. Translation into English with nc_strerror(-49) is "NetCDF: Variable not found"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

Latitude exists in the file, as seen using ncdump -h. ncdump -h from master as well as v4.4.0 show the same information, so I'm not sure what exactly is going on, but I'll dig into it. I'm participating in the EarthCube/ESIP workshop on software quality (small world) so it may be Friday before I can really dig into this.

[DennisHeimbigner]

If memory serves, some (all?) hdf4 files contain
an attribute that contais a lot of metadata (HDF4 EOS?).
The netcdf-c library (and hence also ncdump) makes
no attempt to parse that attribute. Is it possible that ncks
attempts to do that parsing and that the attribute if bad?

[WardF]

I'm torn between conflicting conclusions; the test results for libnetcdf have not changed, suggesting that either nothing has changed or we have a hole in our testing. But the behavior for ncks has changed depending on the version of libnetcdf, suggesting that it's something in libnetcdf. I'm in this EarthCube workshop today and tomorrow but I'll try to figure out what's going wrong, where; hopefully it's a simple issue.

[czender]

@DennisHeimbigner that is possible though it seems disconnected to explaining why nc_inq_varid() returns an error for a variable that does exist.

[czender]

same file hdf.hdf in following three examples.get it at
http://dust.ess.uci.edu/tmp/hdf.hdf
this file contains many variables, including a variable named "one":

NCO with pre-RC2 libnetcdf

zender@grele:~$ ncks -v one hdf.hdf
one: type NC_FLOAT, 0 dimensions, 1 attribute, compressed? no, chunked? no, packed? no
one size (RAM) = 1*sizeof(NC_FLOAT) = 1*4 = 4 bytes
one attribute 0: long_name, size = 3 NC_CHAR, value = one

one = 1 

NCO with RC2 libnetcdf

zender@frazil:~/nco/data$ ncks -v one hdf.hdf
ERROR: nco_inq_varid() reports requested variable "one" is not defined in file
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_inq_varid()
nco_err_exit(): ERROR Error code is -49. Translation into English with nc_strerror(-49) is "NetCDF: Variable not found"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

ncdump with RC2 libnetcdf

zender@frazil:~/nco/data$ ncdump -v one hdf.hdf
ncdump: one: No such variable

[WardF]

Just to keep my ducks in a row; @czender is it ok to integrate that file into a netcdf-side test and/or include it in the distribution (if needed), depending on the ultimate issue?

[WardF]

In any event, I can craft a test script with this file to use with git bisect to figure out where the error was introduced. I'll follow up here once I've had a chance.