HDF5 stride has different semantics than netcdf stride semantics wrt unlimited.

[DennisHeimbigner]

In PR #1001
we converted the netcdf-4 code to use the HDF5 hyperslab
operators to speed up strided access (stride > 1). This replaces
an older element-at-a-time code implementing strided access.

It appears that the HDF5 stride semantics did not exactly match the older
stride semantics in (at least) the following case:

1. A variable has an unlimited dimension
2. The space allocated in the HDF5 dimension for this variable
   is smaller than the current (maximum) UNLIMITED value.
3. A vars (stride > 1) read is performed that accessed values past
   the allocated space for the variable, but inside the space size implied
   by unlimited.
4. For some reason, this causes the last legal value to be misread even though
   it has a defined value. Instead, the fillvalue is returned.

The attached set of files shows this. If you use h5dump on the .nc file,
it shows that the temperature variable has 50 defined values. However,
ncdump on that file shows the temperature variable has the unlimited size
of 55 values. So the netcdf program ncRepro_WeirdStrideIssue.c attempts to
read values past the 50 defined values and somehow the netcdf-c library screws up.
The other program is a pure HDF5 program that attempts to do the same thing,
but if the count is set to 19 (as in the netcdf program) the HDF5 program fails
with an index error.

The fix for this could be ugly.

files.zip

[edhartnett]

I would like to fix this.

[edhartnett]

To elaborate further, the performance speedup is important, and this is all part of libhdf5.

[DennisHeimbigner]

Ok, if you have the time. My thinking was to look at the vara code to see
what it does in a similar situation, and assuming vara works correctlry, there should be
some code fragments that handle the unlimited case that we can transfer to
the vars code.

[edhartnett]

I know what the vara code does. It finds all other datasets that share that unlimited dimension, and finds the maximum extent of the dimension. Then it fakes fill data as needed to support the unlimited dim.

I will take a look at this within the next week and hopefully have a PR with the fix.

[DennisHeimbigner]

Sounds right. Thanks.

[DennisHeimbigner]

Ellen-
There may be a temporary workaround.
In the file libhdf5/hdf5dispatch.c, at about line 58
change the two lines:
NC4_get_vars,
NC4_put_vars,
to
NCDEFAULT_get_vars,
NCDEFAULT_put_vars,

This will revert the code to the previous vars code. However, be warned that
there will be a significant performance penalty.

[edhartnett]

OK, I have managed to duplicate this error in a simple test...

[ellenjohnson]

Hi guys -- is there any update on this? We discovered this bug during an upgrade to netcdf and are blocked until this is resolved, as this bug results in loss of data in this particular workflow. Thanks!

[edhartnett]

I could not get a test that reproduced the error. Do you have such a test?

…

On Tue, Jul 9, 2019 at 10:25 AM ellenjohnson ***@***.***> wrote: Hi guys -- is there any update on this? We discovered this bug during an upgrade to netcdf and are blocked until this is resolved, as this bug results in loss of data in this particular workflow. Thanks! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1380?email_source=notifications&email_token=ABCSXXCWEHFKY7W7IT5EFZDP6S3YLA5CNFSM4HDEV2UKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZQZPRY#issuecomment-509712327>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABCSXXDNTTWHKYMPS6D5CDLP6S3YLANCNFSM4HDEV2UA> .