Multiple test failures on Big Endian/s390x systems

[WardF]

Description

There are currently a number of known test failures observed when running make check or make test on Big-Endian/s390x hardware. Some of these failures have been captured in other Github Issues, but I'm opening this issue to consolidate and track progress; the priority of this issue needs to be elevated.

• ncdump fails on s390x #2670
• nczarr_test_run_ut_mapapi fails on s390x (big-endian) #1987
• netcdf-c-4.7.4 testsuite fails on s390x (big-endian) #1896
• tst_filter fails on s390x #1338

Failures

The following tests FAILED:

	 77 - nc_test4_tst_filter (Failed)
	191 - dap4_test_test_raw (Failed)
	193 - dap4_test_test_data (Failed)
	194 - dap4_test_test_remote (Failed)
	195 - dap4_test_test_hyrax (Failed)
	213 - nczarr_test_run_ut_mapapi (Failed)
	223 - nczarr_test_run_jsonconvention (Failed)
	230 - nczarr_test_run_filter (Failed)

Errors while running CTest

Testng/Debugging via Docker s390x emulation

The following instructions will be updated as I revise the docker image.

Currently, testing can be tricky as we don't have immediate access to the appropriate hardware. I'm working on this with Unidata IT staff. In the meantime, docker to the rescue!

Assuming you have Docker Desktop (download Here) installed (not just Docker Engine, you need Docker Desktop for emulation support), you can run the following command on the command line to get a working environment. This assumes you are starting from the top-level netcdf-c/ directory.

$ docker run -it -v $(pwd):/netcdf-c \
		--platform linux/s390x \
		docker.unidata.ucar.edu/nctests:serial \
		bash

This will give you a command-line in an emulated s390x Ubuntu environment. The dependencies (libhdf5) exist in the /environments directory; for now, you'll need to set CPPFLAGS, LDFLAGS, etc, or use CMAKE_PREFIX_PATH for cmake. I'll be making this easier to access since we'll be using this image for debugging as well as automated testing.

Because the -v $(pwd):/netcdf-c argument maps the host netcdf-c/ directory, you can work on the host machine using your editor of choice, and test/debug in the docker image.

[dschwoerer]

podman run --arch=s390x also works. You need qemu-user-static-s390x.

Unfortunately, I do not have the time to look into this. Did you manage to look into this in the mean time?
If you are interested to get access to real hardware, I did get access in the past and might be able to get you in contact.

[neuschaefer]

I've spent some time looking into nc_test4/tst_h5_endians.c today. On s390x (big endian machine), the little-endian read/write tests fail.

        printf("\tLittle-Endian Int...\t");
        if ((retval = nc_put_var(ncid,le_int_varid,idata_in)))
            return retval;
        if ((retval = nc_get_var(ncid,le_int_varid,idata_le_out)))
            return retval;

After this sequence of operations, idata_le_out contains byteswapped data (0x01000000 instead of 1, etc.). This means that nc_put_var and nc_get_var are inconsistent in whether they apply endian conversion. I can think of two possible explainations:

1. nc_put_var stores the data in the requested endianness (LE in this case), but nc_get_var assumes the data to be in host endianness and simply copies everything
2. nc_put_var stores the data in host endianness (or assumes pre-converted data), but nc_get_var assumes the stored data to be in the requested endianness and does a conversion

The documentation isn't particularly useful in resolving this mismatch. It doesn't mention endianness, and states, for both functions: "[...] the type of the data in memory must match the type of the variable - no data conversion is done."

[edhartnett]

Have you checked the file with h5dump to see what it actually contains and what the endianness is?

Is this just a problem with detecting native endiness?

When you do the nc_put_var(), the data will be written to disk, it is not buffered. So when you call nc_get_var() it does not read a buffer, it reads what was written to disk.

[dschwoerer]

It seems docker.unidata.ucar.edu is gone:

$ podman run --rm -it --arch=s390x docker.unidata.ucar.edu/nctests:serial bash
 Trying to pull docker.unidata.ucar.edu/nctests:serial...

(No further output)

Is there a new url, or is it gone? Are the containers still around some how?

[akrherz]

Is there a new url, or is it gone? Are the containers still around some how?

https://hub.docker.com/u/unidata maybe?

[WardF]

We have moved to unidata/nctests, due to licensing issues for the docker storage we were using.

[edhartnett]

I believe many of these will be fixed by #3285