Byte swapped floats in TMVA sofie on big endian architectures (s390x)

[ellert]

Describe the bug

During a build of root with tmva-sofie enabled, the build fails with the following error:

In file included from /builddir/build/BUILD/root-6.26.00/tmva/sofie/test/TestCustomModelsFromROOT.cxx:3:
/builddir/build/BUILD/root-6.26.00/redhat-linux-build/tmva/sofie/test/Linear_16_FromROOT.hxx:14:123: error: 'nan' was not declared in this scope
14 | float tensor_2bias[800] = {-2.14694423e+20, -2.62987679e-27, -3.31015721e+10, -3.77250566e-25, 2721.82764, -0.185639352, -nan, 4.30533447e-18, 2.46234533e+27, -4.90863758e-14, 9355069, -1.40139204e-31, -4.42654684e+15, 7.01436106e+16, 0.104793049, -3234.07764, -2.00034187e+36, -1.6853868e+38, -7.29983234e+27, 1.04351275e-35, -4.85025149e-26, -1.8755401e-25, -1.1517516e+36, -4.63935318e-31, 0.686275303, 1.24939929e-20, 1.69885186e+27, 2.25105193e+21, -364749.844, 1.5719514e-34, 8.914608, 0.017398471, -1.69850501e+13, -5.68424096e+26, 5.46905411e-28, -4550.52832, 9.75181774e-06, 58461.7383, -9.21360134e-21, 3.14942058e+27, 0.000689532433, -7.38048164e-12, 1.63026903e+36, -23409.6191, 7.75302674e+23, -4.05909378e+21, -1.36839554e-21, nan, 3.00273573e-29, 214934480, -2.14694423e+20, -2.62987679e-27, -3.31015721e+10, -3.77250566e-25, 2721.82764, -0.185639352, -nan, 4.30533447e-18, 2.46234533e+27, -4.90863758e-14, 9355069, -1.40139204e-31, -4.42654684e+15, 7.01436106e+16, 0.104793049, -3234.07764, -2.00034187e+36, .......

The floats in the declared array fluctuate violently, and there are both "nan" and "-nan" in there. This header file is generated during the build.

Compare this to the code generated on x86_64:

float tensor_16bias[800] = {-0.142527401, 0.168874308, -0.0887828916, -0.0631441548, -0.0660232753, 0.0544182472, 0.0641550943, 0.0606994219, -0.0223076269, 0.102970138, 0.0258652028, -0.0809393153, -0.0276761875, 0.0546831638, 0.128886178, -0.0795307532, -0.0189131647, -0.120750055, 0.17368494, -0.0492844023, -0.0578795224, 0.0671775565, 0.0123591702, 0.132642269, -0.0525798798, 0.0173827056, 0.0659839064, -0.0958536118, 0.0788409114, 0.0107072359, 0.0492983423, -0.02552481, ....

Here the numbers vary in a more reasonable manner, and there are no "nan".

The output on s390x looks like what you would get in case you printed out floats that have been stored in the wrong byte order in memory.

Expected behavior

The file generated on s390x should look similar to the file generated on x86_64.

To Reproduce

Steps to reproduce the behavior:

1. Build root for a big endian architecture (e.g. s390x) with -Dtmva-sofie:BOOL=ON
2. Build fails at the point where it tries to include autogenerated headers

Setup

1. ROOT version: 6.26.00
2. Operating system: Fedora rawhide for s390x
3. How you obtained ROOT: Build from source in a package build on Fedora's build server (koji).

[lmoneta]

Thank you @ellert for reporting this issue.
Is the test TestCustomModelsFromONNX having the same issue ?
You can also easily check if the tensor value stored in the file Linear_16_FromONNX.dat are the same of those stored in Linear_16_FromROOT.hxx. If in both cases there wrong values and nan, then the problem is in the ONNX parser or in Protobuf, otherwise it is only in the ROOT I/O storage.

[ellert]

Hi.

Here is a tar file of the complete tmva/sofie/test build directory, which includes the generated .dat, .root and .hxx files.
Most of them look OK, so the issue does not affect all of them.
However, all files beginning with Linear are bad, and that applies to both .dat and .hxx files. (I didn't investigate whether the .root files are also bad, but they are included in the tar file too.)

• Linear_16_FromONNX.dat
• Linear_16_FromROOT.hxx
• Linear_32_FromONNX.dat
• Linear_32_FromROOT.hxx
• Linear_64_FromONNX.dat
• Linear_64_FromROOT.hxx
• LinearWithSelu_FromONNX.dat
• LinearWithSelu_FromROOT.hxx
• LinearWithSigmoid_FromONNX.dat
• LinearWithSigmoid_FromROOT.hxx

build.tar.gz

[ellert]

Although not containing any nan's, some of the others not starting with Linear are also not OK when comparing with the same files created on x86_64. E.g.:

On x86_64:

$ cat ConvWithStridesPadding_FromONNX.dat
tensor_W 9
1 1 1 1 1 1 1 1 1

On s390x:

$ cat ConvWithStridesPadding_FromONNX.dat
tensor_W 9
4.60060299e-41 4.60060299e-41 4.60060299e-41 4.60060299e-41 4.60060299e-41 4.60060299e-41 4.60060299e-41 4.60060299e-41 4.60060299e-41

And it is really byteswapped:

#include <cstring>
#include <iostream>
#include <iomanip>

int main() {
  float f;
  int i;
  f = 1.;
  std::cout << f << std::endl;
  std::memcpy(&i, &f, 4);
  std::cout << std::hex << i << std::endl;
  i = ((i & 0xff000000) >> 24) +
    ((i & 0x00ff0000) >> 8) +
    ((i & 0x0000ff00) << 8) +
    ((i & 0x000000ff) << 24);
  std::cout << std::hex << i << std::endl;
  std::memcpy(&f, &i, 4);
  std::cout << f << std::endl;
}

Gives:

1
3f800000
803f
4.6006e-41

[guitargeek]

Thanks for providing the fix @ellert!