Skip to content

👌🚇 Improve result comparison threshold for spectra #937

@jsnel

Description

@jsnel

Issue identified in #936, comment reproduced below.

After the update we find that the result comparison still fails but only for a singular data point in one of the examples (log below). Digging a little deeper (looking at the plot) we find that the point of failure is at a place were the spectral value in question is very small (~-1.497), compared to the overall maximum of the spectrum (~7500). However, the spectra are compared locally (per wavelength) and the absolute tolerance for the comparison between results is scaled with the local value of the data, meaning that the tolerance for a point close to zero is much tighter than the tolerance for a point near the maximum. This is a point where the result comparison could be further refined, but for now we accept this as a PASS ✔️ and this PR may be merged.

FAILED .github/test_result_consistency.py::test_result_data_var_consistency[study_fluorescence-decay_associated_spectra]
E       AssertionError: Result data_var data mismatch: 'decay_associated_spectra' in 'dataset1.nc'.
E         With sum of absolute difference: 0.0039020775839004873 and shape: (49, 5)
E         Mean difference: 1.592684728122648e-05
E         
E       assert False
E        +  where False = <function allclose.<locals>._allclose at 0x7f40599e5790>(<xarray.DataArray 'decay_associated_spectra' (spectral: 49, component: 5)>\narray([[-145.581345,   52.590453,   21.6330...7\n    rate      (component) float64 ...\n    lifetime  (component) float64 ...\nDimensions without coordinates: component, <xarray.DataArray 'decay_associated_spectra' (spectral: 49, component: 5)>\narray([[-145.581346,   52.590453,   21.6330...7\n    rate      (component) float64 ...\n    lifetime  (component) float64 ...\nDimensions without coordinates: component, atol=array([[1.73546487e-05, 6.26927054e-06, 2.57886253e-06, 2.17881660e-06,\n        6.32769907e-06],\n       [1.50005268e-0...     5.91504505e-06],\n       [5.92400043e-05, 4.45607905e-05, 4.22466039e-06, 1.01596965e-04,\n        6.42848991e-06]]), rtol=1e-05, print_fail=20)

.github/test_result_consistency.py:221: AssertionError
----------------------------- Captured stdout call -----------------------------
allclose first 1 failures:
  (40, 1): -1.4969902223364386 -1.4969635695838823
=========================== short test summary info ============================
FAILED .github/test_result_consistency.py::test_result_data_var_consistency[study_fluorescence-decay_associated_spectra]

Originally posted by @jsnel in #935 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions