Add test execution times and a timeout of 600s

[jklaise]

Addresses #840.

This is primarily to see what is causing mac-os CI tests to periodically run to ~6h before getting killed.

[codecov]

Codecov Report

Merging #849 (e569198) into master (1f627c8) will increase coverage by 0.05%.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #849      +/-   ##
==========================================
+ Coverage   85.22%   85.28%   +0.05%     
==========================================
  Files          74       74              
  Lines        8727     8759      +32     
==========================================
+ Hits         7438     7470      +32     
  Misses       1289     1289              

Flag	Coverage Δ
macos-latest-3.10	85.14% <ø> (-0.04%)	⬇️
ubuntu-latest-3.10	85.23% <ø> (+0.05%)	⬆️
ubuntu-latest-3.7	85.05% <ø> (+0.07%)	⬆️
ubuntu-latest-3.8	85.17% <ø> (+0.05%)	⬆️
ubuntu-latest-3.9	85.17% <ø> (+0.14%)	⬆️
windows-latest-3.9	82.91% <ø> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
alibi/prototypes/protoselect.py	93.36% <0.00%> (ø)
alibi/utils/missing_optional_dependency.py	94.44% <0.00%> (ø)
alibi/explainers/tests/test_simiarlity/conftest.py	100.00% <0.00%> (ø)
.../explainers/similarity/backends/tensorflow/base.py	100.00% <0.00%> (ø)
alibi/explainers/similarity/grad.py	95.65% <0.00%> (+0.33%)	⬆️
...ibi/explainers/similarity/backends/pytorch/base.py	95.74% <0.00%> (+1.80%)	⬆️

[jklaise]

@ascillitoe I would advocate including the --durations=50 --durations-min=1.0 flags by default for alibi and alibi-detect, whilst the pytest-timeout flag should be temporary in case we have frequently hanging tests like we've observed for mac-os on CI here. Otherwise the routine usage of pytest-timeout in a test suite is not particularly recommended, e.g. https://github.com/pytest-dev/pytest-timeout#thread (nevertheless, note that we currently do use it for notebook tests, the overhead hear would be minimal compared to the time it takes to execute the notebooks).

[ascillitoe]

@jklaise including the --durations=50 --durations-min=1.0 by default makes sense to me, as does only using pytest-timeout temporarily given the large number of tests we set-up/teardown.

Just to check, by pytest-timeout being temporary does that mean you will be commenting it out once the CI has run (before merge)?

[ascillitoe]

p.s. just noticed you didn't include pytest-timeout in https://github.com/SeldonIO/alibi-detect/pull/712/files. Maybe it makes sense to include it but have it commented out with a note saying when it should be uncommented?

[jklaise]

@ascillitoe it seems I might have to merge this to master to investigate the mac-os failing as we seem unable to replicate it on regular PRs, but generally yes, the pytest-timout shouldn't be used routinely.

I didn't include it for alibi-detect on purpose as we haven't seen any similar failings there (it's also unwieldy, as for the whole CI to pass need to add to all the tox environments too.

[ascillitoe]

Ah I didn't consider tox. Makes sense to leave out in general then!