em_example upload times out in CI

[SvetiGrin]

Example upload times out in CI

Plugin version: pynxtools-em==0.4.2
NOMAD version: nomad-lab==1.4.1

Description

The em_example example upload (pynxtools_em.nomad.entrypoints:em_example) consistently times out during processing in our GitHub Actions CI environment.

[mkuehbach]

Thank you @SvetiGrin for reporting such issues. On which deployment did you observe this?
I just checked respective example raw data files on Zenodo, here no time out.
Then checked both the NOMAD central and the oasis-b deployment. In both the EM example runs fine.

What am I missing?

[SvetiGrin]

Thanks @mkuehbach for looking into this.
As mentioned in the initial description, this happens in our GitHub Actions CI pipeline on a ubuntu-latest runner. The test spins up the entire NOMAD stack via Docker Compose on that single runner and then triggers the example uploads via the API. All other plugin example uploads finish successfully, but em_example never completes even after 1800 seconds.

Does the em_example involve particularly large or compute-intensive files compared to other plugins?

[mkuehbach]

One seldom reason for timeouts observed has been if the example dataset, which we download from Zenodo, is not retrievable due to Zenodo server downtimes.

More likely is that this is an issue of pynxtools-em and the other plugins plus nomad core having hickups.
Given that there is not something like "the entire NOMAD stack", at the latest with nomad-distro-dev installations can always be heavily customized often the matter is config specific.

I recommend the following: We have just today and over the weekend, related to other updates on pynxtools, composed new releases for all plugins which also changed the connecting of the plugins and its entry points. Personally, I recommend to update your stack configuration to go with pynxtools>=0.13.0 and pynxtools-em>=0.4.3. Here is a link to a possibly relevant action whereby we tested if that all comes together also with the other plugins. To make your installations most up-to-date please also have a look into these
version updates:
https://github.com/FAIRmat-NFDI/pynxtools/blob/b30e14082d88789865ef93770d674dd30691c0e4/pyproject.toml

In the dependencies section you can see all the latest versions we composed together. As pynxtools>=0.13.0 is for several plugins including for pynxtools-em>=0.4.3 a breaking release that seems to me a reason strategy to try.

Please see that we are currently testing these as well so if there are issues we are happy to have them fixed for you and us, i.e., all users also soonish as we are currently working on bringing all these updates also to the NOMAD v1.4.2rc.

[SvetiGrin]

Hi @mkuehbach
After updating to pynxtools>=0.13.0 and all plugins to the versions you recommended, the plugin unit tests fail for pynxtools_mpes v0.2.7:

ERROR: file or directory not found: /tmp/.../tests/test_nomad_examples.py

The test runner is looking for tests/test_nomad_examples.py in the v0.2.7 tag of pynxtools-mpes, but that file doesn't exist there. Is pynxtools-mpes missing a release that includes this test file?

[mkuehbach]

Hi @SvetiGrin can you ping me on the NOMAD discord than we can have look into the matter together via some zoom or so, I recall we have also changed the default location of plugins if the runner reports in the logs though that pynxtools==0.13.1 is used that recall though is irrelevant, these versions are all rather fresh that why its probably for you and me more efficient to have a look into this together.