Skip to content

[CICD] To support temp dgx runner #4175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
wyli merged 6 commits into from
Apr 26, 2022
Merged

[CICD] To support temp dgx runner #4175

wyli merged 6 commits into from
Apr 26, 2022

Conversation

@pxLi
Copy link
Contributor

@pxLi pxLi commented Apr 26, 2022 •
edited
Loading

Signed-off-by: Peixin Li [email protected]

Description

Old self-hosted runners have been removed by host owner.
Try adding a dgx runner (T4 DGX instead of V100) to support multiple-gpus test cases.

Some cases would fail mismatch result w/ the new setup, e.g. https://github.com/Project-MONAI/MONAI/runs/6169176742?check_suite_focus=true
I have no idea about test cases, please help investigate and create follow-up issues if needed, thanks!

Status

Ready

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@pxLi pxLi marked this pull request as draft April 26, 2022 01:22
@pxLi pxLi added the CI/CD label Apr 26, 2022
@pxLi pxLi changed the title [DO NOT REVIEW] Dummy test: new temp dgx runner [CICD] To support temp dgx runner Apr 26, 2022
@pxLi
Copy link
Contributor Author

pxLi commented Apr 26, 2022

/build

@pxLi pxLi marked this pull request as ready for review April 26, 2022 04:20
@pxLi pxLi requested review from Nic-Ma and wyli April 26, 2022 04:20
@pxLi pxLi force-pushed the test-new-runner-only branch from bfe08fe to eaf993f Compare April 26, 2022 04:26
@pxLi
Copy link
Contributor Author

pxLi commented Apr 26, 2022

/build

@wyli
atol 1e-5
9ad18a5 
Signed-off-by: Wenqi Li <[email protected]>
@wyli
Copy link
Contributor

wyli commented Apr 26, 2022

/build

@wyli wyli enabled auto-merge (squash) April 26, 2022 08:42
@wyli
Copy link
Contributor

wyli commented Apr 26, 2022

/integration-test

@wyli
temp tests
071abcf 
Signed-off-by: Wenqi Li <[email protected]>
@wyli
Copy link
Contributor

wyli commented Apr 26, 2022 •
edited
Loading

/integration-test
the integration tests are getting EERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). https://github.com/Project-MONAI/MONAI/runs/6172474180?check_suite_focus=true I'm tuning this arg.

@pxLi
Copy link
Contributor Author

pxLi commented Apr 26, 2022 •
edited
Loading

/integration-test the integration tests are getting EERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). https://github.com/Project-MONAI/MONAI/runs/6172474180?check_suite_focus=true I'm tuning this arg.

The error seems to asking more shm space, I think we may need to adjust it to some reasonable value (4g or even 8g) for integrations tests. The docker daemon only provide 64MB as default, that's why I put up this PR to increase that to 2Gi.

@pxLi
Copy link
Contributor Author

pxLi commented Apr 26, 2022

Let's see if integration could pass w/ default 64m shm-size. We could continually adjust the value here

@wyli
atol 1e-5
5852cc5 
Signed-off-by: Wenqi Li <[email protected]>
@pxLi
Copy link
Contributor Author

pxLi commented Apr 26, 2022

/integration-test

@pxLi
Copy link
Contributor Author

pxLi commented Apr 26, 2022

/build

@wyli
Copy link
Contributor

wyli commented Apr 26, 2022

/build

@wyli wyli merged commit 10cbeff into Project-MONAI:dev Apr 26, 2022
wyli added a commit that referenced this pull request Apr 26, 2022
@wyli @yiheng-wang-nv
@pre-commit-ci @Nic-Ma @dongyang0122 @bhashemian @pxLi @ericspod
sync metatensor branch with dev (#4184)
724f73f 
* 4095 Add bundle download (#4114)

* draft download

Signed-off-by: Yiheng Wang <[email protected]>

* update bundle download

Signed-off-by: Yiheng Wang <[email protected]>

* add url and load

Signed-off-by: Yiheng Wang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rename args and remove a few places

Signed-off-by: Yiheng Wang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix flake8 issue

Signed-off-by: Yiheng Wang <[email protected]>

* enhance with reviews

Signed-off-by: Yiheng Wang <[email protected]>

* add instantiate for load

Signed-off-by: Yiheng Wang <[email protected]>

* fix black error

Signed-off-by: Yiheng Wang <[email protected]>

* add unittest

Signed-off-by: Yiheng Wang <[email protected]>

* add load to docs

Signed-off-by: Yiheng Wang <[email protected]>

* add skip

Signed-off-by: Yiheng Wang <[email protected]>

* add schemaerror

Signed-off-by: Yiheng Wang <[email protected]>

* fix partial places

Signed-off-by: Yiheng Wang <[email protected]>

* download zip bundle

Signed-off-by: Yiheng Wang <[email protected]>

* [DLMED] restore Exception for test

Signed-off-by: Nic Ma <[email protected]>

* update ts features

Signed-off-by: Yiheng Wang <[email protected]>

* add config_files test case

Signed-off-by: Yiheng Wang <[email protected]>

* enhance docstring example for args_file

Signed-off-by: Yiheng Wang <[email protected]>

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Nic Ma <[email protected]>

* Disable pylint error and fix CI tests of new tifffile (#4162)

* workaround

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix tifffile issue

Signed-off-by: Nic Ma <[email protected]>

* Fixed an error in DiNTS model implementation and enabled act and norm layer options (#4157)

* fixed a bug

Signed-off-by: dongy <[email protected]>

* autofix

Signed-off-by: dongy <[email protected]>

* update test case

Signed-off-by: dongy <[email protected]>

Co-authored-by: dongy <[email protected]>

* Split transform (#4153)

* Redesign whole slide image reading (#4107)

* Redesign BaseWSIReader,  WSIReader, CuCIMWSIReader

Signed-off-by: Behrooz <[email protected]>

* Add unittests for WSIReader

Signed-off-by: Behrooz <[email protected]>

* Add image mode for output validation

Signed-off-by: Behrooz <[email protected]>

* Update docs

Signed-off-by: Behrooz <[email protected]>

* Update references to new WSIReader

Signed-off-by: Behrooz <[email protected]>

* Remove legacy WSIReader

Signed-off-by: Behrooz <[email protected]>

* Update unittests

Signed-off-by: Behrooz <[email protected]>

* Update docs

Signed-off-by: Behrooz <[email protected]>

* sort imports

Signed-off-by: Behrooz <[email protected]>

* Clean up imports

Signed-off-by: Behrooz <[email protected]>

* Update docstrings

Signed-off-by: Behrooz <[email protected]>

* Update docs and docstrings

Signed-off-by: Behrooz <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix a typo

Signed-off-by: Behrooz <[email protected]>

* Remove redundant checking

Signed-off-by: Behrooz <[email protected]>

* Update read and other methods

Signed-off-by: Behrooz <[email protected]>

* Update wsireader to support multi image and update docstrings

Signed-off-by: Behrooz <[email protected]>

* Make workaround for CuImage objects

Signed-off-by: Behrooz <[email protected]>

* Add unittests for multi image reading

Signed-off-by: Behrooz <[email protected]>

* Update a note about cucim

Signed-off-by: Behrooz <[email protected]>

* Update type hints and docstrings

Signed-off-by: Behrooz <[email protected]>

* Implement Split transform

Signed-off-by: Behrooz <[email protected]>

* Add unittests

Signed-off-by: Behrooz <[email protected]>

* Update formatting

Signed-off-by: Behrooz <[email protected]>

* Implement SplitDict

Signed-off-by: Behrooz <[email protected]>

* Add unittests for SplitDict

Signed-off-by: Behrooz <[email protected]>

* Add docs

Signed-off-by: Behrooz <[email protected]>

* Remove images from docs

Signed-off-by: Behrooz <[email protected]>

* Address all comments

Signed-off-by: Behrooz <[email protected]>

* Add example and size check

Signed-off-by: Behrooz <[email protected]>

* Update docs

Signed-off-by: Behrooz <[email protected]>

* Revert references to new wsireader

Signed-off-by: Behrooz <[email protected]>

* Add missing comma

Signed-off-by: Behrooz <[email protected]>

* fix bundle download test issue (#4169)

Signed-off-by: Yiheng Wang <[email protected]>

* 4094 Enhance `ckpt_export` to save config files (#4159)

* [DLMED] enhance checkpoint export

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] update according to comments

Signed-off-by: Nic Ma <[email protected]>

* Move RGB/RGBA checks to base class (#4171)

Signed-off-by: Behrooz <[email protected]>

Co-authored-by: Nic Ma <[email protected]>

* [CICD] To support temp dgx runner (#4175)

* Support new temp dgx runner

Signed-off-by: Peixin Li <[email protected]>

* atol 1e-5

Signed-off-by: Wenqi Li <[email protected]>

Co-authored-by: Wenqi Li <[email protected]>

* Test fix for AMP kwargs (#4178)

Signed-off-by: Eric Kerfoot <[email protected]>

Co-authored-by: Yiheng Wang <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Nic Ma <[email protected]>
Co-authored-by: dongyang0122 <[email protected]>
Co-authored-by: dongy <[email protected]>
Co-authored-by: Behrooz <[email protected]>
Co-authored-by: Peixin <[email protected]>
Co-authored-by: Eric Kerfoot <[email protected]>
Can-Zhao pushed a commit to Can-Zhao/MONAI that referenced this pull request May 10, 2022
@pxLi @wyli @Can-Zhao
[CICD] To support temp dgx runner (Project-MONAI#4175)
868da0f 
* Support new temp dgx runner

Signed-off-by: Peixin Li <[email protected]>

* atol 1e-5

Signed-off-by: Wenqi Li <[email protected]>

Co-authored-by: Wenqi Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wyli wyli wyli approved these changes

@Nic-Ma Nic-Ma Awaiting requested review from Nic-Ma Nic-Ma is a code owner

Assignees

No one assigned

Labels

CI/CD

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pxLi @wyli