337 Add config tests (#400)

Fixes #337 .

### Description
This PR adds unit tests scripts, as well as test files on brats
segmentation and spleen ct segmentation bundles. Tests are for both
single GPU and multiple GPUs.
In addition, the PR also enhances the current Blossom CI pipelines. For
PRs that do not need to use GPU or multiple GPUs, the pipeline will not
wait for resources first.

### Status
**Ready**

### Please ensure all the checkboxes:
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Codeformat tests passed locally by running `./runtests.sh
--codeformat`.
- [ ] In-line docstrings updated.
- [ ] Update `version` and `changelog` in `metadata.json` if changing an
existing bundle.
- [ ] Please ensure the naming rules in config files meet our
requirements (please refer to: `CONTRIBUTING.md`).
- [ ] Ensure versions of packages such as `monai`, `pytorch` and `numpy`
are correct in `metadata.json`.
- [ ] Descriptions should be consistent with the content, such as
`eval_metrics` of the provided weights and TorchScript modules.
- [ ] Files larger than 25MB are excluded and replaced by providing
download links in `large_file.yml`.
- [ ] Avoid using path that contains personal information within config
files (such as use `/home/your_name/` for `"bundle_root"`).

---------

Signed-off-by: Yiheng Wang <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: yiheng-wang-nv

CONTRIBUTING.md

@@ -80,7 +80,7 @@ In train config file (if exists), please follow the following requirements in or
 1. If `ValidationHandler` is used, please define the key `val_interval` and use it for the argument `interval`.
 
 
-## Verifying the bundle
+## Verifying a bundle
 
 We prepared several premerge CI tests to verify your bundle.
 
@@ -89,7 +89,7 @@ We prepared several premerge CI tests to verify your bundle.
 For dependencies that support `pip install` command, please put them into `optional_packages_version` in `configs/metadata.json` ([click here for instance](https://github.com/Project-MONAI/model-zoo/blob/dev/models/brats_mri_segmentation/configs/metadata.json)), and the CI test program will extract and install all libraries directly before running tests.
 For dependencies that require multiple steps to install, please prepare a install script in `ci/install_scripts`, and put the bundle name and the script path into `install_dependency_dict` in `ci/bundle_custom_data.py` ([click here for instance](https://github.com/Project-MONAI/model-zoo/tree/dev/ci/install_scripts/)).
 
-### Necessary verifications
+### Necessary tests
 
 1. Check if necessary files are existing in your bundle.
 1. Check if keys naming are consistent with our requirements.
@@ -100,7 +100,7 @@ For dependencies that require multiple steps to install, please prepare a instal
 python -m monai.bundle verify_metadata --meta_file configs/metadata.json --filepath eval/schema.json
 ```
 
-### Optional verifications
+### Optional tests
 
 #### Verify data shape and data type
 Check if the input and output data shape and data type of network defined in the metadata are correct. You can also run the following command locally to verify your bundle before submitting a pull request.
@@ -156,11 +156,15 @@ python -m monai.bundle run --config_file "['configs/inference.json', 'configs/in
 
 If your bundle does not support TensorRT compilation, please mention it in `docs/README.md`.
 
+#### Customized unit tests
+It is recommended to prepare unit tests on your bundle. The main purpose is to ensure each config file is runnable.
+Please create a file called `test_<bundle_name>.py` in `ci/unit_tests/` (like this [example](./ci/unit_tests/test_spleen_ct_segmentation.py)), and you can define the testing scope within the file.
+If multi-gpu config files are also need to be tested, please create a separate file called `test_<bundle_name>_dist.py` in the same directory (like this [example](./ci/unit_tests/test_spleen_ct_segmentation_dist.py)).
 
-## Checking the coding style
+### Code format tests
 
-After verifying your bundle, if there are any `.py` files, coding style is checked and enforced by flake8, black, isort, pytype and mypy.
-Before submitting a pull request, we recommend that all checks should pass, by running the following command locally:
+If there are any `.py` files in your bundle, coding style is checked and enforced by `flake8`, `black`, `isort` and `pytype`.
+Before submitting a pull request, we recommend that all checks should pass by running the following command locally:
 
 ```bash
 # optionally update the dependencies and dev tools
@@ -191,6 +195,7 @@ Ideally, the new branch should be based on the latest `dev` branch.
 
 [monai model zoo issue list]: https://github.com/Project-MONAI/model-zoo/issues
 
+
 ## Validate and release
 
 As for a pull request, a CI program will try to download all large files if mentioned and do several validations. If the pull request is approved and merged, the full bundle (with all large files if exists) will be archived and send to [Releases](https://github.com/Project-MONAI/model-zoo/releases).

ci/get_bundle_requirements.py

@@ -18,6 +18,10 @@
 
 
 def get_requirements(bundle, models_path):
+    """
+    This function is used to produce a requirements txt file, and print a string
+    which shows the filename. The printed string can be used in shell scripts.
+    """
     bundle_path = os.path.join(models_path, bundle)
     meta_file_path = os.path.join(bundle_path, "configs/metadata.json")
     if os.path.exists(meta_file_path):

ci/get_changed_bundle.py

@@ -15,7 +15,11 @@
 from utils import get_changed_bundle_list
 
 
-def main(changed_dirs):
+def get_changed_bundle(changed_dirs):
+    """
+    This function is used to get all changed bundles, a string which
+    contains all bundle names will be printed, and can be used in shell scripts.
+    """
     bundle_names = ""
     root_path = "models"
     bundle_list = get_changed_bundle_list(changed_dirs, root_path=root_path)
@@ -30,4 +34,4 @@ def main(changed_dirs):
     parser.add_argument("-f", "--f", type=str, help="changed files.")
     args = parser.parse_args()
     changed_dirs = args.f.splitlines()
-    main(changed_dirs)
+    get_changed_bundle(changed_dirs)

ci/get_required_resources.py

@@ -0,0 +1,60 @@
+# Copyright (c) MONAI Consortium
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import argparse
+import os
+
+
+def get_changed_bundle_list(changed_dirs, root_path: str = "models"):
+    """
+    This function is used to return all bundle names that have changed files.
+    If a bundle is totally removed, it will be ignored (since it not exists).
+    A similar function is implemented in "utils.py", but that file requires monai.
+    In order to minimize the required libraries when running this file, this function
+    is defined.
+
+    """
+    bundles = [f.name for f in os.scandir(root_path) if f.is_dir()]
+
+    changed_bundle_list = []
+    for sub_dir in changed_dirs:
+        for bundle in bundles:
+            bundle_path = os.path.join(root_path, bundle)
+            if os.path.commonpath([bundle_path]) == os.path.commonpath([bundle_path, sub_dir]):
+                changed_bundle_list.append(bundle)
+
+    return list(set(changed_bundle_list))
+
+
+def get_required_resources(changed_dirs):
+    """
+    This function is used to determine if GPU or multi-GPU resources are needed.
+    A string which contains two flags will be printed, and can be used in shell scripts.
+    """
+    gpu_flag, mgpu_flag = False, False
+    root_path, unit_test_path = "models", "ci/unit_tests"
+    bundle_list = get_changed_bundle_list(changed_dirs, root_path=root_path)
+    if len(bundle_list) > 0:
+        gpu_flag = True
+        for bundle in bundle_list:
+            mgpu_test_file = os.path.join(unit_test_path, f"test_{bundle}_dist.py")
+            if os.path.exists(mgpu_test_file):
+                mgpu_flag = True
+    print(f"{gpu_flag} {mgpu_flag}")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="")
+    parser.add_argument("-f", "--f", type=str, help="changed files.")
+    args = parser.parse_args()
+    changed_dirs = args.f.splitlines()
+    get_required_resources(changed_dirs)

ci/get_required_resources.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+#
+# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+head_ref=$(git rev-parse HEAD)
+git fetch origin dev $head_ref
+changes=$(git diff --name-only $head_ref origin/dev -- models)
+if [ ! -z "$changes" ]
+then
+    flags=$(python $(pwd)/ci/get_required_resources.py --f "$changes")
+else
+    flags=$"False False"
+fi
+echo $flags

ci/run_premerge_gpu.sh

@@ -73,6 +73,8 @@ verify_bundle() {
                 fi
                 # verify bundle
                 pipenv run python $(pwd)/ci/verify_bundle.py --b "$bundle"
+                # do unit tests
+                pipenv run python $(pwd)/ci/unit_tests/runner.py --b "$bundle"
                 remove_pipenv
             done
         else

ci/run_premerge_multi_gpu.sh

@@ -0,0 +1,101 @@
+#!/bin/bash
+#
+# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Argument(s):
+#   BUILD_TYPE:   all/specific_test_name, tests to execute
+
+set -ex
+BUILD_TYPE=all
+
+if [[ $# -eq 1 ]]; then
+    BUILD_TYPE=$1
+
+elif [[ $# -gt 1 ]]; then
+    echo "ERROR: too many parameters are provided"
+    exit 1
+fi
+
+init_pipenv() {
+    echo "initializing pip environment: $1"
+    pipenv install update pip wheel
+    pipenv install --python=3.8 -r $1
+    export PYTHONPATH=$PWD
+}
+
+remove_pipenv() {
+    echo "removing pip environment"
+    pipenv --rm
+    rm Pipfile Pipfile.lock
+}
+
+verify_bundle() {
+    echo 'Run verify bundle...'
+    init_pipenv requirements-dev.txt
+    head_ref=$(git rev-parse HEAD)
+    git fetch origin dev $head_ref
+    # achieve all changed files in 'models'
+    changes=$(git diff --name-only $head_ref origin/dev -- models)
+    if [ ! -z "$changes" ]
+    then
+        # get all changed bundles
+        bundle_list=$(pipenv run python $(pwd)/ci/get_changed_bundle.py --f "$changes")
+        if [ ! -z "$bundle_list" ]
+        then
+            pipenv run python $(pwd)/ci/prepare_schema.py --l "$bundle_list"
+            for bundle in $bundle_list;
+            do
+                init_pipenv requirements-dev.txt
+                # get required libraries according to the bundle's metadata file
+                requirements=$(pipenv run python $(pwd)/ci/get_bundle_requirements.py --b "$bundle")
+                if [ ! -z "$requirements" ]; then
+                    echo "install required libraries for bundle: $bundle"
+                    pipenv install -r "$requirements"
+                fi
+                # get extra install script if exists
+                extra_script=$(pipenv run python $(pwd)/ci/get_bundle_requirements.py --b "$bundle" --get_script True)
+                if [ ! -z "$extra_script" ]; then
+                    echo "install extra libraries with script: $extra_script"
+                    bash $extra_script
+                fi
+                # do multi gpu based unit tests
+                pipenv run python $(pwd)/ci/unit_tests/runner.py --b "$bundle" --dist True
+                remove_pipenv
+            done
+        else
+            echo "this pull request does not change any bundles, skip verify."
+        fi
+    else
+        echo "this pull request does not change any files in 'models', skip verify."
+        remove_pipenv
+    fi
+}
+
+case $BUILD_TYPE in
+
+    all)
+        echo "Run all tests..."
+        verify_bundle
+        ;;
+
+    verify_bundle)
+        verify_bundle
+        ;;
+
+    *)
+        echo "ERROR: unknown parameter: $BUILD_TYPE"
+        ;;
+esac