2310 Add load_csv_datalist utility API

[Nic-Ma]

Fixes #2310 .

Description

This PR added the load_csv_datalist API to load extra information from CSV files.
Users can easily combine this datalist with the image and label, etc. and put in Dataset.

Status

Ready

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[Nic-Ma]

Thanks for Seyed's example code and use cases, I totally changed my previous local code and switched to pandas.
Will complete this PR as soon as possible.

Thanks.

[Nic-Ma]

/black

[Nic-Ma]

/black

[Nic-Ma]

Hi @wyli @ericspod ,

This PR is ready for review, I will also develop a tutorial to show how to use it with load_decathlon_datalist() when this PR merged.

Thanks.

[wyli]

I think the CSV reading should be implemented with the MONAI dataset API, with an option of partially loading large csv files e.g. https://discuss.pytorch.org/t/how-to-use-dataset-larger-than-memory/37785

[Nic-Ma]

Hi @wyli ,

Thanks for your suggestion.
I will investigate how to build a Dataset above this CSV utility and support partially loading.

Thanks.

[Nic-Ma]

Hi @wyli ,

I want to double confirm your suggestion: if we partially load a large CSV in dataset, do you mean to only load chunks of the CSV for a training, or still use the whole dataset for training but don't load data before training, everytime we only open the CSV file to read 1 row based on the shuffled index of dataset?

Thanks.

[Nic-Ma]

pandas.read_csv() provides 2 methods to read a CSV file:

1. default to read the whole content.
2. set chunksize and return an iterator.

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html

@ericspod @rijobro @wyli What's the typical use case for a very big CSV file during training?

Thanks in advance.

[Nic-Ma]

/black

[Nic-Ma]

I didn't figure out why the min_tests of python 3.6 failed, @ericspod @wyli @rijobro do you python experts know something?

Thanks in advance.

[ericspod]

I'd suggest the assert should be changed to have an error message that states the contents of err_mod so we can see what's going on, rather than printing the contents of err_mod every time. It looks like MONAI isn't being loaded at all, it would help to detect if this fails and try to import with import monai again and use the exception from that to see what's going on.

[Nic-Ma]

/black

[Nic-Ma]

/black

[Nic-Ma]

Hi @ericspod ,

Thanks for your suggestions, I tried to print out error message locally and solved the issue.

@wyli The GPU tests failed due to below error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/parameterized/parameterized.py", line 533, in standalone_func
    return func(*(a + p.args), **p.kwargs)
  File "/__w/MONAI/MONAI/tests/test_gmm.py", line 323, in test_cuda
    gmm = GaussianMixtureModel(features_tensor.size(1), mixture_count, class_count)
  File "/__w/MONAI/MONAI/monai/networks/layers/gmm.py", line 42, in __init__
    "gmm", {"CHANNEL_COUNT": channel_count, "MIXTURE_COUNT": mixture_count, "MIXTURE_SIZE": mixture_size}
  File "/__w/MONAI/MONAI/monai/_extensions/loader.py", line 91, in load_module
    verbose=verbose_build,
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/__w/MONAI/MONAI/monai/_extensions/loader.py", line 37, in timeout
    raise TimeoutError(message)
TimeoutError: Build appears to be blocked. Is there a stopped process building the same extension?

Should I wait a while and try again?

Thanks.

[Nic-Ma]

/black

[wyli]

Hi @ericspod ,

Thanks for your suggestions, I tried to print out error message locally and solved the issue.

@wyli The GPU tests failed due to below error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/parameterized/parameterized.py", line 533, in standalone_func
    return func(*(a + p.args), **p.kwargs)
  File "/__w/MONAI/MONAI/tests/test_gmm.py", line 323, in test_cuda
    gmm = GaussianMixtureModel(features_tensor.size(1), mixture_count, class_count)
  File "/__w/MONAI/MONAI/monai/networks/layers/gmm.py", line 42, in __init__
    "gmm", {"CHANNEL_COUNT": channel_count, "MIXTURE_COUNT": mixture_count, "MIXTURE_SIZE": mixture_size}
  File "/__w/MONAI/MONAI/monai/_extensions/loader.py", line 91, in load_module
    verbose=verbose_build,
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/__w/MONAI/MONAI/monai/_extensions/loader.py", line 37, in timeout
    raise TimeoutError(message)
TimeoutError: Build appears to be blocked. Is there a stopped process building the same extension?

Should I wait a while and try again?

Thanks.

looks like multiple ninja builds sharing the same build cache is still creating some issues. any idea @charliebudd ?

[wyli]

thanks, please add some basic support for missing values, we can have another iteration to update the modules

[Nic-Ma]

/black