[OTX] Enable multi-GPU training

[eunwoosh]

Summary

Enable multi-GPU training for classification, detection, segmentation tasks.

This PR includes

• Enable multi-GPU training
• make it available to set output_path in each task class.
• add test code for multi-GPU training

[jaegukhyun]

Generally it looks good to me. Frankly of speaking I'm not expert of multiprocessing, so let's check the test results. I left some comment, actually most of them are questions. Also I have two questions for overall PR.

1. Does using multiprocessing give more benefits than using nn.parallel.DistributedDataParallel? I have read this article
2. Don't we need stressed test for multiprocessing? Multiprocessing works always looks good if they run small amount(training schedule, # of jobs), but when the volume of works grow, they give error out of sudden.

[eunwoosh]

Generally it looks good to me. Frankly of speaking I'm not expert of multiprocessing, so let's check the test results. I left some comment, actually most of them are questions. Also I have two questions for overall PR.

1. Does using multiprocessing give more benefits than using nn.parallel.DistributedDataParallel? I have read this article
2. Don't we need stressed test for multiprocessing? Multiprocessing works always looks good if they run small amount(training schedule, # of jobs), but when the volume of works grow, they give error out of sudden.

Thanks for comment! I think I can answer your questions.

Answer1 : You're right. So I implement to use nn.parallel.DistributedDataParallel.
Answer 2 : Actually, I can't understand your exact point. So Do you mean that we need to make test for multiple multi GPU training?

[jaegukhyun]

Generally it looks good to me. Frankly of speaking I'm not expert of multiprocessing, so let's check the test results. I left some comment, actually most of them are questions. Also I have two questions for overall PR.

1. Does using multiprocessing give more benefits than using nn.parallel.DistributedDataParallel? I have read this article
2. Don't we need stressed test for multiprocessing? Multiprocessing works always looks good if they run small amount(training schedule, # of jobs), but when the volume of works grow, they give error out of sudden.

Thanks for comment! I think I can answer your questions.

Answer1 : You're right. So I implement to use nn.parallel.DistributedDataParallel. Answer 2 : Actually, I can't understand your exact point. So Do you mean that we need to make test for multiple multi GPU training?

To the answer2, I'm not sure, but I think we should check whether multi-gpu training can run multiple training jobs
For example, run below training jobs simultaneously

1. one training job for gpu1, 2
2. one training job for gpu 1,2,3
3. one training job for 2,3
4. one training job for 2,3,4
   Did you check this situation in your environment? Test code can test simple situation, but we should check complex situation in local environment

[eunwoosh]

Generally it looks good to me. Frankly of speaking I'm not expert of multiprocessing, so let's check the test results. I left some comment, actually most of them are questions. Also I have two questions for overall PR.

1. Does using multiprocessing give more benefits than using nn.parallel.DistributedDataParallel? I have read this article
2. Don't we need stressed test for multiprocessing? Multiprocessing works always looks good if they run small amount(training schedule, # of jobs), but when the volume of works grow, they give error out of sudden.

Thanks for comment! I think I can answer your questions.
Answer1 : You're right. So I implement to use nn.parallel.DistributedDataParallel. Answer 2 : Actually, I can't understand your exact point. So Do you mean that we need to make test for multiple multi GPU training?

To the answer2, I'm not sure, but I think we should check whether multi-gpu training can run multiple training jobs For example, run below training jobs simultaneously

1. one training job for gpu1, 2
2. one training job for gpu 1,2,3
3. one training job for 2,3
4. one training job for 2,3,4
   Did you check this situation in your environment? Test code can test simple situation, but we should check complex situation in local environment

I understand. I'll check the situation in the local environment, and I'll update result.

[eunwoosh]

@jaegukhyun I checked that 4 multi GPU training worked well parallelly.

[harimkang]

I left a few comments.

[sungmanc]

Please replace print sentence with logger and also left some comments

[eunwoosh]

There are two fail cases in Pre-Merge Check as bellow.

TestToolsMPAMultilabelClassification.test_otx_eval_openvino[Custom_Image_Classification_EfficientNet-V2-S]
AssertionError: trained_performance[k]=1.0, exported_performance[k]=0.9607843137254902
TestToolsMPAMultilabelClassification.test_otx_eval_openvino[Custom_Image_Classification_EfficientNet-B0]
AssertionError: trained_performance[k]=0.9803921568627451, exported_performance[k]=0.9607843137254902

These are due to gap between exported model performance and trained model performance.
I think that it's acceptable difference. So, could you approve the PR?

[JihwanEom]

Can we use also 3 GPUs or 4 GPUs?

[eunwoosh]

Can we use also 3 GPUs or 4 GPUs?

yes