Auto train type detection (Extension for auto-config functionality)

[kprokofi]

Summary

How to test

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have added e2e tests for validation.
• I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
• I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
• I have linked related issues.

License

• I submit my code changes under the same Apache License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

[kprokofi]

In this PR introduced:

• Auto training type detection based on the given dataset to CLI. Introduced "Auto" default training type
• Added tests for corner cases (Unit + integration)

Currently to start Semi-SL training or Self-SL we should explicitly set training type argument
We can avoid it and make it optional.

otx train MobileNet-V3-large-1x --train-data-roots data/flower_photos/train --val-data-roots data/flower_photos/val --unlabeled-data-roots tests/assets/imagenet_dataset --train-type Semisupervised

We can simplify this command line to:

otx train MobileNet-V3-large-1x --train-data-roots data/flower_photos/train --val-data-roots data/flower_photos/val

1) If directory passed to "--train-data-roots" includes "unlabeled_images" folder and this task supports Semi-SL training -> start Semi-SL training
2) We also can specify --unlabeled-data-roots explicitly, but remove "--train-type" option
3) If we pass to "--train-data-roots" just folder with images or without validation subset -> we start Self-SL training

Implementation:
Modify "_get_train_type()" -> if train_type is not specified then check train-data-roots structure -> if only folder with images or no validation images then start Self-SL, if unlabeled_images are presented in args or in the directory, then Semi-SL, overwise incremental.

Finally, using auto-config functionality users can run Semi-SL and Self-SL in same way as for Supervised training:

otx train --train-data-roots data/flower_photos/train

However, to distinguish between auto_split functionality and Self-SL we should explicitly pass training_type.

The related documentation with all of use cases will be added in the next PR

[harimkang]

Hi @kprokofi ,
I think this is a great idea for a feature.
However, I have a concern about its use in API (Geti as well).
As you know, API also needs configuration.yaml and template.yaml inside our config.
But they don't use ConfigManager. For-example, Geti doesn't use a CLI, they use the API directly.
So Auto in configuration.yaml and template.yaml can be a problem.
I think a different implementation should be proposed.

1. Either recognize Auto in the API and modify it to change (Not in ConfigManager)
   or
2. Revert (configuration.yaml and template.yaml) and use Auto detect only for the CLI. (The default is Incremental, but we suggest giving CLI users an option called Auto.)

I can think of at least two ways to do this.

[eunwoosh]

Yes, I agree with Harim's comment. I think it's a nice feature, but there may be many things to consider including not only what Harim said but also other things.
For example, current PR considers it as semi-SL case to set only training data root, but I think it can be confusing to user who expects auto split.
I think it's better to merge it after having a meeting to share this feature to team. Could you hold a meeting?

[kprokofi]

Yes, I agree with Harim's comment. I think it's a nice feature, but there may be many things to consider including not only what Harim said but also other things. For example, current PR considers it as semi-SL case to set only training data root, but I think it can be confusing to user who expects auto split. I think it's better to merge it after having a meeting to share this feature to team. Could you hold a meeting?

Sure, I will think today how to avoid changing default training_type.
The problem that we have now - it is wrong logic for Self-SL. User should expect run training with Self-SL using only folder with images, but now we should create full dataset structure to run Self-SL. I think it is wrong and we need to reconsider handling this. Moreover documentation for Self-SL also wrong. It says, that it can be launched with images path only, but it is not working...
Let's hold a meeting on the next week

[kprokofi]

• Reverted train type back, make train_type = None by default
• Checked that SupCon runs via additional parameter and it is not related to algo_backend
• Remove overlap with auto-split. Unit tests fixed

We need to add support for running training with only images when training type is Self-SL. This will be done in next PRs after investigation

[harimkang]

Thanks for the great feature. I left a few comments.

[eunwoosh]

Thanks for your work! I left some comments. please take a look

[sungmanc]

Thanks for the good feature, BTW, as you know, this kind of automation could raise unexpected errors and I have a little concern about this. However, it is not crucial and just a minor concern.

[kprokofi]

I found a problem with dataset detection format when using unlabeled data inside dataset folder. Also in imagenet format this folder will be detected as additional label class, so, unfortunately we can't omit path to unlabeled images in CLI

[harimkang]

LGTM, Now we just need to fix it so that the test passes.

[kprokofi]

tests are passed

[kprokofi]

@harimkang @sungmanc could you approve one more time?