Save and load hashkey for explorer

[sooahleex]

Summary

• Ticket no.107264
• Save and load HashKey for dataset after explore command
• Get list of dataset in explorer
• Export HashKey annotation in datumaro format
• Usecase for explorer
  • w/w.o target for explore command
  • explore -> add -> explore
  • explore -> merge -> explore
  • Support versioning

How to test

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have added the description of my changes into CHANGELOG.​
• I have updated the documentation accordingly

License

• I submit my code changes under the same MIT License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
#
# SPDX-License-Identifier: MIT

[codecov-commenter]

Codecov Report

Patch coverage: 40.36% and project coverage change: -0.23 ⚠️

Comparison is base (38bbf0c) 78.75% compared to head (d8e65d4) 78.53%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #981      +/-   ##
===========================================
- Coverage    78.75%   78.53%   -0.23%     
===========================================
  Files          233      233              
  Lines        26626    26749     +123     
  Branches      5283     5320      +37     
===========================================
+ Hits         20969    21007      +38     
- Misses        4424     4497      +73     
- Partials      1233     1245      +12     

Flag	Coverage Δ
macos-11_Python-3.8	77.53% <24.09%> (-0.25%)	⬇️
ubuntu-20.04_Python-3.8	78.51% <40.36%> (-0.22%)	⬇️
windows-2019_Python-3.8	78.41% <40.36%> (-0.23%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
datumaro/plugins/explorer.py	81.81% <0.00%> (-3.64%)	⬇️
datumaro/cli/commands/explore.py	19.48% <7.14%> (-7.80%)	⬇️
datumaro/plugins/data_formats/datumaro/exporter.py	91.21% <25.00%> (-3.95%)	⬇️
...vino_plugin/samples/clip_visual_ViT-B_32_interp.py	29.16% <25.00%> (-2.09%)	⬇️
datumaro/util/meta_file_util.py	64.63% <31.57%> (-28.70%)	⬇️
datumaro/plugins/data_formats/datumaro/base.py	89.03% <42.85%> (-5.34%)	⬇️
datumaro/components/project.py	79.16% <66.66%> (-0.03%)	⬇️
datumaro/components/exporter.py	86.89% <75.00%> (-0.30%)	⬇️
datumaro/components/explorer.py	58.40% <83.72%> (+2.72%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[vinnamkim]

Please move saving hash key functionality to components/exporter.py::Exporter, so that it can be usable for all data formats. This saving function should store the following file with the following directory structure.

dataset_directory/
     - hash_key_meta/
            - hash_keys.json # Hash key data (you can use binary file also)
            - index.bin # FAISS index (future)
     ...

[sooahleex]

I will cover this in future PR.

[vinnamkim]

Please make loading hash key functionality global (all data formats). I think that you need to implement class DatasetExtractor by extending class DatasetBase to take in path to get the hash key checkpoint:

class DatasetExtractor(DatasetBase):
    def __init__(
        self,
        path: str,
        *,
        length: Optional[int] = None,
        subsets: Optional[Sequence[str]] = None,
        media_type: Type[MediaElement] = Image,
        ctx: Optional[ImportContext] = None,
    ):
        ...
        self._load_hash_key(path)
     
     def _load_hash_key(self, path):
        ...

Subsequently, make all data format plugins to inherit DatasetExtractor.

However, I think it can make this PR too huge. Please do it as a separate PR and implement self._load_hash_key(path) function to DatasetExtractor and Datumaro format in this PR.

[sooahleex]

I will cover this in future PR, too.

[vinnamkim]

LGTM.