Skip to content

Download Kaggle datasets#1487

Merged
wonjuleee merged 5 commits intoopen-edge-platform:developfrom
itrushkin:kaggle_datasets_downloading
May 7, 2024
Merged

Download Kaggle datasets#1487
wonjuleee merged 5 commits intoopen-edge-platform:developfrom
itrushkin:kaggle_datasets_downloading

Conversation

@itrushkin
Copy link
Copy Markdown
Contributor

Summary

CVS-137601
Adds the downloading logic for Kaggle datasets.
258 pre-defined configurations are implemented.

Kaggle API has a request limit, so fetching the metadata for each dataset is still unresolved.

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have added the description of my changes into CHANGELOG.​
  • I have updated the documentation accordingly

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2024 Intel Corporation
#
# SPDX-License-Identifier: MIT

Signed-off-by: Ilya Trushkin <ilya.trushkin@intel.com>
@itrushkin itrushkin requested review from a team as code owners May 2, 2024 13:21
@itrushkin itrushkin requested review from jihyeonyi and removed request for a team May 2, 2024 13:21
itrushkin added 4 commits May 2, 2024 16:33
Signed-off-by: Ilya Trushkin <ilya.trushkin@intel.com>
Signed-off-by: Ilya Trushkin <ilya.trushkin@intel.com>
Signed-off-by: Ilya Trushkin <ilya.trushkin@intel.com>
Signed-off-by: Ilya Trushkin <ilya.trushkin@intel.com>
@codecov
Copy link
Copy Markdown

codecov bot commented May 2, 2024

Codecov Report

Attention: Patch coverage is 34.89362% with 153 lines in your changes are missing coverage. Please review.

Project coverage is 80.62%. Comparing base (44cc56a) to head (2ee4699).
Report is 50 commits behind head on develop.

Files Patch % Lines
src/datumaro/cli/commands/downloaders/tfds.py 29.89% 68 Missing ⚠️
src/datumaro/cli/commands/downloaders/kaggle.py 32.85% 47 Missing ⚠️
src/datumaro/plugins/data_formats/kaggle/base.py 29.03% 20 Missing and 2 partials ⚠️
src/datumaro/cli/commands/download.py 33.33% 10 Missing ⚠️
src/datumaro/plugins/data_formats/imagenet.py 33.33% 4 Missing ⚠️
src/datumaro/components/project.py 0.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1487      +/-   ##
===========================================
- Coverage    80.85%   80.62%   -0.24%     
===========================================
  Files          271      276       +5     
  Lines        30689    31320     +631     
  Branches      6197     6303     +106     
===========================================
+ Hits         24815    25251     +436     
- Misses        4489     4656     +167     
- Partials      1385     1413      +28     
Flag Coverage Δ
ubuntu-20.04_Python-3.10 80.60% <34.89%> (-0.24%) ⬇️
windows-2022_Python-3.10 80.59% <34.89%> (-0.24%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Copy Markdown
Contributor

@wonjuleee wonjuleee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but could you add document for download? This could be done in the separate PR.

@yunchu yunchu added this to the 1.7.0 milestone May 7, 2024
Copy link
Copy Markdown

@jihyeonyi jihyeonyi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but could you add test code more later?
It seems that we need to cover more lines in test code.

@wonjuleee wonjuleee merged commit 6e89b4f into open-edge-platform:develop May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants