Skip to content

Set TabularTransform to process clean transform in parallel#1648

Merged
sooahleex merged 5 commits intoopen-edge-platform:releases/1.10.0from
sooahleex:enhance/tabular_transform
Oct 21, 2024
Merged

Set TabularTransform to process clean transform in parallel#1648
sooahleex merged 5 commits intoopen-edge-platform:releases/1.10.0from
sooahleex:enhance/tabular_transform

Conversation

@sooahleex
Copy link
Copy Markdown
Contributor

@sooahleex sooahleex commented Oct 16, 2024

Summary

  • Clean transform took long time due to tabular dataset usually have large dataset
  • To reduce the time for transform, I made TabularTransform to do this process in parallel
  • When I set batch_size as 100 and num_workers as 2, then the total process time reduces in half (about 300s -> 166s)

How to test

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have added the description of my changes into CHANGELOG.​
  • I have updated the documentation accordingly

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2024 Intel Corporation
#
# SPDX-License-Identifier: MIT

@sooahleex sooahleex added the enhancement Enhancement of existing features label Oct 16, 2024
@sooahleex sooahleex marked this pull request as ready for review October 16, 2024 06:47
@sooahleex sooahleex requested review from a team as code owners October 16, 2024 06:47
@sooahleex sooahleex requested review from wonjuleee and removed request for a team October 16, 2024 06:47
@codecov
Copy link
Copy Markdown

codecov bot commented Oct 16, 2024

Codecov Report

Attention: Patch coverage is 93.75000% with 2 lines in your changes missing coverage. Please review.

Please upload report for BASE (releases/1.10.0@9478b9a). Learn more about missing BASE report.

Files with missing lines Patch % Lines
src/datumaro/components/transformer.py 93.10% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@                Coverage Diff                 @@
##             releases/1.10.0    #1648   +/-   ##
==================================================
  Coverage                   ?   81.12%           
==================================================
  Files                      ?      283           
  Lines                      ?    32861           
  Branches                   ?     5295           
==================================================
  Hits                       ?    26659           
  Misses                     ?     4751           
  Partials                   ?     1451           
Flag Coverage Δ
ubuntu-20.04_Python-3.10 81.10% <93.75%> (?)
windows-2022_Python-3.10 81.10% <93.75%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sooahleex sooahleex changed the base branch from develop to releases/1.10.0 October 16, 2024 08:13
@sooahleex sooahleex force-pushed the enhance/tabular_transform branch 2 times, most recently from 0739b12 to 489501c Compare October 17, 2024 01:07
@sooahleex sooahleex force-pushed the enhance/tabular_transform branch from 489501c to 68b45d2 Compare October 21, 2024 07:43
@sooahleex sooahleex merged commit 3b66351 into open-edge-platform:releases/1.10.0 Oct 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Enhancement of existing features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants