Skip to content

fix(tianmu): Avoid OOM when data too large and aggregated (#949)#1355

Merged
mergify[bot] merged 2 commits intostoneatom:stonedb-5.7-devfrom
adofsauron:stonedb-5.7-dev
Mar 10, 2023
Merged

fix(tianmu): Avoid OOM when data too large and aggregated (#949)#1355
mergify[bot] merged 2 commits intostoneatom:stonedb-5.7-devfrom
adofsauron:stonedb-5.7-dev

Conversation

@adofsauron
Copy link
Copy Markdown
Collaborator

@adofsauron adofsauron commented Mar 7, 2023

1. Use memory mapping
2. Set the maximum number of concurrent threads for aggregation

Summary about this PR

Issue Number: close #949 #1291

Tests Check List

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Changelog

  • New Feature
  • Bug Fix
  • Performance Improvement
  • Build/Testing/CI/CD
  • Documentation
  • Not for changelog (changelog entry is not required)

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features

@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Mar 7, 2023

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

@mergify mergify bot added the PR-bug bug for pull request label Mar 7, 2023
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 8, 2023

Codecov Report

Patch coverage: 69.79% and project coverage change: +0.04 🎉

Comparison is base (106431a) 42.89% compared to head (db55498) 42.93%.

❗ Current head db55498 differs from pull request most recent head c2c0676. Consider uploading reports for the commit c2c0676 to get more accurate results

Additional details and impacted files
@@                 Coverage Diff                 @@
##           stonedb-5.7-dev    #1355      +/-   ##
===================================================
+ Coverage            42.89%   42.93%   +0.04%     
===================================================
  Files                 1836     1838       +2     
  Lines               397300   397390      +90     
===================================================
+ Hits                170411   170621     +210     
+ Misses              226889   226769     -120     
Impacted Files Coverage Δ
storage/tianmu/core/aggregation_algorithm.h 60.00% <ø> (ø)
storage/tianmu/core/engine.h 30.43% <ø> (ø)
storage/tianmu/handler/ha_tianmu.cpp 54.24% <0.00%> (+4.29%) ⬆️
storage/tianmu/mm/memory_handling_policy.cpp 47.56% <0.00%> (+0.23%) ⬆️
storage/tianmu/system/configuration.cpp 100.00% <ø> (ø)
storage/tianmu/system/configuration.h 0.00% <ø> (ø)
storage/tianmu/core/aggregation_algorithm.cpp 55.95% <30.00%> (-0.03%) ⬇️
storage/tianmu/mm/huge_heap_policy.cpp 41.66% <44.44%> (+1.66%) ⬆️
storage/tianmu/core/engine.cpp 51.31% <81.81%> (+2.71%) ⬆️
storage/tianmu/mm/memory_statistics.h 86.79% <86.79%> (ø)
... and 1 more

... and 27 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@adofsauron adofsauron requested a review from wisehead March 10, 2023 08:53
    1. Use memory mapping
    2. Set the maximum number of concurrent threads for aggregation
@mergify mergify bot merged commit f5cd1f9 into stoneatom:stonedb-5.7-dev Mar 10, 2023
@Nliver Nliver added this to the StoneDB_5.7_v1.0.3 milestone Apr 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

PR-bug bug for pull request

Projects

Development

Successfully merging this pull request may close these issues.

feature: Cache disk files for intermediate results of aggregate operations to avoid OOM

4 participants