You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
commitment.kv merge producing much storage.kv refferences read. Need try move to MMAP with MADV_WILLNEED
Bloomfilters are stored in RAM (instead mmap). And during merge we can store 2x of Bllomfilters. Such "short peak memory jump" impacting ChainTip
Recsplit.build() -> buildWithWorkers -> GolombRice.Append stored in RAM. Need move to Bufio or MMAP. (2g on some bloatnet file). Such "short peak memory jump" impacting ChainTip. Bloatnet got OOM because of this [performance] OOM on storage .ef indexing #20560
Recsplit.build() -> buildOffsetsEf -> NewEliasFano -> EliasFano.deriveFields storing in RAM. Need move to Bufio or MMAP. (4g on some bloatnet file). Such "short peak memory jump" impacting ChainTip. Bloatnet got OOM because of this [performance] OOM on storage .ef indexing #20560
Problems:
MADV_WILLNEEDRecsplit.build() -> buildWithWorkers -> GolombRice.Appendstored in RAM. Need move to Bufio or MMAP. (2g on some bloatnet file). Such "short peak memory jump" impacting ChainTip. Bloatnet got OOM because of this [performance] OOM on storage .ef indexing #20560Recsplit.build() -> buildOffsetsEf -> NewEliasFano -> EliasFano.deriveFieldsstoring in RAM. Need move to Bufio or MMAP. (4g on some bloatnet file). Such "short peak memory jump" impacting ChainTip. Bloatnet got OOM because of this [performance] OOM on storage .ef indexing #20560.effiles merge have several in-mem buffers/arrays. For example one of them: db/state: inv_index merge remove 1 intermediate buffer to reduce peak memory #20597FYI: recent Merge-related PR's #19430