Refactor histogram reduction using cuco::static_set::insert_and_find#16485
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
histogram reduction using cuco::static_map::insert_or_applyhistogram reduction using cuco::static_set::insert_and_find
| @@ -1,169 +0,0 @@ | |||
| /* | |||
| * Copyright (c) 2022-2024, NVIDIA CORPORATION. | |||
PointKernel
left a comment
There was a problem hiding this comment.
LGTM
Requesting @ttnghia's review as he worked on the original hash reduce-by-row implementation.
| // Hard set the tparam `has_nested_columns` = false for now as we don't yet support nested columns | ||
| auto const key_equal = row_comp.equal_to<false>(has_nulls, null_equality::EQUAL, value_comp); |
There was a problem hiding this comment.
This is error prone. Suppose that now we want to support the nested types, it will be difficult to trace back to this line of code to change it. So I would rather implement the code path ready for nested types here, like what it was before:
if (has_nested_columns) {
auto const key_equal = row_comp.equal_to<true>(has_nulls, null_equality::EQUAL, value_comp);
map.insert(pair_iter, pair_iter + input.num_rows(), key_hasher, key_equal, stream.value());
} else {
auto const key_equal = row_comp.equal_to<false>(has_nulls, null_equality::EQUAL, value_comp);
map.insert(pair_iter, pair_iter + input.num_rows(), key_hasher, key_equal, stream.value());
}
There was a problem hiding this comment.
Good point about the implicit use of non-nested key equality. Instead of adding a code path for nested types that isn't used by any existing code, I suggest leaving the code as is and adding a CUDF_EXPECTS check to ensure no nested types are involved in the calculation. This approach also avoids templating functions, which can help reduce build time.
ttnghia
left a comment
There was a problem hiding this comment.
The performance improvement is fantastic. There's just one small request.
Reverting the added functor which enabled the unused code path in light of Yunsong's comment. |
|
/merge |
Description
Refactors
histogramreduce and groupby aggregations usingcuco::static_set::insert_and_find. Speed improvement results here and here.Checklist