Skip to content

Define pmapreduce as a parallel to mapreduce, and pmapbatch as a parallel to pmap#9

Merged
jishnub merged 4 commits intomasterfrom
reduce_operators
Apr 4, 2021
Merged

Define pmapreduce as a parallel to mapreduce, and pmapbatch as a parallel to pmap#9
jishnub merged 4 commits intomasterfrom
reduce_operators

Conversation

@jishnub
Copy link
Copy Markdown
Owner

@jishnub jishnub commented Apr 2, 2021

This PR makes several breaking changes, but the result is perhaps more consistent now with Base and Distributed.

Now pmapreduce parallels mapreduce, so moving from a serial implementation to a parallel one can be achieved simply through a change in the name of the function.

julia> nworkers()
2

julia> @time mapreduce(x -> (sleep(0.5); x^2), +, 1:10)
  5.084111 seconds (55.17 k allocations: 3.309 MiB, 1.46% compilation time)
385

julia> @time pmapreduce(x -> (sleep(0.5); x^2), +, 1:10)
  2.882450 seconds (1.49 k allocations: 76.635 KiB)
385

The function pmapbatch now parallels pmap, except the values are necessarily passed in contiguous batches to the mapped function.

julia> pmap(identity, 1:5)
5-element Vector{Int64}:
 1
 2
 3
 4
 5

julia> pmapbatch(identity, 1:5)
5-element Vector{Int64}:
 1
 2
 3
 4
 5

Also the commutative functions are removed. An operator may be declared to be commutative through the Commutative tag.

The reduction functions are cleaned up. Several in-place reduction operators are introduced for convenience.

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 2, 2021

Codecov Report

Merging #9 (8ab527e) into master (c0911f8) will increase coverage by 0.16%.
The diff coverage is 99.81%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master       #9      +/-   ##
==========================================
+ Coverage   99.71%   99.88%   +0.16%     
==========================================
  Files           8        6       -2     
  Lines        1060      846     -214     
==========================================
- Hits         1057      845     -212     
+ Misses          3        1       -2     
Impacted Files Coverage Δ
src/ParallelUtilities.jl 100.00% <ø> (ø)
src/clusterquery.jl 100.00% <ø> (ø)
src/trees.jl 99.71% <99.54%> (-0.29%) ⬇️
src/mapreduce.jl 100.00% <100.00%> (+0.51%) ⬆️
src/productsplit.jl 100.00% <100.00%> (ø)
src/reductionfunctions.jl 100.00% <100.00%> (+3.22%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c0911f8...8ab527e. Read the comment docs.

@jishnub jishnub merged commit f7c3163 into master Apr 4, 2021
@jishnub jishnub deleted the reduce_operators branch April 4, 2021 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant