Increase perfomance of BulkAppend and BulkFlush

[ptrendx]

Description

Increase the performance of BulkAppend and BulkFlush methods used in Gluon hybridized models with static_alloc=True, static_shape=False.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• Changes are complete (i.e. I finished coding on this PR)
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• The previous way of bulking ops in Gluon was creating a new lambda during every BulkAppend function in kind of recursive scheme, which involved copying all the environment every time. This PR changes that by instead introducing a vector of lambda functions to the BulkStatus and populating that vector.
• That change improves the time to perform BulkAppend, but BulkFlush still needed to perform the copy of the entire vector of lambdas, including all of the environment. This is alleviated by, instead of passing the vector by value, a shared_ptr is passed instead, increasing the performance of BulkFlush function by ~3.5x from 70us to ~20us.

Comments

• Those changes have the biggest impact in multi-GPU or small model scenario when the speed of scheduling work is the most important (in multi-GPU all work is scheduled by a single Python thread).
• The speed of multiGPU launches should be improved further by going to multiprocess model where each process handles a single GPU.

@eric-haibin-lin

[vandanavk]

@mxnet-label-bot add [pr-awaiting-review, Performance]

[yuxihu]

LGTM. So we should be able to see bigger performance boost when training with Horovod where a process handles a single GPU?

[yuxihu]

@mxnet-label-bot update [pr-awaiting-merge, Performance]

[ptrendx]

@yuxihu It is actually the other way around - single process per GPU alleviates some of those issues, because each GPU is handled independently. This helps the most the single process-multi GPU cases (where single Python thread needs to launch everything on all GPUs) and small batch size scenarios, where you do not have much time to launch your work.

[yuxihu]

@ptrendx Thanks for the explanation. Anyway it is good improvement.

[junrushao]

Why a shared_ptr rather than unique_ptr?

[ptrendx]

@junrushao1994 2 reasons:

• std::move to lambda is not supported before C++14 without some arcane template tricks, which I did not want to put there
• when I tried it (disregarding warnings from GCC that I'm using C++14 feature), I got errors in some other places in the engine that I try to use deleted copy function of those lambdas and did not investigate further whether that's a bigger issue or just a lack of std::move in few places.

[ptrendx]

But I agree unique_ptr would be the ultimate solution there.

[junrushao]

I see. Thanks!

[apeforest]

LGTM