fix(shard-distributor): add context timeout into the shard rebalancing loop

[arzonus]

What changed?

• context.WithTimeout has been added to the rebalancing loop. Default timeout is 1 second

Why?

• We observed long runs in the rebalancing loop caused by the storage operation processing with delays. To ensure that a rebalancing is not stuck by the long running operations, context.WithTimeout has been introduced.

How did you test it?

• Unit tests

Potential risks

• In case of the big number of shards, the rebalancing loop is about to be stuck due to unavailability to at least load the whole state.

Release notes

Documentation Changes