Track preparation can fail on multiple load generator machines with custom track processors

[danielmitterdorfer]

Actual behavior

Steps to reproduce:

1. Implement a track that uses a track processor which adds instances of custom classes (not in Rally's default module load path) to the track in the on_after_load_track phase.
2. Start a benchmark using --load-driver-hosts.
3. Observe that Rally is hanging after starting the benchmark.

Relevant logs

In the actor system log we see:

p29529 I    Pending Actor request received for esrally.driver.driver.TrackPreparationActor reqs {'ip': '10.10.30.79'} from ActorAddr-(T|:42161)
p29529 Warn no system has compatible capabilities for Actor esrally.driver.driver.TrackPreparationActor
p29529 I    Requesting creation of esrally.driver.driver.TrackPreparationActor on remote admin ActorAddr-(T|10.10.30.79:1900)
p30298 I    completion error: ************* TransportIntent(ActorAddr-(T|10.10.30.79:32999)-=-BadPacketError-<class 'esrally.driver.driver.PrepareTrack'>-<esrally.driver.driver.PrepareTrack object at 0x7f04968b5fa0>-quit_0:04:59.841806)

This points us towards an issue during track preparation.

Furthermore we see in the logs:

ERR  OUCH!  Error deserializing received data: Traceback (most recent call last):
  File “/home/esbench/.local/lib/python3.8/site-packages/thespian/system/transport/TCPTransport.py”, line 1433, in _addedDataToIncoming
    rdata, extra = inc.data
  File “/home/esbench/.local/lib/python3.8/site-packages/thespian/system/transport/TCPTransport.py”, line 186, in data
    def data(self): return self._rData.completed()
  File “/home/esbench/.local/lib/python3.8/site-packages/thespian/system/transport/streamBuffer.py”, line 80, in completed
    return self._deserialize(self._buf), self._extra
ModuleNotFoundError: No module named ‘logs’
  (rdata=“”, extra=“”)

Analysis

The issue is that the actor system serializes the PrepareTrack message on the coordinator node, which has also loaded the track and sends it then to all hosts specified via --load-driver-hosts. However, the track including the custom classes has not been loaded yet, thus if the track object in the PrepareTrack message references a class that has not been loaded yet on that machine, deseralizing the message fails.