Fix running tracks across multiple load driver machines#1763
Fix running tracks across multiple load driver machines#1763b-deam merged 5 commits intoelastic:masterfrom
Conversation
b-deam
added
bug
pquentin
left a comment
pquentin
left a comment
There was a problem hiding this comment.
Wow, that's a nice catch! I left a small nit about naming.
esrally/driver/driver.py
Outdated
| self.driver_actor = sender | ||
| # load node-specific config to have correct paths available | ||
| self.cfg = load_local_config(msg.config) |
There was a problem hiding this comment.
Do we really need to attach those to self? If we do we should set them to None in __init__ but it looks they're only used locally.
Additionally, I think having both cfg and msg.config is confusing. What do you think about node_config instead of cfg?
There was a problem hiding this comment.
Good points! I refactored these attributes in 34566f1 as I noticed we were needlessly setting the config twice otherwise.
|
buildkite test this please |
|
Hmm, there's some race condition here related to multiple workers on the same host all loading the track (i.e. running git operations) in parallel: I'll work out a fix. |
|
Confirmed working as of latest commit: |
pquentin
changed the title
3 participants
This commit adds a 'new'
Bootstrapmessage handler to theWorkeractor, which is responsible for ensuring that the new actor has the correct configuration and track modules loaded into its system path. This restores the ability to run tracks across multiple load driver machines.The reason tracks worked across
Workeractors on coordinator machine (where the race was invoked) is because the default behaviour for new actors created withmultiprocTCPBaseis that they fork (in the Unix sense) from their parent process. When a process is forked, it inherits the memory and other properties from the parent, including the system path.When a new
Workerwas created on a remote load driver machine the existing parent was missing paths, meaning there were missing modules for the load path.Notes:
receiveMsg_RallyConfigthat was unused, meaning it was unlikely this ever workedFixes #1752
Relates #1206