Skip to content

Interlocked initialisation of node time could be extended from 10s #326

@justparking

Description

@justparking

This warning that is sometimes genertaed relates to how Nodel spins up the Jython parts of a Node:

The interlocked initialisation sequence of the scripting engines is slow or dead-locked. A new lock has been created to hopefully release any potential dead-lock.

The host does its best to execute these functions only one at a time across ALL nodes:

  • creation of new Interpreter object
  • toolkit injection
  • execution of the scripts themselves, script.py, custom---.py ingredient---.py, etc.
  • execution of main functions, before_main, main and after_main

Originally all nodes were spun up in parralel (constrained by a thread-pool of course) however a race-condition was observed related to a Python XML parsing library (xml.etree.ElementTree class) being used by 2 or more nodes and a serious dead lock was often caused.

If the global sequencing lock stays locked for more than 10s the host simply creates a new one and allows the rest of the nodes to initialise themselves. This prevents a badly written node from permanently preventing others from starting up.

On less powerful hosts I often see hosts not complete the initialisation functions without causing that warning meaning there is the potential for overlapping when new locks are created and I've seen on very rare occasions the dead-lock reoccuring.

10s could be reasonably increased to 1 minute which would provide plenty protection from that race condition.

It would also reduce the likelihood of that very technical message being generated which has been known to cause confusion.

I've marked this as minor given the narrow prerequistites for the bug:

  • heavily CPU constrained host
  • at least 2 nodes that make uses of the Python xml.etree.ElementTree class

Metadata

Metadata

Assignees

Labels

bugSomething isn't working as expected

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions