[train] Fix exception queue race condition in ThreadRunner

[justinvyu]

Summary

This PR fixes a race condition where an exception raised directly from the user target function doesn't get propagated to the TrainController, which results in the run finishing successfully when it shouldn't.

The fix is to join the monitor queue before before considering the target function finished. This ensures that any outstanding exception is processed. If is_running=False, then thread_runner.get_error() always returns the final value.

Problem

1. The user target function which runs in the TrainingThread raises an error. This adds the error to a queue to be processed by the MonitorThread. The training thread exits and sets is_running = False.
2. The controller polls this worker actor, which calls is_running() and get_error() from the main thread.
3. get_error() acquires the lock, and then reads the error which is currently unset.
4. The monitor thread also tries acquiring the lock, but loses the race and only updates the _exc attribute after the poll status call has finished. The controller sees (finished=True, error=None) and thinks that the run succeeded even though the worker errored.

[gemini-code-assist]

Code Review

This pull request effectively resolves a race condition in the ThreadRunner where an exception in the user's target function could be missed, causing a failed run to appear successful. The fix, which involves joining the monitor thread before the target function's thread completes, is correct and robustly implemented. The removal of the manual _is_running flag in favor of thread.is_alive() simplifies the code and improves reliability. The accompanying tests are excellent, using threading.Event to deterministically reproduce the race condition and validate the fix. I have one minor suggestion to improve code clarity in the tests.

[TimothySeah]

Nice, thanks for the fix!