[train] Increase worker group start default timeout to 60s

[liulehui]

Description

1. ray.get(pg_handle.ready(), timeout=self._worker_group_start_timeout_s) includes both start placement group and install runtime env, if the installation takes longer than 30s, it will go into a scheduling/rescheduling phase

2025-10-28 14:49:48.318	Attempting to start training worker group of size 8 with the following resources: [{'GPU': 1}] * 8
I
2025-10-28 14:50:18.369	[FailurePolicy] Decision: FailureDecision.RETRY, Error source: controller, Error count / maximum errors allowed: 1/inf, Error: Training failed due to controller error:
The worker group startup timed out after 30.0 seconds waiting for 8 workers. Potential causes include: (1) temporary insufficient cluster resources while waiting for autoscaling (ignore this warning in this case), (2) infeasible resource request where the provided `ScalingConfig` cannot be satisfied), and (3) transient network issues. Set the RAY_TRAIN_WORKER_GROUP_START_TIMEOUT_S environment variable to increase the timeout.
I
2025-10-28 14:50:18.370	[State Transition] SCHEDULING -> RESCHEDULING.
I
2025-10-28 14:50:18.370	[State Transition] RESCHEDULING -> SCHEDULING.
I
2025-10-28 14:50:18.374	Attempting to start training worker group of size 8 with the following resources: [{'GPU': 1}] * 8
I
2025-10-28 14:50:48.379	[FailurePolicy] Decision: FailureDecision.RETRY, Error source: controller, Error count / maximum errors allowed: 2/inf, Error: Training failed due to controller error:
The worker group startup timed out after 30.0 seconds waiting for 8 workers. Potential causes include: (1) temporary insufficient cluster resources while waiting for autoscaling (ignore this warning in this case), (2) infeasible resource request where the provided `ScalingConfig` cannot be satisfied), and (3) transient network issues. Set the RAY_TRAIN_WORKER_GROUP_START_TIMEOUT_S environment variable to increase the timeout.
I
2025-10-28 14:50:48.379	[State Transition] SCHEDULING -> RESCHEDULING.
I
2025-10-28 14:50:48.379	[State Transition] RESCHEDULING -> SCHEDULING.
I
2025-10-28 14:50:48.382	Attempting to start training worker group of size 8 with the following resources: [{'GPU': 1}] * 8
I
2025-10-28 14:51:18.387	[FailurePolicy] Decision: FailureDecision.RETRY, Error source: controller, Error count / maximum errors allowed: 3/inf, Error: Training failed due to controller error:
The worker group startup timed out after 30.0 seconds waiting for 8 workers. Potential causes include: (1) temporary insufficient cluster resources while waiting for autoscaling (ignore this warning in this case), (2) infeasible resource request where the provided `ScalingConfig` cannot be satisfied), and (3) transient network issues. Set the RAY_TRAIN_WORKER_GROUP_START_TIMEOUT_S environment variable to increase the timeout.
I
2025-10-28 14:51:18.387	[State Transition] SCHEDULING -> RESCHEDULING.
I

2. this change is to change the default timeout to 60s instead to mitigate the fixedScalingPolicy experience.

[gemini-code-assist]

Code Review

This pull request increases the default timeout for worker group startup from 30 seconds to 60 seconds. This change is a practical adjustment to mitigate issues where the installation of runtime environments might exceed the previous timeout, causing premature rescheduling. The increased timeout should improve the reliability of worker group initialization, especially in environments with slower runtime environment setup.

[justinvyu]

Thanks!