[core][autoscaler] fix: Invalid status transition from QUEUED to RAY_STOP_REQUESTED in autoscaler v2

[win5923]

Description

When the autoscaler attempts to terminate QUEUED instances to enforce the max_num_nodes_per_type limit, the reconciler crashes with an assertion error. This happens because QUEUED instances are selected for termination, but the state machine doesn't allow transitioning them to a terminated state.

The reconciler assumes all non-ALLOCATED instances have Ray running and attempts to transition QUEUED → RAY_STOP_REQUESTED, which is invalid.

ray/python/ray/autoscaler/v2/instance_manager/reconciler.py

Lines 1178 to 1197 in ba727da

	for terminate_request in to_terminate:
	instance_id = terminate_request.instance_id
	if terminate_request.instance_status == IMInstance.ALLOCATED:
	# The instance is not yet running, so we can't request to stop/drain Ray.
	# Therefore, we can skip the RAY_STOP_REQUESTED state and directly terminate the node.
	im_instance_to_terminate = im_instances_by_instance_id[instance_id]
	updates[terminate_request.instance_id] = IMInstanceUpdateEvent(
	instance_id=instance_id,
	new_instance_status=IMInstance.TERMINATING,
	cloud_instance_id=im_instance_to_terminate.cloud_instance_id,
	termination_request=terminate_request,
	details=f"terminating ray: {terminate_request.details}",
	)
	else:
	updates[terminate_request.instance_id] = IMInstanceUpdateEvent(
	instance_id=instance_id,
	new_instance_status=IMInstance.RAY_STOP_REQUESTED,
	termination_request=terminate_request,
	details=f"draining ray: {terminate_request.details}",
	)

This occurs when max_workers configuration is dynamically reduced or when instances exceed the limit.

2025-12-04 06:21:55,298	INFO event_logger.py:77 -- Removing 167 nodes of type elser-v2-ingest (max number of worker nodes per type reached).
2025-12-04 06:21:55,307 - INFO - Update instance QUEUED->RAY_STOP_REQUESTED (id=e13a1528-ffd9-403b-9fd1-b54e9c2698a0, type=elser-v2-ingest, cloud_instance_id=, ray_id=): draining ray: Terminating node due to MAX_NUM_NODE_PER_TYPE: max_num_nodes=None, max_num_nodes_per_type=220
2025-12-04 06:21:55,307	INFO instance_manager.py:263 -- Update instance QUEUED->RAY_STOP_REQUESTED (id=e13a1528-ffd9-403b-9fd1-b54e9c2698a0, type=elser-v2-ingest, cloud_instance_id=, ray_id=): draining ray: Terminating node due to MAX_NUM_NODE_PER_TYPE: max_num_nodes=None, max_num_nodes_per_type=220
2025-12-04 06:21:55,307 - ERROR - Invalid status transition from QUEUED to RAY_STOP_REQUESTED

This PR add a valid transition QUEUED -> TERMINATED to allow canceling queued instances.

Related issues

Closes #59219

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

[gemini-code-assist]

Code Review

This pull request correctly fixes a bug where QUEUED instances were being transitioned to an invalid state (RAY_STOP_REQUESTED) upon a termination request. The introduction of the QUEUED -> TERMINATED state transition is a logical and clean solution for canceling instance allocations before any cloud resources are provisioned. The changes are well-implemented across the state machine definition, the reconciler logic, and the protobuf comments. The addition of a dedicated test case ensures the new behavior is verified and prevents future regressions. Overall, this is a high-quality contribution that improves the robustness of the autoscaler.

[rueian]

Thanks @win5923! Looks good to me!

[rueian]

@edoakes, please take a look and merge when you get a chance.