Skip to content

fix: error out early if ray cluster does not have resources #89

Merged
parthchadha merged 2 commits intomainfrom
pchadha/ray-cluster-validation
Mar 26, 2025
Merged

fix: error out early if ray cluster does not have resources #89
parthchadha merged 2 commits intomainfrom
pchadha/ray-cluster-validation

Conversation

@parthchadha
Copy link
Copy Markdown
Contributor

What does this PR do ?

Throw error early if resources are not available.

ValueError: Not enough GPUs available. Requested 16 GPUs, but only 8 are available in the cluster.
Traceback (most recent call last):
  File "/core/gh/NeMo-Reinforcer/./examples/run_grpo_math.py", line 293, in <module>
    main()
  File "/core/gh/NeMo-Reinforcer/./examples/run_grpo_math.py", line 275, in main
    ) = setup(config, tokenizer, dataset, val_dataset)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/core/gh/NeMo-Reinforcer/nemo_reinforcer/algorithms/grpo.py", line 205, in setup
    cluster = RayVirtualCluster(
              ^^^^^^^^^^^^^^^^^^
  File "/core/gh/NeMo-Reinforcer/nemo_reinforcer/distributed/virtual_cluster.py", line 162, in __init__
    self._init_placement_groups(placement_group_strategy)
  File "/core/gh/NeMo-Reinforcer/nemo_reinforcer/distributed/virtual_cluster.py", line 189, in _init_placement_groups
    raise ValueError(
ValueError: Not enough GPUs available. Requested 16 GPUs, but only 8 are available in the cluster.

Issues

List issues that this PR closes (syntax):

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this 

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

… requested

Signed-off-by: Parth Chadha <pchadha@nvidia.com>
@parthchadha parthchadha merged commit ab52ca6 into main Mar 26, 2025
18 of 21 checks passed
@parthchadha parthchadha deleted the pchadha/ray-cluster-validation branch March 26, 2025 23:22
yfw pushed a commit that referenced this pull request Apr 2, 2025
Signed-off-by: Parth Chadha <pchadha@nvidia.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
KiddoZhu pushed a commit that referenced this pull request May 6, 2025
Signed-off-by: Parth Chadha <pchadha@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants