[fsdp,trainer,vllm_omni,algo] feat: support FlowGRPO training for QwenImage

[zhtmike]

What does this PR do?

Follow-up Work for #4639

• A trainable script for the FlowGRPO algorithm for Qwen-Image is provided.
• Support for vLLM-Omni has been added to the rollout engine.
• Diffusers has been integrated as the training engine for the diffusion model.

This is currently a draft PR and contains repeated or redundant code/configurations. A pruned version will be available once it is ready for review.

Add concise overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review.

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: ...
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, veomni, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data, cfg, reward
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

We use Levenshtein distance for OCR reward calculation. Qwen2.5-VL-3B is employed as the reward model. The following figure shows the scores for the testing dataset.

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc.

API and Usage Example

Demonstrate how the API changes if any, and provide usage example(s) if possible.

# Add code snippet or script demonstrating how to use this

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)
• If your PR is related to the recipe submodule, please also update the reference to the submodule commit via git submodule update --remote or cd recipe && git pull origin main.

[CLAassistant]

All committers have signed the CLA.

[gemini-code-assist]

Code Review

This pull request introduces significant new functionality to support FlowGRPO training, including a new trainer, core algorithm implementations, and integrations with vLLM-Omni and diffusers. The changes are extensive and well-structured. My review primarily focuses on the new test files, where I've identified several instances of hardcoded paths. These paths make the tests non-portable and likely to fail in CI environments or for other developers. Addressing these will be crucial for ensuring the long-term maintainability and reliability of the new features.

[gemini-code-assist]

The test configuration hardcodes a model path using os.path.expanduser. This makes the test non-portable and dependent on a specific local file structure, which will cause it to fail in CI environments or for other developers. Tests should be self-contained and not rely on external, user-specific files.

Suggested change

	model_path = os.path.expanduser("~/models/Qwen/Qwen-Image")
	# It's recommended to use a mock model or a small, downloadable test model.
	# For example, you could use a fixture to create a temporary model directory.
	model_path = "path/to/test/model" # Replace with a portable path solution

[gemini-code-assist]

This test produces side effects by saving generated images to the filesystem. Tests should be hermetic and not write files to the working directory, as this can interfere with the test environment and other tests. The TODO comment indicates this is likely temporary debugging code that should be removed.

[gemini-code-assist]

The function compute_score_ocr contains a hardcoded model path using os.path.expanduser. This makes the function difficult to reuse and test in different environments. The model path should be provided as a configuration parameter rather than being hardcoded.

    model_name = model_name

[gemini-code-assist]

The test hardcodes model paths using os.path.expanduser, which makes it non-portable and likely to fail in CI or for other developers. Tests should be self-contained.

Suggested change

	rollout_model_name = os.path.expanduser("~/models/Qwen/Qwen-Image")
	reward_model_name = os.path.expanduser("~/models/Qwen/Qwen2.5-VL-3B-Instruct")
	# Consider using a fixture to provide a path to a small, downloadable test model
	# or mocking the model loading process entirely.
	rollout_model_name = "path/to/test/rollout_model"
	reward_model_name = "path/to/test/reward_model"

[gemini-code-assist]

This test function computes results but lacks assertions to verify their correctness. It only prints the output. A test without assertions does not validate the behavior of the code and can only confirm that it runs without crashing. Please add assertions to check the properties of the outputs, such as shape, type, or value ranges.

    # Example assertion:
    # assert len(outputs) > 0
    # assert "score" in outputs[0].non_tensor_batch
    # assert isinstance(outputs[0].non_tensor_batch["score"], float)

[gemini-code-assist]

The test configuration hardcodes a model path using ~/models/Qwen/Qwen-Image. This makes the test non-portable and dependent on a specific user's local file setup. Tests should be hermetic and use mock objects or small, self-contained test artifacts.

        model="path/to/test/model", # Replace with a portable path solution

[gemini-code-assist]

The test hardcodes a data path using os.path.expanduser. This makes the test non-portable and reliant on a specific local directory structure. Test data should be created programmatically within the test or included as a small test artifact.

[gemini-code-assist]

This test hardcodes the path to the tokenizer using os.path.expanduser. This makes the test non-portable and will cause it to fail for other users or in CI environments. Please use a mock tokenizer or a small, self-contained test artifact.

[gemini-code-assist]

Catching ValueError here is a good addition, as AutoConfig.from_pretrained can raise it for certain model types like diffusers that don't have a standard config.json. This makes the function more robust.