Add gguf dependency for cpu/xpu

[ZailiWang]

gguf dependency is introduced in #11964

[gemini-code-assist]

Summary of Changes

Hello @ZailiWang, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates the gguf library into the project's dependency management for both CPU and XPU configurations. This change ensures that environments targeting these platforms will automatically include gguf, which was previously introduced in a related pull request, facilitating its use across different hardware setups.

Highlights

• Dependency Addition: The gguf dependency has been added to the pyproject_cpu.toml and pyproject_xpu.toml files, ensuring its availability for both CPU and XPU environments.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This PR adds the gguf dependency for CPU and XPU environments. While the changes themselves are correct, the underlying GGUF feature implementation appears to be CUDA-only. As seen in python/sglang/srt/layers/quantization/gguf.py, the necessary kernels are only imported for CUDA builds, which will lead to runtime errors on CPU and XPU. I've left critical comments on the dependency files suggesting to either add CPU/XPU support or remove these dependencies to prevent breaking these environments.

[gemini-code-assist]

Adding the gguf dependency for CPU environments seems premature. The current GGUF implementation in python/sglang/srt/layers/quantization/gguf.py is CUDA-specific and will cause runtime NameErrors on CPU-only systems because the required kernels are not imported. This will break functionality for users trying to load GGUF models on CPU. It's recommended to either implement a CPU-compatible fallback for GGUF operations or remove this dependency until CPU support is complete.

[gemini-code-assist]

Similar to the CPU build, adding the gguf dependency for XPU is problematic. The GGUF implementation in python/sglang/srt/layers/quantization/gguf.py only contains CUDA kernels and lacks any implementation for XPU. This will result in runtime errors when attempting to use GGUF models on XPU. Please consider adding XPU support or deferring this dependency to avoid shipping broken functionality.