[Bug]: Missing detection of BFloat16 for CPU ARM

[wallashss]

Your current environment

The output of `python collect_env.py`

Collecting environment information...
PyTorch version: 2.5.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (aarch64)
GCC version: (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
Clang version: Could not collect
CMake version: version 3.31.2
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.9.12-200.fc40.aarch64-aarch64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         aarch64
CPU op-mode(s):                       64-bit
Byte Order:                           Little Endian
CPU(s):                               4
On-line CPU(s) list:                  0-3
Vendor ID:                            Apple
Model:                                0
Thread(s) per core:                   1
Core(s) per cluster:                  4
Socket(s):                            -
Cluster(s):                           1
Stepping:                             0x0
BogoMIPS:                             48.00
Flags:                                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-3
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; __user pointer sanitization
Vulnerability Spectre v2:             Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] pyzmq==26.2.0
[pip3] torch==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.47.1
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.4.post2.dev543+g313608dab.d20250102
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
Could not collect

LD_LIBRARY_PATH=/usr/local/lib/python3.10/dist-packages/cv2/../../lib64:

Model Input Dumps

No response

🐛 Describe the bug

On CPU-ARM not all processors have support for bfloat16. In those case trying to run inference will crash like in the following stacktrace:

ERROR 01-07 18:11:29 engine.py:135] RuntimeError('"rms_norm_impl" not implemented for \'BFloat16\'')
ERROR 01-07 18:11:29 engine.py:135] Traceback (most recent call last):
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 133, in start
ERROR 01-07 18:11:29 engine.py:135]     self.run_engine_loop()
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 196, in run_engine_loop
ERROR 01-07 18:11:29 engine.py:135]     request_outputs = self.engine_step()
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 214, in engine_step
ERROR 01-07 18:11:29 engine.py:135]     raise e
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 205, in engine_step
ERROR 01-07 18:11:29 engine.py:135]     return self.engine.step()
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 1394, in step
ERROR 01-07 18:11:29 engine.py:135]     outputs = self.model_executor.execute_model(
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/cpu_executor.py", line 201, in execute_model
ERROR 01-07 18:11:29 engine.py:135]     output = self.driver_method_invoker(self.driver_worker,
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/cpu_executor.py", line 298, in _driver_method_invoker
ERROR 01-07 18:11:29 engine.py:135]     return getattr(driver, method)(*args, **kwargs)
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 344, in execute_model
ERROR 01-07 18:11:29 engine.py:135]     output = self.model_runner.execute_model(
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 01-07 18:11:29 engine.py:135]     return func(*args, **kwargs)
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/cpu_model_runner.py", line 530, in execute_model
ERROR 01-07 18:11:29 engine.py:135]     hidden_states = model_executable(
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 01-07 18:11:29 engine.py:135]     return self._call_impl(*args, **kwargs)
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 01-07 18:11:29 engine.py:135]     return forward_call(*args, **kwargs)
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen2.py", line 477, in forward
ERROR 01-07 18:11:29 engine.py:135]     hidden_states = self.model(input_ids, positions, kv_caches,
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/decorators.py", line 168, in __call__
ERROR 01-07 18:11:29 engine.py:135]     return self.forward(*args, **kwargs)
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen2.py", line 340, in forward
ERROR 01-07 18:11:29 engine.py:135]     hidden_states, residual = layer(
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 01-07 18:11:29 engine.py:135]     return self._call_impl(*args, **kwargs)
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 01-07 18:11:29 engine.py:135]     return forward_call(*args, **kwargs)
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen2.py", line 243, in forward
ERROR 01-07 18:11:29 engine.py:135]     hidden_states = self.input_layernorm(hidden_states)
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 01-07 18:11:29 engine.py:135]     return self._call_impl(*args, **kwargs)
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 01-07 18:11:29 engine.py:135]     return forward_call(*args, **kwargs)
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/custom_op.py", line 24, in forward
ERROR 01-07 18:11:29 engine.py:135]     return self._forward_method(*args, **kwargs)
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/custom_op.py", line 48, in forward_cpu
ERROR 01-07 18:11:29 engine.py:135]     return self.forward_cuda(*args, **kwargs)
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/layernorm.py", line 94, in forward_cuda
ERROR 01-07 18:11:29 engine.py:135]     ops.rms_norm(
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/vllm/_custom_ops.py", line 182, in rms_norm
ERROR 01-07 18:11:29 engine.py:135]     torch.ops._C.rms_norm(out, input, weight, epsilon)
ERROR 01-07 18:11:29 engine.py:135]   File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 1116, in __call__
ERROR 01-07 18:11:29 engine.py:135]     return self._op(*args, **(kwargs or {}))
ERROR 01-07 18:11:29 engine.py:135] RuntimeError: "rms_norm_impl" not implemented for 'BFloat16'

Server

vllm serve Qwen/Qwen2.5-0.5B-Instruct

Request

  curl http://localhost:8002/v1/completions -H "Content-Type: application/json"   -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "prompt": ["How to make pizza"],
    "max_tokens": 100,
    "temperature": 0 
  }'

Fix suggestion

To support or not bfloat16 is a device dependent issue. The ideal solution is to check if the device support or not in vLLM code. I see to two ways to address this problem:

• At build time: The easiest way. We need to check during the build for CPU if the host has support to it like in cpu_extension.cpu and add some bindings to the python side to check if the build support the feature and properly handle cases where this feature is used to allow it or not and avoid crashes. This solution works nice when the user build and install vLLM from the source.
• At runtime: Part of the implementation is similar to the build time, however we should always build with support of bfloat64, and we have to check not only if the build support it but also if bfloat64 is available in the host at runtime (probably it is possible to address the latter with python only). This solution is proper for distribution if we plan to do that in the future.

This issue also helps in the case of Mac with Apple chips, M1 does not support bfloat64 but newer version shall support it and they could take advantage of this feature.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[racheldevops]

Same error; stack below.
For me at least its a Critical and kills the server/the local docker container

Apple Silicon docker ARM build for VLLM
machine: Apple M4 Pro
So as referenced in other item; I'm one of those daft new Apple chip people.

curl --location 'http://localhost:8000/v1/chat/completions'
--header 'Content-Type: application/json'
--data '{
"model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
"messages": [
{
"role": "assistant",
"content": "What is the capital of indonesia?"
}
],
"temperature": 0.6
}'

INFO:     172.17.0.1:61032 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 291, in app
    solved_result = await solve_dependencies(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/dependencies/utils.py", line 666, in solve_dependencies
    ) = await request_body_to_args(  # body_params checked above
  File "/usr/local/lib/python3.10/dist-packages/fastapi/dependencies/utils.py", line 898, in request_body_to_args
    v_, errors_ = _validate_value_with_model_field(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/dependencies/utils.py", line 706, in _validate_value_with_model_field
    v_, errors_ = field.validate(value, values, loc=loc)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/_compat.py", line 129, in validate
    self._type_adapter.validate_python(value, from_attributes=True),
  File "/usr/local/lib/python3.10/dist-packages/pydantic/type_adapter.py", line 412, in validate_python
    return self.validator.validate_python(
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/protocol.py", line 52, in __log_extra_fields__
    result = handler(data)
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/protocol.py", line 628, in check_generation_prompt
    if data.get("continue_final_message") and data.get(
AttributeError: 'bytes' object has no attribute 'get'
INFO 01-29 17:15:27 chat_utils.py:330] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
INFO 01-29 17:15:27 logger.py:37] Received request chatcmpl-50a43c935f3a42b782d2bf3f0312d2b0: prompt: '<｜begin▁of▁sentence｜><｜Assistant｜>What is the capital of indonesia?<｜end▁of▁sentence｜><｜Assistant｜>', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.rature=0.6, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32757, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 01-29 17:15:27 engine.py:273] Added request chatcmpl-50a43c935f3a42b782d2bf3f0312d2b0.
CRITICAL 01-29 17:15:27 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO:     172.17.0.1:55394 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR 01-29 17:15:27 engine.py:137] RuntimeError('"rms_norm_impl" not implemented for \'BFloat16\'')
ERROR 01-29 17:15:27 engine.py:137] Traceback (most recent call last):
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 135, in start
ERROR 01-29 17:15:27 engine.py:137]     self.run_engine_loop()
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 198, in run_engine_loop
ERROR 01-29 17:15:27 engine.py:137]     request_outputs = self.engine_step()
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 216, in engine_step
ERROR 01-29 17:15:27 engine.py:137]     raise e
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 207, in engine_step
ERROR 01-29 17:15:27 engine.py:137]     return self.engine.step()
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 1384, in step
ERROR 01-29 17:15:27 engine.py:137]     outputs = self.model_executor.execute_model(
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 136, in execute_model
ERROR 01-29 17:15:27 engine.py:137]     output = self.collective_rpc("execute_model",
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 49, in collective_rpc
ERROR 01-29 17:15:27 engine.py:137]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 2208, in run_method
ERROR 01-29 17:15:27 engine.py:137]     return func(*args, **kwargs)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 411, in execute_model
ERROR 01-29 17:15:27 engine.py:137]     output = self.model_runner.execute_model(
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 01-29 17:15:27 engine.py:137]     return func(*args, **kwargs)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/cpu_model_runner.py", line 655, in execute_model
ERROR 01-29 17:15:27 engine.py:137]     hidden_states = model_executable(
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 01-29 17:15:27 engine.py:137]     return self._call_impl(*args, **kwargs)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 01-29 17:15:27 engine.py:137]     return forward_call(*args, **kwargs)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen2.py", line 484, in forward
ERROR 01-29 17:15:27 engine.py:137]     hidden_states = self.model(input_ids, positions, kv_caches,
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/decorators.py", line 170, in __call__
ERROR 01-29 17:15:27 engine.py:137]     return self.forward(*args, **kwargs)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen2.py", line 346, in forward
ERROR 01-29 17:15:27 engine.py:137]     hidden_states, residual = layer(
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 01-29 17:15:27 engine.py:137]     return self._call_impl(*args, **kwargs)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 01-29 17:15:27 engine.py:137]     return forward_call(*args, **kwargs)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen2.py", line 241, in forward
ERROR 01-29 17:15:27 engine.py:137]     hidden_states = self.input_layernorm(hidden_states)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 01-29 17:15:27 engine.py:137]     return self._call_impl(*args, **kwargs)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 01-29 17:15:27 engine.py:137]     return forward_call(*args, **kwargs)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/custom_op.py", line 23, in forward
ERROR 01-29 17:15:27 engine.py:137]     return self._forward_method(*args, **kwargs)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/custom_op.py", line 47, in forward_cpu
ERROR 01-29 17:15:27 engine.py:137]     return self.forward_cuda(*args, **kwargs)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/layernorm.py", line 94, in forward_cuda
ERROR 01-29 17:15:27 engine.py:137]     ops.rms_norm(
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/vllm/_custom_ops.py", line 151, in rms_norm
ERROR 01-29 17:15:27 engine.py:137]     torch.ops._C.rms_norm(out, input, weight, epsilon)
ERROR 01-29 17:15:27 engine.py:137]   File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 1116, in __call__
ERROR 01-29 17:15:27 engine.py:137]     return self._op(*args, **(kwargs or {}))
ERROR 01-29 17:15:27 engine.py:137] RuntimeError: "rms_norm_impl" not implemented for 'BFloat16'
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [1]

[INNKCake]

Oracle Cloud, Ubuntu 20.04, Ampere A1 ARM processor, build from source in docker, same error.

According to the thread it was fixed, but for MacOS; any ideas what needed to be done to fix it for other ARM?
I dirty fixed it by overriding torch_dtype as torch.float16 in config.py

curl http://localhost:8888/v1/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen2.5-Coder-3B-Instruct", "prompt": "San Francisco is a", "max_tokens": 7, "temperature": 0 }'

vllm-cpu  | INFO 03-08 14:03:38 [engine.py:289] Added request cmpl-0e8a90ddef5243af9c51a89a5b946d05-0.
vllm-cpu  | CRITICAL 03-08 14:03:38 [launcher.py:116] MQLLMEngine is already dead, terminating server process
vllm-cpu  | INFO:     192.168.64.1:51888 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141] RuntimeError('"rms_norm_impl" not implemented for \'BFloat16\'')
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141] Traceback (most recent call last):
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 139, in start
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     self.run_engine_loop()
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 202, in run_engine_loop
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     request_outputs = self.engine_step()
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 228, in engine_step
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     raise e
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 211, in engine_step
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     return self.engine.step()
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 1407, in step
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     outputs = self.model_executor.execute_model(
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 284, in execute_model
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     driver_outputs = self._driver_execute_model(execute_model_req)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/mp_distributed_executor.py", line 144, in _driver_execute_model
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     return self.driver_worker.execute_model(execute_model_req)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 420, in execute_model
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     output = self.model_runner.execute_model(
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     return func(*args, **kwargs)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/cpu_model_runner.py", line 654, in execute_model
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     hidden_states = model_executable(
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     return self._call_impl(*args, **kwargs)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     return forward_call(*args, **kwargs)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen2.py", line 462, in forward
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/decorators.py", line 172, in __call__
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     return self.forward(*args, **kwargs)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen2.py", line 338, in forward
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     hidden_states, residual = layer(
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     return self._call_impl(*args, **kwargs)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     return forward_call(*args, **kwargs)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen2.py", line 239, in forward
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     hidden_states = self.input_layernorm(hidden_states)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     return self._call_impl(*args, **kwargs)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     return forward_call(*args, **kwargs)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/custom_op.py", line 25, in forward
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     return self._forward_method(*args, **kwargs)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/custom_op.py", line 49, in forward_cpu
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     return self.forward_cuda(*args, **kwargs)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/layernorm.py", line 95, in forward_cuda
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     ops.rms_norm(
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/vllm/_custom_ops.py", line 153, in rms_norm
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     torch.ops._C.rms_norm(out, input, weight, epsilon)
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]   File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 1116, in __call__
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141]     return self._op(*args, **(kwargs or {}))
vllm-cpu  | ERROR 03-08 14:03:38 [engine.py:141] RuntimeError: "rms_norm_impl" not implemented for 'BFloat16'
vllm-cpu  | INFO:     Shutting down
vllm-cpu  | INFO:     Waiting for application shutdown.
vllm-cpu  | INFO:     Application shutdown complete.
vllm-cpu  | INFO:     Finished server process [1]
vllm-cpu exited with code 0

[sramkrishna]

Oracle Cloud, Ubuntu 20.04, Ampere A1 ARM processor, build from source in docker, same error.

According to the thread it was fixed, but for MacOS; any ideas what needed to be done to fix it for other ARM? I dirty fixed it by overriding torch_dtype as torch.float16 in config.py

I"m getting the same issue. Thanks for the work around. I also did a similar workaround which works but probably should follow the guidance above.

[zhengkezhou1]

I follow QuicklyStart example and same issue here.

MacOS M3

INFO 05-10 18:19:25 [engine.py:310] Added request cmpl-0d75eb1d0c3442d789e0f59240c2b9eb-0.
INFO:     127.0.0.1:49297 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
INFO:     Shutting down
ERROR 05-10 18:19:25 [engine.py:160] RuntimeError('"rms_norm_impl" not implemented for \'BFloat16\'')
ERROR 05-10 18:19:25 [engine.py:160] Traceback (most recent call last):
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/engine/multiprocessing/engine.py", line 158, in start
ERROR 05-10 18:19:25 [engine.py:160]     self.run_engine_loop()
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/engine/multiprocessing/engine.py", line 221, in run_engine_loop
ERROR 05-10 18:19:25 [engine.py:160]     request_outputs = self.engine_step()
ERROR 05-10 18:19:25 [engine.py:160]                       ^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/engine/multiprocessing/engine.py", line 247, in engine_step
ERROR 05-10 18:19:25 [engine.py:160]     raise e
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/engine/multiprocessing/engine.py", line 230, in engine_step
ERROR 05-10 18:19:25 [engine.py:160]     return self.engine.step()
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/engine/llm_engine.py", line 1398, in step
ERROR 05-10 18:19:25 [engine.py:160]     outputs = self.model_executor.execute_model(
ERROR 05-10 18:19:25 [engine.py:160]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/executor/executor_base.py", line 299, in execute_model
ERROR 05-10 18:19:25 [engine.py:160]     driver_outputs = self._driver_execute_model(execute_model_req)
ERROR 05-10 18:19:25 [engine.py:160]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/executor/mp_distributed_executor.py", line 144, in _driver_execute_model
ERROR 05-10 18:19:25 [engine.py:160]     return self.driver_worker.execute_model(execute_model_req)
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/worker/worker_base.py", line 420, in execute_model
ERROR 05-10 18:19:25 [engine.py:160]     output = self.model_runner.execute_model(
ERROR 05-10 18:19:25 [engine.py:160]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 05-10 18:19:25 [engine.py:160]     return func(*args, **kwargs)
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/worker/cpu_model_runner.py", line 639, in execute_model
ERROR 05-10 18:19:25 [engine.py:160]     hidden_states = model_executable(
ERROR 05-10 18:19:25 [engine.py:160]                     ^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
ERROR 05-10 18:19:25 [engine.py:160]     return self._call_impl(*args, **kwargs)
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
ERROR 05-10 18:19:25 [engine.py:160]     return forward_call(*args, **kwargs)
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/model_executor/models/qwen2.py", line 466, in forward
ERROR 05-10 18:19:25 [engine.py:160]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
ERROR 05-10 18:19:25 [engine.py:160]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/compilation/decorators.py", line 172, in __call__
ERROR 05-10 18:19:25 [engine.py:160]     return self.forward(*args, **kwargs)
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/model_executor/models/qwen2.py", line 343, in forward
ERROR 05-10 18:19:25 [engine.py:160]     hidden_states, residual = layer(
ERROR 05-10 18:19:25 [engine.py:160]                               ^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
ERROR 05-10 18:19:25 [engine.py:160]     return self._call_impl(*args, **kwargs)
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
ERROR 05-10 18:19:25 [engine.py:160]     return forward_call(*args, **kwargs)
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/model_executor/models/qwen2.py", line 238, in forward
ERROR 05-10 18:19:25 [engine.py:160]     hidden_states = self.input_layernorm(hidden_states)
ERROR 05-10 18:19:25 [engine.py:160]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
ERROR 05-10 18:19:25 [engine.py:160]     return self._call_impl(*args, **kwargs)
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
ERROR 05-10 18:19:25 [engine.py:160]     return forward_call(*args, **kwargs)
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/model_executor/custom_op.py", line 25, in forward
ERROR 05-10 18:19:25 [engine.py:160]     return self._forward_method(*args, **kwargs)
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/model_executor/custom_op.py", line 49, in forward_cpu
ERROR 05-10 18:19:25 [engine.py:160]     return self.forward_cuda(*args, **kwargs)
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/model_executor/layers/layernorm.py", line 164, in forward_cuda
ERROR 05-10 18:19:25 [engine.py:160]     return norm_func(x, self.weight.data, self.variance_epsilon)
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/model_executor/layers/layernorm.py", line 23, in rms_norm
ERROR 05-10 18:19:25 [engine.py:160]     ops.rms_norm(
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/workplace/vllm/vllm/_custom_ops.py", line 194, in rms_norm
ERROR 05-10 18:19:25 [engine.py:160]     torch.ops._C.rms_norm(out, input_contiguous, weight, epsilon)
ERROR 05-10 18:19:25 [engine.py:160]   File "/Users/zhengkezhou/miniconda3/envs/vllm/lib/python3.12/site-packages/torch/_ops.py", line 1158, in __call__
ERROR 05-10 18:19:25 [engine.py:160]     return self._op(*args, **(kwargs or {}))
ERROR 05-10 18:19:25 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-10 18:19:25 [engine.py:160] RuntimeError: "rms_norm_impl" not implemented for 'BFloat16'
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [75557]

[reidliu41]

try to specify --dtype float16 work:
e.g.

vllm serve qwen/Qwen1.5-0.5B-Chat --dtype float16

[FireballDWF]

I get the same error on a graviton2 ( c6gd.8xlarge ) instance which doesn't support bf16.

NFO 05-18 21:53:04 [importing.py:16] Triton not installed or not compatible; certain GPU-related functions will not be available.
WARNING 05-18 21:53:04 [importing.py:28] Triton is not installed. Using dummy decorators. Install it via `pip install triton` to enable kernel compilation.
INFO 05-18 21:53:06 [__init__.py:248] Automatically detected platform cpu.
Collecting environment information...
==============================
        System Info
==============================
OS                           : Amazon Linux 2023.7.20250512 (aarch64)
GCC version                  : (GCC) 14.2.1 20250110 (Red Hat 14.2.1-7)
Clang version                : Could not collect
CMake version                : version 4.0.2
Libc version                 : glibc-2.34

==============================
       PyTorch Info
==============================
PyTorch version              : 2.7.0+cpu
Is debug build               : False
CUDA used to build PyTorch   : None
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.10 (main, Apr 29 2025, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-5)] (64-bit runtime)
Python platform              : Linux-6.12.25-32.101.amzn2023.aarch64-aarch64-with-glibc2.34

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : False
CUDA runtime version         : No CUDA
CUDA_MODULE_LOADING set to   : N/A
GPU models and configuration : No CUDA
Nvidia driver version        : No CUDA
cuDNN version                : No CUDA
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         aarch64
CPU op-mode(s):                       32-bit, 64-bit
Byte Order:                           Little Endian
CPU(s):                               32
On-line CPU(s) list:                  0-31
Vendor ID:                            ARM
Model name:                           Neoverse-N1
Model:                                1
Thread(s) per core:                   1
Core(s) per socket:                   32
Socket(s):                            1
Stepping:                             r3p1
BogoMIPS:                             243.75
Flags:                                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
L1d cache:                            2 MiB (32 instances)
L1i cache:                            2 MiB (32 instances)
L2 cache:                             32 MiB (32 instances)
L3 cache:                             32 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-31
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; CSV2, BHB
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.5
[pip3] pyzmq==26.4.0
[pip3] torch==2.7.0+cpu
[pip3] torchaudio==2.7.0
[pip3] torchvision==0.22.0
[pip3] transformers==4.51.3
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
Neuron SDK Version           : N/A
vLLM Version                 : 0.9.1.dev53+g9214e6063.d20250517 (git sha: 9214e6063, date: 20250517)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
  Could not collect

==============================
     Environment Variables
==============================
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1

[FireballDWF]

Note, model's which natively default to float16 seem to work fine, like "python -m vllm.entrypoints.openai.api_server —device cpu —model KaranKKR/Llama-3.2-3B-float16-finetuned"

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!