Question about supporting Float16Array

[xmcp]

Question

I am trying transformers.js with WebGPU. The performance is great, but I found that transformers.js returns a Float32Array where the model is quantized to fp16:

const extractor = await pipeline(
    "feature-extraction",
    "bge-small-zh-v1.5",
    {
        device: "webgpu",
        dtype: "fp16",
        local_files_only: true,
    },
);
// ...
const embeddings = await extractor(texts, {pooling: "mean", normalize: true});
console.log(embeddings.data);
// -> Float32Array(5120000) [...]

Since the model itself has only 16-bit precision, returning a Float32Array (instead of Float16Array that is supported in latest browsers) seems a waste of performance. Is this comment correct, and do we have plans to support Float16Array for better performance? Thanks!

[sbrzz]

Hello @xmcp,

Before jumping to any hypotheses about this behavior, could you share the specs of your testing machine?

A few months ago, I noticed some issues related to fp16 here: #1205

It might be worth checking WebGPUReport to confirm if shader-f16 is supported.

I’m also curious if the automatic casting to Float16Array might be related to this.

Cheers,

[xmcp]

Hi @sbrzz,

I tested with latest Chrome on both a MacBook Pro with M3 Pro and another Windows laptop with Intel Core Ultra 185H iGPU.

I can confirm shader-f16 exists on https://webgpureport.org/ and the performance of using dtype: "fp16" is better than fp32. So it should be using fp16 internally, but the output is somehow converted to Float32Array?

I'm unaware of the "automatic casting to Float16Array" thing. Is this a feature of onnxruntime?

[sbrzz]

So it seems in fact using fp16 internally.

I am courius to have a look at the onnx graph (last nodes), are you able to export it after having imported as in #1338 (comment) ?

I don't say the automatic casting is a feature but maybe needed, due to the graph?

[xmcp]

The model is https://huggingface.co/Xenova/bge-small-zh-v1.5 without any modification.

[xenova]

Hi there 👋 There are a couple of reasons why many fp16 models currently output Float32Array outputs:

1. When those models were exported, Float16Array wasn't baselines across browsers (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Float16Array). Before this, we'd use Uint8Array buffers to achieve the same thing.
2. For backwards compatibility reasons, many newly exported models also have Float32Array outputs... even although all computation happens in fp16, we add a fp32 cast right at the end.
3. When models are exported, we need to define the input and output signature beforehand (we can't dynamically select which output type we want at runtime).

That said, many models, especially those with kv cache outputs do output Float16Array values for present key values, but this is because users don't typically access these directly. I'm open to hearing if users would benefit from new model exports with fp16 inputs though!

[sbrzz]

Thanks @xenova.

I'm open to hearing if users would benefit from new model exports with fp16 inputs though!

No suggestion for this topic since I don't work to much with this kind of models.

@xmcp two ways to achieve the optmization you want:

• remove the Cast layer just before the output and export again with Optimum
• modify the - already converted - onnx graph with a surgery operation

[philippn]

Not sure if this is related at all, but the demo at https://huggingface.co/spaces/Xenova/experimental-phi3-webgpu also throws some error related to fp16.

worker-C9kOGc1L.js:8 Uncaught (in promise) TypeError: A float16 tensor's data must be type of function Float16Array() { [native code] }
    at new Q (worker-C9kOGc1L.js:8:19095)
    at new Q (worker-C9kOGc1L.js:2659:70081)
    at z.addPastKeyValues (worker-C9kOGc1L.js:2656:36185)
    at z.U [as _forward] (worker-C9kOGc1L.js:2656:21382)
    at z.forward (worker-C9kOGc1L.js:2656:28124)
    at z.generate (worker-C9kOGc1L.js:2656:34433)
    at of (worker-C9kOGc1L.js:2661:51284)