4bit ONNX models support

[Mihaiii]

Feature request

It look like ONNX now supports 4bit: https://onnx.ai/onnx/technical/int4.html

It would be nice if we could use 4bit models with transformers.js .

Motivation

Make models for semantic-autocomplete that are even smaller when it comes to disk size

Your contribution

Maybe, but for the moment is not even clear to me how to convert models to 4bit. I think ONNX needs to add more docs.

[bil-ash]

That would be a very nice feature. It will then be possible to get quite small whisper and nllb models. Since, W4A4 quant is supported in onnx 1.17.0, I guess it will be available in onnxruntime 1.19.0. Till then, @xenova please finish the refactoring of the conversion and quantization script.

[flatsiedatsie]

It will then be possible to get quite small whisper and nllb models

Ooh, perhaps it will then be able to run on mobile too?

[bil-ash]

@xenova Any updates? It seems to be well suited for encoder-decoder & encoder-only models, so whisper models running on browser should be the biggest beneficiary of this conversion.

[bil-ash]

The docs also got updated recently. @xenova Please make it available in transformers.js . Will be very helpful for encoder-decoder models like whisper and nllb.

[xenova]

Phew, I should have updated this issue a while ago. We have supported 4-bit models for over a year! We also have additional 4-bit support for ops as of Transformers.js v4 (#1382). Feel free to try it out!