[Feature] microsoft/Phi-4-multimodal-instruct

[avinash31d]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Model architectures ['Phi4MMForCausalLM'] are not supported for now. When can we expect support for this model. As Phi4-mini-instruct is already supported, this can be en extension to it.

Related resources

No response

[lifuhuang]

We are also interested in phi4-mm support. Thank you @avinash31d for creating it.

Let us know if there is anything we can help. Would love to collaborate over and learn from this. (cc @hebiao064, @lhcavalcanti, @temporalartichoke).

[hebiao064]

Refer to #4693 about how to add new model

And refer to existing Phi model as well: https://github.com/sgl-project/sglang/blob/141a45964453b7a3b827303ca7847d5902f974c7/python/sglang/srt/models/phi3_small.py

HF Impl: https://github.com/huggingface/transformers/tree/main/src/transformers/models/phi4_multimodal

[hebiao064]

Related PR: fix phi3 #5991 with @lifuhuang

[lifuhuang]

Started working on it. Created a parent issue to track progress and break down tasks: #6544