[Deepseek R1][v0] Porting deepseek r1 to habana_main

[xuechendi]

JIRA: https://jira.habana-labs.com/browse/SW-227174

cherry-pick #1030 and fixed conflicts after rebase
Dependency: HabanaAI/vllm-hpu-extension#161
HabanaAI/vllm-hpu-extension#170

Verified with below 3 methods:

1. test with deepseek-v2 BF16 weight. => Passed
2. evaluate acc on deepseek-r1 with out of box block fp8 weight => Passed
3. evaluate acc on deepseek-r1 with out of box block fp8 weight + INC calibrated per-channel scale => Passed acc check, performance reach goal(number is in jira ticket)

== Details ==

1. test with deepseek-v2 BF16 weight:

PT_HPU_LAZY_MODE=1 python run_example_tp.py --model DeepSeek-V2-Lite --tokenizer DeepSeek-V2-Lite --osl 32 

(VllmWorkerProcess pid=1039) WARNING 04-25 03:01:53 [hpu_model_runner.py:1039] Configuration: ('decode', 4, 128) was not warmed-up!
(VllmWorkerProcess pid=1038) WARNING 04-25 03:01:53 [hpu_model_runner.py:1039] Configuration: ('decode', 4, 128) was not warmed-up!
(VllmWorkerProcess pid=1041) WARNING 04-25 03:01:53 [hpu_model_runner.py:1039] Configuration: ('decode', 4, 128) was not warmed-up!
WARNING 04-25 03:01:53 [hpu_model_runner.py:1039] Configuration: ('decode', 4, 128) was not warmed-up!
Processed prompts: 100%|████████████████████████████████████████████████████████████████████████████| 4/4 [00:02<00:00,  1.57it/s, est. speed input: 12.59 toks/s, output: 50.37 toks/s]
e2e took 2.5509743690199684 seconds
====================================
Prompt: 'Hello, my name is'
Generated text: '\nI am a 20 year old student from the UK. I am currently studying for a degree in English Literature and Creative Writing at the University of East'
Ground truth: None
====================================
====================================
Prompt: '0.999 compares to 0.9 is '
Generated text: '100%\n0.9999999999999999999999999'
Ground truth: None
====================================
====================================
Prompt: 'The capital of France is'
Generated text: ' Paris, which is also the largest city in the country. The city is located on the Seine River and is known for its beautiful architecture, museums, and art'
Ground truth: None
====================================
====================================
Prompt: 'The future of AI is'
Generated text: ' in the hands of the people\nThe future of AI is in the hands of the people\nThe future of AI is in the hands of the people\nThe'
Ground truth: None
====================================

2. evaluate acc on deepseek-r1 with out of box block fp8 weight - limit 256

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.9648	±	0.0115
		strict-match	5	exact_match	↑	0.9648	±	0.0115

3. evaluate acc on deepseek-r1 with out of box block fp8 weight + INC calibrated per-channel scale

run calibration

{
    "method": "HOOKS",
    "mode": "MEASURE",
    "observer": "maxabs",
    "whitelist": {
        "types": [],
        "names": []
    },
    "blocklist": {
        "types": [],
        "names": ["lm_head", "mlp\\.gate\\b"]
    },
    "quantize_weight": false,
    "dump_stats_path": "./nc_workspace_measure_kvache/inc_measure_output"
}

OFFICIAL_FP8_MODEL=DeepSeek-R1

PT_HPU_LAZY_MODE=1 \
VLLM_SKIP_WARMUP=true \
PT_HPU_ENABLE_LAZY_COLLECTIVES=true \
PT_HPU_WEIGHT_SHARING=0 \
QUANT_CONFIG=inc_measure_with_fp8kv_config.json \
python run_example_tp.py --model ${OFFICIAL_FP8_MODEL} --tokenizer ${OFFICIAL_FP8_MODEL} --osl 32 --max_num_seqs 1 --nprompts 512 --dataset pile

run test

QUANT_CONFIG=inc_quant_with_fp8kv_config.json \
PT_HPU_LAZY_MODE=1 \
VLLM_DISABLE_MARK_SCALES_AS_CONST=1 \
VLLM_SKIP_WARMUP=true \
PT_HPU_ENABLE_LAZY_COLLECTIVES=true \
PT_HPU_WEIGHT_SHARING=0 \
lm_eval --model vllm \
  --model_args "pretrained=/mnt/weka/data/pytorch/DeepSeek-R1/,tensor_parallel_size=8,distributed_executor_backend=mp,trust_remote_code=true,max_model_len=4096,use_v2_block_manager=True,dtype=bfloat16,kv_cache_dtype=fp8_inc,enable_expert_parallel=True,max_num_seqs=256" \
  --tasks gsm8k --num_fewshot "5" \
  --batch_size "256" --limit 16 --log_samples --output_path fp8_gsm8k.json

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.9688	±	0.0109
		strict-match	5	exact_match	↑	0.9688	±	0.0109

[xuechendi]

@kzawora-intel @michalkuligowski @kwisniewski98 @madamczyk-intel

[michalkuligowski]

/run-gaudi-tests

[xuechendi]

I verified failed CI locally, and they are working OK
And I realized that since we haven't get HabanaAI/vllm-hpu-extension#161 merged firstly, so it is lacking vllm-hpu-extension functions in this PR as well.

Please help to review HabanaAI/vllm-hpu-extension#161 firstly

[michalkuligowski]

@xuechendi you can update requiremets/hpu.txt for the related extension change to be utilized. We cant merge it before it is tested in this PR

[xuechendi]

/run-gaudi-tests

[xuechendi]

CI failed on getting resource:

[michalkuligowski]

/skip-gaudi-tests - two multimodal jobs failing is false negative, the undrlying tests are passing

[kwisniewski98]

There is a lot of changes that I'm not sure if are upstreamable, we'd have to ask maintainers.
Also, I see workarounds that were introduced by me, about which I'm also not sure if we can introduce to main, i. e. MLA not supporting V1 and issues with torch compile

[xuechendi]

There is a lot of changes that I'm not sure if are upstreamable, we'd have to ask maintainers. Also, I see workarounds that were introduced by me, about which I'm also not sure if we can introduce to main, i. e. MLA not supporting V1 and issues with torch compile

This PR is aming to cherry-pick 1.21 deepseek support to habana_main. I don't want to push more changes to this PR to make the impl being very different to the one we merged for 1.21.
Let's aim V1 support and torch compile in next PR.
I hope to get this PR merged, so llama4 and Qwen3 can share the same MOE changes.

[xuechendi]

/run-gaudi-tests

[xuechendi]

/run-gaudi-tests

[jikunshang]

q_proj removed. need change.

[xuechendi]

/run-gaudi-tests