[QUESTION]NVFP4 Post SFT Training & Model Accuracy

[deepak-vij]

Your question
Hi @Phlip79 , this is regarding the issue I created earlier (see below) related to nvfp4. One of the follow on activities I am trying to do is to gauge the accuracy of the SFT trained model using nvfp4. After completion of the SFT training run using nvfp4, I tried accessing the model using SgLang inference serving engine. The output of the inference request seems to be all gibberish. It all seems to work fine without any quantization. Is there anything I am missing that is messing up the model while doing post SFT training using nvfp4? Thanks.

#3470 (comment)

[Phlip79]

I would need a lot more information to help you out. But I would start investigating by starting with higher precision formats and working your way down.

[deepak-vij]

Hi @Phlip79 , as I mentioned that I wanted to gauge the accuracy of the nvfp4 quantization aware SFT generated model versus non-quantization aware SFT model. My baseline model Qwen3-VL-8B-Instruct is of type bf16 and I am using the following input dataset for SFT training. The accuracy test I have, that uses SGLang as the inference server, works fine for the non-quantization aware SFT generated model. However, when I run the same accuracy script against the nvfp4 quantization aware SFT generated model, I see gibberish coming out of the model. I am wondering if there are any issues with bf16->nvfp4->bf16 conversion process and the model weights stored as bf16 at the completion of SFT process. I am getting this issue even for a very simple prompt - "weather in France".

Is there a way to isolate the issue, meaning is there a way to ascertain that the model weights are fine (no datatype issues etc.) without using Sglang? Let me know if I can provide more information. Thanks.

https://huggingface.co/datasets/chenhegu/geo3k_imgurl/viewer/default/train?row=0

[deepak-vij]

In bypassed SGLang and generated directly using transformers, the serving stack is eliminated (see below). This seems to indicate that The NVFP4 SFT model weights are degraded for some reason. So, this does not seem to be SGLang issue, I guess.

This is NOT an SGLang issue anymore.
+++++++++++++++++++++++
python - <<'PY'
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

path="/data04/deepak/final_model_nvfp4" # <-- change to your nvfp4 export dir
proc = AutoProcessor.from_pretrained(path, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
path, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda"
)

messages = [{"role":"user","content":[{"type":"text","text":"weather in france"}]}]
prompt = proc.apply_chat_template(messages, add_generation_prompt=True)
inputs = proc.tokenizer(prompt, return_tensors="pt").to("cuda")

out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(proc.tokenizer.decode(out[0], skip_special_tokens=True))
PY

[deepak-vij]

Hi @Phlip79 , I wanted to give you an update on this. After digging through this issue further, I have made some further somewhat positive progress on this issue. As I mentioned earlier that the non-quantization aware SFT generated model works fine but quantization aware SFT training generated model was generating gibberish output. I noticed this behavior even for a very simple prompt - "Weather in France".

As brief summary of all this, I was successfully able to resolve the gibberish output behavior of the nvfp4 model by revising the hyper-parameters. However, the accuracy of the nvfp4 model is still really bad as compared to the non-quantization aware model. It would be good to get yours and community feedback on all of this in order to ensure that I am on the right track.

In any case, in order to troubleshoot the issue, I went through the following step-by-step process in order to pinpoint the actual cause for this issue. As you can see that at the end, I was able to resolve this issue by revising the hyperparameters at the time of SFT training.

%%%%%%%% Verifying the generated nvfp4 aware model weights datatype %%%%%%%
I used the following Python program to list out the generated model datatype. As can be seen in the output, the generated model weights are of correct datatype bf16.

import os, glob
from collections import Counter
from safetensors.torch import safe_open

MODEL_DIR = "/data05/deepak/final_model_nvfp4_fine" # <-- change to your nvfp4 export dir

files = sorted(glob.glob(os.path.join(MODEL_DIR, "*.safetensors")))
assert files, f"No safetensors found in {MODEL_DIR}"

dtype_counts = Counter()
example = []

for f in files:
with safe_open(f, framework="pt", device="cpu") as sf:
for k in sf.keys():
t = sf.get_tensor(k)
dtype_counts[str(t.dtype)] += 1
if len(example) < 20:
example.append((k, tuple(t.shape), str(t.dtype)))

print("Dtype counts:", dtype_counts)
print("\nExample tensors:")
for k, shape, dt in example:
print(f"{dt:>12} {shape!s:<18} {k}")

++++++++++Output++++++++
Example tensors:
torch.bfloat16 (151936, 4096) model.language_model.embed_tokens.weight
torch.bfloat16 (4096,) model.language_model.layers.0.input_layernorm.weight
torch.bfloat16 (4096, 12288) model.language_model.layers.0.mlp.down_proj.weight
torch.bfloat16 (12288, 4096) model.language_model.layers.0.mlp.gate_proj.weight
torch.bfloat16 (12288, 4096) model.language_model.layers.0.mlp.up_proj.weight
torch.bfloat16 (4096,) model.language_model.layers.0.post_attention_layernorm.weight
torch.bfloat16 (128,) model.language_model.layers.0.self_attn.k_norm.weight
torch.bfloat16 (1024, 4096) model.language_model.layers.0.self_attn.k_proj.weight
torch.bfloat16 (4096, 4096) model.language_model.layers.0.self_attn.o_proj.weight
torch.bfloat16 (128,) model.language_model.layers.0.self_attn.q_norm.weight
torch.bfloat16 (4096, 4096) model.language_model.layers.0.self_attn.q_proj.weight
torch.bfloat16 (1024, 4096) model.language_model.layers.0.self_attn.v_proj.weight
torch.bfloat16 (4096,) model.language_model.layers.1.input_layernorm.weight
torch.bfloat16 (4096, 12288) model.language_model.layers.1.mlp.down_proj.weight
torch.bfloat16 (12288, 4096) model.language_model.layers.1.mlp.gate_proj.weight
torch.bfloat16 (12288, 4096) model.language_model.layers.1.mlp.up_proj.weight
torch.bfloat16 (4096,) model.language_model.layers.1.post_attention_layernorm.weight
torch.bfloat16 (128,) model.language_model.layers.1.self_attn.k_norm.weight
torch.bfloat16 (1024, 4096) model.language_model.layers.1.self_attn.k_proj.weight
torch.bfloat16 (4096, 4096) model.language_model.layers.1.self_attn.o_proj.weight

%%%%%%%%Verifying if the gibberish output is due to Sglang Inference Engine%%%%%%%
I used the following Python program to access the nvfp4 model with a simple prompt using transformers APIs, bypassing SGLang inference serving altogether. As can be seen in the output, the output is all gibberish. So this proves that this issue is not due to SgLang. Instead, the model seems to collapse in our case.

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

path="/data05/deepak/final_model" # <-- change to your nvfp4 export dir
proc = AutoProcessor.from_pretrained(path, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
path, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda"
)

messages = [{"role":"user","content":[{"type":"text","text":"weather in france"}]}]
prompt = proc.apply_chat_template(messages, add_generation_prompt=True)
inputs = proc.tokenizer(prompt, return_tensors="pt").to("cuda")

out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(proc.tokenizer.decode(out[0], skip_special_tokens=True))

++++++++++Output++++++++
user
weather in france
assistant
Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer:

%%%%%%%% Verifying the next token distribution %%%%%%%
I ran a sanity-check / probe that runs the nvfp4 model on a prompt and then prints the model’s next-token distribution (top-30 + entropy) for the very next token after the prompt. As can be seen in the it’s a textbook logit collapse.

• Next token 'Answer' has 0.994 probability
• Entropy 0.055
• Even 'OK' is only 8.4e-5

So the NVFP4 QAT model has effectively learned “always start with Answer” and is extremely overconfident. That’s why it loops “Answer Answer…”. It’s a training/quantization configuration failure (soft collapse).

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

path="/data05/deepak/final_model"
proc = AutoProcessor.from_pretrained(path, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
path, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda"
).eval()

prompt="<|im_start|>user\nReply with exactly: OK<|im_end|>\n<|im_start|>assistant\n"
inputs = proc.tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
out = model(**inputs)
logits = out.logits[0, -1].float()
probs = torch.softmax(logits, dim=-1)

topk = torch.topk(probs, k=30)
ids = topk.indices.tolist()
vals = topk.values.tolist()
tokens = [proc.tokenizer.decode([i]) for i in ids]

print("Top-30 next tokens:")
for t, p in zip(tokens, vals):
print(f"{repr(t):>20} {float(p):.6f}")

entropy = float(-(probs * (probs + 1e-12).log()).sum())
print("Entropy:", entropy)

++++++++++Output++++++++
Top-30 next tokens:
'Answer' 0.993991
' Answer' 0.003585
'答案' 0.000294
' answer' 0.000229
'answer' 0.000178
'Reply' 0.000123
'OK' 0.000084
'Question' 0.000079
'_answer' 0.000040
'1' 0.000040
'回答' 0.000037
'A' 0.000035
'2' 0.000035
' ANSW' 0.000033
'答' 0.000026
'-' 0.000023
'Response' 0.000021
'**' 0.000020
'답' 0.000016
' OK' 0.000013
' ' 0.000011
' �' 0.000011
'Final' 0.000011
' �' 0.000010
' ответ' 0.000009
'Answers' 0.000009
' -' 0.000009
'(' 0.000008
'Output' 0.000008
'3' 0.000008
Entropy: 0.05465629696846008

%%%%%%%% Resolving the Model Collapse issue %%%%%%%
I was successfully able to resolve the gibberish output behavior of the nvfp4 model by revising the following hyperparameters. However, the accuracy of the nvfp4 model was really bad as compared to the non-quantization aware model. Maybe, more fine-tuning of hyper-parameters can be done to achieve better results. But the way we stand now, it looks quite bad.

++++ Old Hyperparameters ++++++
OPTIMIZER_ARGS=(
--optimizer adam
--lr 1e-5
--lr-decay-style cosine
--min-lr 1e-6
--lr-warmup-fraction 0.1

++++ New nvfp4 Hyperparameters ++++++
OPTIMIZER_ARGS=(
--lr 2e-6
--lr-decay-style cosine
--min-lr 2e-7
--lr-warmup-fraction 0.2