gemma-3

[terrykong]

To support gemma-3 it looks like we need a few changes:

• bump transformers to latest (feat: streaming each dtensor in refit #176 bumps from 4.49.0 -> 4.51.3, which should be enough)
• use eager attention / make it configurable (related)

(FSDP1PolicyWorker[rank=0] pid=1810886) It is strongly recommended to train Gemma3 models with the eager attention implementation instead of sdpa. Use eager with AutoModelForCausalLM.from_pretrained('<path-to-checkpoint>', attn_implementation='eager').

[terrykong]

The model looks like it doesn't generate properly. Here is the current training behavior + the first generation:

generated_texts[0]:  Closer Closer Closer Closer Closer Alps Closer Closerδία Agencies Closer帮 Alps Closer Alpsδία Alps Alps Closer Closerríklad अमृताjQuery Closer Alps Alps Closer Alps Alps Freeway Closer Closer Closer Alps FreewayjQueryjQuery Alps ситуация Closer Plugin खुशखबरीTruthy Closer Closer elixir cavalry Closer Closer Alps Closerδίαδία अपराTruthy CloserTruthy Alps rétréci Closer Closer Closer Closer Closer Closer Alpsδία AlpsThetaδία situasiTruthy CloserGPIO֥ kucchDebuggingjQueryforminδία元件 Closer Closer lousy Closerformin Closerшкиríklad अपरा Passport éléments Closer সালের Closer Closer individuijQuery Closerδία Closer individui Closerayin Closererol吲 Direction變化 ahuv Alpsδίαδία吲δία individuiδία backdoorδίαδίαłkaδίαδίαδίαδία吲ంటీ Ambroseଣ୍jQueryδίαDrew அற்புதTruthyphusTruthy Closer ficheroంటీ yeh kuchh吲ríklad ماسکوδία吲ంటీ verhaalδία goatee verhaal Closer डॉलरTruthy yehంటీshovelScenario吲 здравоохраненияδία仆formin douaactoryδίαScenario吲edayTruthy吲formin Almightyంటీríkladంటీ Closer$-$,jQuery positifVIDENCEంటీంటీ individui yehformin Traction Middlewareformin ماشيంటీ উপাদান पादुంటీTruthy吲ంటీంటీ mismo Closerంటీ yeh吲ంటీ Production অ্যাকাউন্ট jQueryProductionδία mesmaδίαformin譬ంటీFTR Freeway goateeంటీ CloserంటీjQuery goatee goatee yehంటీ ماشي yehScenario吲ISHforminంటీ ماشيদৃশ্যBedroomшки individui吲吲 Closerంటీ吲 mesmas吲 productionforminంటీంటీ `.farandejQuery Sandyforminంటీ ماشيంటీ goateeδία جریانంటీ mesmas individui vergleichంటీOfStringంటీ `. mesmajQueryríklad Horseshoe mesmofarande吲ంటీంటీfarande Almighty டீ adware yehδίαంటీ estradiol goatee andar yeh constituencyంటీ yra쩔 yeh டீ goateeforminScenarioంటీ mesmas verhaal仆ంటీ Funny தயంటీ ماشي Closer cierformin HezbollahScenario יד comps BlackBerry यूरोδία Closer吲吲Scenario ماشي constituency கிஷோர் yehformin mesma吲ంటీ fateful{- yeh ferramentas吲 mesmasforminంటీScenario Hezbollah吲吲 zaken吲吲 mesmas Almightyంటీ yeh yeh yeh yeh떻 yehfarande கூட பிரபு ماشي吲 mesmas吲 goatee仆 yeh பிரபு па仆ய் individui吲 goatee吲 IGF mesmasfarande吲吲吲Scenarioంటీ犇吲ଣ୍kosť

I tried VLLM_USE_V1=0 from here but that doesn't seem to help. attn_implementation=eager also doesn't help.

[terrykong]

generating via vllm serve seems to generate something sensible, and the weights refit seem to match on the very first step. probably points to something in our integration

[yfw]

Another data point is that I notice the same thing when trying to load DeepSeek-V2-Lite. In reinforcer, vllm doesn't generate properly, but separately manually running vllm with the same prompt generates reasonable responses.

[yfw]

DeepSeek-V2-Lite issue is a separate issue tracked in #275