[BUG] can't place models close to memory limits

[leanzero-srl]

Thisi s really a question possibly a bug, i am not sure however when I load something like a Step 3.5 flash for eg. even from the beginning, without any messages being sent, i get:

60gb occupied on M3 ultra studio with 96gb
64gb occupied on M4 max 128gb

In total that is clearly 124gb. The model I am using is the 4bit which only has 107gb. I have tried with lower context or higher and even at the start the size is always around 120gb.

After vibe coding for a while, around 30-40 messages in and around 90k context it reaches 150-160gb, which i do find kind of extreme. From what I've messed around withi n Inferencer for eg. Step 3.5 never goes above 125gb in full load at around 200k context.

Any idea? is this known?

[leanzero-srl]

I tried step 3.5 flash 6bit, a 148gb model. It doesn't even load on the 2 devices. this is a bit extreme if you ask me.

if you look in the image theoretically I would have at least 76gb on my macbook and at least 80gb on my studio (doesn't have that many apps on it). So all in all it should hold up an easy 156gb.

[Evanev7]

ah - we're pretty hesitant to force out other memory at the moment after encountering some ooms. i think we need some manual overrides of "just let me try anyway". you won't get much context with only on 8gb of kv cache though!

[leanzero-srl]

hmmm still I have to press on with this question because something is not making sense. Why would a 107gb model consume north of 150gb? Why can't I even fit a 120gb model on these 2 devices?
LE: on LM studio I can fit and work with 82gb model on 128gb machine alone with no problems at up to 128k context.
I am not trying to be aggressive here, but I am trying to figure out whether my baseline is incorrect and RDMA is supposed to consume much more memory than normal.

[Evanev7]

sorry is the model 107gb? step-3.5-flash-6bit's safetensors model.safetensors.index.json
suggests it's ~160gb

[leanzero-srl]

Ok, so the 107 model, the 4bit consumes an easy 150gb memory. In full load at 128k it consumes even more, ranging to about 160-165....
I can't even load the 6bit version.
And after a few messages for eg. minimax-m2.5-4bit which is a 120gb model fails to continue and unloads or crashes the entire cluster. I do think that is might be slightly buggy.

[Evanev7]

ah the 4bit. at full load it will definitely consume more (kv cache isn't free!) but 150gb for the model alone is too high. will look into it.

[leanzero-srl]

I tried 1.0.69 with qwen3.5-27b which is 29gb. Running in a cluster, memory consumption rose to over 150gb. This is incredible. It's clearly a memory leak of epic proportions.

And it also crashed at the end ...