About SEG token

[2kxx]

Thanks for your good work. Using two losses, the embedding of the SEG token will change, which may cause the logits to not necessarily increase toward generating the SEG token. How do you solve this problem?

[MaverickRen]

I sincerely apologize for overlooking your issue. I believe this is a common problem in works like LISA/PixelLM, as they primarily rely on carefully designed data formats to provide downstream task capabilities, which to some extent inevitably introduces this bias. In the context of 2025, I think approaches that integrate Chain-of-Thought (CoT) and Reinforcement Learning (RL) may better mitigate this bias.

[zhitongcui]

I sincerely apologize for overlooking your issue. I believe this is a common problem in works like LISA/PixelLM, as they primarily rely on carefully designed data formats to provide downstream task capabilities, which to some extent inevitably introduces this bias. In the context of 2025, I think approaches that integrate Chain-of-Thought (CoT) and Reinforcement Learning (RL) may better mitigate this bias.

Do you have recommended papers that have tried to mitigate this bias?

[MaverickRen]

Perhaps you could take a look at recent work in "thinking with images," for example, TreeVGR and PixelReasoner.

[zhitongcui]

Thanks so much for your quick response!
I still have one more question. Do you think CoT or RL based methods would increase the inference speed compared to LISA/PixelLM?

Btw, do you think papers "Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement" and "Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learning" are practices that might possible alliviate such bias?

[MaverickRen]

In my view, the primary role of RL at present is to further elevate the performance of models post-SFT. A well-designed reward function can effectively regulate the model's outputs, for example, to reduce this kind of bias

[zhitongcui]

I am not sure if my understanding is correct, Do you think the methods related to Lisa will inevitably reduce the chat capabilities of MLLMs? When I tested PixelLM 7B, I found that when the model outputs segmentation results, the amount of textual content it generates becomes very limited.