Question about the training on the expanded vocabulary

[KennyNH]

Thanks for your exciting work.

However, I have a question regarding the potential instability of directly expanding the LLM's decoding space by incorporating visual reference prototypes. Given that the semantics and number of these prototypes are input-dependent, will a unified cross-entropy training objective lead to performance fluctuations?

Was an alternative approach investigated, such as separating the classification of visual patches from the LLM's original vocabulary and optimizing it with a distinct, auxiliary loss function?

[Dyb3438]

@KennyNH Thank you for your insightful opinion. To answer your question, I have broken it down into the following three sub-questions:

1. Does combining VRTs and text embeddings into a unified embedding table affect convergence?

No. I have included the training curves below, showing the trend of change in each sub-loss and the IoU of the predicted boxes and masks. It shows that PaDT converges steadily.

2. How can PaDT place image patch features (VRTs) and text embeddings in the same embedding table without them interfering with each other?

That's a good question. As we observed in the CLIP and SigLIP vision-language models, our prior knowledge showed that, although CLIP could perfectly classify each image with different language prompts, the image and text features showed almost no overlap and were very dissimilar to each other (see our previous paper: ECALP (https://github.com/Yushu-Li/ECALP), as can be seen in the t-SNE plot for dimensionality reduction of features. Therefore, we hypothesise that, in a high-dimensional feature space, features from different modalities are non-overlapping, which is why they can be decoded into the same codebook. Indeed, while we aim to implicitly strengthen the association between text and image patch features within the context of large language models through the PaDT paradigm, this does not imply that features from these two modalities will interfere with one another. For example, the text space for 'apple' only stores semantic information for 'apple', whereas the 'apple' image patches also store patch location, texture information, and so on.

3. What are the advantages of unifying them into a single embedding table?

Instead of using a separate embedding table, combining them into a single table could enable PaDT to predict text and VRT freely, allowing them to be mixed and interleaved into the sentences (whether prompt input or response output). It will support our subsequent work on multi-modal reasoning and generate more multi-modal outputs.

[KennyNH]

Thank you for sharing your valuable insights.

1. I apologize if I missed them, but could you please point me to the training curves you mentioned?

2. Thank you for sharing your previous work. I agree that VRTs and text tokens do not interfere with each other.

3. I would be very interested to hear your perspective on the effectiveness of different training strategies for joint text and visual decoding. Specifically, one approach is to unify them into a single embedding table, as done in PaDT. Another strategy, seen in a similar work like MINT-CoT (NeurIPS'2025)[https://arxiv.org/pdf/2506.05331], applies a separate loss on the visual tokens in the second training stage. Was this separate loss paradigm something you considered?
   As mentioned in my last comment, my intuition is that since the semantics and quantity of these visual prototypes are input-dependent, it might lead to semantic inconsistency in the softmax distributions of the LLM's outputs. In other words, as the last $N^\prime$ parameters of the classifier in Equation (3) have inconsistent meanings and the value of $N^\prime$ can vary across inputs, do you think this could affect model performance? It seems that applying a separate loss, as in MINT-CoT, could potentially avoid this issue.

[Dyb3438]

@KennyNH The training curves are also at here #4.