Does "input_ids“ contain tokens such as [SEG] in your code ?

[epiphany051077]

I noticed when generating input_ids. Prompt does not include tokens such as [SEG]. Will this cause seg_toked_idx to be unable to identify the index of the [SEG] token?

code：
if isinstance(self.seg_token_idx, list):
seg_token_num = self.seg_token_num
seg_token_mask = torch.zeros_like(input_ids[:, 1:]).bool()
for seg_token_idx in self.seg_token_idx:
seg_token_mask = seg_token_mask | (input_ids[:, 1:] == seg_token_idx)
else:
seg_token_num = self.seg_token_num
seg_token_mask = input_ids[:, 1:] == self.seg_token_idx

[epiphany051077]

I would greatly appreciate your response!

[zhixuanli]

I also found this problem. And I believe this may be the reason of this problem: #20

[zhixuanli]

This problem still exists. BTW, if seg_token_num is 3, according to train_ds.py, the seg tokens will be [SEG0], [SEG1] and [SEG2]. However, these three seg tokens have the same seg_token_idx after using the code args.seg_token_idx = [tokenizer(token, add_special_tokens=False).input_ids[0] for token in new_tokens]. This makes the training impossible.

[zhixuanli]

Hope the authors could release the final correct version. Thanks! @MaverickRen

[jifeng35]

After reading the source code, I find there is almost nothing different with LISA's design about the <seg> token, except the number of the <seg> tokens is different.
So basically, this research's diff from LISA is just in the part of the mask decoder(or let's say this work is working without SAM).
Am I right?

[audreyeternal]

@zhixuanli @epiphany051077 I think your assumption of the 'input_ids' is incorrect. Actually during the training, the input_ids also contains the text_answers's information: collate_fn.

[audreyeternal]

For those who are still bothered with the 0 pred_masks (e.g. issue #20) , actually the solution is quite simple. The reason why sometimes the seg_token_mask is all False (thus leads to 0 pred_masks) is because the author still uses the default model_max_length config (512) set in the LISA's project. However, the MUSE's text_answers are much longer than those in the LISA's ReasonSeg dataset (Sure, it is [SEG], etc.). When the final tokens are longer than model_max_length, the remaining tokens will be truncated. Unfortunately, it is often the case that all the [SEG] tokens are at last.

So, just need to increase the model_max_length to 2048 and the error will gone. (definitely there are some other small bugs in the code...)