Can point prompts (including negative points) be used instead of a bounding box?

[Projector3567]

Hi, thanks for the great work on SAMURAI, the tracking works really well with the default bounding box input.

However, in sequences where the target is very small, densely packed with other objects, or becomes heavily occluded, drawing a tight box is awkward and sometimes leads to mask leakage onto neighboring objects. It would be helpful to use point prompts instead, especially with the ability to add negative points to exclude nearby objects.

Is it currently possible to use point prompts (positive and negative) with SAMURAI?

Thanks.

[wngkj]

This can be achieved through the add_new_points_or_box function in sam_video_predictor, you can refer to my repo

[Projector3567]

This can be achieved through the add_new_points_or_box function in sam_video_predictor, you can refer to my repo

Thanks for your reply. I’m a bit confused: the authors of SAMURAI clearly state that it only supports single-object tracking and that multi-object tracking isn’t possible at this time. My concern is that when you provide multiple points, it automatically falls back to standard SAM2 tracking. Could that be the case?

[wngkj]

I think it should support multi-target tracking. Although the internal method of kalman_filter.py of samurai is designed for a single target, the original sam2 supports multi-target tracking at the bottom level. It will instantiate a tracking state for each new object. When a new target starts to be tracked, the _forward_sam_heads method will be called. Inside this method, the initiate method of kalman_filter.py will be called to create the initial self.kf_mean and self.kf_covariance for a new high_res_bbox. Then the tracking state of each object is updated in a loop. This means that the motion state of each object is predicted and updated separately. It's just that samurai has not implemented this function in the high-level tracking logic.