sam2 real time inference?

[Karmik25]

can someone explain to me how to use sam2 for real-time inference, such that the model can predict objects while my webcam is open.

[heyoeyo]

The existing code doesn't support live/streaming video. Instead you'd have to first record a video on webcam and then process that video using the existing video predictor and/or web demo. It is possible to handle streaming video by modifying the code. It's mostly a matter of re-implementing the existing frame loop outside of the sam2_video_predictor.py script, and handling the frame reading (e.g. from a webcam) in that new loop.

If you don't want to implement this manually, there is an existing repo that has done this:
https://github.com/Gy920/segment-anything-2-real-time
I also have a script that works with webcams (though it's worth noting the repo above has code much more similar to the original, in case that's important):
https://github.com/heyoeyo/muggled_sam?tab=readme-ov-file#run-video-or-webcam

[277845265]

https://www.bilibili.com/video/BV1TNkEYrEnb/?spm_id_from=333.33.top_right_bar_window_dynamic.content.click&vd_source=1aa85c9fa1298c2b3a033e3bd0932f3a

[asakhare-zdata]

We also adapted the code in video_predictor.py for real-time tracking. This implementation can easily be paired with YOLO for real-time tracking of detected objects. However, the number of objects to track must be specified during initialization.

The code can be found here:
https://github.com/zdata-inc/sam2_realtime

The core tracking logic is implemented in object_tracker.py

Check out the following notebook for a usage example:
https://github.com/zdata-inc/sam2_realtime/blob/main/notebooks/realtime_detect_and_track.ipynb

[Grpab]

The existing code doesn't support live/streaming video. Instead you'd have to first record a video on webcam and then process that video using the existing video predictor and/or web demo. It is possible to handle streaming video by modifying the code. It's mostly a matter of re-implementing the existing frame loop outside of the sam2_video_predictor.py script, and handling the frame reading (e.g. from a webcam) in that new loop.

If you don't want to implement this manually, there is an existing repo that has done this: https://github.com/Gy920/segment-anything-2-real-time I also have a script that works with webcams (though it's worth noting the repo above has code much more similar to the original, in case that's important): https://github.com/heyoeyo/muggled_sam?tab=readme-ov-file#run-video-or-webcam

I would like to ask if you have implemented the real-time samurai algorithm?

[heyoeyo]

I would like to ask if you have implemented the real-time samurai algorithm?

No, though I think samurai is just a post-processing step for picking masks/handling the memory bank. It should be fairly easy to support. I can maybe take a look at it at some point.

[Grpab]

OK, thanks

[heyoeyo]

Here's an example script for running samurai on the mask predictions, which should work in a real-time/webcam setting:
https://github.com/heyoeyo/muggled_sam/blob/main/simple_examples/video_segmentation_samurai.py

You can set the video & model path as well as prompts near the top of the script.
Setting video_path to an integer value (typically 0) will run it on a webcam.

Here's an example of it on the '1917' battlefield test video. The box shows the kalman filter prediction from samurai. It's green if the mask passes the memory acceptance thresholds, red otherwise:

samurai_demo.mp4

[Grpab]

以下是在掩码预测上运行 samurai 的示例脚本，该脚本应在实时/网络摄像头设置中工作： https://github.com/heyoeyo/muggled_sam/blob/main/simple_examples/video_segmentation_samurai.py

您可以在脚本顶部附近设置视频和模型路径以及提示。 设置video_path为整数值（通常为0）将在网络摄像头上运行它。

以下是“1917”战场测试视频中的一个例子。框中显示了武士的卡尔曼滤波器预测。如果掩码通过了内存接受阈值，则为绿色，否则为红色：

武士_演示.mp4

Thank you very much for your reply

[Grpab]

Here's an example script for running samurai on the mask predictions, which should work in a real-time/webcam setting: https://github.com/heyoeyo/muggled_sam/blob/main/simple_examples/video_segmentation_samurai.py

You can set the video & model path as well as prompts near the top of the script. Setting video_path to an integer value (typically 0) will run it on a webcam.

Here's an example of it on the '1917' battlefield test video. The box shows the kalman filter prediction from samurai. It's green if the mask passes the memory acceptance thresholds, red otherwise:

samurai_demo.mp4

Can you help me make changes in the SAMURAI algorithm library? If you can, I would be very grateful!

[heyoeyo]

Can you help me make changes in the SAMURAI algorithm library? If you can, I would be very grateful!

I think the SAMURAI code makes use of the original SAM2 code for running videos. So the same fix mentioned earlier is needed (moving the frame loop outside of the model) to support webcams.

It's probably easiest to reference code from the segment-anything-2-real-time repo, which is very similar to the original SAM2 (and therefore SAMURAI) code. Specifically, you'd want add something like the track function (and the related functions it uses) to the SAMURAI video predictor class. That track function can then be used in a frame loop outside of the model, like the 'Camera Prediction' example code from the real-time repo.

[Grpab]

您能帮我修改一下 SAMURAI 算法库吗？如果可以的话，我将不胜感激！

我认为 SAMURAI 代码利用了原始的 SAM2 代码来播放视频。因此需要进行前面提到的相同修复（将帧循环移出模型）来支持网络摄像头。

引用来自segment-anything-2-real-time repo 的代码可能是最简单的方法，它与原始 SAM2（因此也是 SAMURAI）代码非常相似。具体来说，您需要将类似于跟踪函数（及其使用的相关函数）的内容添加到 SAMURAI视频预测器类。track然后可以在模型外部的帧循环中使用该函数，就像来自实时​​ repo 的“相机预测”示例代码一样。

Thanks for the ideas! However, I don't know Python, so could you help me modify it? I would be very grateful!

[heyoeyo]

so could you help me modify it?

I don't have editing access on the samurai repo, so I can't make modifications directly. It might be best to check through the existing forks of the repo to see if someone has already made a version with real-time support.