远程推理中机器人端的client如何从机器人的摄像头获取视频流作为输入？

[zbzyjya]

感谢你们在该项目的经常贡献与开源。我用的是智元的轮式双臂机器人，目前已经实现了模型下载和远程推理的简单demo测试，但是我注意到client中的payload中需要传入的是图片，

测试中我给payload输入的是随机数作为三个相机的输入
payload = {
"top": np.random.randint(0, 255, (224, 224, 3), dtype=np.uint8),
"right": np.random.randint(0, 255, (224, 224, 3), dtype=np.uint8),
"left": np.random.randint(0, 255, (224, 224, 3), dtype=np.uint8),
"instruction": "pick the box",
"state": np.random.uniform(-3.14, 3.14, (16,)).astype(np.float32),
"ctrl_freqs": np.array([30], dtype=np.float32)
}
那么我将如何改动才能将轮式双臂机器人的视频流输入到client中呢？请问你们是否有实现相关的组件可供我参考？万分感谢

[ModiShi]

The implementation depends on your camera hardware. You can typically find usage instructions in the camera's SDK documentation. For example, if you're using RealSense cameras, please refer to: https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/opencv_viewer_example.py

[zbzyjya]

我使用的机器型号是智元精灵G01，使用的输入相机编号分别是head hand_left hand_right ,输出动作维度为16维度（七个关节角+夹爪+七个关节角+夹爪），从硬平台使用和输入输出来看，和你们原始模型训练的是完全一致的。我希望能在智元精灵G01端实现这三个相机输入payload，通过client到server获取返回的16维度action，再通过关节角下发给机械臂执行动作。请问是否有基于智元精灵G01已经实现这一套流程的软件可供我们参考？

[zbzyjya]

The implementation depends on your camera hardware. You can typically find usage instructions in the camera's SDK documentation. For example, if you're using RealSense cameras, please refer to: https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/opencv_viewer_example.py

我是否可以参考这个？https://github.com/AgibotTech/genie_sim/blob/main/model/demo_infer.py 望回复

[JinChen0056]

https://github.com/OpenDriveLab/AgiBot-World/blob/manipulation-challenge/UniVLA/scripts/infer_real.py
Maybe you can refer to this.

[zbzyjya]

https://github.com/OpenDriveLab/AgiBot-World/blob/manipulation-challenge/UniVLA/scripts/infer_real.py Maybe you can refer to this.

Thank you for your reply. This is extremely important to me as an independent developer. Thank you again.