Teleoperation Device

[kylehuo8]

Thanks for sharing the nice work. I have succeed in control a single arm in Mujoco with teleoperation. However, when I try to teleoperate dual arms (dual UR5, Aloha), Hsr, and G1 with keyboard, if raise the error. It seems like that these kinds of teleoperation only support space mouse. Can you update the code to support keyboard? By the way, are two space mouses must be connected to control G1 and dual arms? Thanks.

[mmurooka]

Thank you for trying our code!

are two space mouses must be connected to control G1 and dual arms?

Yes, a single SpaceMouse does not cause an error, but both arms move in the same way and no meaningful movement is possible.

It seems like that these kinds of teleoperation only support space mouse. Can you update the code to support keyboard?

This was not on our current to-do list, but will be considered in the future.

[kylehuo8]

Thanks for the reply. I am curious about the action implementation during the teleoperation and the rollout.

During teleoperation, I guess you are using mocap to move the end-effector of the arm, right? And then, you will collect the position of each joint in the dataset.

During rollout, the policy will output the joint position given the observation. However, I notice that the arm is controlled by:

self.data.ctrl[:] = ctrl
mujoco.mj_step(self.model, self.data, nstep=n_frames)

I think this mode controls the torque of each joint. How can you ensure the position of the joint achieve the desired value of the policy output? Please correct me if I am wrong since I am new to MuJoCo, Thanks.

[mmurooka]

During teleoperation, I guess you are using mocap to move the end-effector of the arm, right? And then, you will collect the position of each joint in the dataset.

Yes, we use a 3D mouse to command the end-effector pose and calculate the joint angles using inverse kinematics.

I think this mode controls the torque of each joint. How can you ensure the position of the joint achieve the desired value of the policy output?

In our MuJoCo settings, joint angles are commanded instead of joint torques. According to the following references and implementation, I believe that is the case.

https://mujoco.readthedocs.io/en/3.1.6/XMLreference.html#actuator-general

RoboManipBaselines/robo_manip_baselines/envs/assets/mujoco/robots/ur5e/ur5e_integrated_shared_config.xml

Lines 6 to 7 in 01a1350

	<general gaintype="fixed" biastype="affine" ctrlrange="-6.2831 6.2831" gainprm="2000" biasprm="0 -2000 -400"
	forcerange="-150 150"/>