Skeleton-Aware Hand-Arm Retargeting for Contact-Rich Dexterous Teleoperation

Dexterous manipulation is essential for robots performing complex, real-world tasks that require coordinated arm-hand motions and contact-rich interactions. Consequently, high-fidelity teleoperation is critical for both real-time robot control and acquiring human demonstrations for imitation learning. However, existing dexterous teleoperation systems often struggle to simultaneously achieve stable arm trajectories and natural, high-Degree-of-Freedom (DoF) hand retargeting due to the inherent embodiment gap and hardware constraints. This paper presents a practical, low-cost, yet highly accurate arm-hand teleoperation framework, validated on a setup comprising a UR5e robotic arm and a 22-DoF Sharpa Wave dexterous hand. The framework retargets vision-based human hand motion through an inverse-kinematics-based formulation derived from the relative skeleton relationships between human hand landmarks and robotic hand links. Concurrently, operator wrist motion is mapped to the robotic arm via a relative pose self-calibration and tool-frame compensation mechanism. Empirical evaluations on real-world manipulation tasks demonstrate that the proposed method achieves superior grasp stability and more reliable task execution compared to traditional direct joint-angle mapping and fingertip-only IK retargeting baselines. This framework provides a robust and accessible teleoperation solution for high-DoF dexterous manipulation.

Recommended citation: K. Li, Y. Jiang, M. Zhao, Q. Wang, and H. Liu, "Skeleton-Aware Hand-Arm Retargeting for Contact-Rich Dexterous Teleoperation," in Proceedings of the 2026 International Conference on Embodied Intelligence and Robotics (EIR 2026), 2026, pp. 1-9.