DexTeleop-0 Read paper

Robotics research / arXiv · 2026

DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy

Haichao Liu1, Yuyao Jiang1, Hyunsun Park2, Yuanjiang Xue1, Mingze Sun1, Ziwei Liu1, Ziwei Wang1,†

1 Nanyang Technological University, Singapore

2 OOJU, USA · † Corresponding author

Paper arXiv

Robot-side fingertip feedback refines retargeted hand motion while the operator supplies the manipulation strategy.

DexTeleop-0 in action. Simulation replay, hardware manipulation, and contact assistance.Open video ↗

01 / Overview

Preserve human intent. Refine the contact.

Egocentric tracking makes dexterous teleoperation accessible, but matching human motion does not directly regulate robot contact forces. Differences in hand geometry and tracking accuracy can leave contact errors that are difficult to correct through visual feedback alone.

DexTeleop-0 adds a local feedback loop around the operator’s retargeted hand commands. Estimated contact positions and fingertip forces inform small joint corrections during contact, while arm motion remains nominal.

Shared autonomy here means autonomous robot-side contact assistance. The operator receives visual feedback; the interface does not reflect forces back to the human.

  1. 01

    Contact-aware motion correction

    A local stiffness–Jacobian model predicts how hand-joint changes affect individual fingertip forces.

  2. 02

    Bounded residual optimization

    Per-contact vector force tracking and soft moment regularization refine each hand’s target independently.

  3. 03

    Complementary evaluation protocols

    Controlled simulation, fixed-input hardware replay, a seven-operator study, and manual-disturbance traces examine different aspects of assistance.

91%Ball Assembly · final-stage successSimulation · 100 trials per method62%Stir in Cup · final-stage successSimulation · lift and insertion scoring7Operators in the live user study5 novice · 2 experienced

02 / Method

A contact-feedback loop around teleoperation.

The human supplies nominal motion through a Meta Quest 3 headset. The controller changes hand targets during contact, with a separate optimization for each hand.

Architecture linking egocentric tracking, arm inverse kinematics, hand retargeting, and fingertip-feedback residual control.
Arm commands follow wrist tracking; bounded residuals modify hand targets. The two hands are optimized independently. Full-resolution figure ↗
  1. 01

    Track & retarget

    Wrist poses drive arm IK. Vector-based retargeting maps tracked hand keypoints to nominal robot-hand targets.

  2. 02

    Estimate contacts

    A hand-specific adapter supplies valid contact positions, force vectors, and confidence in the hand-base frame.

  3. 03

    Optimize residuals

    A bounded quadratic program tracks per-contact forces and penalizes predicted contact moment outside a tolerance.

  4. 04

    Execute at 30 Hz

    Add the residual to the nominal hand target, respect joint limits, and repeat with fresh fingertip feedback.

Hand commandqcommand = qnominal + Δq

Residuals apply to hand joints. Arm commands remain nominal.

Local contact model

Track individual forces, then coordinate their moments.

The kinematic Jacobian predicts contact displacement from a joint residual. A local stiffness approximation maps that displacement to a predicted force change. Separate vector errors retain each contact’s load: opposing forces can cancel in the resultant while individual forces remain large.

The moment term is a soft regularizer about the current contact centroid. It does not impose hard equilibrium or model every external load. The formulation uses a moment tolerance of 10 N·m (10 Nest·m for Wuji), with at least two valid contacts.

Conceptual comparison of fingertip posture and contact forces before and after contact correction.
Posture changes alter contact forces and their combined moment. Full-resolution figure ↗
Direct feedback

Sharpa Wave 22 DoF per hand

Tactile contact locations, finger geometry, and force-vector channels supply the contact state.

Estimated feedback

Wuji Hand 2 20 joints per hand

Joint position, velocity, and current inform a quasi-static, normal-force estimate. Its empirical Nest values are distinct from calibrated force measurements.

03 / Demonstrations

From recorded commands to live assistance.

The protocols separate repeatable nominal inputs from operator-guided interaction on physical hardware.

Watch the main video ↑

Vary physical parameters around recorded inputs.

Single-arm Ball Assembly and bimanual Stir in Cup reuse nominal commands across randomized object mass and friction. Stir evaluation scores lift and insertion events.

100trials per method
for each simulation task

Ball Assembly and Stir in Cup simulation sequences, with four snapshots in each row.
Simulation replay tests contact correction under mass and friction variation. Full-resolution figure ↗
Physical dual UR7e arms and Sharpa Wave hands alongside the IsaacSim counterpart, both driven by Meta Quest 3.
The Sharpa platform pairs two 6-DoF UR7e arms with two 22-DoF hands; IsaacSim provides the corresponding simulation setup.

Compare feedback corrections under shared inputs.

Within each hand and task, methods receive identical recorded nominal commands while initial object placement varies. Recordings and sensing adapters are specific to each hand configuration.

2hand configurations
Sharpa Wave & Wuji Hand 2

Wuji Hand 2 executing prism placement, orange placement, and bimanual pouring.
Wuji Hand 2: prism placement, food placement, and pouring. Full-resolution figure ↗
Sharpa Wave executing prism placement, orange placement, and bimanual pouring.
Sharpa Wave: the same task families with direct fingertip feedback. Full-resolution figure ↗

These trials examine two implementations with hand-specific sensing and kinematics. They do not establish transfer without embodiment adaptation.

Assist single-arm assembly and bimanual tasks.

Five novices and two experienced operators perform Gear Mesh, Peg Insertion, Food Sorting, and Tube Operation. Each makes five attempts per task and method after a basic functional introduction, without task-operation training.

7operators
method order drawn by lot

Live teleoperation sequences for Gear Mesh, Peg Insertion, Food Sorting, and Tube Operation, with enlarged contact regions.
Four user-study tasks cover precise assembly, object handling, and multi-step bimanual coordination. Full-resolution figure ↗

04 / Experimental findings

Benefits depend on the task and feedback.

Selected results across complementary evaluation protocols. Simulation replay, hardware replay, and live teleoperation have different conditions and denominators.

Simulation / Comparison & ablation

Separate contact tracking from moment regularization.

No Residual follows retargeted positions. Linear adjusts fingers independently. Force Tracking retains the full controller’s force objective, residual regularization, and bounds, with the moment weight set to zero.

100 trials per method and task. Stir stages assess lift and insertion; these rates do not measure stirring quality.
MethodBall Assembly · final stageStir in Cup · final stage
No Residual11%49%
Linear1%44%
DexForce38%3%
Force Tracking88%49%
DexTeleop-091%62%

Moment-term ablation. DexTeleop-0 reaches 91% versus 88% for Force Tracking on Ball Assembly, and 62% versus 49% on Stir in Cup. The incremental benefit differs across tasks.

DexForce adaptation. Force-informed targets come from one successful demonstration per task and are tracked with Cartesian impedance. Its demonstration and initialization conditions differ.

Force magnitude alone does not characterize completion: DexForce’s lower mean thumb forces coincide with lower success and contact coverage. Simulation force statistics summarize trial-average positive-force magnitudes, including failed trials.

Hardware / Identical nominal commands

Contact assistance across two hand configurations.

DexTeleop-0 has the highest displayed success in all six hand–task conditions, with a smaller margin on Sharpa pouring. Initial object-placement variation differs from a force disturbance applied during grasping.

Successful attempts out of ten per method, hand, and task. Recordings are shared within each condition and are hand-specific across platforms.
Hand / taskNo ResidualDexForceDexTeleop-0
Wuji Hand 2
Food placement4/103/108/10
Prism placement4/105/109/10
Bimanual pouring0/101/104/10
Sharpa Wave
Food placement3/104/109/10
Prism placement5/104/108/10
Bimanual pouring4/105/106/10
User study / 5 novice & 2 experienced operators

Assistance varies across tasks and experience levels.

Each method has 25 novice attempts and 10 experienced attempts per task. Final-stage rates improve in several tasks; novice Peg Insertion favors Force Tracking.

Final-stage success rates from the user-study tables. Each stage is scored over all group attempts; success at the final stage is distinct from completing every stage.
TaskNo ResidualDexForceForce TrackingDexTeleop-0
Novice operators · 25 attempts per method and task
Gear Mesh12%20%28%44%
Peg Insertion28%24%64%52%
Food Sorting48%36%48%68%
Tube Operation28%20%52%76%
Experienced operators · 10 attempts per method and task
Gear Mesh10%80%80%90%
Peg Insertion20%60%60%80%
Food Sorting50%90%80%90%
Tube Operation50%40%70%80%

Peg Insertion exception. Novice final-stage success is 64% for Force Tracking and 52% for DexTeleop-0.

Stage versus full task. Novice Tube Operation reaches 76% at the final stage, while 72% of attempts complete every stage.

User-study force statistics include complete successful and failed records. Progression and contact duration differ, so contact conditions are not strictly matched and force differences do not establish safety or reduced object damage.

Completion over time

Read completion and execution time together.

The curves include failed attempts in their denominators. DexTeleop-0 leads at later displayed times in Gear Mesh, Food Sorting, and Tube Operation; Force Tracking leads in Peg Insertion.

Recorded completion rate versus elapsed time for Peg Insertion, Gear Mesh, Tube Operation, and Food Sorting.
Fraction of recorded attempts completed by each time, with failed attempts retained in the denominator. Full-resolution figure ↗
Inspect contact-force and manual-disturbance analyses

A small resultant can hide large individual loads.

These traces share recorded nominal commands and elapsed time in a Ball Assembly replay. Opposing contact forces can cancel in the resultant; separate per-contact tracking retains the individual errors.

Magnitude of the sum of fingertip force vectors across five methods in one Ball Assembly replay.
Contact-force resultant magnitude. Full-resolution figure ↗
Individual thumb, index, and middle contact-force magnitudes for the same replay and methods.
Individual fingertip forces on the same elapsed-time interval. Full-resolution figure ↗

Manual disturbance provides a descriptive contact test.

In the illustrated watermelon-model grasp, No Residual and DexForce contacts decay near zero at 12–14 s; DexTeleop-0 retains thumb and middle contact until about 21 s. Manual disturbance amplitude and timing vary, so these traces do not quantify calibrated disturbance rejection.

Finger-force and joint-residual traces during manual disturbance of a watermelon-model grasp for No Residual, DexForce, and DexTeleop-0.
Common manual-disturbance procedure on the fixed-arm Sharpa configuration. Full-resolution figure ↗

Citation

DexTeleop-0

Download .bib

Haichao Liu, Yuyao Jiang, Hyunsun Park, Yuanjiang Xue, Mingze Sun, Ziwei Liu, and Ziwei Wang. DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy. arXiv:2606.23431, 2026. doi:10.48550/arXiv.2606.23431

@misc{liu2026dexteleop0,
  title={DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy},
  author={Haichao Liu and Yuyao Jiang and Hyunsun Park and Yuanjiang Xue and Mingze Sun and Ziwei Liu and Ziwei Wang},
  year={2026},
  eprint={2606.23431},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  doi={10.48550/arXiv.2606.23431},
  url={https://arxiv.org/abs/2606.23431}
}