Robotics research / arXiv · 2026

UniManip: Open-World Robotic Manipulation via Agentic Operational Graphs

Haichao Liu1, Yuanjiang Xue1,Yuheng Zhou1,Li Yang1,Haoyuan Deng1,Yinan Liang2, Lihua Xie1,Ziwei Wang1

1 School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore

2 Department of Automation, Tsinghua University, China

Paper arXiv

Connecting language-guided task reasoning to physical interaction through persistent scene state, grounded motion, and verified recovery.

UniManip in action. Task execution, verification, and recovery across manipulation settings.Open video ↗

01 / Overview

From task intent to verified action.

Following an instruction requires more than choosing the next action. A robot must identify the right objects, account for geometry, and recognize when an interaction changes the scene or fails.

UniManip uses a Bi-level Agentic Operational Graph (AOG) to retain this context throughout execution. Language-guided planning and an object-centric 3D state representation form a closed loop: perceive, plan, act, verify, and recover.

Here, zero-shot means execution without task-specific demonstrations or fine-tuning in the evaluated settings. The system uses pretrained perception and language models, a tool library, and calibrated embodiment adapters.

  1. 01

    Persistent operation context

    Couple task sequencing with time-indexed object states, tool outputs, and verification history.

  2. 02

    Geometrically grounded execution

    Turn wrist-camera observations into affordances and end-effector trajectories using conservative occupancy and safety-aware planning.

  3. 03

    Recovery within the task loop

    Use observed outcomes to repair an operation locally or replan the remaining sequence.

02 / Method

One graph connects reasoning and execution.

The AOG couples a procedural layer for operations with an episodic layer for the scene. Each verified outcome informs the next operation.

Procedural memory

Agent Logic Graph ALG

Operation order, tool selection, expected outcomes, and recovery conditions.

Episodic memory

Semantic-Operational State Graph SOSG

Object and part geometry, spatial relations, affordances, and verified state history.

Bi-level AOG architecture linking agent operations and recovery decisions to object-centric scene states.
The two graph layers preserve operation context across perception, execution, and reflection.Full-resolution figure ↗
  1. 01

    Perceive

    Ground the instruction in objects, parts, poses, and relations from a wrist-mounted RGB-D camera.

  2. 02

    Plan

    Compose operations and choose tools using the current scene state and task objective.

  3. 03

    Execute

    Synthesize end-effector waypoints and track them through the robot’s motion interface.

  4. 04

    Verify & recover

    Commit verified state changes; diagnose mismatches and repair or replan within a bounded retry budget.

Grounding motion

Plan around what a single view cannot see.

Conservative reconstruction fills occluded space below observed occupied voxels. A Euclidean signed distance field guides clearance-aware trajectory planning, followed by relaxed inverse kinematics when strict pose tracking is infeasible.

The tool interface separates moving to an object from operating on it, such as pulling a drawer or tilting a bottle. Mobile embodiments also expose a navigation primitive.

Three-dimensional motion-planning example showing a trajectory through a reconstructed scene.
Geometry constrains the trajectory before physical execution. View figure ↗
Recovery case studies: regrasping after an object slips, and correcting a distractor selection after checking object attributes.
Observed failures trigger a regrasp or a correction of the selected target. View cases ↗

Closing the loop

Verify the outcome before moving on.

After an operation, the verifier checks the observed state against its expected outcome. A success commits the scene update. A failure records a diagnostic label and marks affected graph elements as stale.

The agent retrieves this context to adjust the current operation or restructure the remaining plan. Recovery is bounded; persistent perception errors and unreachable targets can still prevent completion.

See the recovery ablation ↗

Compose interactions over multiple steps.

A Galaxea A1 arm performs rigid, deformable, and articulated-object tasks, including sorting, insertion, pouring, breakfast preparation, button pressing, and drawer operation.

90trials per configuration
in the interaction-rich ablation suite

Sequences of real robot demonstrations including trash sorting, insertion, pouring, breakfast preparation, and drawer interaction.
Representative execution sequences on the fixed-base tabletop platform. View sequences ↗

Retain the graph; adapt the robot interface.

On the UR7e, arm transfer uses the robot’s kinematics, joint limits, calibrated tool frame, and controller interface. The Inspire hand additionally uses a multi-fingered grasp generator and hand-specific action adapter.

2end-effector configurations
parallel-jaw gripper & multi-fingered hand

UR7e demonstrations with a Robotiq parallel-jaw gripper and an Inspire multi-fingered hand performing pouring, meal preparation, and drawer placement.
The AOG retains task sequencing and state verification across the two end effectors. View sequences ↗
UR7e hardware evaluation · manuscript Table VIII
End effectorTaskTrialsTask success
Robotiq RF-85Pour water2065%
Robotiq RF-85Prepare meal2080%
Inspire handOpen drawer & place block1553%

Reposition, observe, and manipulate.

A Realman RM65 arm on a differential-drive base performs office meal preparation and cleanup from randomized docking configurations. A mapless visual-servo navigation primitive aligns the base before the shared manipulation tools execute.

79%end-to-end task success
80 object episodes · 20 per target

Mobile robot sequences for meal preparation and office cleanup, combining navigation, docking, grasping, and placement.
Navigation and manipulation are coordinated through the target object’s scene-graph state. View sequences ↗

Manuscript Table IX reports 85% grasp success and 93% placement success conditioned on successful grasps. Mean execution time is 29.37 ± 6.77 s. Transfer requires a navigation adapter, calibrated hardware interfaces, and tuned docking gains.

04 / Experimental findings

What changes when execution retains context?

Selected results from the manuscript. Rates below are percentages converted from the reported, rounded decimal values; comparisons use the evaluation conditions described here.

Real world / Table I

Objects, instructions, and distractors

16 task settings · 10 randomized trials per setting and method

UniManip94%
NORA-1.571%
NORA63%
π042%

VLA baselines are fine-tuned on 1,000 Galaxea A1 demonstrations over nine tasks and use wrist and external cameras. UniManip uses a wrist camera and no task-specific demonstrations. “Seen” and “unseen” refer to the VLA training object set.

Target selection: across the nine settings with distractors, the reported mean distractor-grasp rate is 8% for UniManip and 16% for NORA-1.5.

Real world / Table III

Cluttered desktop manipulation

8 tasks · 80 randomized trials per method

UniManip83%
MOKA58%
VoxPoser55%
ReKep48%
UniManip w/o Graph40%
RoboEXP39%

The matched no-graph control replaces ALG/SOSG with a linear text plan and transient tool state. It retains perception, motion generation, verification, recovery policy, and retry budget.

Persistent state: the 83% versus 40% comparison tests the contribution of the graph representation with the low-level stack retained.

Ablation / Tables VI–VII

Recovery improves completion; planning reduces contact.

Across the interaction-rich suite, collision-free planning trades a small change in completion rate for fewer edge collisions. Adding failure detection and recovery improves multi-step completion.

90 trials per configuration · all three variants include relaxed inverse kinematics
ConfigurationCollision-free planningFailure detection & recoveryTask success
V1NoNo52%
V2YesNo51%
V3 · full systemYesYes80%

Planning tradeoff. Reported edge collisions decrease from 38.9% without collision-free planning to 5.6% with it.

Recovery overhead. Up to three task-level repair or replanning attempts are allowed; average execution time increases by 14.4% relative to V2.

Simulation / Table V

LIBERO-Pro: geometric and task perturbations.

UniManip leads seven of the eight reported conditions. Goal position perturbation remains an exception: π0.5 reaches 38%, compared with UniManip’s 33%.

Selected comparison · position perturbations test pose generalization; task perturbations test task-level reasoning
SuiteUniManip · positionUniManip · taskπ0.5 · positionπ0.5 · task
Object77%85%17%1%
Goal33%29%38%0%
Spatial32%37%20%1%
Long21%17%8%1%

The paper includes the full comparison with MolmoAct, NORA, X-VLA, OpenVLA, π0, and CaP-Agent0. Simulation and real-world rates describe different evaluations and should be interpreted separately.

Citation

UniManip

Download .bib

Haichao Liu, Yuanjiang Xue, Yuheng Zhou, Li Yang, Haoyuan Deng, Yinan Liang, Lihua Xie, and Ziwei Wang. UniManip: Open-World Robotic Manipulation via Agentic Operational Graphs. arXiv:2602.13086, 2026. doi:10.48550/arXiv.2602.13086

@misc{liu2026unimanip,
  title={UniManip: Open-World Robotic Manipulation via Agentic Operational Graphs},
  author={Haichao Liu and Yuanjiang Xue and Yuheng Zhou and Li Yang and Haoyuan Deng and Yinan Liang and Lihua Xie and Ziwei Wang},
  year={2026},
  eprint={2602.13086},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  doi={10.48550/arXiv.2602.13086},
  url={https://arxiv.org/abs/2602.13086}
}