Haichao Liu

Haichao Liu

Postdoc at NTU, Singapore. Research interests include robot motion planning, autonomous driving, embodied AI, etc.

Publications

You can also find my articles on my Google Scholar profile.

All Papers

DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy

Published in arXiv, 2026

Tags: robotic manipulation, motion planning and control

Fine-grained, bimanual dexterous manipulation remains a foundational challenge in robotics. Traditional teleoperation systems often fail in contact-rich tasks because embodiment gaps hinder accurate kinematic mapping, while tactile and force feedback remain absent. Consequently, data collection efficiency for highprecision tasks remains prohibitively low. To address these limitations, we propose a tactile-driven adaptation strategy designed to enable fine-grained manipulation on top of teleoperation pipelines. Instantiated within our bimanual dexterous framework, DexTeleop-0, this strategy introduces a real-time optimization loop that bridges the embodiment gap by translating coarse human tracking intents into precise, force-compliant robotic commands with tactile sensing. By estimating accurate contact points and leveraging a tactile-enabled fingertip force-sensing profile, the system dynamically computes localized corrections using the operational space Jacobian with respect to joint angle updates. We rigorously evaluate this tactile-driven adaptation strategy across both simulated environments and real-world hardware. Compared with representative baselines, the proposed method consistently achieves higher task success rates and improved execution efficiency in robust grasping, disturbance-resilient manipulation, and complex dexterous tasks. Our open-source project page can be found at https://henryhcliu.github.io/dexteleop-0.

Recommended citation: H. Liu, Y. Jiang, H. Park, Y. Xue, Z. Wang, "DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy," arXiv preprint arXiv:2606.23431, 2026. https://arxiv.org/pdf/2606.23431

Attention Distribution via Vision Language Models for Interactive Decision-Making and Motion Planning in Autonomous Driving

Published in 2026 IEEE 29th International Conference on Intelligent Transportation Systems (ITSC), 2026

Tags: autonomous driving

TBA

Recommended citation: B. Ma, H. Liu, and J. Ma, "Attention Distribution via Vision Language Models for Interactive Decision-Making and Motion Planning in Autonomous Driving," in Proceedings of the 2026 IEEE 29th International Conference on Intelligent Transportation Systems (ITSC), 2026, pp. 1-8.

An Intention-Driven Lane Change Framework Considering Heterogeneous Dynamic Cooperation in Mixed-Traffic Environments

Published in IEEE Transactions on Intelligent Transportation Systems, 2026

Tags: autonomous driving, motion planning and control

In mixed-traffic environments, autonomous vehicles (AVs) must interact with heterogeneous human-driven vehicles (HVs) whose intentions and driving styles vary across individuals and scenarios. Such variability introduces uncertainty into lane change interactions, where safety and efficiency critically depend on accurately anticipating surrounding drivers’ cooperative responses. Existing methods often oversimplify these interactions by assuming uniform or fixed behavioral patterns. To address this limitation, we propose an intention-driven lane change framework that integrates driving-style recognition with cooperationaware decision-making and motion-planning. A deep learningbased classifier identifies distinct human driving styles in real time. We then introduce a dual-perspective cooperation score composed of intrinsic style-dependent tendencies and interactive dynamic components, enabling interpretable and adaptive intention prediction and quantitative inference. A decision-making module combines behavior cloning (BC) and inverse reinforcement learning (IRL) to determine lane change feasibility. Later, a coordinated motion-planning architecture integrating IRLbased intention inference with model predictive control (MPC) is established to generate collision-free and socially compliant trajectories. Experiments on the NGSIM dataset show that the proposed decision-making model outperforms representative rule-based and learning-based baselines, achieving 96.98% accuracy in lane change classification. Motion-planning evaluations further demonstrate improved maneuver success and execution stability in mixed-traffic environments. These results validate the effectiveness of structured cooperation modeling for intentiondriven autonomous lane changes.

Recommended citation: X. Qiu, H. Liu, Y. Pan, J. Ma, and X. Zheng, "An Intention-Driven Lane Change Framework Considering Heterogeneous Dynamic Cooperation in Mixed-Traffic Environments," IEEE Transactions on Intelligent Transportation Systems, vol. 0, no. 0, pp. 1-13, 2026. https://arxiv.org/pdf/2509.22550

RoCo Challenge at AAAI 2026: Benchmarking Robotic Collaborative Manipulation for Assembly Towards Industrial Automation

Published in arXiv, 2026

Tags: robotic manipulation, motion planning and control

Embodied Artificial Intelligence (EAI) is rapidly developing, gradually subverting previous autonomous systems’ paradigms from isolated perception to integrated, continuous action. This transition is highly significant for industrial robotic manipulation, promising to free human workers from repetitive, dangerous daily labor. To benchmark and advance this capability, we introduce the Robotic Collaborative Assembly Assistance (RoCo) Challenge with a dataset towards simulation and real-world assembly manipulation. Set against the backdrop of human-centered manufacturing, this challenge focuses on a high-precision planetary gearbox assembly task, a demanding yet highly representative operation in modern industry. Built upon a self-developed data collection, training, and evaluation system in Isaac Sim, and utilizing a dual-arm robot for real-world deployment, the challenge operates in two phases. The Simulation Round defines fine-grained task phases for step-wise scoring to handle the long-horizon nature of the assembly. The Real-World Round mirrors this evaluation with physical gearbox components and high-quality teleoperated datasets. The core tasks require assembling an epicyclic gearbox from scratch, including mounting three planet gears, a sun gear, and a ring gear. Attracting over 60 teams and 170+ participants from more than 10 countries, the challenge yielded highly effective solutions, most notably ARC-VLA and RoboCola. Results demonstrate that a dual-model framework for long-horizon multi-task learning is highly effective, and the strategic utilization of recovery-from-failure curriculum data is a critical insight for successful deployment. This report outlines the competition setup, evaluation approach, key findings, and future directions for industrial EAI. Our dataset, CAD files, code, and evaluation results can be found at: https://rocochallenge.github.io/RoCo2026/.

Recommended citation: H. Liu, Y. Zhou, Z. Wu, Z. Ji, Z. Shan, Q. Wang, R. Liu, Z. Yang, Y. Gu, S. Khan, S. Yan, J. Liu, H. Zhu, C. Liu, J. Yang, J. Zhang, Z. Wang, "RoCo Challenge at AAAI 2026: Benchmarking Robotic Collaborative Manipulation for Assembly Towards Industrial Automation," arXiv preprint arXiv:2603.15469, 2026. https://arxiv.org/pdf/2603.15469

MoE-ACT: Scaling Multi-Task Bimanual Manipulation with Sparse Language-Conditioned Mixture-of-Experts Transformers

Published in arXiv, 2026

Tags: robotic manipulation, motion planning and control

The ability of robots to handle multiple tasks under a unified policy is critical for deploying embodied intelligence in real-world household and industrial applications. However, out-of-distribution variation across tasks often causes severe task interference and negative transfer when training general robotic policies. To address this challenge, we propose a lightweight multi-task imitation learning framework for bimanual manipulation, termed Mixture-of-Experts-Enhanced Action Chunking Transformer (MoE-ACT), which integrates sparse Mixture-of-Experts (MoE) modules into the Transformer encoder of ACT. The MoE layer decomposes a unified task policy into independently invoked expert components. Through adaptive activation, it naturally decouples multi-task action distributions in latent space. During decoding, Feature-wise Linear Modulation (FiLM) dynamically modulates action tokens to improve consistency between action generation and task instructions. In parallel, multi-scale cross-attention enables the policy to simultaneously focus on both low-level and high-level semantic features, providing rich visual information for robotic manipulation. We further incorporate textual information, transitioning the framework from a purely vision-based model to a vision-centric, language-conditioned action generation system. Experimental validation in both simulation and a real-world dual-arm setup shows that MoE-ACT substantially improves multi-task performance. Specifically, MoE-ACT outperforms vanilla ACT by an average of 33% in success rate. These results indicate that MoE-ACT provides stronger robustness and generalization in complex multi-task bimanual manipulation environments. Our open-source project page can be found at https://j3k7.github.io/MoE-ACT/.

Recommended citation: K. Guo, H. Liu, Y. Sun, R. Zhao, J. Zhou, J. Ma, "MoE-ACT: Scaling Multi-Task Bimanual Manipulation with Sparse Language-Conditioned Mixture-of-Experts Transformers," arXiv preprint arXiv:2603.15265, 2026. https://arxiv.org/pdf/2603.15265

UniManip: General-Purpose Zero-Shot Robotic Manipulation with Agentic Operational Graph

Published in arXiv, 2026

Tags: robotic manipulation, motion planning and control

Achieving general-purpose robotic manipulation requires robots to seamlessly bridge high-level semantic intent with low-level physical interaction in unstructured environments. However, existing approaches falter in zero-shot generalization: end-to-end Vision-Language-Action (VLA) models often lack the precision required for long-horizon tasks, while traditional hierarchical planners suffer from semantic rigidity when facing open-world variations. To address this, we present UniManip, a framework grounded in a Bi-level Agentic Operational Graph (AOG) that unifies semantic reasoning and physical grounding. By coupling a high-level Agentic Layer for task orchestration with a low-level Scene Layer for dynamic state representation, the system continuously aligns abstract planning with geometric constraints, enabling robust zero-shot execution. Unlike static pipelines, UniManip operates as a dynamic agentic loop: it actively instantiates object-centric scene graphs from unstructured perception, parameterizes these representations into collision-free trajectories via a safety-aware local planner, and exploits structured memory to autonomously diagnose and recover from execution failures. Extensive experiments validate the system’s robust zero-shot capability on unseen objects and tasks, demonstrating a 22.5% and 25.0% higher success rate compared to state-of-the-art VLA and hierarchical baselines, respectively. Notably, the system enables direct zero-shot transfer from fixed-base setups to mobile manipulation without fine-tuning or reconfiguration. Our open-source project page can be found at https://henryhcliu.github.io/unimanip.

Recommended citation: H. Liu, Y. Xue, Y. Zhou, H. Deng, Y. Liang, L. Xie, Z. Wang, "UniManip: General-Purpose Zero-Shot Robotic Manipulation with Agentic Operational Graph," arXiv preprint arXiv:2602.13086, 2026. https://arxiv.org/pdf/2602.13086

VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion

Published in 2026 IEEE International Conference on Robotics and Automation (ICRA), 2026

Tags: autonomous driving

Human drivers adeptly navigate complex scenarios by utilizing rich attentional semantics, but the current autonomous systems struggle to replicate this ability, as they often lose critical semantic information when converting 2D observations into 3D space. In this sense, it hinders their effective deployment in dynamic and complex environments. Leveraging the superior scene understanding and reasoning abilities of Vision-Language Models (VLMs), we propose VLM-E2E, a novel framework that uses the VLMs to enhance training by providing attentional cues. Our method integrates textual representations into Bird’s-Eye-View (BEV) features for semantic supervision, which enables the model to learn richer feature representations that explicitly capture the driver’s attentional semantics. By focusing on attentional semantics, VLM-E2E better aligns with human-like driving behavior, which is critical for navigating dynamic and complex environments. Furthermore, we introduce a BEV-Text learnable weighted fusion strategy to address the issue of modality importance imbalance in fusing multimodal information. This approach dynamically balances the contributions of BEV and text features, ensuring that the complementary information from visual and textual modality is effectively utilized. By explicitly addressing the imbalance in multimodal fusion, our method facilitates a more holistic and robust representation of driving environments. We evaluate VLM-E2E on the nuScenes dataset and demonstrate its superiority over state-of-the-art approaches, showcasing significant improvements in performance.

Recommended citation: P. Liu, H. Liu, H. Liu, X. Liu, J. Ni, and J. Ma, "VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion," in Proceedings of the 2026 IEEE International Conference on Robotics and Automation (ICRA), 2026, pp. 1-8. https://arxiv.org/abs/2502.18042

A Survey on Reinforcement Learning of Vision-Language-Action Models for Robotic Manipulation

Published in TechRxiv, 2025

Tags: robotic manipulation, survey

The vision of building generalist robotic systems capable of performing diverse manipulation tasks has been significantly advanced by Vision-Language-Action models (VLAs), which leverage large-scale pretraining to acquire general visuomotor priors via imitation learning. Current pre-trained VLAs still require fine-tuning to adapt to real-world deployment, where conventional imitation learning struggles with out-of-distribution (OOD) generalization due to the dependence on collected datasets with limited coverage of states and actions. Reinforcement learning (RL) leverages self-exploration and result-driven optimization to enhance OOD generalization in VLAs. This survey outlines how RL can bridge the gap between pretraining and real-world deployment, offering a comprehensive overview of the RL-VLA training paradigm. Our taxonomy is organized along four core dimensions that reflect the full learning-to-deployment lifecycle: RL-VLA architecture, training paradigms, real-world deployment, and benchmarking and evaluation.

Recommended citation: H. Deng, Z. Wu, H. Liu, W. Guo, Y. Xue, Z. Shan, C. Zhang, B. Jia, Y. Ling, G. Lu, and Z. Wang, "A Survey on Reinforcement Learning of Vision-Language-Action Models for Robotic Manipulation," TechRxiv, 2025. doi: 10.36227/techrxiv.176531955.54563920/v1 https://www.techrxiv.org/doi/full/10.36227/techrxiv.176531955.54563920/v1

An Autonomous Mobility on Demand Platform Supporting Cooperative Driving with Visual Context Generation for LMMs

Published in 2025 The IEEE International Conference on Intelligent Transportation Systems (ITSC), 2025

Tags: autonomous driving, motion planning and control

As Connected Autonomous Vehicles (CAVs) emerge as a cornerstone of future transportation systems, Autonomous Mobility on Demand (AMoD) platforms stand to revolutionize urban mobility by mitigating traffic congestion and enhancing passenger comfort and convenience. However, existing research primarily addresses either vehicle dispatch using real-world datasets to estimate the time cost for each CAV to its destination, or emphasizes cooperative motion planning of the CAVs for obstacle avoidance. Moreover, these studies often overlook the integration of the following two aspects: CAV destination assignments and cooperative motion planning in AMoD systems. To bridge this gap, our proposed CoDriveVis offers an integrated platform that integrates microscopic motion planning and control with macroscopic vehicle scheduling and dispatching. CoDriveVis leverages CARLA’s perception capabilities to produce multimodal outputs such as LiDAR-generated point clouds, images, and videos from diverse perspectives, which can inform traffic system operations, especially when coupled with large multimodal models (LMMs). Additionally, it features the Bird’s Eye View (BEV) image generation to visualize vehicle dispatching and request distribution across the map. The scheduling integration of the platform is streamlined through organized dictionaries for CAVs and passenger requests, while its control modes support both position transition and throttle/steering inputs. CoDriveVis also facilitates destination changes for updating the coordination of CAVs. Experimental deployment of representative methods within CoDriveVis demonstrates its efficacy and practicality as a robust simulation platform for advancing AMoD systems. The code is available at https://github.com/henryhcliu/CoDriveVis.git.

Recommended citation: H. Liu, H. Liu, and J. Ma, "An Autonomous Mobility on Demand Platform Supporting Cooperative Driving with Visual Context Generation for LMMs," in Proceedings of IEEE International Conference on Intelligent Transportation Systems (ITSC), IEEE, 2025, pp. 1-8. https://ieeexplore.ieee.org/document/11423430/

Human-like Driving: A Comprehensive Survey from Depth and Breadth

Published in Artificial Intelligence for Transportation, 2025

Tags: autonomous driving

Recent years have witnessed a burgeoning paradigm shift in transportation, in which autonomous vehicles (AVs) are reshaping conventional commuting modes with an overwhelming momentum. Yet we are not ready to embrace this change fully. The ”unfamiliar behavior of AVs is rendering themselves so unpredictable for conventional road users that the public distrusts or even deliberately refuses them, which many severe conflicts could surge accordingly. Therefore, human-like driving is proposed to help AVs seamlessly integrate into such long-standing transportation by behaving like real drivers. However, a comprehensive survey about human-like driving is still lacking today. In this paper, we explore current human-like driving studies from two brand-new perspectives: 1) depth, which focuses on the affordance of humans themselves to answer where human-like driving stems from, and 2) breadth, which grasps a comprehensive practical roadmap through the lens of technological actualization. In doing so, we can inform potential developers with a holistic picture of how to develop human-like driving and what aspects should be considered. This will contribute to a deeper understanding of human-like driving technologies in the future. This illuminates an in-depth discussion in human-like driving—the ‘reality alignment between AVs and human drivers. To the end, we underscore a multi-disciplinary avenue that holds great promise for reaching this target.

Recommended citation: H. Lu, J. Yang, Y. Liu, S. Shen, H. Liu, X. Zheng, and H. Yang, "Human-like driving: A comprehensive survey from depth and breadth," Artificial Intelligence for Transportation, vol. 3, p. 100033, 2025. https://www.sciencedirect.com/science/article/pii/S305086062500033X

Diffeomorphism-Transformed Iterative Linear Quadratic Regulator for Constrained Motion Planning in Autonomous Driving

Published in IEEE Transactions on Intelligent Transportation Systems, 2025

Tags: autonomous driving, motion planning and control

Ensuring safe driving and real-time execution is a crucial requirement in the motion planning process for autonomous vehicles. Hence, there is a compelling demand for advanced motion planning algorithms that exhibit effective management of inequality constraints and exceptional computational performance. This paper investigates a diffeomorphism-transformed iterative linear quadratic regulator (DTiLQR) algorithm for addressing constrained motion planning problems in autonomous vehicles with nonlinear dynamics and multiple inequality constraints. With regard to the state and input constraints, a novel state-and-input diffeomorphism is proposed to transform the constrained state/input space into an unconstrained one. Subsequently, these inequality constraints are systematically incorporated into the vehicle dynamics, thereby leading to the newly constructed system in this context. Then, we reformulate and incorporate the obstacle avoidance constraint into the objective function using state diffeomorphism and logarithmic barrier function. With this, the original optimization problem is converted to the unconstrained counterpart, adhering only to the constructed system dynamics. In this sense, featuring a streamlined single-loop architecture (which is essentially different from the dual-loop algorithmic design of existing constrained iLQR algorithms), DTiLQR is used to solve the optimization problem effectively while maintaining motion performance and constraint satisfaction for the resulting optimal trajectory. Ultimately, case studies across various driving situations showcase the effectiveness and exceptional computational efficiency of the proposed DTiLQR algorithm.

Recommended citation: Z. Zhu, H. Liu, W. Wang, J. Duan, H. Zhao, and J. Ma, "Diffeomorphism-Transformed Iterative Linear Quadratic Regulator for Constrained Motion Planning in Autonomous Driving," IEEE Transactions on Intelligent Transportation Systems, vol. 0, no. 0, pp. 1-15, 2025. https://ieeexplore.ieee.org/document/11157916

VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving

Published in arXiv, 2025

Tags: autonomous driving, motion planning and control

Scene understanding and risk-aware attentions are crucial for human drivers to make safe and effective driving decisions. To imitate this cognitive ability in urban autonomous driving while ensuring the transparency and interpretability, we propose a vision-language model (VLM)-enhanced unified decision-making and motion control framework, named VLM-UDMC. This framework incorporates scene reasoning and risk-aware insights into an upper-level slow system, which dynamically reconfigures the optimal motion planning for the downstream fast system. The reconfiguration is based on real-time environmental changes, which are encoded through context-aware potential functions. More specifically, the upper-level slow system employs a two-step reasoning policy with Retrieval-Augmented Generation (RAG), leveraging foundation models to process multimodal inputs and retrieve contextual knowledge, thereby generating risk-aware insights. Meanwhile, a lightweight multi-kernel decomposed LSTM provides real-time trajectory predictions for heterogeneous traffic participants by extracting smoother trend representations for short-horizon trajectory prediction. The effectiveness of the proposed VLM-UDMC framework is verified via both simulations and real-world experiments with a full-size autonomous vehicle. It is demonstrated that the presented VLM-UDMC effectively leverages scene understanding and attention decomposition for rational driving decisions, thus improving the overall urban driving performance. Our open-source project is available at https://github.com/henryhcliu/vlmudmc.git.

Recommended citation: H. Liu, H. Guo, P. Liu, B. Ma, Y. Zhang, J. Ma, and T. H. Lee, "VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving," arXiv preprint arXiv:2507.15266, 2025. https://www.arxiv.org/abs/2507.15266

RoboDexVLM: Visual Language Model-Enabled Task Planning and Motion Control for Dexterous Robot Manipulation

Published in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2025), 2025

Tags: robotic manipulation, motion planning and control

This paper introduces RoboDexVLM, an innovative framework for robot task planning and grasp detection tailored for a collaborative manipulator equipped with a dexterous hand. Previous methods focus on simplified and limited manipulation tasks, which often neglect the complexities associated with grasping a diverse array of objects in a long-horizon manner. In contrast, our proposed framework utilizes a dexterous hand capable of grasping objects of varying shapes and sizes while executing tasks based on natural language commands. The proposed approach has the following core components: First, a robust task planner with a task-level recovery mechanism that leverages vision-language models (VLMs) is designed, which enables the system to interpret and execute open-vocabulary commands for long sequence tasks. Second, a language-guided dexterous grasp perception algorithm is presented based on robot kinematics and formal methods, tailored for zero-shot dexterous manipulation with diverse objects and commands. Comprehensive experimental results validate the effectiveness, adaptability, and robustness of RoboDexVLM in handling long-horizon scenarios and performing dexterous grasping. These results highlight the framework’s ability to operate in complex environments, showcasing its potential for open-vocabulary dexterous manipulation. Our open-source project page can be found at https://henryhcliu.github.io/robodexvlm.

Recommended citation: H. Liu, S. Guo, P. Mai, J. Cao, H. Li, and J. Ma, "RoboDexVLM: Visual Language Model-Enabled Task Planning and Motion Control for Dexterous Robot Manipulation," in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2025, pp. 1-8. https://arxiv.org/abs/2503.01616

LMMCoDrive: Cooperative Driving with Large Multimodal Models

Published in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2025), 2025

Tags: autonomous driving, motion planning and control

To address the intricate challenges of decentralized cooperative scheduling and motion planning in Autonomous Mobility-on-Demand (AMoD) systems, this paper introduces LMMCoDrive, a novel cooperative driving framework that leverages a Large Multimodal Model (LMM) to enhance traffic efficiency in dynamic urban environments. This framework seamlessly integrates scheduling and motion planning processes to ensure the effective operation of Cooperative Autonomous Vehicles (CAVs). The spatial relationship between CAVs and passenger requests is abstracted into a Bird’s-Eye View (BEV) to fully exploit the potential of the LMM. Besides, trajectories are cautiously refined for each CAV while ensuring collision avoidance through safety constraints. A decentralized optimization strategy, facilitated by the Alternating Direction Method of Multipliers (ADMM) within the LMM framework, is proposed to drive the graph evolution of CAVs. Simulation results demonstrate the pivotal role and significant impact of LMM in optimizing CAV scheduling and enhancing decentralized cooperative optimization process for each vehicle. This marks a substantial stride towards achieving practical, efficient, and safe AMoD systems that are poised to revolutionize urban transportation. The code is available at https://github.com/henryhcliu/LMMCoDrive.

Recommended citation: H. Liu, R. Yao, Z. Huang, S. Shen, and J. Ma, "LMMCoDrive: Cooperative Driving with Large Multimodal Model," in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2025, pp. 1-8. https://arxiv.org/abs/2409.11981

UDMC: Unified Decision-Making and Control Framework for Urban Autonomous Driving with Motion Prediction of Traffic Participants

Published in IEEE Transactions on Intelligent Transportation Systems, 2025

Tags: autonomous driving, motion planning and control

Current autonomous driving systems often struggle to balance decision-making and motion control while ensuring safety and traffic rule compliance, especially in complex urban environments. Existing methods may fall short due to separate handling of these functionalities, leading to inefficiencies and safety compromises. To address these challenges, we introduce UDMC, an interpretable and unified Level 4 autonomous driving framework. UDMC integrates decision-making and motion control into a single optimal control problem (OCP), considering the dynamic interactions with surrounding vehicles, pedestrians, road lanes, and traffic signals. By employing innovative potential functions to model traffic participants and regulations, and incorporating a specialized motion prediction module, our framework enhances on-road safety and rule adherence. The integrated design allows for real-time execution of flexible maneuvers suited to diverse driving scenarios. High-fidelity simulations conducted in CARLA exemplify computational efficiency, robustness, and safety of the framework, resulting in superior driving performance when compared against various baseline models. Our open-source project is available at https://github.com/henryhcliu/udmc_carla.git.

Recommended citation: H. Liu, K. Chen, Y. Li, Z. Huang, M. Liu, and J. Ma, "UDMC: Unified Decision-Making and Control Framework for Urban Autonomous Driving with Motion Prediction of Traffic Participants," IEEE Transactions on Intelligent Transportation Systems, , vol. 26, no. 5, pp. 5856-871, 2025. https://ieeexplore.ieee.org/document/10942456/

Robot Navigation in Unknown and Cluttered Workspace with Dynamical System Modulation in Starshaped Roadmap

Published in 2025 IEEE International Conference on Robotics and Automation (ICRA 2025), 2025

Tags: robot navigation, motion planning and control

Compared to conventional decomposition methods that use ellipses or polygons to represent free space, starshaped representation can better capture the natural distribution of sensor data, thereby exploiting a larger portion of traversable space. This paper introduces a novel motion planning and control framework for navigating robots in unknown and cluttered environments using a dynamically constructed starshaped roadmap. Our approach generates a starshaped representation of the surrounding free space from real-time sensor data using piece-wise polynomials. Additionally, an incremental roadmap maintaining the connectivity information is constructed, and a searching algorithm efficiently selects short-term goals on this roadmap. Importantly, this framework addresses deadend situations with a graph updating mechanism. To ensure safe and efficient movement within the starshaped roadmap, we propose a reactive controller based on Dynamic System Modulation (DSM). This controller facilitates smooth motion within starshaped regions and their intersections, avoiding conservative and short-sighted behaviors and allowing the system to handle intricate obstacle configurations in unknown and cluttered environments. Comprehensive evaluations in both simulations and real-world experiments show that the proposed method achieves higher success rates and reduced travel times compared to other methods. It effectively manages intricate obstacle configurations, avoiding conservative and myopic behaviors.

Recommended citation: K. Chen, H. Liu, Y. Li, J. Duan, L. Zhu, and J. Ma, "Robot navigation in unknown and cluttered workspace with dynamical system modulation in starshaped roadmap," in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2025, pp. 1-7. https://arxiv.org/abs/2403.11484

CoDriveVLM: VLM-Enhanced Urban Cooperative Dispatching and Motion Planning for Future Autonomous Mobility on Demand Systems

Published in arXiv, 2025

Tags: autonomous driving, motion planning and control

The increasing demand for flexible and efficient urban transportation solutions has spotlighted the limitations of traditional Demand Responsive Transport (DRT) systems, particularly in accommodating diverse passenger needs and dynamic urban environments. Autonomous Mobility-on-Demand (AMoD) systems have emerged as a promising alternative, leveraging connected and autonomous vehicles (CAVs) to provide responsive and adaptable services. However, existing methods primarily focus on either vehicle scheduling or path planning, which often simplify complex urban layouts and neglect the necessity for simultaneous coordination and mutual avoidance among CAVs. This oversimplification poses significant challenges to the deployment of AMoD systems in real-world scenarios. To address these gaps, we propose CoDriveVLM, a novel framework that integrates high-fidelity simultaneous dispatching and cooperative motion planning for future AMoD systems. Our method harnesses Vision-Language Models (VLMs) to enhance multi-modality information processing, and this enables comprehensive dispatching and collision risk evaluation. The VLM-enhanced CAV dispatching coordinator is introduced to effectively manage complex and unforeseen AMoD conditions, thus supporting efficient scheduling decision-making. Furthermore, we propose a scalable decentralized cooperative motion planning method via consensus alternating direction method of multipliers (ADMM) focusing on collision risk evaluation and decentralized trajectory optimization. Simulation results demonstrate the feasibility and robustness of CoDriveVLM in various traffic conditions, showcasing its potential to significantly improve the fidelity and effectiveness of AMoD systems in future urban transportation networks. The code is available at https://github.com/henryhcliu/CoDriveVLM.git.

Recommended citation: H. Liu, R. Yao, W. Liu, Z. Huang, S. Shen, and J. Ma, "CoDriveVLM: VLM-enhanced urban cooperative dispatching and motion planning for future autonomous mobility on demand systems," arXiv preprint arXiv:2501.06132, 2025. https://arxiv.org/abs/2501.06132

Synergizing Decision Making and Trajectory Planning Using Two-Stage Optimization for Autonomous Vehicles

Published in IEEE Transactions on Vehicular Technology, 2024

Tags: autonomous driving, motion planning and control

This paper introduces a local planner that synergizes the decision making and trajectory planning modules towards autonomous driving. The decision making and trajectory planning tasks are jointly formulated as a nonlinear programming problem with an integrated objective function. However, integrating the discrete decision variables into the continuous trajectory optimization leads to a mixed-integer programming (MIP) problem with inherent nonlinearity and nonconvexity. To address the challenge in solving the problem, the original problem is decomposed into two sub-stages, and a two-stage optimization (TSO) based approach is presented to ensure the coherence in outcomes for the two stages. The optimization problem in the first stage determines the optimal decision sequence that acts as an informed initialization. With the outputs from the first stage, the second stage necessitates the use of a high-fidelity vehicle model and strict enforcement of the collision avoidance constraints as part of the trajectory planning problem. We evaluate the effectiveness of our proposed planner across diverse multi-lane scenarios. The results demonstrate that the proposed planner simultaneously generates a sequence of optimal decisions and the corresponding trajectory that significantly improves driving performance in terms of driving safety and traveling efficiency as compared to alternative methods. Additionally, we implement the closed-loop simulation in CARLA, and the results showcase the effectiveness of the proposed planner to adapt to changing driving situations with high computational efficiency.

Recommended citation: W. Liu, H. Liu, L. Zheng, Z. Huang, and J. Ma, "Synergizing Decision Making and Trajectory Planning Using Two-Stage Optimization for Autonomous Vehicles," IEEE Transactions on Vehicular Technology, vol. 74, no. 4, pp. 5489-503, 2025. https://arxiv.org/abs/2411.18974

Real-Time Teleoperation in Motion Mapping for Medical Robot Based on Robot Manipulator and Haptic Device

Published in 2024 IEEE International Conference on Robotics and Biomimetics (ROBIO 2024), 2024

Tags: robotic manipulation

Access to specialized medical care, particularly in rural or underserved areas, remains a significant challenge, with limited availability of healthcare professionals and diagnostic equipment. Teleoperated robotic systems offer a solution by enabling remote procedures, such as ultrasound examinations, where doctors can operate robotic arms from distant locations. In response to these healthcare challenges, we propose a real-time teleoperation system for controlling a UR5 robot arm using a Geomagic Touch haptic device. This system maps the operator’s hand movements to the robot arm, ensuring precise control and providing real-time haptic feedback, which is critical for delicate medical tasks. A force-torque sensor at the robot’s end-effector measures the force applied during operation, relaying this information back to the haptic device to simulate tactile sensations. Synchronization between the devices at a frequency of 500 Hz ensures smooth and responsive control. Additionally, a Kalman filter is applied to reduce noise in the force data, enhancing system accuracy. Experimental results demonstrate the system’s effectiveness with minimal delay and robust force feedback. This work represents a step forward in bridging the gap between remote healthcare providers and patients, offering a scalable solution for teleoperated medical procedures. All the codes and implementation details for this work can be found in the following repository: https://github.com/primpunn/Master.git.

Recommended citation: P. Peuchpen, H. Liu, and J. Ma, "Real-Time Teleoperation in Motion Mapping for Medical Robot Based on Robot Manipulator and Haptic Device," in Proceedings of IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, 2024, pp. 1-6. https://ieeexplore.ieee.org/document/10907567/

Enhance Planning with Physics-informed Safety Controller for End-to-end Autonomous Driving

Published in 2024 IEEE International Conference on Robotics and Biomimetics (ROBIO 2024), 2024

Tags: autonomous driving, motion planning and control

Recent years have seen a growing research interest in applications of Deep Neural Networks (DNN) on autonomous vehicle technology. The trend started with perception and prediction a few years ago and it is gradually being applied to motion planning tasks. Despite the performance of networks improve over time, DNN planners inherit the natural drawbacks of Deep Learning. Learning-based planners have limitations in achieving perfect accuracy on the training dataset and network performance can be affected by out-of-distribution problem. In this paper, we propose FusionAssurance, a novel trajectory-based end-to-end driving fusion framework which combines physics-informed control for safety assurance. By incorporating Potential Field into Model Predictive Control, FusionAssurance is capable of navigating through scenarios that are not included in the training dataset and scenarios where neural network fail to generalize. The effectiveness of the approach is demonstrated by extensive experiments under various scenarios on the CARLA benchmark.

Recommended citation: H. Zhou, H. Liu, H. Lu, J. Ma, and Y. Ji, "Enhance planning with physics-informed safety controller for end-to-end autonomous driving," in Proceedings of IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, 2024, pp. 1-8. https://arxiv.org/abs/2405.00316

Parallel optimization with hard safety constraints for cooperative planning of connected autonomous vehicles

Published in 2024 IEEE International Conference on Robotics and Automation (ICRA 2024), 2024

Tags: autonomous driving, motion planning and control

The development of connected autonomous vehicles (CAVs) facilitates the enhancement of traffic efficiency in complicated scenarios. In unsignalized roundabout scenarios, difficulties remain unsolved in developing an effective and efficient coordination strategy for CAVs. In this paper, we formulate the cooperative autonomous driving problem of CAVs in the roundabout scenario as a constrained optimal control problem, and propose a computationally-efficient parallel optimization framework to generate strategies for CAVs such that the travel efficiency is improved with hard safety guarantees. All constraints involved in the roundabout scenario are addressed appropriately with convex approximation, such that the convexity property of the reformulated optimization problem is exhibited. Then, a parallel optimization algorithm is presented to solve the reformulated optimization problem, where an embodied iterative nearest neighbor search strategy to determine the optimal passing sequence in the roundabout scenario. It is noteworthy that the travel efficiency in the roundabout scenario is enhanced and the computation burden is considerably alleviated with the innovation development. We also examine the proposed method in CARLA simulator and perform thorough comparisons with a rule-based baseline and the commonly used IPOPT optimization solver to demonstrate the effectiveness and efficiency of the proposed approach.

Recommended citation: Z. Huang, H. Liu, S. Shen, and J. Ma, "Parallel optimization with hard safety constraints for cooperative planning of connected autonomous vehicles," in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2024, pp. 1-7. https://arxiv.org/abs/2303.03090

Improved consensus ADMM for cooperative motion planning of large-scale connected autonomous vehicles with limited communication

Published in IEEE Transactions on Intelligent Vehicles, 2024

Tags: autonomous driving, motion planning and control

This paper investigates a cooperative motion planning problem for large-scale connected autonomous vehicles (CAVs) under limited communications, which addresses the challenges of high communication and computing resource requirements. Our proposed methodology incorporates a parallel optimization algorithm with improved consensus ADMM considering a more realistic locally connected topology network, and time complexity of $\mathcal{O}(N)$ is achieved by exploiting the sparsity in the dual update process. To further enhance the computational efficiency, we employ a lightweight evolution strategy for the dynamic connectivity graph of CAVs, and each sub-problem split from the consensus ADMM only requires managing a small group of CAVs. The proposed method implemented with the receding horizon scheme is validated thoroughly, and comparisons with existing numerical solvers and approaches demonstrate the efficiency of our proposed algorithm. Also, simulations on large-scale cooperative driving tasks involving up to 100 vehicles are performed in the high-fidelity CARLA simulator, which highlights the remarkable computational efficiency, scalability, and effectiveness of our proposed development. Demonstration videos are available at https://henryhcliu.github.io/icadmm_cmp_carla.

Recommended citation: H. Liu, Z. Huang, Z. Zhu, Y. Li, S. Shen, and J. Ma, "Improved consensus ADMM for cooperative motion planning of large-scale connected autonomous vehicles with limited communication," IEEE Transactions on Intelligent Vehicles, vol. 9, no. 11, pp. 7222-238, 2024. https://arxiv.org/abs/2401.09032

Incremental learning-based real-time trajectory prediction for autonomous driving via sparse Gaussian process regression

Published in 2024 IEEE Intelligent Vehicles Symposium (IV 2024), 2024

Tags: autonomous driving

In the context of spatial-temporal autonomous driving, the accurate and real-time trajectory prediction of the surrounding vehicle (SV) is crucial. This paper aims to design an efficient, accurate, and interpretable unimodal trajectory prediction approach. To achieve this objective, we employ Sparse Gaussian Process Regression (SGPR), which enables large dataset learning and efficient inference of future trajectories. This approach ensures accurate predictions while maintaining high computational efficiency. To further enhance the robustness of the prediction module, we propose the translation and rotation transformation strategy, which effectively simplifies the prediction problem. {Additionally, we utilize an instant evaluation algorithm to assess the prediction performance and maintain a streaming dataset for incremental learning, capable of adapting to dynamic driving environments.} In our experimental evaluation, we compare our proposed trajectory prediction approach with a series of existing methods. The results demonstrate that our work achieves superior prediction accuracy while requiring less inference time. It is noteworthy that, the proposed SGPR-based trajectory prediction approach with rotation equivalence is able to swiftly infer and incrementally learn from dynamic environments, which makes it a promising tool for enhancing safety and efficiency in autonomous driving systems.

Recommended citation: H. Liu, K. Chen, and J. Ma, "Incremental learning-based real-time trajectory prediction for autonomous driving via sparse Gaussian process regression," in Proceedings of Intelligent Vehicles Symposium(IV), IEEE, 2024, pp. 1-7. https://ieeexplore.ieee.org/document/10588687

Geometry-aware safety-critical local reactive controller for robot navigation in unknown and cluttered environments

Published in IEEE Robotics and Automation Letters, 2024

Tags: robot navigation, motion planning and control

This work proposes a safety-critical local reactive controller that enables the robot to navigate in unknown and cluttered environments. In particular, the trajectory tracking task is formulated as a constrained polynomial optimization problem. Then, safety constraints are imposed on the control variables invoking the notion of polynomial positivity certificates in conjunction with their Sum-of-Squares (SOS) approximation, thereby confining the robot motion inside the locally extracted convex free region. It is noteworthy that, in the process of devising the proposed safety constraints, the geometry of the robot can be approximated using any shape that can be characterized with a set of polynomial functions. The optimization problem is further convexified into a semidefinite program (SDP) leveraging truncated multi-sequences (tms) and moment relaxation, which favorably facilitates the effective use of off-the-shelf conic programming solvers, such that real-time performance is attainable. Various robot navigation tasks are investigated to demonstrate the effectiveness of the proposed approach in terms of safety and tracking performance.

Recommended citation: Y. Li, X. Tang, K. Chen, C. Zheng, H. Liu, and J. Ma, "Geometry-aware safety-critical local reactive controller for robot navigation in unknown and cluttered environments," IEEE Robotics and Automation Letters, vol. 9, no. 4, pp. 3419 - 3426, 2024. http://ieeexplore.ieee.org/document/10417140

Integrated behavior planning and motion control for autonomous vehicles with traffic rules compliance

Published in 2023 IEEE International Conference on Robotics and Biomimetics (ROBIO 2023), 2023

Tags: autonomous driving, motion planning and control

In this article, we propose an optimization-based integrated behavior planning and motion control scheme, which is an interpretable and adaptable urban autonomous driving solution that complies with complex traffic rules while ensuring driving safety. Inherently, to ensure compliance with traffic rules, an innovative design of potential functions (PFs) is presented to characterize various traffic rules related to traffic lights, traversable and non-traversable traffic line markings, etc. These PFs are further incorporated as part of the model predictive control (MPC) formulation. In this sense, high-level behavior planning is attained implicitly along with motion control as an integrated architecture, facilitating flexible maneuvers with safety guarantees. Due to the well-designed objective function of the MPC scheme, our integrated behavior planning and motion control scheme is competent for various urban driving scenarios and able to generate versatile behaviors, such as overtaking with adaptive cruise control, turning in the intersection, and merging in and out of the roundabout. As demonstrated from a series of simulations with challenging scenarios in CARLA, it is noteworthy that the proposed framework admits real-time performance and high generalizability.

Recommended citation: H. Liu, K. Chen, Y. Li, Z. Huang, J. Duan, and J. Ma, "Integrated behavior planning and motion control for autonomous vehicles with traffic rules compliance," in Proceedings of IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, 2023, pp. 1-7. https://ieeexplore.ieee.org/document/10354858/

Real-time parallel trajectory optimization with spatiotemporal safety constraints for autonomous driving in congested traffic

Published in The 26th IEEE International Conference on Intelligent Transportation Systems (ITSC 2023), 2023

Tags: autonomous driving

Multi-modal behaviors exhibited by surrounding vehicles (SVs) can typically lead to traffic congestion and reduce the travel efficiency of autonomous vehicles (AVs) in dense traffic. This paper proposes a real-time parallel trajectory optimization method for the AV to achieve high travel efficiency in dynamic and congested environments. A spatiotemporal safety module is developed to facilitate the safe interaction between the AV and SVs in the presence of trajectory prediction errors resulting from the multi-modal behaviors of the SVs. By leveraging multiple shooting and constraint transcription, we transform the trajectory optimization problem into a nonlinear programming problem, which allows for the use of optimization solvers and parallel computing techniques to generate multiple feasible trajectories in parallel. Subsequently, these spatiotemporal trajectories are fed into a multi-objective evaluation module considering both safety and efficiency objectives, such that the optimal feasible trajectory corresponding to the optimal target lane can be selected. The proposed framework is validated through simulations in a dense and congested driving scenario with multiple uncertain SVs. The results demonstrate that our method enables the AV to safely navigate through a dense and congested traffic scenario while achieving high travel efficiency and task accuracy in real time.

Recommended citation: L. Zheng, R. Yang, Z. Peng, H. Liu, M. Y. Wang, and J. Ma, "Real-time parallel trajectory optimization with spatiotemporal safety constraints for autonomous driving in congested traffic," in Proceedings of 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), IEEE, 2023, pp. 1186-1193. https://arxiv.org/abs/2309.05298

Bio-inspired robot swarm path formation with local sensor scope

Published in Applied Intelligence, 2022

Tags: robot navigation

Creating a robot network to connect the targets while exploring an unknown map is the goal of the path formation issue. It is difficult to maximize the efficiency of exploring a multi-target maze by a robot swarm with limited sensor capabilities. In this study, a novel behavior-based path formation approach (BPFM) that incorporates an artificial potential field and a bio-model inspired by slime mold is proposed for this problem. The robot’s controller can operate more sensibly while using the heuristic term from particle swarm. In order to maintain a dynamic multi-source network, a series of mechanisms and transition rules have been designed for the multi-target maze. Grid maps with obstacle density from sparse to dense are utilized in simulations to compare the proposed method with other algorithms. The results indicate that the performance of collective exploration, which is examined in diverse circumstances, is unexpectedly efficient and robust.

Recommended citation: Y. Zhao, Z. Qu, H. Liu, and R. Zhu, “Bio-inspired robot swarm path formation with local sensor scope, Applied Intelligence, vol. 53, no. 1, pp. 17310-17326, 2022. https://link.springer.com/article/10.1007/s10489-022-04356-9

Solving a Multi-robot Search Problem with Bionic Sarsa Algorithm and Artificial Potential Field

Published in 2021 China Automation Congress (CAC), 2021

Tags: robot navigation

Safe and effective path planning of multiple combat vehicles engaged in antagonistic environments keeps a challenging problem. Based on the application background of multi-robots in cooperative reconnaissance of enemy camps in environment with traps, this paper studies the multi-agent path planning based on bionic algorithms and artificial potential field method. The proposed bionic PP-AP Sarsa Scheme is inspired by food-finding scheme of Physarum Polycephalum (PP), which can effectively solve the dimensional explosion problem of traditional multi-agent reinforcement learning methods. This paper first studies the single-agent bionic planning problem with the PP algorithm to initialize the Q table used in Sarsa-based reinforcement learning, which effectively reduces the search space and accelerates the convergence speed of the early stage of reinforcement learning. After the Q tables in the same map are obtained through the training of different single agents, the Q tables of every agents are extended to multi-agents scenario by the assistance of simplified artificial potential field, hence a composite parallel path planner named RL-APCP 3 is constructed to synchronously update the actions of all of the agents, which allows us to complete the coordinated and efficient search of enemy camps by multiple agents. Compared with the Sarsa path planning algorithm of single agent, the efficiency of this scheme is improved up to 55.22%.

Recommended citation: H. Liu, Z. Qu and R. Zhu, "Solving a Multi-robot Search Problem with Bionic Sarsa Algorithm and Artificial Potential Field," 2021 China Automation Congress (CAC), Beijing, China, 2021, pp. 1830-1835. https://ieeexplore.ieee.org/document/9728613

Research on Robot Visual Grabbing Based on Mechanism Analysis

Published in 2021 IEEE 11th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), 2021

Tags: robotic manipulation

This paper mainly studies the problem of grasping objects by manipulator based on vision, and a model-based visual grabbing strategy is proposed. Compared with the existing classical image processing methods including the Sobel operator edge extraction method, the superiority of the corrosion operation edge extraction method used in this strategy has been verified, through several fruit image processing experiments. In order to solve the lack of sufficient number of labeled object recognition samples required by machine learning methods, a model-based image classifier is also established, which is based on artificially extracted object features. Hence, it can be interpreted strongly and does not require training using a large number of data samples. Finally, a visual robot grabbing experiment has been constructed and carried out. The results show that efficiency and accuracy of the image recognition algorithm are proved, and this algorithm is efficient, light and interpretable.

Recommended citation: H. Liu, Z. Liu, H. Liu and W. Lin, "Research on Robot Visual Grabbing Based on Mechanism Analysis," 2021 IEEE 11th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), Jiaxing, China, 2021, pp. 181-186. https://ieeexplore.ieee.org/document/9588176

Robotic manipulation

DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy

Published in arXiv, 2026

Tags: robotic manipulation, motion planning and control

Fine-grained, bimanual dexterous manipulation remains a foundational challenge in robotics. Traditional teleoperation systems often fail in contact-rich tasks because embodiment gaps hinder accurate kinematic mapping, while tactile and force feedback remain absent. Consequently, data collection efficiency for highprecision tasks remains prohibitively low. To address these limitations, we propose a tactile-driven adaptation strategy designed to enable fine-grained manipulation on top of teleoperation pipelines. Instantiated within our bimanual dexterous framework, DexTeleop-0, this strategy introduces a real-time optimization loop that bridges the embodiment gap by translating coarse human tracking intents into precise, force-compliant robotic commands with tactile sensing. By estimating accurate contact points and leveraging a tactile-enabled fingertip force-sensing profile, the system dynamically computes localized corrections using the operational space Jacobian with respect to joint angle updates. We rigorously evaluate this tactile-driven adaptation strategy across both simulated environments and real-world hardware. Compared with representative baselines, the proposed method consistently achieves higher task success rates and improved execution efficiency in robust grasping, disturbance-resilient manipulation, and complex dexterous tasks. Our open-source project page can be found at https://henryhcliu.github.io/dexteleop-0.

Recommended citation: H. Liu, Y. Jiang, H. Park, Y. Xue, Z. Wang, "DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy," arXiv preprint arXiv:2606.23431, 2026. https://arxiv.org/pdf/2606.23431

RoCo Challenge at AAAI 2026: Benchmarking Robotic Collaborative Manipulation for Assembly Towards Industrial Automation

Published in arXiv, 2026

Tags: robotic manipulation, motion planning and control

Embodied Artificial Intelligence (EAI) is rapidly developing, gradually subverting previous autonomous systems’ paradigms from isolated perception to integrated, continuous action. This transition is highly significant for industrial robotic manipulation, promising to free human workers from repetitive, dangerous daily labor. To benchmark and advance this capability, we introduce the Robotic Collaborative Assembly Assistance (RoCo) Challenge with a dataset towards simulation and real-world assembly manipulation. Set against the backdrop of human-centered manufacturing, this challenge focuses on a high-precision planetary gearbox assembly task, a demanding yet highly representative operation in modern industry. Built upon a self-developed data collection, training, and evaluation system in Isaac Sim, and utilizing a dual-arm robot for real-world deployment, the challenge operates in two phases. The Simulation Round defines fine-grained task phases for step-wise scoring to handle the long-horizon nature of the assembly. The Real-World Round mirrors this evaluation with physical gearbox components and high-quality teleoperated datasets. The core tasks require assembling an epicyclic gearbox from scratch, including mounting three planet gears, a sun gear, and a ring gear. Attracting over 60 teams and 170+ participants from more than 10 countries, the challenge yielded highly effective solutions, most notably ARC-VLA and RoboCola. Results demonstrate that a dual-model framework for long-horizon multi-task learning is highly effective, and the strategic utilization of recovery-from-failure curriculum data is a critical insight for successful deployment. This report outlines the competition setup, evaluation approach, key findings, and future directions for industrial EAI. Our dataset, CAD files, code, and evaluation results can be found at: https://rocochallenge.github.io/RoCo2026/.

Recommended citation: H. Liu, Y. Zhou, Z. Wu, Z. Ji, Z. Shan, Q. Wang, R. Liu, Z. Yang, Y. Gu, S. Khan, S. Yan, J. Liu, H. Zhu, C. Liu, J. Yang, J. Zhang, Z. Wang, "RoCo Challenge at AAAI 2026: Benchmarking Robotic Collaborative Manipulation for Assembly Towards Industrial Automation," arXiv preprint arXiv:2603.15469, 2026. https://arxiv.org/pdf/2603.15469

MoE-ACT: Scaling Multi-Task Bimanual Manipulation with Sparse Language-Conditioned Mixture-of-Experts Transformers

Published in arXiv, 2026

Tags: robotic manipulation, motion planning and control

The ability of robots to handle multiple tasks under a unified policy is critical for deploying embodied intelligence in real-world household and industrial applications. However, out-of-distribution variation across tasks often causes severe task interference and negative transfer when training general robotic policies. To address this challenge, we propose a lightweight multi-task imitation learning framework for bimanual manipulation, termed Mixture-of-Experts-Enhanced Action Chunking Transformer (MoE-ACT), which integrates sparse Mixture-of-Experts (MoE) modules into the Transformer encoder of ACT. The MoE layer decomposes a unified task policy into independently invoked expert components. Through adaptive activation, it naturally decouples multi-task action distributions in latent space. During decoding, Feature-wise Linear Modulation (FiLM) dynamically modulates action tokens to improve consistency between action generation and task instructions. In parallel, multi-scale cross-attention enables the policy to simultaneously focus on both low-level and high-level semantic features, providing rich visual information for robotic manipulation. We further incorporate textual information, transitioning the framework from a purely vision-based model to a vision-centric, language-conditioned action generation system. Experimental validation in both simulation and a real-world dual-arm setup shows that MoE-ACT substantially improves multi-task performance. Specifically, MoE-ACT outperforms vanilla ACT by an average of 33% in success rate. These results indicate that MoE-ACT provides stronger robustness and generalization in complex multi-task bimanual manipulation environments. Our open-source project page can be found at https://j3k7.github.io/MoE-ACT/.

Recommended citation: K. Guo, H. Liu, Y. Sun, R. Zhao, J. Zhou, J. Ma, "MoE-ACT: Scaling Multi-Task Bimanual Manipulation with Sparse Language-Conditioned Mixture-of-Experts Transformers," arXiv preprint arXiv:2603.15265, 2026. https://arxiv.org/pdf/2603.15265

UniManip: General-Purpose Zero-Shot Robotic Manipulation with Agentic Operational Graph

Published in arXiv, 2026

Tags: robotic manipulation, motion planning and control

Achieving general-purpose robotic manipulation requires robots to seamlessly bridge high-level semantic intent with low-level physical interaction in unstructured environments. However, existing approaches falter in zero-shot generalization: end-to-end Vision-Language-Action (VLA) models often lack the precision required for long-horizon tasks, while traditional hierarchical planners suffer from semantic rigidity when facing open-world variations. To address this, we present UniManip, a framework grounded in a Bi-level Agentic Operational Graph (AOG) that unifies semantic reasoning and physical grounding. By coupling a high-level Agentic Layer for task orchestration with a low-level Scene Layer for dynamic state representation, the system continuously aligns abstract planning with geometric constraints, enabling robust zero-shot execution. Unlike static pipelines, UniManip operates as a dynamic agentic loop: it actively instantiates object-centric scene graphs from unstructured perception, parameterizes these representations into collision-free trajectories via a safety-aware local planner, and exploits structured memory to autonomously diagnose and recover from execution failures. Extensive experiments validate the system’s robust zero-shot capability on unseen objects and tasks, demonstrating a 22.5% and 25.0% higher success rate compared to state-of-the-art VLA and hierarchical baselines, respectively. Notably, the system enables direct zero-shot transfer from fixed-base setups to mobile manipulation without fine-tuning or reconfiguration. Our open-source project page can be found at https://henryhcliu.github.io/unimanip.

Recommended citation: H. Liu, Y. Xue, Y. Zhou, H. Deng, Y. Liang, L. Xie, Z. Wang, "UniManip: General-Purpose Zero-Shot Robotic Manipulation with Agentic Operational Graph," arXiv preprint arXiv:2602.13086, 2026. https://arxiv.org/pdf/2602.13086

A Survey on Reinforcement Learning of Vision-Language-Action Models for Robotic Manipulation

Published in TechRxiv, 2025

Tags: robotic manipulation, survey

The vision of building generalist robotic systems capable of performing diverse manipulation tasks has been significantly advanced by Vision-Language-Action models (VLAs), which leverage large-scale pretraining to acquire general visuomotor priors via imitation learning. Current pre-trained VLAs still require fine-tuning to adapt to real-world deployment, where conventional imitation learning struggles with out-of-distribution (OOD) generalization due to the dependence on collected datasets with limited coverage of states and actions. Reinforcement learning (RL) leverages self-exploration and result-driven optimization to enhance OOD generalization in VLAs. This survey outlines how RL can bridge the gap between pretraining and real-world deployment, offering a comprehensive overview of the RL-VLA training paradigm. Our taxonomy is organized along four core dimensions that reflect the full learning-to-deployment lifecycle: RL-VLA architecture, training paradigms, real-world deployment, and benchmarking and evaluation.

Recommended citation: H. Deng, Z. Wu, H. Liu, W. Guo, Y. Xue, Z. Shan, C. Zhang, B. Jia, Y. Ling, G. Lu, and Z. Wang, "A Survey on Reinforcement Learning of Vision-Language-Action Models for Robotic Manipulation," TechRxiv, 2025. doi: 10.36227/techrxiv.176531955.54563920/v1 https://www.techrxiv.org/doi/full/10.36227/techrxiv.176531955.54563920/v1

RoboDexVLM: Visual Language Model-Enabled Task Planning and Motion Control for Dexterous Robot Manipulation

Published in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2025), 2025

Tags: robotic manipulation, motion planning and control

This paper introduces RoboDexVLM, an innovative framework for robot task planning and grasp detection tailored for a collaborative manipulator equipped with a dexterous hand. Previous methods focus on simplified and limited manipulation tasks, which often neglect the complexities associated with grasping a diverse array of objects in a long-horizon manner. In contrast, our proposed framework utilizes a dexterous hand capable of grasping objects of varying shapes and sizes while executing tasks based on natural language commands. The proposed approach has the following core components: First, a robust task planner with a task-level recovery mechanism that leverages vision-language models (VLMs) is designed, which enables the system to interpret and execute open-vocabulary commands for long sequence tasks. Second, a language-guided dexterous grasp perception algorithm is presented based on robot kinematics and formal methods, tailored for zero-shot dexterous manipulation with diverse objects and commands. Comprehensive experimental results validate the effectiveness, adaptability, and robustness of RoboDexVLM in handling long-horizon scenarios and performing dexterous grasping. These results highlight the framework’s ability to operate in complex environments, showcasing its potential for open-vocabulary dexterous manipulation. Our open-source project page can be found at https://henryhcliu.github.io/robodexvlm.

Recommended citation: H. Liu, S. Guo, P. Mai, J. Cao, H. Li, and J. Ma, "RoboDexVLM: Visual Language Model-Enabled Task Planning and Motion Control for Dexterous Robot Manipulation," in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2025, pp. 1-8. https://arxiv.org/abs/2503.01616

Real-Time Teleoperation in Motion Mapping for Medical Robot Based on Robot Manipulator and Haptic Device

Published in 2024 IEEE International Conference on Robotics and Biomimetics (ROBIO 2024), 2024

Tags: robotic manipulation

Access to specialized medical care, particularly in rural or underserved areas, remains a significant challenge, with limited availability of healthcare professionals and diagnostic equipment. Teleoperated robotic systems offer a solution by enabling remote procedures, such as ultrasound examinations, where doctors can operate robotic arms from distant locations. In response to these healthcare challenges, we propose a real-time teleoperation system for controlling a UR5 robot arm using a Geomagic Touch haptic device. This system maps the operator’s hand movements to the robot arm, ensuring precise control and providing real-time haptic feedback, which is critical for delicate medical tasks. A force-torque sensor at the robot’s end-effector measures the force applied during operation, relaying this information back to the haptic device to simulate tactile sensations. Synchronization between the devices at a frequency of 500 Hz ensures smooth and responsive control. Additionally, a Kalman filter is applied to reduce noise in the force data, enhancing system accuracy. Experimental results demonstrate the system’s effectiveness with minimal delay and robust force feedback. This work represents a step forward in bridging the gap between remote healthcare providers and patients, offering a scalable solution for teleoperated medical procedures. All the codes and implementation details for this work can be found in the following repository: https://github.com/primpunn/Master.git.

Recommended citation: P. Peuchpen, H. Liu, and J. Ma, "Real-Time Teleoperation in Motion Mapping for Medical Robot Based on Robot Manipulator and Haptic Device," in Proceedings of IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, 2024, pp. 1-6. https://ieeexplore.ieee.org/document/10907567/

Research on Robot Visual Grabbing Based on Mechanism Analysis

Published in 2021 IEEE 11th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), 2021

Tags: robotic manipulation

This paper mainly studies the problem of grasping objects by manipulator based on vision, and a model-based visual grabbing strategy is proposed. Compared with the existing classical image processing methods including the Sobel operator edge extraction method, the superiority of the corrosion operation edge extraction method used in this strategy has been verified, through several fruit image processing experiments. In order to solve the lack of sufficient number of labeled object recognition samples required by machine learning methods, a model-based image classifier is also established, which is based on artificially extracted object features. Hence, it can be interpreted strongly and does not require training using a large number of data samples. Finally, a visual robot grabbing experiment has been constructed and carried out. The results show that efficiency and accuracy of the image recognition algorithm are proved, and this algorithm is efficient, light and interpretable.

Recommended citation: H. Liu, Z. Liu, H. Liu and W. Lin, "Research on Robot Visual Grabbing Based on Mechanism Analysis," 2021 IEEE 11th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), Jiaxing, China, 2021, pp. 181-186. https://ieeexplore.ieee.org/document/9588176

Robot navigation

Robot Navigation in Unknown and Cluttered Workspace with Dynamical System Modulation in Starshaped Roadmap

Published in 2025 IEEE International Conference on Robotics and Automation (ICRA 2025), 2025

Tags: robot navigation, motion planning and control

Compared to conventional decomposition methods that use ellipses or polygons to represent free space, starshaped representation can better capture the natural distribution of sensor data, thereby exploiting a larger portion of traversable space. This paper introduces a novel motion planning and control framework for navigating robots in unknown and cluttered environments using a dynamically constructed starshaped roadmap. Our approach generates a starshaped representation of the surrounding free space from real-time sensor data using piece-wise polynomials. Additionally, an incremental roadmap maintaining the connectivity information is constructed, and a searching algorithm efficiently selects short-term goals on this roadmap. Importantly, this framework addresses deadend situations with a graph updating mechanism. To ensure safe and efficient movement within the starshaped roadmap, we propose a reactive controller based on Dynamic System Modulation (DSM). This controller facilitates smooth motion within starshaped regions and their intersections, avoiding conservative and short-sighted behaviors and allowing the system to handle intricate obstacle configurations in unknown and cluttered environments. Comprehensive evaluations in both simulations and real-world experiments show that the proposed method achieves higher success rates and reduced travel times compared to other methods. It effectively manages intricate obstacle configurations, avoiding conservative and myopic behaviors.

Recommended citation: K. Chen, H. Liu, Y. Li, J. Duan, L. Zhu, and J. Ma, "Robot navigation in unknown and cluttered workspace with dynamical system modulation in starshaped roadmap," in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2025, pp. 1-7. https://arxiv.org/abs/2403.11484

Geometry-aware safety-critical local reactive controller for robot navigation in unknown and cluttered environments

Published in IEEE Robotics and Automation Letters, 2024

Tags: robot navigation, motion planning and control

This work proposes a safety-critical local reactive controller that enables the robot to navigate in unknown and cluttered environments. In particular, the trajectory tracking task is formulated as a constrained polynomial optimization problem. Then, safety constraints are imposed on the control variables invoking the notion of polynomial positivity certificates in conjunction with their Sum-of-Squares (SOS) approximation, thereby confining the robot motion inside the locally extracted convex free region. It is noteworthy that, in the process of devising the proposed safety constraints, the geometry of the robot can be approximated using any shape that can be characterized with a set of polynomial functions. The optimization problem is further convexified into a semidefinite program (SDP) leveraging truncated multi-sequences (tms) and moment relaxation, which favorably facilitates the effective use of off-the-shelf conic programming solvers, such that real-time performance is attainable. Various robot navigation tasks are investigated to demonstrate the effectiveness of the proposed approach in terms of safety and tracking performance.

Recommended citation: Y. Li, X. Tang, K. Chen, C. Zheng, H. Liu, and J. Ma, "Geometry-aware safety-critical local reactive controller for robot navigation in unknown and cluttered environments," IEEE Robotics and Automation Letters, vol. 9, no. 4, pp. 3419 - 3426, 2024. http://ieeexplore.ieee.org/document/10417140

Bio-inspired robot swarm path formation with local sensor scope

Published in Applied Intelligence, 2022

Tags: robot navigation

Creating a robot network to connect the targets while exploring an unknown map is the goal of the path formation issue. It is difficult to maximize the efficiency of exploring a multi-target maze by a robot swarm with limited sensor capabilities. In this study, a novel behavior-based path formation approach (BPFM) that incorporates an artificial potential field and a bio-model inspired by slime mold is proposed for this problem. The robot’s controller can operate more sensibly while using the heuristic term from particle swarm. In order to maintain a dynamic multi-source network, a series of mechanisms and transition rules have been designed for the multi-target maze. Grid maps with obstacle density from sparse to dense are utilized in simulations to compare the proposed method with other algorithms. The results indicate that the performance of collective exploration, which is examined in diverse circumstances, is unexpectedly efficient and robust.

Recommended citation: Y. Zhao, Z. Qu, H. Liu, and R. Zhu, “Bio-inspired robot swarm path formation with local sensor scope, Applied Intelligence, vol. 53, no. 1, pp. 17310-17326, 2022. https://link.springer.com/article/10.1007/s10489-022-04356-9

Solving a Multi-robot Search Problem with Bionic Sarsa Algorithm and Artificial Potential Field

Published in 2021 China Automation Congress (CAC), 2021

Tags: robot navigation

Safe and effective path planning of multiple combat vehicles engaged in antagonistic environments keeps a challenging problem. Based on the application background of multi-robots in cooperative reconnaissance of enemy camps in environment with traps, this paper studies the multi-agent path planning based on bionic algorithms and artificial potential field method. The proposed bionic PP-AP Sarsa Scheme is inspired by food-finding scheme of Physarum Polycephalum (PP), which can effectively solve the dimensional explosion problem of traditional multi-agent reinforcement learning methods. This paper first studies the single-agent bionic planning problem with the PP algorithm to initialize the Q table used in Sarsa-based reinforcement learning, which effectively reduces the search space and accelerates the convergence speed of the early stage of reinforcement learning. After the Q tables in the same map are obtained through the training of different single agents, the Q tables of every agents are extended to multi-agents scenario by the assistance of simplified artificial potential field, hence a composite parallel path planner named RL-APCP 3 is constructed to synchronously update the actions of all of the agents, which allows us to complete the coordinated and efficient search of enemy camps by multiple agents. Compared with the Sarsa path planning algorithm of single agent, the efficiency of this scheme is improved up to 55.22%.

Recommended citation: H. Liu, Z. Qu and R. Zhu, "Solving a Multi-robot Search Problem with Bionic Sarsa Algorithm and Artificial Potential Field," 2021 China Automation Congress (CAC), Beijing, China, 2021, pp. 1830-1835. https://ieeexplore.ieee.org/document/9728613

Motion planning and control

DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy

Published in arXiv, 2026

Tags: robotic manipulation, motion planning and control

Fine-grained, bimanual dexterous manipulation remains a foundational challenge in robotics. Traditional teleoperation systems often fail in contact-rich tasks because embodiment gaps hinder accurate kinematic mapping, while tactile and force feedback remain absent. Consequently, data collection efficiency for highprecision tasks remains prohibitively low. To address these limitations, we propose a tactile-driven adaptation strategy designed to enable fine-grained manipulation on top of teleoperation pipelines. Instantiated within our bimanual dexterous framework, DexTeleop-0, this strategy introduces a real-time optimization loop that bridges the embodiment gap by translating coarse human tracking intents into precise, force-compliant robotic commands with tactile sensing. By estimating accurate contact points and leveraging a tactile-enabled fingertip force-sensing profile, the system dynamically computes localized corrections using the operational space Jacobian with respect to joint angle updates. We rigorously evaluate this tactile-driven adaptation strategy across both simulated environments and real-world hardware. Compared with representative baselines, the proposed method consistently achieves higher task success rates and improved execution efficiency in robust grasping, disturbance-resilient manipulation, and complex dexterous tasks. Our open-source project page can be found at https://henryhcliu.github.io/dexteleop-0.

Recommended citation: H. Liu, Y. Jiang, H. Park, Y. Xue, Z. Wang, "DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy," arXiv preprint arXiv:2606.23431, 2026. https://arxiv.org/pdf/2606.23431

An Intention-Driven Lane Change Framework Considering Heterogeneous Dynamic Cooperation in Mixed-Traffic Environments

Published in IEEE Transactions on Intelligent Transportation Systems, 2026

Tags: autonomous driving, motion planning and control

In mixed-traffic environments, autonomous vehicles (AVs) must interact with heterogeneous human-driven vehicles (HVs) whose intentions and driving styles vary across individuals and scenarios. Such variability introduces uncertainty into lane change interactions, where safety and efficiency critically depend on accurately anticipating surrounding drivers’ cooperative responses. Existing methods often oversimplify these interactions by assuming uniform or fixed behavioral patterns. To address this limitation, we propose an intention-driven lane change framework that integrates driving-style recognition with cooperationaware decision-making and motion-planning. A deep learningbased classifier identifies distinct human driving styles in real time. We then introduce a dual-perspective cooperation score composed of intrinsic style-dependent tendencies and interactive dynamic components, enabling interpretable and adaptive intention prediction and quantitative inference. A decision-making module combines behavior cloning (BC) and inverse reinforcement learning (IRL) to determine lane change feasibility. Later, a coordinated motion-planning architecture integrating IRLbased intention inference with model predictive control (MPC) is established to generate collision-free and socially compliant trajectories. Experiments on the NGSIM dataset show that the proposed decision-making model outperforms representative rule-based and learning-based baselines, achieving 96.98% accuracy in lane change classification. Motion-planning evaluations further demonstrate improved maneuver success and execution stability in mixed-traffic environments. These results validate the effectiveness of structured cooperation modeling for intentiondriven autonomous lane changes.

Recommended citation: X. Qiu, H. Liu, Y. Pan, J. Ma, and X. Zheng, "An Intention-Driven Lane Change Framework Considering Heterogeneous Dynamic Cooperation in Mixed-Traffic Environments," IEEE Transactions on Intelligent Transportation Systems, vol. 0, no. 0, pp. 1-13, 2026. https://arxiv.org/pdf/2509.22550

RoCo Challenge at AAAI 2026: Benchmarking Robotic Collaborative Manipulation for Assembly Towards Industrial Automation

Published in arXiv, 2026

Tags: robotic manipulation, motion planning and control

Embodied Artificial Intelligence (EAI) is rapidly developing, gradually subverting previous autonomous systems’ paradigms from isolated perception to integrated, continuous action. This transition is highly significant for industrial robotic manipulation, promising to free human workers from repetitive, dangerous daily labor. To benchmark and advance this capability, we introduce the Robotic Collaborative Assembly Assistance (RoCo) Challenge with a dataset towards simulation and real-world assembly manipulation. Set against the backdrop of human-centered manufacturing, this challenge focuses on a high-precision planetary gearbox assembly task, a demanding yet highly representative operation in modern industry. Built upon a self-developed data collection, training, and evaluation system in Isaac Sim, and utilizing a dual-arm robot for real-world deployment, the challenge operates in two phases. The Simulation Round defines fine-grained task phases for step-wise scoring to handle the long-horizon nature of the assembly. The Real-World Round mirrors this evaluation with physical gearbox components and high-quality teleoperated datasets. The core tasks require assembling an epicyclic gearbox from scratch, including mounting three planet gears, a sun gear, and a ring gear. Attracting over 60 teams and 170+ participants from more than 10 countries, the challenge yielded highly effective solutions, most notably ARC-VLA and RoboCola. Results demonstrate that a dual-model framework for long-horizon multi-task learning is highly effective, and the strategic utilization of recovery-from-failure curriculum data is a critical insight for successful deployment. This report outlines the competition setup, evaluation approach, key findings, and future directions for industrial EAI. Our dataset, CAD files, code, and evaluation results can be found at: https://rocochallenge.github.io/RoCo2026/.

Recommended citation: H. Liu, Y. Zhou, Z. Wu, Z. Ji, Z. Shan, Q. Wang, R. Liu, Z. Yang, Y. Gu, S. Khan, S. Yan, J. Liu, H. Zhu, C. Liu, J. Yang, J. Zhang, Z. Wang, "RoCo Challenge at AAAI 2026: Benchmarking Robotic Collaborative Manipulation for Assembly Towards Industrial Automation," arXiv preprint arXiv:2603.15469, 2026. https://arxiv.org/pdf/2603.15469

MoE-ACT: Scaling Multi-Task Bimanual Manipulation with Sparse Language-Conditioned Mixture-of-Experts Transformers

Published in arXiv, 2026

Tags: robotic manipulation, motion planning and control

The ability of robots to handle multiple tasks under a unified policy is critical for deploying embodied intelligence in real-world household and industrial applications. However, out-of-distribution variation across tasks often causes severe task interference and negative transfer when training general robotic policies. To address this challenge, we propose a lightweight multi-task imitation learning framework for bimanual manipulation, termed Mixture-of-Experts-Enhanced Action Chunking Transformer (MoE-ACT), which integrates sparse Mixture-of-Experts (MoE) modules into the Transformer encoder of ACT. The MoE layer decomposes a unified task policy into independently invoked expert components. Through adaptive activation, it naturally decouples multi-task action distributions in latent space. During decoding, Feature-wise Linear Modulation (FiLM) dynamically modulates action tokens to improve consistency between action generation and task instructions. In parallel, multi-scale cross-attention enables the policy to simultaneously focus on both low-level and high-level semantic features, providing rich visual information for robotic manipulation. We further incorporate textual information, transitioning the framework from a purely vision-based model to a vision-centric, language-conditioned action generation system. Experimental validation in both simulation and a real-world dual-arm setup shows that MoE-ACT substantially improves multi-task performance. Specifically, MoE-ACT outperforms vanilla ACT by an average of 33% in success rate. These results indicate that MoE-ACT provides stronger robustness and generalization in complex multi-task bimanual manipulation environments. Our open-source project page can be found at https://j3k7.github.io/MoE-ACT/.

Recommended citation: K. Guo, H. Liu, Y. Sun, R. Zhao, J. Zhou, J. Ma, "MoE-ACT: Scaling Multi-Task Bimanual Manipulation with Sparse Language-Conditioned Mixture-of-Experts Transformers," arXiv preprint arXiv:2603.15265, 2026. https://arxiv.org/pdf/2603.15265

UniManip: General-Purpose Zero-Shot Robotic Manipulation with Agentic Operational Graph

Published in arXiv, 2026

Tags: robotic manipulation, motion planning and control

Achieving general-purpose robotic manipulation requires robots to seamlessly bridge high-level semantic intent with low-level physical interaction in unstructured environments. However, existing approaches falter in zero-shot generalization: end-to-end Vision-Language-Action (VLA) models often lack the precision required for long-horizon tasks, while traditional hierarchical planners suffer from semantic rigidity when facing open-world variations. To address this, we present UniManip, a framework grounded in a Bi-level Agentic Operational Graph (AOG) that unifies semantic reasoning and physical grounding. By coupling a high-level Agentic Layer for task orchestration with a low-level Scene Layer for dynamic state representation, the system continuously aligns abstract planning with geometric constraints, enabling robust zero-shot execution. Unlike static pipelines, UniManip operates as a dynamic agentic loop: it actively instantiates object-centric scene graphs from unstructured perception, parameterizes these representations into collision-free trajectories via a safety-aware local planner, and exploits structured memory to autonomously diagnose and recover from execution failures. Extensive experiments validate the system’s robust zero-shot capability on unseen objects and tasks, demonstrating a 22.5% and 25.0% higher success rate compared to state-of-the-art VLA and hierarchical baselines, respectively. Notably, the system enables direct zero-shot transfer from fixed-base setups to mobile manipulation without fine-tuning or reconfiguration. Our open-source project page can be found at https://henryhcliu.github.io/unimanip.

Recommended citation: H. Liu, Y. Xue, Y. Zhou, H. Deng, Y. Liang, L. Xie, Z. Wang, "UniManip: General-Purpose Zero-Shot Robotic Manipulation with Agentic Operational Graph," arXiv preprint arXiv:2602.13086, 2026. https://arxiv.org/pdf/2602.13086

An Autonomous Mobility on Demand Platform Supporting Cooperative Driving with Visual Context Generation for LMMs

Published in 2025 The IEEE International Conference on Intelligent Transportation Systems (ITSC), 2025

Tags: autonomous driving, motion planning and control

As Connected Autonomous Vehicles (CAVs) emerge as a cornerstone of future transportation systems, Autonomous Mobility on Demand (AMoD) platforms stand to revolutionize urban mobility by mitigating traffic congestion and enhancing passenger comfort and convenience. However, existing research primarily addresses either vehicle dispatch using real-world datasets to estimate the time cost for each CAV to its destination, or emphasizes cooperative motion planning of the CAVs for obstacle avoidance. Moreover, these studies often overlook the integration of the following two aspects: CAV destination assignments and cooperative motion planning in AMoD systems. To bridge this gap, our proposed CoDriveVis offers an integrated platform that integrates microscopic motion planning and control with macroscopic vehicle scheduling and dispatching. CoDriveVis leverages CARLA’s perception capabilities to produce multimodal outputs such as LiDAR-generated point clouds, images, and videos from diverse perspectives, which can inform traffic system operations, especially when coupled with large multimodal models (LMMs). Additionally, it features the Bird’s Eye View (BEV) image generation to visualize vehicle dispatching and request distribution across the map. The scheduling integration of the platform is streamlined through organized dictionaries for CAVs and passenger requests, while its control modes support both position transition and throttle/steering inputs. CoDriveVis also facilitates destination changes for updating the coordination of CAVs. Experimental deployment of representative methods within CoDriveVis demonstrates its efficacy and practicality as a robust simulation platform for advancing AMoD systems. The code is available at https://github.com/henryhcliu/CoDriveVis.git.

Recommended citation: H. Liu, H. Liu, and J. Ma, "An Autonomous Mobility on Demand Platform Supporting Cooperative Driving with Visual Context Generation for LMMs," in Proceedings of IEEE International Conference on Intelligent Transportation Systems (ITSC), IEEE, 2025, pp. 1-8. https://ieeexplore.ieee.org/document/11423430/

Diffeomorphism-Transformed Iterative Linear Quadratic Regulator for Constrained Motion Planning in Autonomous Driving

Published in IEEE Transactions on Intelligent Transportation Systems, 2025

Tags: autonomous driving, motion planning and control

Ensuring safe driving and real-time execution is a crucial requirement in the motion planning process for autonomous vehicles. Hence, there is a compelling demand for advanced motion planning algorithms that exhibit effective management of inequality constraints and exceptional computational performance. This paper investigates a diffeomorphism-transformed iterative linear quadratic regulator (DTiLQR) algorithm for addressing constrained motion planning problems in autonomous vehicles with nonlinear dynamics and multiple inequality constraints. With regard to the state and input constraints, a novel state-and-input diffeomorphism is proposed to transform the constrained state/input space into an unconstrained one. Subsequently, these inequality constraints are systematically incorporated into the vehicle dynamics, thereby leading to the newly constructed system in this context. Then, we reformulate and incorporate the obstacle avoidance constraint into the objective function using state diffeomorphism and logarithmic barrier function. With this, the original optimization problem is converted to the unconstrained counterpart, adhering only to the constructed system dynamics. In this sense, featuring a streamlined single-loop architecture (which is essentially different from the dual-loop algorithmic design of existing constrained iLQR algorithms), DTiLQR is used to solve the optimization problem effectively while maintaining motion performance and constraint satisfaction for the resulting optimal trajectory. Ultimately, case studies across various driving situations showcase the effectiveness and exceptional computational efficiency of the proposed DTiLQR algorithm.

Recommended citation: Z. Zhu, H. Liu, W. Wang, J. Duan, H. Zhao, and J. Ma, "Diffeomorphism-Transformed Iterative Linear Quadratic Regulator for Constrained Motion Planning in Autonomous Driving," IEEE Transactions on Intelligent Transportation Systems, vol. 0, no. 0, pp. 1-15, 2025. https://ieeexplore.ieee.org/document/11157916

VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving

Published in arXiv, 2025

Tags: autonomous driving, motion planning and control

Scene understanding and risk-aware attentions are crucial for human drivers to make safe and effective driving decisions. To imitate this cognitive ability in urban autonomous driving while ensuring the transparency and interpretability, we propose a vision-language model (VLM)-enhanced unified decision-making and motion control framework, named VLM-UDMC. This framework incorporates scene reasoning and risk-aware insights into an upper-level slow system, which dynamically reconfigures the optimal motion planning for the downstream fast system. The reconfiguration is based on real-time environmental changes, which are encoded through context-aware potential functions. More specifically, the upper-level slow system employs a two-step reasoning policy with Retrieval-Augmented Generation (RAG), leveraging foundation models to process multimodal inputs and retrieve contextual knowledge, thereby generating risk-aware insights. Meanwhile, a lightweight multi-kernel decomposed LSTM provides real-time trajectory predictions for heterogeneous traffic participants by extracting smoother trend representations for short-horizon trajectory prediction. The effectiveness of the proposed VLM-UDMC framework is verified via both simulations and real-world experiments with a full-size autonomous vehicle. It is demonstrated that the presented VLM-UDMC effectively leverages scene understanding and attention decomposition for rational driving decisions, thus improving the overall urban driving performance. Our open-source project is available at https://github.com/henryhcliu/vlmudmc.git.

Recommended citation: H. Liu, H. Guo, P. Liu, B. Ma, Y. Zhang, J. Ma, and T. H. Lee, "VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving," arXiv preprint arXiv:2507.15266, 2025. https://www.arxiv.org/abs/2507.15266

RoboDexVLM: Visual Language Model-Enabled Task Planning and Motion Control for Dexterous Robot Manipulation

Published in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2025), 2025

Tags: robotic manipulation, motion planning and control

This paper introduces RoboDexVLM, an innovative framework for robot task planning and grasp detection tailored for a collaborative manipulator equipped with a dexterous hand. Previous methods focus on simplified and limited manipulation tasks, which often neglect the complexities associated with grasping a diverse array of objects in a long-horizon manner. In contrast, our proposed framework utilizes a dexterous hand capable of grasping objects of varying shapes and sizes while executing tasks based on natural language commands. The proposed approach has the following core components: First, a robust task planner with a task-level recovery mechanism that leverages vision-language models (VLMs) is designed, which enables the system to interpret and execute open-vocabulary commands for long sequence tasks. Second, a language-guided dexterous grasp perception algorithm is presented based on robot kinematics and formal methods, tailored for zero-shot dexterous manipulation with diverse objects and commands. Comprehensive experimental results validate the effectiveness, adaptability, and robustness of RoboDexVLM in handling long-horizon scenarios and performing dexterous grasping. These results highlight the framework’s ability to operate in complex environments, showcasing its potential for open-vocabulary dexterous manipulation. Our open-source project page can be found at https://henryhcliu.github.io/robodexvlm.

Recommended citation: H. Liu, S. Guo, P. Mai, J. Cao, H. Li, and J. Ma, "RoboDexVLM: Visual Language Model-Enabled Task Planning and Motion Control for Dexterous Robot Manipulation," in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2025, pp. 1-8. https://arxiv.org/abs/2503.01616

LMMCoDrive: Cooperative Driving with Large Multimodal Models

Published in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2025), 2025

Tags: autonomous driving, motion planning and control

To address the intricate challenges of decentralized cooperative scheduling and motion planning in Autonomous Mobility-on-Demand (AMoD) systems, this paper introduces LMMCoDrive, a novel cooperative driving framework that leverages a Large Multimodal Model (LMM) to enhance traffic efficiency in dynamic urban environments. This framework seamlessly integrates scheduling and motion planning processes to ensure the effective operation of Cooperative Autonomous Vehicles (CAVs). The spatial relationship between CAVs and passenger requests is abstracted into a Bird’s-Eye View (BEV) to fully exploit the potential of the LMM. Besides, trajectories are cautiously refined for each CAV while ensuring collision avoidance through safety constraints. A decentralized optimization strategy, facilitated by the Alternating Direction Method of Multipliers (ADMM) within the LMM framework, is proposed to drive the graph evolution of CAVs. Simulation results demonstrate the pivotal role and significant impact of LMM in optimizing CAV scheduling and enhancing decentralized cooperative optimization process for each vehicle. This marks a substantial stride towards achieving practical, efficient, and safe AMoD systems that are poised to revolutionize urban transportation. The code is available at https://github.com/henryhcliu/LMMCoDrive.

Recommended citation: H. Liu, R. Yao, Z. Huang, S. Shen, and J. Ma, "LMMCoDrive: Cooperative Driving with Large Multimodal Model," in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2025, pp. 1-8. https://arxiv.org/abs/2409.11981

UDMC: Unified Decision-Making and Control Framework for Urban Autonomous Driving with Motion Prediction of Traffic Participants

Published in IEEE Transactions on Intelligent Transportation Systems, 2025

Tags: autonomous driving, motion planning and control

Current autonomous driving systems often struggle to balance decision-making and motion control while ensuring safety and traffic rule compliance, especially in complex urban environments. Existing methods may fall short due to separate handling of these functionalities, leading to inefficiencies and safety compromises. To address these challenges, we introduce UDMC, an interpretable and unified Level 4 autonomous driving framework. UDMC integrates decision-making and motion control into a single optimal control problem (OCP), considering the dynamic interactions with surrounding vehicles, pedestrians, road lanes, and traffic signals. By employing innovative potential functions to model traffic participants and regulations, and incorporating a specialized motion prediction module, our framework enhances on-road safety and rule adherence. The integrated design allows for real-time execution of flexible maneuvers suited to diverse driving scenarios. High-fidelity simulations conducted in CARLA exemplify computational efficiency, robustness, and safety of the framework, resulting in superior driving performance when compared against various baseline models. Our open-source project is available at https://github.com/henryhcliu/udmc_carla.git.

Recommended citation: H. Liu, K. Chen, Y. Li, Z. Huang, M. Liu, and J. Ma, "UDMC: Unified Decision-Making and Control Framework for Urban Autonomous Driving with Motion Prediction of Traffic Participants," IEEE Transactions on Intelligent Transportation Systems, , vol. 26, no. 5, pp. 5856-871, 2025. https://ieeexplore.ieee.org/document/10942456/

Robot Navigation in Unknown and Cluttered Workspace with Dynamical System Modulation in Starshaped Roadmap

Published in 2025 IEEE International Conference on Robotics and Automation (ICRA 2025), 2025

Tags: robot navigation, motion planning and control

Compared to conventional decomposition methods that use ellipses or polygons to represent free space, starshaped representation can better capture the natural distribution of sensor data, thereby exploiting a larger portion of traversable space. This paper introduces a novel motion planning and control framework for navigating robots in unknown and cluttered environments using a dynamically constructed starshaped roadmap. Our approach generates a starshaped representation of the surrounding free space from real-time sensor data using piece-wise polynomials. Additionally, an incremental roadmap maintaining the connectivity information is constructed, and a searching algorithm efficiently selects short-term goals on this roadmap. Importantly, this framework addresses deadend situations with a graph updating mechanism. To ensure safe and efficient movement within the starshaped roadmap, we propose a reactive controller based on Dynamic System Modulation (DSM). This controller facilitates smooth motion within starshaped regions and their intersections, avoiding conservative and short-sighted behaviors and allowing the system to handle intricate obstacle configurations in unknown and cluttered environments. Comprehensive evaluations in both simulations and real-world experiments show that the proposed method achieves higher success rates and reduced travel times compared to other methods. It effectively manages intricate obstacle configurations, avoiding conservative and myopic behaviors.

Recommended citation: K. Chen, H. Liu, Y. Li, J. Duan, L. Zhu, and J. Ma, "Robot navigation in unknown and cluttered workspace with dynamical system modulation in starshaped roadmap," in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2025, pp. 1-7. https://arxiv.org/abs/2403.11484

CoDriveVLM: VLM-Enhanced Urban Cooperative Dispatching and Motion Planning for Future Autonomous Mobility on Demand Systems

Published in arXiv, 2025

Tags: autonomous driving, motion planning and control

The increasing demand for flexible and efficient urban transportation solutions has spotlighted the limitations of traditional Demand Responsive Transport (DRT) systems, particularly in accommodating diverse passenger needs and dynamic urban environments. Autonomous Mobility-on-Demand (AMoD) systems have emerged as a promising alternative, leveraging connected and autonomous vehicles (CAVs) to provide responsive and adaptable services. However, existing methods primarily focus on either vehicle scheduling or path planning, which often simplify complex urban layouts and neglect the necessity for simultaneous coordination and mutual avoidance among CAVs. This oversimplification poses significant challenges to the deployment of AMoD systems in real-world scenarios. To address these gaps, we propose CoDriveVLM, a novel framework that integrates high-fidelity simultaneous dispatching and cooperative motion planning for future AMoD systems. Our method harnesses Vision-Language Models (VLMs) to enhance multi-modality information processing, and this enables comprehensive dispatching and collision risk evaluation. The VLM-enhanced CAV dispatching coordinator is introduced to effectively manage complex and unforeseen AMoD conditions, thus supporting efficient scheduling decision-making. Furthermore, we propose a scalable decentralized cooperative motion planning method via consensus alternating direction method of multipliers (ADMM) focusing on collision risk evaluation and decentralized trajectory optimization. Simulation results demonstrate the feasibility and robustness of CoDriveVLM in various traffic conditions, showcasing its potential to significantly improve the fidelity and effectiveness of AMoD systems in future urban transportation networks. The code is available at https://github.com/henryhcliu/CoDriveVLM.git.

Recommended citation: H. Liu, R. Yao, W. Liu, Z. Huang, S. Shen, and J. Ma, "CoDriveVLM: VLM-enhanced urban cooperative dispatching and motion planning for future autonomous mobility on demand systems," arXiv preprint arXiv:2501.06132, 2025. https://arxiv.org/abs/2501.06132

Synergizing Decision Making and Trajectory Planning Using Two-Stage Optimization for Autonomous Vehicles

Published in IEEE Transactions on Vehicular Technology, 2024

Tags: autonomous driving, motion planning and control

This paper introduces a local planner that synergizes the decision making and trajectory planning modules towards autonomous driving. The decision making and trajectory planning tasks are jointly formulated as a nonlinear programming problem with an integrated objective function. However, integrating the discrete decision variables into the continuous trajectory optimization leads to a mixed-integer programming (MIP) problem with inherent nonlinearity and nonconvexity. To address the challenge in solving the problem, the original problem is decomposed into two sub-stages, and a two-stage optimization (TSO) based approach is presented to ensure the coherence in outcomes for the two stages. The optimization problem in the first stage determines the optimal decision sequence that acts as an informed initialization. With the outputs from the first stage, the second stage necessitates the use of a high-fidelity vehicle model and strict enforcement of the collision avoidance constraints as part of the trajectory planning problem. We evaluate the effectiveness of our proposed planner across diverse multi-lane scenarios. The results demonstrate that the proposed planner simultaneously generates a sequence of optimal decisions and the corresponding trajectory that significantly improves driving performance in terms of driving safety and traveling efficiency as compared to alternative methods. Additionally, we implement the closed-loop simulation in CARLA, and the results showcase the effectiveness of the proposed planner to adapt to changing driving situations with high computational efficiency.

Recommended citation: W. Liu, H. Liu, L. Zheng, Z. Huang, and J. Ma, "Synergizing Decision Making and Trajectory Planning Using Two-Stage Optimization for Autonomous Vehicles," IEEE Transactions on Vehicular Technology, vol. 74, no. 4, pp. 5489-503, 2025. https://arxiv.org/abs/2411.18974

Enhance Planning with Physics-informed Safety Controller for End-to-end Autonomous Driving

Published in 2024 IEEE International Conference on Robotics and Biomimetics (ROBIO 2024), 2024

Tags: autonomous driving, motion planning and control

Recent years have seen a growing research interest in applications of Deep Neural Networks (DNN) on autonomous vehicle technology. The trend started with perception and prediction a few years ago and it is gradually being applied to motion planning tasks. Despite the performance of networks improve over time, DNN planners inherit the natural drawbacks of Deep Learning. Learning-based planners have limitations in achieving perfect accuracy on the training dataset and network performance can be affected by out-of-distribution problem. In this paper, we propose FusionAssurance, a novel trajectory-based end-to-end driving fusion framework which combines physics-informed control for safety assurance. By incorporating Potential Field into Model Predictive Control, FusionAssurance is capable of navigating through scenarios that are not included in the training dataset and scenarios where neural network fail to generalize. The effectiveness of the approach is demonstrated by extensive experiments under various scenarios on the CARLA benchmark.

Recommended citation: H. Zhou, H. Liu, H. Lu, J. Ma, and Y. Ji, "Enhance planning with physics-informed safety controller for end-to-end autonomous driving," in Proceedings of IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, 2024, pp. 1-8. https://arxiv.org/abs/2405.00316

Parallel optimization with hard safety constraints for cooperative planning of connected autonomous vehicles

Published in 2024 IEEE International Conference on Robotics and Automation (ICRA 2024), 2024

Tags: autonomous driving, motion planning and control

The development of connected autonomous vehicles (CAVs) facilitates the enhancement of traffic efficiency in complicated scenarios. In unsignalized roundabout scenarios, difficulties remain unsolved in developing an effective and efficient coordination strategy for CAVs. In this paper, we formulate the cooperative autonomous driving problem of CAVs in the roundabout scenario as a constrained optimal control problem, and propose a computationally-efficient parallel optimization framework to generate strategies for CAVs such that the travel efficiency is improved with hard safety guarantees. All constraints involved in the roundabout scenario are addressed appropriately with convex approximation, such that the convexity property of the reformulated optimization problem is exhibited. Then, a parallel optimization algorithm is presented to solve the reformulated optimization problem, where an embodied iterative nearest neighbor search strategy to determine the optimal passing sequence in the roundabout scenario. It is noteworthy that the travel efficiency in the roundabout scenario is enhanced and the computation burden is considerably alleviated with the innovation development. We also examine the proposed method in CARLA simulator and perform thorough comparisons with a rule-based baseline and the commonly used IPOPT optimization solver to demonstrate the effectiveness and efficiency of the proposed approach.

Recommended citation: Z. Huang, H. Liu, S. Shen, and J. Ma, "Parallel optimization with hard safety constraints for cooperative planning of connected autonomous vehicles," in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2024, pp. 1-7. https://arxiv.org/abs/2303.03090

Improved consensus ADMM for cooperative motion planning of large-scale connected autonomous vehicles with limited communication

Published in IEEE Transactions on Intelligent Vehicles, 2024

Tags: autonomous driving, motion planning and control

This paper investigates a cooperative motion planning problem for large-scale connected autonomous vehicles (CAVs) under limited communications, which addresses the challenges of high communication and computing resource requirements. Our proposed methodology incorporates a parallel optimization algorithm with improved consensus ADMM considering a more realistic locally connected topology network, and time complexity of $\mathcal{O}(N)$ is achieved by exploiting the sparsity in the dual update process. To further enhance the computational efficiency, we employ a lightweight evolution strategy for the dynamic connectivity graph of CAVs, and each sub-problem split from the consensus ADMM only requires managing a small group of CAVs. The proposed method implemented with the receding horizon scheme is validated thoroughly, and comparisons with existing numerical solvers and approaches demonstrate the efficiency of our proposed algorithm. Also, simulations on large-scale cooperative driving tasks involving up to 100 vehicles are performed in the high-fidelity CARLA simulator, which highlights the remarkable computational efficiency, scalability, and effectiveness of our proposed development. Demonstration videos are available at https://henryhcliu.github.io/icadmm_cmp_carla.

Recommended citation: H. Liu, Z. Huang, Z. Zhu, Y. Li, S. Shen, and J. Ma, "Improved consensus ADMM for cooperative motion planning of large-scale connected autonomous vehicles with limited communication," IEEE Transactions on Intelligent Vehicles, vol. 9, no. 11, pp. 7222-238, 2024. https://arxiv.org/abs/2401.09032

Geometry-aware safety-critical local reactive controller for robot navigation in unknown and cluttered environments

Published in IEEE Robotics and Automation Letters, 2024

Tags: robot navigation, motion planning and control

This work proposes a safety-critical local reactive controller that enables the robot to navigate in unknown and cluttered environments. In particular, the trajectory tracking task is formulated as a constrained polynomial optimization problem. Then, safety constraints are imposed on the control variables invoking the notion of polynomial positivity certificates in conjunction with their Sum-of-Squares (SOS) approximation, thereby confining the robot motion inside the locally extracted convex free region. It is noteworthy that, in the process of devising the proposed safety constraints, the geometry of the robot can be approximated using any shape that can be characterized with a set of polynomial functions. The optimization problem is further convexified into a semidefinite program (SDP) leveraging truncated multi-sequences (tms) and moment relaxation, which favorably facilitates the effective use of off-the-shelf conic programming solvers, such that real-time performance is attainable. Various robot navigation tasks are investigated to demonstrate the effectiveness of the proposed approach in terms of safety and tracking performance.

Recommended citation: Y. Li, X. Tang, K. Chen, C. Zheng, H. Liu, and J. Ma, "Geometry-aware safety-critical local reactive controller for robot navigation in unknown and cluttered environments," IEEE Robotics and Automation Letters, vol. 9, no. 4, pp. 3419 - 3426, 2024. http://ieeexplore.ieee.org/document/10417140

Integrated behavior planning and motion control for autonomous vehicles with traffic rules compliance

Published in 2023 IEEE International Conference on Robotics and Biomimetics (ROBIO 2023), 2023

Tags: autonomous driving, motion planning and control

In this article, we propose an optimization-based integrated behavior planning and motion control scheme, which is an interpretable and adaptable urban autonomous driving solution that complies with complex traffic rules while ensuring driving safety. Inherently, to ensure compliance with traffic rules, an innovative design of potential functions (PFs) is presented to characterize various traffic rules related to traffic lights, traversable and non-traversable traffic line markings, etc. These PFs are further incorporated as part of the model predictive control (MPC) formulation. In this sense, high-level behavior planning is attained implicitly along with motion control as an integrated architecture, facilitating flexible maneuvers with safety guarantees. Due to the well-designed objective function of the MPC scheme, our integrated behavior planning and motion control scheme is competent for various urban driving scenarios and able to generate versatile behaviors, such as overtaking with adaptive cruise control, turning in the intersection, and merging in and out of the roundabout. As demonstrated from a series of simulations with challenging scenarios in CARLA, it is noteworthy that the proposed framework admits real-time performance and high generalizability.

Recommended citation: H. Liu, K. Chen, Y. Li, Z. Huang, J. Duan, and J. Ma, "Integrated behavior planning and motion control for autonomous vehicles with traffic rules compliance," in Proceedings of IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, 2023, pp. 1-7. https://ieeexplore.ieee.org/document/10354858/

Autonomous driving

Attention Distribution via Vision Language Models for Interactive Decision-Making and Motion Planning in Autonomous Driving

Published in 2026 IEEE 29th International Conference on Intelligent Transportation Systems (ITSC), 2026

Tags: autonomous driving

TBA

Recommended citation: B. Ma, H. Liu, and J. Ma, "Attention Distribution via Vision Language Models for Interactive Decision-Making and Motion Planning in Autonomous Driving," in Proceedings of the 2026 IEEE 29th International Conference on Intelligent Transportation Systems (ITSC), 2026, pp. 1-8.

An Intention-Driven Lane Change Framework Considering Heterogeneous Dynamic Cooperation in Mixed-Traffic Environments

Published in IEEE Transactions on Intelligent Transportation Systems, 2026

Tags: autonomous driving, motion planning and control

In mixed-traffic environments, autonomous vehicles (AVs) must interact with heterogeneous human-driven vehicles (HVs) whose intentions and driving styles vary across individuals and scenarios. Such variability introduces uncertainty into lane change interactions, where safety and efficiency critically depend on accurately anticipating surrounding drivers’ cooperative responses. Existing methods often oversimplify these interactions by assuming uniform or fixed behavioral patterns. To address this limitation, we propose an intention-driven lane change framework that integrates driving-style recognition with cooperationaware decision-making and motion-planning. A deep learningbased classifier identifies distinct human driving styles in real time. We then introduce a dual-perspective cooperation score composed of intrinsic style-dependent tendencies and interactive dynamic components, enabling interpretable and adaptive intention prediction and quantitative inference. A decision-making module combines behavior cloning (BC) and inverse reinforcement learning (IRL) to determine lane change feasibility. Later, a coordinated motion-planning architecture integrating IRLbased intention inference with model predictive control (MPC) is established to generate collision-free and socially compliant trajectories. Experiments on the NGSIM dataset show that the proposed decision-making model outperforms representative rule-based and learning-based baselines, achieving 96.98% accuracy in lane change classification. Motion-planning evaluations further demonstrate improved maneuver success and execution stability in mixed-traffic environments. These results validate the effectiveness of structured cooperation modeling for intentiondriven autonomous lane changes.

Recommended citation: X. Qiu, H. Liu, Y. Pan, J. Ma, and X. Zheng, "An Intention-Driven Lane Change Framework Considering Heterogeneous Dynamic Cooperation in Mixed-Traffic Environments," IEEE Transactions on Intelligent Transportation Systems, vol. 0, no. 0, pp. 1-13, 2026. https://arxiv.org/pdf/2509.22550

VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion

Published in 2026 IEEE International Conference on Robotics and Automation (ICRA), 2026

Tags: autonomous driving

Human drivers adeptly navigate complex scenarios by utilizing rich attentional semantics, but the current autonomous systems struggle to replicate this ability, as they often lose critical semantic information when converting 2D observations into 3D space. In this sense, it hinders their effective deployment in dynamic and complex environments. Leveraging the superior scene understanding and reasoning abilities of Vision-Language Models (VLMs), we propose VLM-E2E, a novel framework that uses the VLMs to enhance training by providing attentional cues. Our method integrates textual representations into Bird’s-Eye-View (BEV) features for semantic supervision, which enables the model to learn richer feature representations that explicitly capture the driver’s attentional semantics. By focusing on attentional semantics, VLM-E2E better aligns with human-like driving behavior, which is critical for navigating dynamic and complex environments. Furthermore, we introduce a BEV-Text learnable weighted fusion strategy to address the issue of modality importance imbalance in fusing multimodal information. This approach dynamically balances the contributions of BEV and text features, ensuring that the complementary information from visual and textual modality is effectively utilized. By explicitly addressing the imbalance in multimodal fusion, our method facilitates a more holistic and robust representation of driving environments. We evaluate VLM-E2E on the nuScenes dataset and demonstrate its superiority over state-of-the-art approaches, showcasing significant improvements in performance.

Recommended citation: P. Liu, H. Liu, H. Liu, X. Liu, J. Ni, and J. Ma, "VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion," in Proceedings of the 2026 IEEE International Conference on Robotics and Automation (ICRA), 2026, pp. 1-8. https://arxiv.org/abs/2502.18042

An Autonomous Mobility on Demand Platform Supporting Cooperative Driving with Visual Context Generation for LMMs

Published in 2025 The IEEE International Conference on Intelligent Transportation Systems (ITSC), 2025

Tags: autonomous driving, motion planning and control

As Connected Autonomous Vehicles (CAVs) emerge as a cornerstone of future transportation systems, Autonomous Mobility on Demand (AMoD) platforms stand to revolutionize urban mobility by mitigating traffic congestion and enhancing passenger comfort and convenience. However, existing research primarily addresses either vehicle dispatch using real-world datasets to estimate the time cost for each CAV to its destination, or emphasizes cooperative motion planning of the CAVs for obstacle avoidance. Moreover, these studies often overlook the integration of the following two aspects: CAV destination assignments and cooperative motion planning in AMoD systems. To bridge this gap, our proposed CoDriveVis offers an integrated platform that integrates microscopic motion planning and control with macroscopic vehicle scheduling and dispatching. CoDriveVis leverages CARLA’s perception capabilities to produce multimodal outputs such as LiDAR-generated point clouds, images, and videos from diverse perspectives, which can inform traffic system operations, especially when coupled with large multimodal models (LMMs). Additionally, it features the Bird’s Eye View (BEV) image generation to visualize vehicle dispatching and request distribution across the map. The scheduling integration of the platform is streamlined through organized dictionaries for CAVs and passenger requests, while its control modes support both position transition and throttle/steering inputs. CoDriveVis also facilitates destination changes for updating the coordination of CAVs. Experimental deployment of representative methods within CoDriveVis demonstrates its efficacy and practicality as a robust simulation platform for advancing AMoD systems. The code is available at https://github.com/henryhcliu/CoDriveVis.git.

Recommended citation: H. Liu, H. Liu, and J. Ma, "An Autonomous Mobility on Demand Platform Supporting Cooperative Driving with Visual Context Generation for LMMs," in Proceedings of IEEE International Conference on Intelligent Transportation Systems (ITSC), IEEE, 2025, pp. 1-8. https://ieeexplore.ieee.org/document/11423430/

Human-like Driving: A Comprehensive Survey from Depth and Breadth

Published in Artificial Intelligence for Transportation, 2025

Tags: autonomous driving

Recent years have witnessed a burgeoning paradigm shift in transportation, in which autonomous vehicles (AVs) are reshaping conventional commuting modes with an overwhelming momentum. Yet we are not ready to embrace this change fully. The ”unfamiliar behavior of AVs is rendering themselves so unpredictable for conventional road users that the public distrusts or even deliberately refuses them, which many severe conflicts could surge accordingly. Therefore, human-like driving is proposed to help AVs seamlessly integrate into such long-standing transportation by behaving like real drivers. However, a comprehensive survey about human-like driving is still lacking today. In this paper, we explore current human-like driving studies from two brand-new perspectives: 1) depth, which focuses on the affordance of humans themselves to answer where human-like driving stems from, and 2) breadth, which grasps a comprehensive practical roadmap through the lens of technological actualization. In doing so, we can inform potential developers with a holistic picture of how to develop human-like driving and what aspects should be considered. This will contribute to a deeper understanding of human-like driving technologies in the future. This illuminates an in-depth discussion in human-like driving—the ‘reality alignment between AVs and human drivers. To the end, we underscore a multi-disciplinary avenue that holds great promise for reaching this target.

Recommended citation: H. Lu, J. Yang, Y. Liu, S. Shen, H. Liu, X. Zheng, and H. Yang, "Human-like driving: A comprehensive survey from depth and breadth," Artificial Intelligence for Transportation, vol. 3, p. 100033, 2025. https://www.sciencedirect.com/science/article/pii/S305086062500033X

Diffeomorphism-Transformed Iterative Linear Quadratic Regulator for Constrained Motion Planning in Autonomous Driving

Published in IEEE Transactions on Intelligent Transportation Systems, 2025

Tags: autonomous driving, motion planning and control

Ensuring safe driving and real-time execution is a crucial requirement in the motion planning process for autonomous vehicles. Hence, there is a compelling demand for advanced motion planning algorithms that exhibit effective management of inequality constraints and exceptional computational performance. This paper investigates a diffeomorphism-transformed iterative linear quadratic regulator (DTiLQR) algorithm for addressing constrained motion planning problems in autonomous vehicles with nonlinear dynamics and multiple inequality constraints. With regard to the state and input constraints, a novel state-and-input diffeomorphism is proposed to transform the constrained state/input space into an unconstrained one. Subsequently, these inequality constraints are systematically incorporated into the vehicle dynamics, thereby leading to the newly constructed system in this context. Then, we reformulate and incorporate the obstacle avoidance constraint into the objective function using state diffeomorphism and logarithmic barrier function. With this, the original optimization problem is converted to the unconstrained counterpart, adhering only to the constructed system dynamics. In this sense, featuring a streamlined single-loop architecture (which is essentially different from the dual-loop algorithmic design of existing constrained iLQR algorithms), DTiLQR is used to solve the optimization problem effectively while maintaining motion performance and constraint satisfaction for the resulting optimal trajectory. Ultimately, case studies across various driving situations showcase the effectiveness and exceptional computational efficiency of the proposed DTiLQR algorithm.

Recommended citation: Z. Zhu, H. Liu, W. Wang, J. Duan, H. Zhao, and J. Ma, "Diffeomorphism-Transformed Iterative Linear Quadratic Regulator for Constrained Motion Planning in Autonomous Driving," IEEE Transactions on Intelligent Transportation Systems, vol. 0, no. 0, pp. 1-15, 2025. https://ieeexplore.ieee.org/document/11157916

VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving

Published in arXiv, 2025

Tags: autonomous driving, motion planning and control

Scene understanding and risk-aware attentions are crucial for human drivers to make safe and effective driving decisions. To imitate this cognitive ability in urban autonomous driving while ensuring the transparency and interpretability, we propose a vision-language model (VLM)-enhanced unified decision-making and motion control framework, named VLM-UDMC. This framework incorporates scene reasoning and risk-aware insights into an upper-level slow system, which dynamically reconfigures the optimal motion planning for the downstream fast system. The reconfiguration is based on real-time environmental changes, which are encoded through context-aware potential functions. More specifically, the upper-level slow system employs a two-step reasoning policy with Retrieval-Augmented Generation (RAG), leveraging foundation models to process multimodal inputs and retrieve contextual knowledge, thereby generating risk-aware insights. Meanwhile, a lightweight multi-kernel decomposed LSTM provides real-time trajectory predictions for heterogeneous traffic participants by extracting smoother trend representations for short-horizon trajectory prediction. The effectiveness of the proposed VLM-UDMC framework is verified via both simulations and real-world experiments with a full-size autonomous vehicle. It is demonstrated that the presented VLM-UDMC effectively leverages scene understanding and attention decomposition for rational driving decisions, thus improving the overall urban driving performance. Our open-source project is available at https://github.com/henryhcliu/vlmudmc.git.

Recommended citation: H. Liu, H. Guo, P. Liu, B. Ma, Y. Zhang, J. Ma, and T. H. Lee, "VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving," arXiv preprint arXiv:2507.15266, 2025. https://www.arxiv.org/abs/2507.15266

LMMCoDrive: Cooperative Driving with Large Multimodal Models

Published in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2025), 2025

Tags: autonomous driving, motion planning and control

To address the intricate challenges of decentralized cooperative scheduling and motion planning in Autonomous Mobility-on-Demand (AMoD) systems, this paper introduces LMMCoDrive, a novel cooperative driving framework that leverages a Large Multimodal Model (LMM) to enhance traffic efficiency in dynamic urban environments. This framework seamlessly integrates scheduling and motion planning processes to ensure the effective operation of Cooperative Autonomous Vehicles (CAVs). The spatial relationship between CAVs and passenger requests is abstracted into a Bird’s-Eye View (BEV) to fully exploit the potential of the LMM. Besides, trajectories are cautiously refined for each CAV while ensuring collision avoidance through safety constraints. A decentralized optimization strategy, facilitated by the Alternating Direction Method of Multipliers (ADMM) within the LMM framework, is proposed to drive the graph evolution of CAVs. Simulation results demonstrate the pivotal role and significant impact of LMM in optimizing CAV scheduling and enhancing decentralized cooperative optimization process for each vehicle. This marks a substantial stride towards achieving practical, efficient, and safe AMoD systems that are poised to revolutionize urban transportation. The code is available at https://github.com/henryhcliu/LMMCoDrive.

Recommended citation: H. Liu, R. Yao, Z. Huang, S. Shen, and J. Ma, "LMMCoDrive: Cooperative Driving with Large Multimodal Model," in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2025, pp. 1-8. https://arxiv.org/abs/2409.11981

UDMC: Unified Decision-Making and Control Framework for Urban Autonomous Driving with Motion Prediction of Traffic Participants

Published in IEEE Transactions on Intelligent Transportation Systems, 2025

Tags: autonomous driving, motion planning and control

Current autonomous driving systems often struggle to balance decision-making and motion control while ensuring safety and traffic rule compliance, especially in complex urban environments. Existing methods may fall short due to separate handling of these functionalities, leading to inefficiencies and safety compromises. To address these challenges, we introduce UDMC, an interpretable and unified Level 4 autonomous driving framework. UDMC integrates decision-making and motion control into a single optimal control problem (OCP), considering the dynamic interactions with surrounding vehicles, pedestrians, road lanes, and traffic signals. By employing innovative potential functions to model traffic participants and regulations, and incorporating a specialized motion prediction module, our framework enhances on-road safety and rule adherence. The integrated design allows for real-time execution of flexible maneuvers suited to diverse driving scenarios. High-fidelity simulations conducted in CARLA exemplify computational efficiency, robustness, and safety of the framework, resulting in superior driving performance when compared against various baseline models. Our open-source project is available at https://github.com/henryhcliu/udmc_carla.git.

Recommended citation: H. Liu, K. Chen, Y. Li, Z. Huang, M. Liu, and J. Ma, "UDMC: Unified Decision-Making and Control Framework for Urban Autonomous Driving with Motion Prediction of Traffic Participants," IEEE Transactions on Intelligent Transportation Systems, , vol. 26, no. 5, pp. 5856-871, 2025. https://ieeexplore.ieee.org/document/10942456/

CoDriveVLM: VLM-Enhanced Urban Cooperative Dispatching and Motion Planning for Future Autonomous Mobility on Demand Systems

Published in arXiv, 2025

Tags: autonomous driving, motion planning and control

The increasing demand for flexible and efficient urban transportation solutions has spotlighted the limitations of traditional Demand Responsive Transport (DRT) systems, particularly in accommodating diverse passenger needs and dynamic urban environments. Autonomous Mobility-on-Demand (AMoD) systems have emerged as a promising alternative, leveraging connected and autonomous vehicles (CAVs) to provide responsive and adaptable services. However, existing methods primarily focus on either vehicle scheduling or path planning, which often simplify complex urban layouts and neglect the necessity for simultaneous coordination and mutual avoidance among CAVs. This oversimplification poses significant challenges to the deployment of AMoD systems in real-world scenarios. To address these gaps, we propose CoDriveVLM, a novel framework that integrates high-fidelity simultaneous dispatching and cooperative motion planning for future AMoD systems. Our method harnesses Vision-Language Models (VLMs) to enhance multi-modality information processing, and this enables comprehensive dispatching and collision risk evaluation. The VLM-enhanced CAV dispatching coordinator is introduced to effectively manage complex and unforeseen AMoD conditions, thus supporting efficient scheduling decision-making. Furthermore, we propose a scalable decentralized cooperative motion planning method via consensus alternating direction method of multipliers (ADMM) focusing on collision risk evaluation and decentralized trajectory optimization. Simulation results demonstrate the feasibility and robustness of CoDriveVLM in various traffic conditions, showcasing its potential to significantly improve the fidelity and effectiveness of AMoD systems in future urban transportation networks. The code is available at https://github.com/henryhcliu/CoDriveVLM.git.

Recommended citation: H. Liu, R. Yao, W. Liu, Z. Huang, S. Shen, and J. Ma, "CoDriveVLM: VLM-enhanced urban cooperative dispatching and motion planning for future autonomous mobility on demand systems," arXiv preprint arXiv:2501.06132, 2025. https://arxiv.org/abs/2501.06132

Synergizing Decision Making and Trajectory Planning Using Two-Stage Optimization for Autonomous Vehicles

Published in IEEE Transactions on Vehicular Technology, 2024

Tags: autonomous driving, motion planning and control

This paper introduces a local planner that synergizes the decision making and trajectory planning modules towards autonomous driving. The decision making and trajectory planning tasks are jointly formulated as a nonlinear programming problem with an integrated objective function. However, integrating the discrete decision variables into the continuous trajectory optimization leads to a mixed-integer programming (MIP) problem with inherent nonlinearity and nonconvexity. To address the challenge in solving the problem, the original problem is decomposed into two sub-stages, and a two-stage optimization (TSO) based approach is presented to ensure the coherence in outcomes for the two stages. The optimization problem in the first stage determines the optimal decision sequence that acts as an informed initialization. With the outputs from the first stage, the second stage necessitates the use of a high-fidelity vehicle model and strict enforcement of the collision avoidance constraints as part of the trajectory planning problem. We evaluate the effectiveness of our proposed planner across diverse multi-lane scenarios. The results demonstrate that the proposed planner simultaneously generates a sequence of optimal decisions and the corresponding trajectory that significantly improves driving performance in terms of driving safety and traveling efficiency as compared to alternative methods. Additionally, we implement the closed-loop simulation in CARLA, and the results showcase the effectiveness of the proposed planner to adapt to changing driving situations with high computational efficiency.

Recommended citation: W. Liu, H. Liu, L. Zheng, Z. Huang, and J. Ma, "Synergizing Decision Making and Trajectory Planning Using Two-Stage Optimization for Autonomous Vehicles," IEEE Transactions on Vehicular Technology, vol. 74, no. 4, pp. 5489-503, 2025. https://arxiv.org/abs/2411.18974

Enhance Planning with Physics-informed Safety Controller for End-to-end Autonomous Driving

Published in 2024 IEEE International Conference on Robotics and Biomimetics (ROBIO 2024), 2024

Tags: autonomous driving, motion planning and control

Recent years have seen a growing research interest in applications of Deep Neural Networks (DNN) on autonomous vehicle technology. The trend started with perception and prediction a few years ago and it is gradually being applied to motion planning tasks. Despite the performance of networks improve over time, DNN planners inherit the natural drawbacks of Deep Learning. Learning-based planners have limitations in achieving perfect accuracy on the training dataset and network performance can be affected by out-of-distribution problem. In this paper, we propose FusionAssurance, a novel trajectory-based end-to-end driving fusion framework which combines physics-informed control for safety assurance. By incorporating Potential Field into Model Predictive Control, FusionAssurance is capable of navigating through scenarios that are not included in the training dataset and scenarios where neural network fail to generalize. The effectiveness of the approach is demonstrated by extensive experiments under various scenarios on the CARLA benchmark.

Recommended citation: H. Zhou, H. Liu, H. Lu, J. Ma, and Y. Ji, "Enhance planning with physics-informed safety controller for end-to-end autonomous driving," in Proceedings of IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, 2024, pp. 1-8. https://arxiv.org/abs/2405.00316

Parallel optimization with hard safety constraints for cooperative planning of connected autonomous vehicles

Published in 2024 IEEE International Conference on Robotics and Automation (ICRA 2024), 2024

Tags: autonomous driving, motion planning and control

The development of connected autonomous vehicles (CAVs) facilitates the enhancement of traffic efficiency in complicated scenarios. In unsignalized roundabout scenarios, difficulties remain unsolved in developing an effective and efficient coordination strategy for CAVs. In this paper, we formulate the cooperative autonomous driving problem of CAVs in the roundabout scenario as a constrained optimal control problem, and propose a computationally-efficient parallel optimization framework to generate strategies for CAVs such that the travel efficiency is improved with hard safety guarantees. All constraints involved in the roundabout scenario are addressed appropriately with convex approximation, such that the convexity property of the reformulated optimization problem is exhibited. Then, a parallel optimization algorithm is presented to solve the reformulated optimization problem, where an embodied iterative nearest neighbor search strategy to determine the optimal passing sequence in the roundabout scenario. It is noteworthy that the travel efficiency in the roundabout scenario is enhanced and the computation burden is considerably alleviated with the innovation development. We also examine the proposed method in CARLA simulator and perform thorough comparisons with a rule-based baseline and the commonly used IPOPT optimization solver to demonstrate the effectiveness and efficiency of the proposed approach.

Recommended citation: Z. Huang, H. Liu, S. Shen, and J. Ma, "Parallel optimization with hard safety constraints for cooperative planning of connected autonomous vehicles," in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2024, pp. 1-7. https://arxiv.org/abs/2303.03090

Improved consensus ADMM for cooperative motion planning of large-scale connected autonomous vehicles with limited communication

Published in IEEE Transactions on Intelligent Vehicles, 2024

Tags: autonomous driving, motion planning and control

This paper investigates a cooperative motion planning problem for large-scale connected autonomous vehicles (CAVs) under limited communications, which addresses the challenges of high communication and computing resource requirements. Our proposed methodology incorporates a parallel optimization algorithm with improved consensus ADMM considering a more realistic locally connected topology network, and time complexity of $\mathcal{O}(N)$ is achieved by exploiting the sparsity in the dual update process. To further enhance the computational efficiency, we employ a lightweight evolution strategy for the dynamic connectivity graph of CAVs, and each sub-problem split from the consensus ADMM only requires managing a small group of CAVs. The proposed method implemented with the receding horizon scheme is validated thoroughly, and comparisons with existing numerical solvers and approaches demonstrate the efficiency of our proposed algorithm. Also, simulations on large-scale cooperative driving tasks involving up to 100 vehicles are performed in the high-fidelity CARLA simulator, which highlights the remarkable computational efficiency, scalability, and effectiveness of our proposed development. Demonstration videos are available at https://henryhcliu.github.io/icadmm_cmp_carla.

Recommended citation: H. Liu, Z. Huang, Z. Zhu, Y. Li, S. Shen, and J. Ma, "Improved consensus ADMM for cooperative motion planning of large-scale connected autonomous vehicles with limited communication," IEEE Transactions on Intelligent Vehicles, vol. 9, no. 11, pp. 7222-238, 2024. https://arxiv.org/abs/2401.09032

Incremental learning-based real-time trajectory prediction for autonomous driving via sparse Gaussian process regression

Published in 2024 IEEE Intelligent Vehicles Symposium (IV 2024), 2024

Tags: autonomous driving

In the context of spatial-temporal autonomous driving, the accurate and real-time trajectory prediction of the surrounding vehicle (SV) is crucial. This paper aims to design an efficient, accurate, and interpretable unimodal trajectory prediction approach. To achieve this objective, we employ Sparse Gaussian Process Regression (SGPR), which enables large dataset learning and efficient inference of future trajectories. This approach ensures accurate predictions while maintaining high computational efficiency. To further enhance the robustness of the prediction module, we propose the translation and rotation transformation strategy, which effectively simplifies the prediction problem. {Additionally, we utilize an instant evaluation algorithm to assess the prediction performance and maintain a streaming dataset for incremental learning, capable of adapting to dynamic driving environments.} In our experimental evaluation, we compare our proposed trajectory prediction approach with a series of existing methods. The results demonstrate that our work achieves superior prediction accuracy while requiring less inference time. It is noteworthy that, the proposed SGPR-based trajectory prediction approach with rotation equivalence is able to swiftly infer and incrementally learn from dynamic environments, which makes it a promising tool for enhancing safety and efficiency in autonomous driving systems.

Recommended citation: H. Liu, K. Chen, and J. Ma, "Incremental learning-based real-time trajectory prediction for autonomous driving via sparse Gaussian process regression," in Proceedings of Intelligent Vehicles Symposium(IV), IEEE, 2024, pp. 1-7. https://ieeexplore.ieee.org/document/10588687

Integrated behavior planning and motion control for autonomous vehicles with traffic rules compliance

Published in 2023 IEEE International Conference on Robotics and Biomimetics (ROBIO 2023), 2023

Tags: autonomous driving, motion planning and control

In this article, we propose an optimization-based integrated behavior planning and motion control scheme, which is an interpretable and adaptable urban autonomous driving solution that complies with complex traffic rules while ensuring driving safety. Inherently, to ensure compliance with traffic rules, an innovative design of potential functions (PFs) is presented to characterize various traffic rules related to traffic lights, traversable and non-traversable traffic line markings, etc. These PFs are further incorporated as part of the model predictive control (MPC) formulation. In this sense, high-level behavior planning is attained implicitly along with motion control as an integrated architecture, facilitating flexible maneuvers with safety guarantees. Due to the well-designed objective function of the MPC scheme, our integrated behavior planning and motion control scheme is competent for various urban driving scenarios and able to generate versatile behaviors, such as overtaking with adaptive cruise control, turning in the intersection, and merging in and out of the roundabout. As demonstrated from a series of simulations with challenging scenarios in CARLA, it is noteworthy that the proposed framework admits real-time performance and high generalizability.

Recommended citation: H. Liu, K. Chen, Y. Li, Z. Huang, J. Duan, and J. Ma, "Integrated behavior planning and motion control for autonomous vehicles with traffic rules compliance," in Proceedings of IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, 2023, pp. 1-7. https://ieeexplore.ieee.org/document/10354858/

Real-time parallel trajectory optimization with spatiotemporal safety constraints for autonomous driving in congested traffic

Published in The 26th IEEE International Conference on Intelligent Transportation Systems (ITSC 2023), 2023

Tags: autonomous driving

Multi-modal behaviors exhibited by surrounding vehicles (SVs) can typically lead to traffic congestion and reduce the travel efficiency of autonomous vehicles (AVs) in dense traffic. This paper proposes a real-time parallel trajectory optimization method for the AV to achieve high travel efficiency in dynamic and congested environments. A spatiotemporal safety module is developed to facilitate the safe interaction between the AV and SVs in the presence of trajectory prediction errors resulting from the multi-modal behaviors of the SVs. By leveraging multiple shooting and constraint transcription, we transform the trajectory optimization problem into a nonlinear programming problem, which allows for the use of optimization solvers and parallel computing techniques to generate multiple feasible trajectories in parallel. Subsequently, these spatiotemporal trajectories are fed into a multi-objective evaluation module considering both safety and efficiency objectives, such that the optimal feasible trajectory corresponding to the optimal target lane can be selected. The proposed framework is validated through simulations in a dense and congested driving scenario with multiple uncertain SVs. The results demonstrate that our method enables the AV to safely navigate through a dense and congested traffic scenario while achieving high travel efficiency and task accuracy in real time.

Recommended citation: L. Zheng, R. Yang, Z. Peng, H. Liu, M. Y. Wang, and J. Ma, "Real-time parallel trajectory optimization with spatiotemporal safety constraints for autonomous driving in congested traffic," in Proceedings of 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), IEEE, 2023, pp. 1186-1193. https://arxiv.org/abs/2309.05298