Publications

LMMCoDrive: Cooperative Driving with Large Multimodal Model

Published in arXiv, 2024

To address the intricate challenges of decentralized cooperative scheduling and motion planning in Autonomous Mobility-on-Demand (AMoD) systems, this paper introduces LMMCoDrive, a novel cooperative driving framework that leverages a Large Multimodal Model (LMM) to enhance traffic efficiency in dynamic urban environments. This framework seamlessly integrates scheduling and motion planning processes to ensure the effective operation of Cooperative Autonomous Vehicles (CAVs). The spatial relationship between CAVs and passenger requests is abstracted into a Bird’s-Eye View (BEV) to fully exploit the potential of the LMM. Besides, trajectories are cautiously refined for each CAV while ensuring collision avoidance through safety constraints. A decentralized optimization strategy, facilitated by the Alternating Direction Method of Multipliers (ADMM) within the LMM framework, is proposed to drive the graph evolution of CAVs. Simulation results demonstrate the pivotal role and significant impact of LMM in optimizing CAV scheduling and enhancing decentralized cooperative optimization process for each vehicle. This marks a substantial stride towards achieving practical, efficient, and safe AMoD systems that are poised to revolutionize urban transportation. The code is available at https://github.com/henryhcliu/LMMCoDrive.

Recommended citation: H. Liu, R. Yao, Z. Huang, S. Shen, and J. Ma, “LMMCoDrive: Cooperative Driving with Large Multimodal Model,” arXiv preprint arXiv: 2409.11981, 2024. https://arxiv.org/abs/2409.11981

Robot Navigation in Unknown and Cluttered Workspace with Dynamical System Modulation in Starshaped Roadmap

Published in arXiv, 2024

Compared to conventional decomposition methods that use ellipses or polygons to represent free space, starshaped representation can better capture the natural distribution of sensor data, thereby exploiting a larger portion of traversable space. This paper introduces a novel motion planning and control framework for navigating robots in unknown and cluttered environments using a dynamically constructed starshaped roadmap. Our approach generates a starshaped representation of the surrounding free space from real-time sensor data using piece-wise polynomials. Additionally, an incremental roadmap maintaining the connectivity information is constructed, and a searching algorithm efficiently selects short-term goals on this roadmap. Importantly, this framework addresses deadend situations with a graph updating mechanism. To ensure safe and efficient movement within the starshaped roadmap, we propose a reactive controller based on Dynamic System Modulation (DSM). This controller facilitates smooth motion within starshaped regions and their intersections, avoiding conservative and short-sighted behaviors and allowing the system to handle intricate obstacle configurations in unknown and cluttered environments. Comprehensive evaluations in both simulations and real-world experiments show that the proposed method achieves higher success rates and reduced travel times compared to other methods. It effectively manages intricate obstacle configurations, avoiding conservative and myopic behaviors.

Recommended citation: K. Chen, H. Liu, Y. Li, J. Duan, L. Zhu, and J. Ma, “Robot navigation in unknown and cluttered workspace with dynamical system modulation in starshaped roadmap,” arXiv preprint arXiv: 2403.11484, 2024 https://arxiv.org/abs/2405.00316

Parallel optimization with hard safety constraints for cooperative planning of connected autonomous vehicles

Published in 2024 IEEE International Conference on Robotics and Automation (ICRA 2024), 2024

The development of connected autonomous vehicles (CAVs) facilitates the enhancement of traffic efficiency in complicated scenarios. In unsignalized roundabout scenarios, difficulties remain unsolved in developing an effective and efficient coordination strategy for CAVs. In this paper, we formulate the cooperative autonomous driving problem of CAVs in the roundabout scenario as a constrained optimal control problem, and propose a computationally-efficient parallel optimization framework to generate strategies for CAVs such that the travel efficiency is improved with hard safety guarantees. All constraints involved in the roundabout scenario are addressed appropriately with convex approximation, such that the convexity property of the reformulated optimization problem is exhibited. Then, a parallel optimization algorithm is presented to solve the reformulated optimization problem, where an embodied iterative nearest neighbor search strategy to determine the optimal passing sequence in the roundabout scenario. It is noteworthy that the travel efficiency in the roundabout scenario is enhanced and the computation burden is considerably alleviated with the innovation development. We also examine the proposed method in CARLA simulator and perform thorough comparisons with a rule-based baseline and the commonly used IPOPT optimization solver to demonstrate the effectiveness and efficiency of the proposed approach.

Recommended citation: Z. Huang, H. Liu, S. Shen, and J. Ma, “Parallel optimization with hard safety constraints for cooperative planning of connected autonomous vehicles,” in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2024, pp. 1-7. https://arxiv.org/abs/2303.03090

Enhance Planning with Physics-informed Safety Controller for End-to-end Autonomous Driving

Published in arXiv, 2024

Recent years have seen a growing research interest in applications of Deep Neural Networks (DNN) on autonomous vehicle technology. The trend started with perception and prediction a few years ago and it is gradually being applied to motion planning tasks. Despite the performance of networks improve over time, DNN planners inherit the natural drawbacks of Deep Learning. Learning-based planners have limitations in achieving perfect accuracy on the training dataset and network performance can be affected by out-of-distribution problem. In this paper, we propose FusionAssurance, a novel trajectory-based end-to-end driving fusion framework which combines physics-informed control for safety assurance. By incorporating Potential Field into Model Predictive Control, FusionAssurance is capable of navigating through scenarios that are not included in the training dataset and scenarios where neural network fail to generalize. The effectiveness of the approach is demonstrated by extensive experiments under various scenarios on the CARLA benchmark.

Recommended citation: H. Zhou, H. Liu, H. Lu, D. Xu, J. Ma, and Y. Ji, “Enhance planning with physics-informed safety controller for end-to-end autonomous driving,” arXiv preprint arXiv: 2405.00316, 2024 https://arxiv.org/abs/2405.00316

Improved consensus ADMM for cooperative motion planning of large-scale connected autonomous vehicles with limited communication

Published in IEEE Transactions on Intelligent Vehicles, 2024

This paper investigates a cooperative motion planning problem for large-scale connected autonomous vehicles (CAVs) under limited communications, which addresses the challenges of high communication and computing resource requirements. Our proposed methodology incorporates a parallel optimization algorithm with improved consensus ADMM considering a more realistic locally connected topology network, and time complexity of $\mathcal{O}(N)$ is achieved by exploiting the sparsity in the dual update process. To further enhance the computational efficiency, we employ a lightweight evolution strategy for the dynamic connectivity graph of CAVs, and each sub-problem split from the consensus ADMM only requires managing a small group of CAVs. The proposed method implemented with the receding horizon scheme is validated thoroughly, and comparisons with existing numerical solvers and approaches demonstrate the efficiency of our proposed algorithm. Also, simulations on large-scale cooperative driving tasks involving up to 100 vehicles are performed in the high-fidelity CARLA simulator, which highlights the remarkable computational efficiency, scalability, and effectiveness of our proposed development. Demonstration videos are available at https://henryhcliu.github.io/icadmm_cmp_carla.

Recommended citation: H. Liu, Z. Huang, Z. Zhu, Y. Li, S. Shen, and J. Ma, “Improved consensus ADMM for cooperative motion planning of large-scale connected autonomous vehicles with limited communication,” IEEE Transactions on Intelligent Vehicles, vol. 0, no. 0, pp. 1-16, 2024. https://arxiv.org/abs/2401.09032

Incremental learning-based real-time trajectory prediction for autonomous driving via sparse Gaussian process regression

Published in 2024 IEEE Intelligent Vehicles Symposium (IV 2024), 2024

In the context of spatial-temporal autonomous driving, the accurate and real-time trajectory prediction of the surrounding vehicle (SV) is crucial. This paper aims to design an efficient, accurate, and interpretable unimodal trajectory prediction approach. To achieve this objective, we employ Sparse Gaussian Process Regression (SGPR), which enables large dataset learning and efficient inference of future trajectories. This approach ensures accurate predictions while maintaining high computational efficiency. To further enhance the robustness of the prediction module, we propose the translation and rotation transformation strategy, which effectively simplifies the prediction problem. {Additionally, we utilize an instant evaluation algorithm to assess the prediction performance and maintain a streaming dataset for incremental learning, capable of adapting to dynamic driving environments.} In our experimental evaluation, we compare our proposed trajectory prediction approach with a series of existing methods. The results demonstrate that our work achieves superior prediction accuracy while requiring less inference time. It is noteworthy that, the proposed SGPR-based trajectory prediction approach with rotation equivalence is able to swiftly infer and incrementally learn from dynamic environments, which makes it a promising tool for enhancing safety and efficiency in autonomous driving systems.

Recommended citation: H. Liu, K. Chen, and J. Ma, “Incremental learning-based real-time trajectory prediction for autonomous driving via sparse Gaussian process regression,” in Proceedings of Intelligent Vehicles Symposium(IV), IEEE, 2024, pp. 1-7. https://ieeexplore.ieee.org/document/10588687

Geometry-aware safety-critical local reactive controller for robot navigation in unknown and cluttered environments

Published in IEEE Robotics and Automation Letters, 2024

This work proposes a safety-critical local reactive controller that enables the robot to navigate in unknown and cluttered environments. In particular, the trajectory tracking task is formulated as a constrained polynomial optimization problem. Then, safety constraints are imposed on the control variables invoking the notion of polynomial positivity certificates in conjunction with their Sum-of-Squares (SOS) approximation, thereby confining the robot motion inside the locally extracted convex free region. It is noteworthy that, in the process of devising the proposed safety constraints, the geometry of the robot can be approximated using any shape that can be characterized with a set of polynomial functions. The optimization problem is further convexified into a semidefinite program (SDP) leveraging truncated multi-sequences (tms) and moment relaxation, which favorably facilitates the effective use of off-the-shelf conic programming solvers, such that real-time performance is attainable. Various robot navigation tasks are investigated to demonstrate the effectiveness of the proposed approach in terms of safety and tracking performance.

Recommended citation: Y. Li, X. Tang, K. Chen, C. Zheng, H. Liu, and J. Ma, “Geometry-aware safety-critical local reactive controller for robot navigation in unknown and cluttered environments,” IEEE Robotics and Automation Letters, vol. 9, no. 4, pp. 3419 - 3426, 2024. http://ieeexplore.ieee.org/document/10417140

Integrated behavior planning and motion control for autonomous vehicles with traffic rules compliance

Published in 2023 IEEE International Conference on Robotics and Biomimetics (ROBIO 2023), 2023

In this article, we propose an optimization-based integrated behavior planning and motion control scheme, which is an interpretable and adaptable urban autonomous driving solution that complies with complex traffic rules while ensuring driving safety. Inherently, to ensure compliance with traffic rules, an innovative design of potential functions (PFs) is presented to characterize various traffic rules related to traffic lights, traversable and non-traversable traffic line markings, etc. These PFs are further incorporated as part of the model predictive control (MPC) formulation. In this sense, high-level behavior planning is attained implicitly along with motion control as an integrated architecture, facilitating flexible maneuvers with safety guarantees. Due to the well-designed objective function of the MPC scheme, our integrated behavior planning and motion control scheme is competent for various urban driving scenarios and able to generate versatile behaviors, such as overtaking with adaptive cruise control, turning in the intersection, and merging in and out of the roundabout. As demonstrated from a series of simulations with challenging scenarios in CARLA, it is noteworthy that the proposed framework admits real-time performance and high generalizability.

Recommended citation: H. Liu, K. Chen, Y. Li, Z. Huang, J. Duan, and J. Ma, “Integrated behavior planning and motion control for autonomous vehicles with traffic rules compliance,” in Proceedings of IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, 2023, pp. 1-7. https://ieeexplore.ieee.org/document/10354858/

Real-time parallel trajectory optimization with spatiotemporal safety constraints for autonomous driving in congested traffic

Published in The 26th IEEE International Conference on Intelligent Transportation Systems (ITSC 2023), 2023

Multi-modal behaviors exhibited by surrounding vehicles (SVs) can typically lead to traffic congestion and reduce the travel efficiency of autonomous vehicles (AVs) in dense traffic. This paper proposes a real-time parallel trajectory optimization method for the AV to achieve high travel efficiency in dynamic and congested environments. A spatiotemporal safety module is developed to facilitate the safe interaction between the AV and SVs in the presence of trajectory prediction errors resulting from the multi-modal behaviors of the SVs. By leveraging multiple shooting and constraint transcription, we transform the trajectory optimization problem into a nonlinear programming problem, which allows for the use of optimization solvers and parallel computing techniques to generate multiple feasible trajectories in parallel. Subsequently, these spatiotemporal trajectories are fed into a multi-objective evaluation module considering both safety and efficiency objectives, such that the optimal feasible trajectory corresponding to the optimal target lane can be selected. The proposed framework is validated through simulations in a dense and congested driving scenario with multiple uncertain SVs. The results demonstrate that our method enables the AV to safely navigate through a dense and congested traffic scenario while achieving high travel efficiency and task accuracy in real time.

Recommended citation: L. Zheng, R. Yang, Z. Peng, H. Liu, M. Y. Wang, and J. Ma, “Real-time parallel trajectory optimization with spatiotemporal safety constraints for autonomous driving in congested traffic,” in Proceedings of 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), IEEE, 2023, pp. 1186-1193. https://arxiv.org/abs/2309.05298

Bio-inspired robot swarm path formation with local sensor scope

Published in Applied Intelligence, 2022

Creating a robot network to connect the targets while exploring an unknown map is the goal of the path formation issue. It is difficult to maximize the efficiency of exploring a multi-target maze by a robot swarm with limited sensor capabilities. In this study, a novel behavior-based path formation approach (BPFM) that incorporates an artificial potential field and a bio-model inspired by slime mold is proposed for this problem. The robot’s controller can operate more sensibly while using the heuristic term from particle swarm. In order to maintain a dynamic multi-source network, a series of mechanisms and transition rules have been designed for the multi-target maze. Grid maps with obstacle density from sparse to dense are utilized in simulations to compare the proposed method with other algorithms. The results indicate that the performance of collective exploration, which is examined in diverse circumstances, is unexpectedly efficient and robust.

Recommended citation: Y. Zhao, Z. Qu, H. Liu, and R. Zhu, “Bio-inspired robot swarm path formation with local sensor scope,” Applied Intelligence, vol. 53, no. 1, pp. 17310-17326, 2022. https://link.springer.com/article/10.1007/s10489-022-04356-9

Solving a Multi-robot Search Problem with Bionic Sarsa Algorithm and Artificial Potential Field

Published in 2021 China Automation Congress (CAC), 2021

Safe and effective path planning of multiple combat vehicles engaged in antagonistic environments keeps a challenging problem. Based on the application background of multi-robots in cooperative reconnaissance of enemy camps in environment with traps, this paper studies the multi-agent path planning based on bionic algorithms and artificial potential field method. The proposed bionic PP-AP Sarsa Scheme is inspired by food-finding scheme of Physarum Polycephalum (PP), which can effectively solve the dimensional explosion problem of traditional multi-agent reinforcement learning methods. This paper first studies the single-agent bionic planning problem with the PP algorithm to initialize the Q table used in Sarsa-based reinforcement learning, which effectively reduces the search space and accelerates the convergence speed of the early stage of reinforcement learning. After the Q tables in the same map are obtained through the training of different single agents, the Q tables of every agents are extended to multi-agents scenario by the assistance of simplified artificial potential field, hence a composite parallel path planner named RL-APCP 3 is constructed to synchronously update the actions of all of the agents, which allows us to complete the coordinated and efficient search of enemy camps by multiple agents. Compared with the Sarsa path planning algorithm of single agent, the efficiency of this scheme is improved up to 55.22%.

Recommended citation: H. Liu, Z. Qu and R. Zhu, "Solving a Multi-robot Search Problem with Bionic Sarsa Algorithm and Artificial Potential Field," 2021 China Automation Congress (CAC), Beijing, China, 2021, pp. 1830-1835. https://ieeexplore.ieee.org/document/9728613

Research on Robot Visual Grabbing Based on Mechanism Analysis

Published in 2021 IEEE 11th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), 2021

This paper mainly studies the problem of grasping objects by manipulator based on vision, and a model-based visual grabbing strategy is proposed. Compared with the existing classical image processing methods including the Sobel operator edge extraction method, the superiority of the corrosion operation edge extraction method used in this strategy has been verified, through several fruit image processing experiments. In order to solve the lack of sufficient number of labeled object recognition samples required by machine learning methods, a model-based image classifier is also established, which is based on artificially extracted object features. Hence, it can be interpreted strongly and does not require training using a large number of data samples. Finally, a visual robot grabbing experiment has been constructed and carried out. The results show that efficiency and accuracy of the image recognition algorithm are proved, and this algorithm is efficient, light and interpretable.

Recommended citation: H. Liu, Z. Liu, H. Liu and W. Lin, "Research on Robot Visual Grabbing Based on Mechanism Analysis," 2021 IEEE 11th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), Jiaxing, China, 2021, pp. 181-186. https://ieeexplore.ieee.org/document/9588176

Haichao Liu

Publications