Behavioral Uncertainty-Aware Attention Allocation via VLMs for Interactive Autonomous Driving

To navigate interactive traffics with limited onboard computational resources, an autonomous vehicle (AV) must selectively allocate its attention to the surrounding vehicles (SVs) that pose the greatest impact on their future motion. Existing attention allocation approaches fail to fully account for the SVs’ behavioral uncertainty, leading to their poor performance in complex interactive scenarios. To address these issues, this paper proposes a behavioral uncertainty-aware attention allocation method via vision-language models (VLMs) for autonomous driving. Behavioral uncertainty is first categorized into micro-dimensional cooperativeness as well as macro-dimensional intentions, and the interactive behaviors of the SVs are predicted with comprehensive consideration of such uncertainties. Subsequently, the behavioral uncertainty data, combined with multimodal information of the current situation and attention allocation memories of historical situation, is elaborately arranged in the designed template as the prompt for VLMs. Leveraging the multimodal information in the prompt, VLMs are required to conduct situation understanding, explicitly analyze the impact of the SVs on the AV’s motion, and allocate attention to the SVs that pose real threats to the AV’s motion. Comprehensive results demonstrate that the proposed approach outperforms baselines in success rates across a wide range of complex interactive scenarios. Furthermore, our method maintains remarkable resilience and exhibits excellent robustness as traffic density increases.

Recommended citation: B. Ma, H. Liu, R. Zhong, P. Liu, X. Zhou, and J. Ma, "Behavioral Uncertainty-Aware Attention Allocation via VLMs for Interactive Autonomous Driving," IEEE Transactions on Vehicular Technology, vol. 0, no. 0, pp. 1-13, 2026. https://ieeexplore.ieee.org/abstract/document/11608941/