OmniScene: Attention-Augmented Multimodal 4D Scene Understanding for Autonomous Driving
Published in IEEE Transactions on Vehicular Technology, 2026
OmniScene combines vision-language modeling with spatial and temporal perception to support human-like scene understanding for autonomous driving. Its OmniVLM model processes observations from multiple viewpoints over time. A teacher-student framework transfers language-based knowledge into 3D instance features to capture attention cues relevant to driving behavior. A hierarchical fusion strategy adjusts the contributions of geometric and semantic features at different levels. Evaluations on nuScenes report improvements over competing methods in perception, prediction, planning, and visual question answering.
Recommended citation: P. Liu, H. Lu, H. Liu, H. Liu, X. Liu, R. Yao, S. E. Li, and J. Ma, "OmniScene: Attention-Augmented Multimodal 4D Scene Understanding for Autonomous Driving," IEEE Transactions on Vehicular Technology, accepted for publication, 2026. https://arxiv.org/abs/2509.19973