Enhancing User Acceptance in Autonomous Vehicles: Integrating Scene Understanding and Emotion Recognition

Autonomous vehicles (AVs) have the potential to transform transportation by improving road safety and enhancing mobility. However, widespread user acceptance remains a significant challenge due to persistent concerns about trust, system transparency, and the lack of human-centered communication. Existing AV systems often fail to provide real-time, interpretable explanations of their actions, particularly during unexpected or abrupt maneuvers. Furthermore, passenger discomfort is frequently overlooked, resulting in reduced user confidence and engagement. To address these challenges, this paper presents an integrated framework that combines hierarchical VisionLanguage Model (VLM)-based scene understanding with passenger emotion recognition to improve AV decision transparency. A structured question-answering pipeline interprets the driving environment, while an in-cabin emotion recognition module monitors passenger reactions. When a negative emotional response is detected during a critical driving event, such as an emergency stop, the system generates a natural language explanation to clarify the AV's behavior. Experimental results from the test scenarios demonstrate that the proposed framework is feasible for generating real-time explanations and detecting passenger emotional responses during critical driving events. The findings highlight the importance of affective feedback in advancing human-centered AV systems.