An End-to-End Motion Planner Using Sensor Fusion for Autonomous Driving

Autonomous driving vehicles and advanced driver-assistance systems are gaining tremendous attention with the hope of providing a new transportation mode that is more convenient and ensures road safety. Different types of sensors are deployed together with the aid of deep learning (DL) techniques to help the vehicle perceive the surrounding environment and navigate toward the destination. In this paper, we implemented a deep learning-based motion planner using sensor fusion from LiDAR point clouds and camera RGB images to predict future waypoints. The model is trained in an end-to-end manner in which input are the multimodal sensor data, and output is the predicted future waypoints. A transformer module with a self-attention mechanism is used to integrate the representation of the two sensor modalities. During training, auxiliary tasks including depth estimation and bird-eye-view semantic segmentation are carried out to provide an intermediate representation of the perception process as well as to enhance the performance of the motion planning task. Experimental results obtained from different model configurations on the Longest6 benchmark have shown that our proposed model achieves competitive performance compared to baselines.