A sensor fusion system with thermal infrared camera and LiDAR for autonomous vehicles and deep learning based object detection

Vision, Radar, and LiDAR sensors are widely used for autonomous vehicle perception technology. Especially object detection and classification are primarily dependent on vision sensors. However, under poor lighting conditions, dazzling sunlight, or bad weather an object might be difficult to be identified with general vision sensors. In this paper, we propose a sensor fusion system that combines a thermal infrared camera and a LiDAR sensor that can reliably detect and identify objects even in environments with poor visibility, such as day or night. The proposed method obtains the external parameters of the two sensors by designing and manufacturing a 3D calibration target to externally calibrate the thermal infrared camera and the LiDAR sensor. To verify the performance, experiments were conducted in day and night environments. The proposed sensor system and fusion algorithm show that it can reliably detect and identify objects even in environments with poor visibility, such as day or night.