Spatial representation of human poses and object locations via AI-driven LiDAR-vision image fusion

Situational awareness is crucial for effective decision-making during human-robot collaboration (HRC), particularly in industrial environments. This study introduces a method for generating real-time spatial representations of human poses and object locations by integrating AI-driven image analysis with point cloud data. RGB camera images are utilized to extract visual information, including object classifications, regions of interest, and human poses, via object detection and pose estimation. Through a frustum-based clustering approach to segment point clouds, objects and human poses are represented by volumetric bounding boxes and geometric skeleton models, respectively. This is achieved by reprojecting planar joint data into the spatial domain and aligning it with corresponding human point cloud data. The effectiveness of the proposed method in enhancing situational awareness for HRC is validated through a pick-and-place task. Thus, by facilitating real-time task monitoring, the proposed method can improve safety and efficiency in collaborative robotics and smart-manufacturing environments.