Deep Learning-Based Calibration Method for An Augmented Reality Surgical Navigation System without Head-mounted Optical Markers

2D medical visualization techniques often fall short in adequately representing complex 3D anatomical structures. 2D surgical navigation lacks depth information, which is a significant drawback. Additionally, displaying medical data on a 2D screen during surgery is suboptimal because it necessitates the surgeon to constantly shift their focus. Augmented Reality (AR) compensates for the significant drawback of 2D surgical navigation, which lacks depth information. AR is a technology that overlays computer-generated information onto the real world, providing users with an enhanced visual experience. By integrating digital information with the physical environment in real-time, AR offers more intuitive and useful information. Currently, research on surgical navigation using AR is actively progressing. This innovative technology is being explored and developed to enhance the precision, efficiency, and safety of surgical procedures. In this paper, we utilize a snapshot from the built-in forward camera of the OST-HMDs, capturing both virtual points and real marker balls, to automatically calculate the transformation matrix between the virtual and real world. This method requires precise positions of both the virtual points and the real markers to successfully overlay anatomical information onto the real world. we use the YOLOv8 model and Virtual Aruco Marker to precisely determine the positions of both real and virtual points, and to automatically identify their points, ensuring an enhanced AR Navigation System.