Intraoperative surgical photoacoustic microscopy (IS-PAM) using augmented reality

We have developed an intraoperative surgical photoacoustic microscopy (IS-PAM) system by integrating an optical resolution photoacoustic microscopy (OR-PAM) and conventional surgical microscope. Based on the common optical path in the OR-PAM and microscope system, we can acquire the PAM and microscope images at the same time. Furthermore, by utilizing a mini-sized beam projector, 2D PAM images are back-projected onto the microscope view plane as augmented reality. Thus, both the conventional microscopic and 2D cross-sectional PAM images are displayed on the plane through an eyepiece lens of the microscope. In our method, additional image display tool is not required to show the PAM image. Therefore, it potentially offers significant convenience to surgeons without movement of their sights during surgeries. In order to demonstrate the performance of our IS-PAM system, first, we successfully monitored needle intervention in phantoms. Moreover, we successfully guided needle insertion into mice skins in vivo by visualizing surrounding blood vessels from the PAM images and the magnified skin surfaces from the conventional microscopic images simultaneously.