Creating Hyperrealistic Neuroanatomical Volumetric Models with Artificial Intelligence: A Fusion of Photogrammetry Concepts and Instant NeRF

Introduction: Neuroanatomical volumetric models acquired with surface scanning have emerged as a powerful tool in surgical neuroanatomy education, enabling detailed and immersive representations of complex structures. This technical note presents the use of previously documented photogrammetry concepts in combination with Instant NeRF (Neural Radiance Fields) for acquiring hyperrealistic neuroanatomical models.

Methods: Twenty-four datasets generated with a local photogrammetry workflow, acquired from 2017 to 2023, and comprising an average of 128 high-resolution photos of anatomical dissections in the skull base and cerebrovascular laboratory at UCSF, were processed as NeRF 3D scenes with instant neural graphic primitives (NGP) (NVIDIA, Santa Clara, California, United States) (local computing) and Luma AI (Palo Alto, California, United States) (cloud computing; [Fig. 1]). Upon successful 3D reconstruction, cinematic renders of the reconstructions were generated using original applications and a 3D render engine (Unreal Engine 5.2 Cari, North Carolina, United States). Additionally, the outputs obtained with NGP were explored using a virtual reality headset (HTC Vive, New Xindian District, Taipei City, Taiwan).

Fig. 1

Results: All datasets were successfully processed and converted into NeRF scenes. NeRF reconstructions represent a significant advancement in 3D modeling technology, employing deep learning algorithms to generate a 3D representation from an image dataset. NeRF then uses this representation to estimate the model’s appearance from various angles and under varying light intensities, resulting in the creation of hyperrealistic 3D models capable of depicting anatomical structures and surgical environments with an accurate representation of the original lightning. This novel technique is the first approach to utilizing artificial intelligence for the improvement of 3D reconstructions of surface anatomy and elevates the resources for surgical planning and anatomical education to an unprecedented level.

Conclusion: The combination of photogrammetry concepts and Instant NeRF enables the creation of hyperrealistic neuroanatomical models with significant potential in surgical neuroanatomy education and training. These models offer enhanced visuospatial understanding, virtual preservation of anatomical specimens, and immersive learning experiences. Further advancements in image acquisition, algorithm optimization, interactivity, and comprehensive evaluations are necessary to fully harness the benefits of this technology. By addressing these challenges, the field can advance toward utilizing hyperrealistic neuroanatomical models to improve surgical education, enhance surgical planning, and, ultimately, improve patient outcomes.

Publikationsverlauf

Artikel online veröffentlicht:
05. Februar 2024