3D Holographic Visualization for Pre-procedural Planning of Spatially Complex Brain Aneurysm Treatment

doi:10.21203/rs.3.rs-60213/v1

Download PDF

Research

3D Holographic Visualization for Pre-procedural Planning of Spatially Complex Brain Aneurysm Treatment

https://doi.org/10.21203/rs.3.rs-60213/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

The purpose of this study was to develop and demonstrate a novel imaging platform that post-processes volumetric imaging data (e.g. Computed Tomography (CT) or Magnetic Resonance angiography) to provide holographic visualization for pre-procedural treatment planning of a morphologically complex brain aneurysm. Digital CT images were segmented, using the Materialise Mimics software, into three-dimensional digital models that were imported into a prototype application for the Microsoft HoloLens. Feedback from testing the prototype indicated potential for augmented reality to assist an interventionalist in spatial understanding and depth perception of spatially complex anatomy and could increase confidence in pre-procedural planning. Future studies will be conducted with additional cases to further validate the utility of the platform in surgical planning decisions and to expand the platform for patient/resident education, telemedicine, and intra-operative use.

Nuclear Medicine & Medical Imaging

Augmented reality

brain aneurysm

pre-operative planning

visualization

holography

segmentation

Brain aneurysms (otherwise known as cerebral or intracranial aneurysms) occur when the wall of a cerebral artery is weakened, resulting in excessive bulging of the blood vessel. Upon rupture, a patient will be exposed to subarachnoid hemorrhaging (SAH), in which a stroke can occur. According to Johns Hopkins Medicine, the fatality rate for patients who experience SAH is around 46%, but can rise up to 80% if left untreated.³ Currently, the major treatment options for patients include endovascular therapies (including coiling and flow diversion devices) and surgical clipping. Endovascular coiling is a less-invasive treatment that involves a surgeon guiding a catheter from the patient’s groin to the location of the aneurysm, where they deploy detachable devices that block or redirect blood flow (as shown in Fig. 1a).⁴ Surgical clipping is a more invasive surgical treatment in which the surgeon gains access to the aneurysm site and blocks (i.e., occludes) blood flow by attaching a metal clip to the neck of the aneurysm (see Fig. 1b).⁵

Although these treatments are the current standard of care for hospitals across the United States, there are many problems that have been unaddressed. Current methodologies for endovascular therapy involve imaging with cerebral angiography, a two-dimensional (2D) imaging modality for an inherently three-dimensional (3D) problem. If an interventional neuroradiologist (INR) cannot get good visualization utilizing biplane angiography, there is risk of complications. Additionally, angiographic imaging (which is based on fluoroscopy) produces X-rays that pass (i.e., transmit) through the patient, exposing both the patient and the caregiver to radiation.⁶ To produce a high-quality image, INRs have to align the C-arm of the fluoroscopy machine at the correct angle co-axially to the tumor, which can prolong procedure time (especially for less experienced INRs). Furthermore, according to Pierot et. al, one of the main difficulties that was evident through endovascular coiling was complex aneurysm shape and location in proximity to the main cerebral artery.⁴

One major drawback to the surgical clipping method is that it is more invasive. When this procedure is performed, a part of the patient’s skull is removed for the surgeon to gain access to the aneurysm. With surgical clipping, exposure and visualization of the cerebral aneurysm, the parent artery, and important branches is also important. If care is not taken to understand the 3D anatomy, complications can also result. Overall, many of the issues that arise from current standard-of-care procedures are a result of spatially complex aneurysm anatomy and location, as well as determining what angle to image the patient from.

Augmented reality (AR) offers a solution to current standard-of-care issues by providing surgeons with 3D visualization of a patient’s intracranial vasculature. The proposed platform system utilizes processed patient-specific virtual models from Digital Imaging & Communications in Medicine (DICOM) imaging data that are segmented with Materialise Mimics software (Leuven, Belgium) and projected three-dimensionally in space for pre-operative planning. The use of an untethered head-mounted display (HMD), such as the Microsoft HoloLens (1st generation; Redmond, WA),⁷ has the potential to assist an INR in 3D spatial understanding of complex aneurysm shapes and the location of the aneurysm in relation to critical structures of the brain. The use of the HoloLens could also provide in-depth pre-procedural insight as to the morphology of the vasculature and the most efficient camera placements for angiography. The Microsoft HoloLens is a completely free-standing, untethered HMD for viewing augmented/mixed reality (AR/MR). It utilizes six degree-of-freedom positional head tracking and spatial mapping to enable the user to view and interact with 3D virtual images overlaid on the physical environment in the user’s field of view (F.O.V.).⁷ The HoloLens uses diffractive planar waveguides—which are fiber optic devices that transmit visual data to the user’s eyes—to allow for visualization of the 3D images.^{8, 9} In turn, this technology would have the potential to assist in eye-hand coordination as a result of enhanced 3D visualization, as well as the potential to benefit education for patients and less experienced surgeons. Furthermore, the use of AR could potentially decrease overall procedure time and radiation exposure to clinicians and patients, while simultaneously confirming pre-procedural insight (which will be determined in a future study).

The use of the Microsoft HoloLens in combination with registration techniques, such as Vuforia SDK markers (for tracking and alignment of virtual models with the physical world), has been studied and proven to be accurate, with a mean hologram drift of 1.41 mm in a study by Frantz et al.¹⁰ In a previous study at the Cleveland Clinic, similar AR-based technology exhibited potential to be able to assist interventional radiologists in the performance of abdominal tumor ablation procedures in both a clinical and pre-procedural setting.¹¹ The objective of this project is to evaluate the use of a novel prototype imaging system for assisting an INR in visualization and performance of pre-procedural planning for treatment of a morphologically complex brain aneurysm.

Sets of computed tomography (CT) data in DICOM imaging format (Digital Imaging and Communications in Medicine) were collected from a patient who was diagnosed with a cerebral aneurysm between the right internal carotid and the right-sided middle cerebral artery off of the Circle of Willis. The 2D image slices of the CT were segmented into 3D digital models using Materialise Mimics software for the skull and a section of the cerebral artery that encompassed the aneurysm (as shown in Figure 2). The skull was segmented through thresholding grayscale values that optimized reconstruction of the bone. As shown in Figure 3, a smooth and anatomically accurate model of the skull was created, including its thin structures. The walls of the model were slightly thickened to improve durability of a planned 3D-printed model of the patient’s skull and vasculature, without sacrificing any anatomical accuracy. A separate mask was added using arterial phase data. The cerebral arteries and aneurysm were segmented in a similar manner. Individual mask files were exported in stereolithographic (STL) file format and imported into MeshLab for decimation. The different meshes were decimated to 120k faces (for optimized hologram stability) and exported as 3D object (OBJ) files, as shown in Figure 3. From here, OBJs were imported into Unity Engine as assets, where an app for the Microsoft HoloLens was built with the ARToolKit. The app was built in Visual Studio 2019 and deployed to the Microsoft HoloLens.

The projected holograms are registered to a specific location in space using the Vuforia SDK, which is a proprietary image-processing software integration into Unity Engine (San Francisco, CA) that allows the projected holograms to be tracked in real-time to a physical image target.¹² The Vuforia engine utilizes the camera on the HoloLens to detect contrast-enhanced features of an image target within a user’s F.O.V. and compares them to the same image in a digital database (stored in the Unity app). Figure 4 exhibits the image target pattern used, with and without the feature detection. The difference between the physical and virtual allows the software to calculate the target’s position in HoloLens coordinates, and will project the virtual objects aligned to the image target. This results in hologram stability with little drift,¹³ even when the caregiver walks around the projection and views the holographic scene from various angles. A 3D-printed Vuforia image target was used to track the position of the head and to align the virtual model to the physical marker. An image target using a test cube can be seen in the Unity scene shown in Figure 5.

After initial prototyping, feedback was collected from an INR at the Cleveland Clinic to determine further features that would benefit both the clinicians and the patients pre- and intra-procedurally.

An initial prototype app was successfully built and deployed to the HoloLens. The prototype app consisted of the anatomical segmented digital models of the skull and a region of interest surrounding the aneurysm vasculature from Mimics that could be registered and tracked

with a Vuforia marker. One feature enabled in this prototype was a clipping plane, so that as the user moves their head closer to the holograms, the layers of the digital models are pealed back slice by slice – revealing an in-depth look at the inside of the skull.

The prototype app sets an origin based on the user’s position in the room when the app is started. It then registers the anatomical holograms to the physical Vuforia marker. The registration occurs when the user gazes at the image target on a flat surface from above. Once the holograms are registered to the marker, the user can move freely about the room within a five-meter radius of the app’s origin¹³ and view the holograms from different angles to understand the 3D nature of the aneurysm in relation to the skull and cerebral vasculature. One observation was that the registration was sometimes slow at the start, and occasional hologram “drift” would occur. Measurement of this accuracy and drift was outside the scope of this study, but there are many ways in which this can be improved (see DISCUSSION). Figure 6 shows screen captures from the HoloLens running the prototype app.

Upon evaluation of this prototype to an INR at the Cleveland Clinic, useful feedback was collected for both how it may assist doctors in the operating room and angiography suite, as well as potential future improvements that could improve the patient-caregiver experience. One comment on the prototype regarded what areas that this AR technology would help, specifically for 3D visualization with eye-hand coordination and spatial understanding, as well as for patient and resident education.

Feedback from a Cleveland Clinic caregiver regarding future improvements of the prototype involve the addition of a scale-up function, in which a voice command/user-interface (UI) could raise the holograms anteriorly and scale them by a factor of two. In addition, a zoom and rotate function with hand gestures could potentially help doctors see the anatomy up close to assist with pre-operative planning. Voice commands could also be used to toggle holograms on/off, which could further ease-of-use for the end-user.

For patient and resident education, it was suggested that possibly two HoloLenses could be utilized cooperatively so that patients could better understand the procedure and disease, and so that residents-in-training or younger surgeons would be able to walk through a complex procedure with an experienced surgeon – even remotely from different institutions.

In this project, an AR application for the Microsoft HoloLens was built to enable pre-procedural planning and spatial understanding of a cerebral aneurysm in relation to other critical structures of the brain. Anatomical digital models were built using segmented DICOM data in Materialise Mimics that were processed and incorporated into the Unity app as 3D digital models. While the prototype build was successful, there were many observations and feedback of features that could be improved or required future investigation.

One observation was that the registration of the anatomical holograms to the Vuforia image target was sometimes delayed, and the user experienced occasional hologram drift after the anatomical holograms were registered. This may have resulted because another user registered the anatomical holograms before the INR donned the HoloLens. When the HoloLens was passed to the INR, the inertial measurement unit (IMU) within the HMD may have needed recalibration/reregistration. This drift could potentially be avoided by holding a training session for those using the app with simple voice commands that could easily register the holograms to the desired location.

The question of accuracy (i.e. the accuracy of the registration of the virtual images to the patient’s real anatomy via the Vuforia tracking system) is important—especially in a clinical setting. The accuracy of the Vuforia SDK integration with the Microsoft HoloLens has been previously studied for a neuronavigational app, and is reported to have a mean hologram drift of 1.41 mm.¹⁰ While accuracy was outside the scope of this current study, these results could be verified through methods described by Frantz et al in a future study.¹⁰ Future accuracy measures could verify patient safety and pre-procedural plan confidence for INRs.

In a future study, the prototype app could be integrated with the second generation of the Microsoft HoloLens (HoloLens 2). The HoloLens 2 has many upgraded features, such as an increased field-of-view, a more ergonomic and comfortable design, and real-time eye, hand, and head tracking (as opposed to the first generation HoloLens, head-tracking only).⁷

Furthermore, this early feasibility project lends to plenty of future work in which the prototype could be built out to examine feasibility for patient/surgeon education, remote collaboration between caregivers, or even intra-operative use with the anatomical holograms projected directly to the patient. While the current feasibility study worked directly with one INR at the Cleveland Clinic, a standard system usability survey with a Likert-style questionnaire (1 = strongly disagree, 5 = strongly agree) could collect data in the future from many clinicians to better shape the needs of the end-user and functionality of the prototype application.¹⁴

Overall, this evaluation provides a first step towards development of an augmented reality imaging platform for neuronavigational procedures, such as cerebral aneurysms of complex morphologies. The development of a prototype app for pre-procedural planning and spatial understanding of complex anatomy showcases the potential for AR technology to benefit clinicians in many ways. Some benefits of this technology include 3D visualization of the anatomy to visualize locations of an aneurysm in relation to critical structures of the cerebral vasculature and skull base, as well as providing insight for a clinical plan during the pre-operative phase.

This project opens the door to various future studies that would build out the platform to be a multi-functional application that could be used not only pre-operatively, but also possibly intra-operatively to guide surgical decisions, alignment of the C-arm for imaging, and to act as an adjunct to standard-of-care methodologies. Future work to further validate the utility of the platform in surgical planning decisions can provide clinicians with the confidence to implement this novel technology into their daily procedures.

computed tomography

three-dimensional

SAH

subarachnoid hemorrhaging

two-dimensional

INR

interventional neuroradiologist

augmented reality

HMD

head-mounted display

mixed reality

F.O.V.

field-of-view

DICOM

Digital Imaging and Communications in Medicine

STL

stereolithographic

OBJ

3D object file

IMU

Inertial Measurement Unit [of the HoloLens]

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

Competing interests

C.W., A.H., S.A., S.H., and J.Y. declare that they have no competing interests.

Funding

All components of this project were funded internally at the Cleveland Clinic through departmental funds provided by the Cerebrovascular Center.

Authors’ Contributions

CW performed the background research, segmented the patient data, built the AR app, received feedback from an INR, and drafted the manuscript. AH & SA provided key feedback throughout segmentation and building the prototype app—and they also substantially revised the manuscript. JY provided guidance on building the prototype app, received feedback from an INR, and made substantial revisions to the manuscript. SH provided feedback and suggestions for the prototype app and made significant contribution to the manuscript. All authors read and approved the final manuscript.

Acknowledgements

We would like to thank Karl West of the Holographic Interventional Therapy (HIT) Lab at the Cleveland Clinic Lerner Research Institute for his support, resources, time, and help throughout this project.

Endovascular Coiling. Mayo Clinic; [cited 2020 April 29]. Available from: https://www.mayoclinic.org/diseases-conditions/brain-aneurysm/multimedia/aneurysm-coil/img-20007617.
3D-Printed Replica of Brain Aneurysm Helps Guide Surgical Repair Consult QD. Cleveland Clinic; [cited 2020 April 29]. Available from: https://consultqd.clevelandclinic.org/3d-printed-replica-of-brain-aneurysm-helps-guide-surgical-repair/.
Brain Aneurysms C, Center. Johns Hopkins Medicine; [cited 2020 April 20]. Available from: https://www.hopkinsmedicine.org/neurology_neurosurgery/centers_clinics/cerebrovascular/conditions/aneurysms.html.
Pierot L, Wakhloo AK. Endovascular treatment of intracranial aneurysms: current status. Stroke. 2013;44(7):2046–54. doi:10.1161/STROKEAHA.113.000733. PubMed PMID: 23798560. Epub 2013/06/27.
Microsurgical Clipping for Brain Aneurysms Health. Johns Hopkins Medicine; [cited 2020 April 29]. Available from: https://www.hopkinsmedicine.org/health/treatment-tests-and-therapies/microsurgical-clipping-for-brain-aneurysms.
Feygelman VM, Huda W, Peters KR. Effective Dose Equivalents to Patients Undergoing Cerebral Angiography. AJNR. 1992;13:845–9.
HoloLens 2: Microsoft; 2020 [cited 2020 March 24]. Available from: https://www.microsoft.com/en-us/hololens/hardware#.
HoloLens (1st gen) hardware Microsoft Documentation: Microsoft; 2019 [cited 2020 April 23]. Available from: https://docs.microsoft.com/en-us/hololens/hololens1-hardware.
Erdenebat M-U, Lim Y-T, Kwon K-C, Darkhanbaatar N, Kim N. Waveguide-Type Head-Mounted Display System for AR Application. State of the Art Virtual Reality and Augmented Reality Knowhow. 2018:41–58. doi: 10.5772/intechopen.75172.
Frantz T, Jansen B, Duerinck J, Vandemeulebroucke J. Augmenting Microsoft's HoloLens with Vuforia tracking for neuronavigation. Healthc Technol Lett. 2018;5(5):221–5. doi:10.1049/htl.2018.5079. PubMed PMID: 30464854; PMCID: PMC6222243. Epub 2018/11/23.
Al-Nimer S, Hanlon A, Cho K, Kalra-Lall A, Weunski C, Yanof J, West K, Martin C. 3rd. 3D Holographic Guidance and Navigation for Percutaneous Ablation of Solid Tumor. J Vasc Interv Radiol. 2020;31(3):526–8. doi:10.1016/j.jvir.2019.09.027. PubMed PMID: 32008844. Epub 2020/02/06.
Getting Started with Vuforia Engine in Unity; Vuforia Developer Library. Vuforia; [cited 2020 April 20]. Available from: https://library.vuforia.com/articles/Training/getting-started-with-vuforia-in-unity.html.
Hologram stability, Documentation M: Microsoft; 2018 [updated March 21, 2018; cited 2020 April 20]. Available from: https://docs.microsoft.com/en-us/windows/mixed-reality/hologram-stability.
Brooke J. SUS: A ‘quick and dirty’ usability scale 1996. Available from: https://hell.meiert.org/core/pdf/sus.pdf.

Download PDF

Editorial decision: Minor revision
10 Nov, 2020
Review #3 received at journal
09 Nov, 2020
Review #2 received at journal
07 Nov, 2020
Reviewer #3 agreed at journal
22 Oct, 2020
Review #1 received at journal
29 Sep, 2020
Reviewer #2 agreed at journal
22 Sep, 2020
Reviewers invited by journal
15 Sep, 2020
Reviewer #1 agreed at journal
15 Sep, 2020
Editor assigned by journal
19 Aug, 2020
First submitted to journal
18 Aug, 2020
Submission checks completed at journal
18 Aug, 2020
Editor invited by journal
18 Aug, 2020

You are reading this latest preprint version

3D Holographic Visualization for Pre-procedural Planning of Spatially Complex Brain Aneurysm Treatment

Status:

Version 1

Abstract

Figures

Background

Methods

Results

Discussion

Conclusions

Abbreviations

Declarations

References

Status:

Version 1