The simulator developed and reported here is the first described MR endoscopic simulator using the HL for ETNS. The concept of this simulator involves MR and VR being combined as one application. Actually, the view from the tip of the replica of the endoscope is a VR image, and the VR image projection combined with MR tracks creates the realistic feedback as an MR simulator. The HL is a stand-alone head-mounted display that contains four environmental cameras, a depth camera, and Windows OS system. Thus, this device enables one to immerse in the MR environment, with hands free and a wireless connection. Long Qian et al. evaluated which choice among HL, ODG R-7, and Epson Moverio BT-200 was the most accessible for medical use as a head-mounted display. They concluded the HL best performed among the three [10]. Among these head-mounted displays, the HL’s advantage is its versatility.
One of the most important advantages of computer simulators for surgical training is the opportunity they afford for independent learning [10]. Most of the commercially available simulators so far have proved costly because of their complex computer hardware and software and are not easily accessible or convenient [11]. Our simulator needs only the HL, two MR markers, the replica of the endoscope, and the training model, which are all easily portable, so they can be used anywhere, at any time. In the process of creating our MR, all software was free and could be created inexpensively. Additionally, creating an application was easier with the Unity engine than with other programming systems, and after the application was created as an asset, functional options could be added to it. For example, in this study, MR markers and the replica of the endoscope were among the functional options.
European Association of Endoscopic Surgeons guidelines for validation of virtual reality surgical simulators specify that they must be able to mimic visual–spatial and real-time characteristics of the procedure, and preferably provide realistic haptic feedback [12]. A previous study of a VR simulator showed that the face validity was assessed in three separate domains: visual appearance (photorealism), haptic feedback, and user-friendliness [13]. The simulator developed and described in this report provides user-friendliness with versatility. We consider that this simulator achieved photorealism, which is demonstrated by the participants’ improvement in understanding of anatomical structures. Regarding the haptic feedback, commercially available ETNS simulators created a sense of reality as judged by haptic feedback[14] [15, 16]. On the other hand, our developed simulator has no haptic feedback function for surgical manipulation, such as mucosal dissection, drilling bony structures, and tumor removal. This is a disadvantage of our simulator compared with the commercially available simulators. However, in our simulator, the projection of the MR to the real model could create interactive feedback. The endoscopic training model created a limitation of motion and real contact resistance for the replica of the endoscope. It is not necessary to use the endoscopic training model as we described; we could also apply whatever would reproduce the restriction of the nasal cavity. Additionally, for this study, we decided it was more important to simulate a limited operative field of view with a narrow surgical corridor, rather than to reproduce surgical manipulation like a surgical trainer. There was an innate learning curve associated with VR itself for previous VE simulators, because of using the unnatural haptic device [11]. Because we would like to assess whether there is a learning curve with our simulator, we analyzed the scores of validities in the first half term and the second half term of assessment. There was a statistically significant difference in each group. There was also an innate learning curve for manipulation of the replica of the endoscope and understanding the VR surgical fields.
The HL has problems with processing speed and the device’s viewing angle. A stable MR in the HoloLens is expressed by 60 frames per second (FPS). The HL has a limit to the number of polygons that could display without FPS delay. If the 3D model has more than 100,000 polygons, the processing speed becomes slow, and there is FPS delay. This causes loss of orientation of the MR marker and a time lag between operation and display. To avoid these factors, a minimum segmentation of 3D model was required; however, this might take more time. Furthermore, the reconstructed 3D image of the paranasal sinus from CT parameters applicable in clinical routine data was inefficient [17]. However, cutting off polygons causes a loss of reproducibility of MR images and might result in the face validity being adversely affected. As with other wearable computers, there is some potential for nausea and vertigo based on the predisposition of the users. These points are disadvantages of our simulator.
MR simulator for the future
Our concept for the MR simulator would apply intraoperative real-time navigation for the ETNS. However, the HL continuously uses spatial mapping to provide us with stable MR imaging. Spatial mapping means creation of a spatial surface mesh of surroundings and adaptation of MR images to that mesh. However, the spatial mesh is often too coarse to merge AR with real objects precisely. Vuforia supports the stability of MR, but it remains in need of improvement. Thus, at this point, the accuracy of registration and tracking with the HL was not good enough to use for neuro-navigation, especially in microscopic and endoscopic neurosurgery.
MR technology is making remarkable progress. In fact, we had used the HL since 2017, and in the beginning, the projection of MR and MR tracking were unstable, with delays and mismatch. The setup was found to be problematic for medical application. But the continuous update of software and SDK gradually improved these issues. Additionally, some developers published usage tips and pitfalls on the Internet. There is strength in using a versatile game engine and Microsoft production. The disadvantages that we have reported as limitations can be improved over time. This development and improvement may also provide MR navigation with the HL that satisfies microscopic or endoscopic neurosurgeons.