Research

An enhanced VR medical training system

The proposed enhanced VR medical training system will embody an Intelligent Digital Surgeon (IDS) within MAGES virtual OR, supported by social awareness and perceptual function, that identifies the trainee’s behavioral model by recognizing and analyzing the trainee’s hand/arm gestures and provides personalized assessment, real-time feedback, instructions, and recommendations. Based on an optimal behavioral model, the IDS will present the training module’s actions using the correct gestures.

Medical-specific operations will be performed using physics-based 3D representation of deformable soft body objects (skin, tissue, etc.) to achieve high-realism and immersion. The proposed system will adopt a distributed software architecture taking advantage of edge-cloud resources and 5G networking to enable ultra-low latency untethered VR.

Solution

Creating such an IDS involves two main aspects:

1. Creation and animation of a Cognitive and Intelligent Digital Surgeon

In this project, we will use a virtual assistant for acting as a skilled digital surgeon. We will model its gestures and behavior according to the situation. This is quite new and to our knowledge, no one has provided the presence of the digital surgeon while the trainee does a surgery. In addition, the intelligent digital surgeon will be able to teach the trainee to do the right gestures.

2. Analysis of the trainee’s hand/arm gestures

3D hand pose estimation is one of the core technologies for human computer interaction in virtual reality and augmented reality applications since this technology provides a natural way for users to interact with virtual environments and virtual objects.

Novelties

The main novelty of our solution is to define a VR medical training system with the presence of an embodied Intelligent Digital Surgeon (IDS). The IDS is a cognitive agent tightly coupled with the trainee within the virtual environment, supported by social awareness and perceptual functions.

Its main role is to build the trainee’s behavioral profile, by recognizing and analyzing the trainee’s hand/arm gestures while performing the training module’s actions, and providing personalized assessment, real-time feedback, instructions, and recommendations. Based on an optimal behavioral profile, the IDS will be able to present the surgical module’s sequence of actions with the correct hand/arm gestures.