Surgical Workflow-Recognition

Automatic detection of surgical phases in the OR to improve and facilitate the surgical procedure

Modern assistance systems in the operating room should not only make individual steps easier, but also understand the course of an operation in its entirety. The automatic recognition of surgical phases aims to identify the current progress of an operation. Such systems provide contextual knowledge about the current phase of the procedure and thus enable intelligent assistance functions, such as the targeted provision of instruments or automated surgical documentation.

Multimodal phase detection in surgery using fused image data from in-room and laparoscopic cameras using different gallbladder phantoms
Multimodal phase detection in surgery using fused image data from in-room and laparoscopic cameras using different gallbladder phantoms

At the IMT, we are researching how artificial intelligence methods can be applied to video data from real operations in order to develop models that reliably and robustly classify surgical procedures - even with limited data. The aim is to use such systems to increase safety in the OR, reduce the cognitive load on the OR team and create the basis for automated assistance in the future.

Visualization of various XAI methods for interpreting an AI model during Callot triangle dissection. The heatmaps are superimposed on the original image, with red marking particularly relevant and blue less relevant image areas from the model's perspective.
Visualization of various XAI methods for interpreting an AI model during Callot triangle dissection. The heatmaps are superimposed on the original image, with red marking particularly relevant and blue less relevant image areas from the model's perspective.
Original image from cholecystectomy (left) with variants based on different data augmentation methods, including photometric, geometric and AI-generated changes.
Original image from cholecystectomy (left) with variants based on different data augmentation methods, including photometric, geometric and AI-generated changes.
To the top of the page