Master thesis: Surgical tool instance segmentation based on deep learning for minimally invasive surgery

Minimally Invasive Surgery (MIS) has revolutionized modern medical practice, offering significant patient benefits such as reduced pain and faster recovery. However, the reliance on endoscopic visualization presents considerable challenges for surgeons, particularly in precisely identifying and tracking surgical instruments. This constraint directly impacts surgical precision, patient safety, and the efficacy of Computer-Assisted Surgery (CAS) and robotic-assisted procedures. Addressing this critical need, this study focuses on developing advanced deep learning models for robust, real-time surgical tool instance segmentation, aiming to enhance intraoperative guidance and objective skill assessment in MIS environments.

Researchers and practitioners in medical imaging, deep learning, computer vision, robotics, and minimally invasive surgery, as well as medical professionals involved in surgical training and computer-assisted interventions.

This study tackles the pressing challenge of accurately detecting and segmenting surgical instruments in real-time within Minimally Invasive Surgery (MIS) environments. Recognizing the limitations of current methods, the research employs a structured experimental approach to optimize YOLOv8 and YOLOv11 deep learning models, crucial for enhancing surgical precision and patient safety. Key architectural modifications include integrating GhostConvolutions and Depthwise Convolution (DWConv) for efficiency, alongside Mish and GELU activation functions to boost learning capabilities and non-linearity. Leveraging the M2CAI16-Tool dataset, the methodology balances detection accuracy with computational performance. Findings demonstrate that YOLOv11-DWConv achieves substantial parameter reduction (26%) while maintaining competitive detection accuracy (mAP@0.5: 0.906), making it highly suitable for resource-constrained settings. Conversely, YOLOv11-GELU excels in detection accuracy (mAP@0.5: 0.910) due to its enhanced localization. Crucially, both models achieve real-time inference speeds (up to 81 FPS), validating their practical applicability for intraoperative guidance. Beyond real-time assistance, the generated instance segmentation data proves invaluable for objective surgical skill assessment by analyzing instrument usage patterns, thereby supporting personalized training and standardizing surgical quality. This research significantly advances AI-driven surgical instrument recognition, offering optimized solutions that enhance safety, efficiency, and evaluation in MIS, ultimately leading to improved surgical workflows and patient outcomes.