During laparoscopic liver resection, the limited access to the organ, the small field of view and lack of palpation can obstruct a surgeon’s workflow. Automatic navigation systems could use the images from preoperative volumetric organ scans to help the surgeons find their target (tumors) and risk-structures (vessels) more efficiently. This requires the preoperative data to be fused (or registered) with the intraoperative scene in order to display information at the correct intraoperative position.
One key challenge in this setting is the automatic estimation of the organ’s current intra-operative deformation, which is required in order to predict the position of internal structures. Parameterizing the many patient-specific unknowns (tissue properties, boundary conditions, interactions with other tissues, direction of gravity) is very difficult. Instead, this work explores how to employ deep neural networks to solve the registration problem in a data-driven manner. To this end, convolutional neural networks are trained on synthetic data to estimate an organ’s intraoperative displacement field and thus its current deformation. To drive this estimation, visible surface cues from the intraoperative camera view must be supplied to the networks. Since reliable surface features are very difficult to find, the networks are adapted to also find correspondences between the pre- and intraoperative liver geometry automatically. This combines the search for correspondences with the biomechanical behavior estimation and allows the networks to tackle the full non-rigid registration problem in one single step. The result is a model which can quickly predict the volume deformation of a liver, given only sparse surface information. The model combines the advantages of a physically accurate biomechanical simulation with the speed and powerful feature extraction capabilities of deep neural networks.
To test the method intraoperatively, a registration pipeline is developed which constructs a map of the liver and its surroundings from the laparoscopic video and then uses the neural networks to fuse the preoperative volume data into this map. The deformed organ volume can then be rendered as an overlay directly onto the laparoscopic video stream. The focus of this pipeline is to be applicable to real surgery, where everything should be quick and non-intrusive. To meet these requirements, a SLAM system is used to localize the laparoscopic camera (avoiding setup of an external tracking system), various neural networks are used to quickly interpret the scene and semi-automatic tools let the surgeons guide the system.
Beyond the concrete advantages of the data-driven approach for intraoperative registration, this work also demonstrates general benefits of training a registration system preoperatively on synthetic data. The method lets the engineer decide which values need to be known explicitly and which should be estimated implicitly by the networks, which opens the door to many new possibilities.:1 Introduction
1.1 Motivation
1.1.1 Navigated Liver Surgery
1.1.2 Laparoscopic Liver Registration
1.2 Challenges in Laparoscopic Liver Registration
1.2.1 Preoperative Model
1.2.2 Intraoperative Data
1.2.3 Fusion/Registration
1.2.4 Data
1.3 Scope and Goals of this Work
1.3.1 Data-Driven, Biomechanical Model
1.3.2 Data-Driven Non-Rigid Registration
1.3.3 Building a Working Prototype
2 State of the Art
2.1 Rigid Registration
2.2 Non-Rigid Liver Registration
2.3 Neural Networks for Simulation and Registration
3 Theoretical Background
3.1 Liver
3.2 Laparoscopic Liver Resection
3.2.1 Staging Procedure
3.3 Biomechanical Simulation
3.3.1 Physical Balance Principles
3.3.2 Material Models
3.3.3 Numerical Solver: The Finite Element Method (FEM)
3.3.4 The Lagrangian Specification
3.4 Variables and Data in Liver Registration
3.4.1 Observable
3.4.2 Unknowns
4 Generating Simulations of Deforming Organs
4.1 Organ Volume
4.2 Forces and Boundary Conditions
4.2.1 Surface Forces
4.2.2 Zero-Displacement Boundary Conditions
4.2.3 Surrounding Tissues and Ligaments
4.2.4 Gravity
4.2.5 Pressure
4.3 Simulation
4.3.1 Static Simulation
4.3.2 Dynamic Simulation
4.4 Surface Extraction
4.4.1 Partial Surface Extraction
4.4.2 Surface Noise
4.4.3 Partial Surface Displacement
4.5 Voxelization
4.5.1 Voxelizing the Liver Geometry
4.5.2 Voxelizing the Displacement Field
4.5.3 Voxelizing Boundary Conditions
4.6 Pruning Dataset - Removing Unwanted Results
4.7 Data Augmentation
5 Deep Neural Networks for Biomechanical Simulation
5.1 Training Data
5.2 Network Architecture
5.3 Loss Functions and Training
6 Deep Neural Networks for Non-Rigid Registration
6.1 Training Data
6.2 Architecture
6.3 Loss
6.4 Training
6.5 Mesh Deformation
6.6 Example Application
7 Intraoperative Prototype
7.1 Image Acquisition
7.2 Stereo Calibration
7.3 Image Rectification, Disparity- and Depth- estimation
7.4 Liver Segmentation
7.4.1 Synthetic Image Generation
7.4.2 Automatic Segmentation
7.4.3 Manual Segmentation Modifier
7.5 SLAM
7.6 Dense Reconstruction
7.7 Rigid Registration
7.8 Non-Rigid Registration
7.9 Rendering
7.10 Robotic Operating System
8 Evaluation
8.1 Evaluation Datasets
8.1.1 In-Silico
8.1.2 Phantom Torso and Liver
8.1.3 In-Vivo, Human, Breathing Motion
8.1.4 In-Vivo, Human, Laparoscopy
8.2 Metrics
8.2.1 Mean Displacement Error
8.2.2 Target Registration Error (TRE)
8.2.3 Champfer Distance
8.2.4 Volumetric Change
8.3 Evaluation of the Synthetic Training Data
8.4 Data-Driven Biomechanical Model (DDBM)
8.4.1 Amount of Intraoperative Surface
8.4.2 Dynamic Simulation
8.5 Volume to Surface Registration Network (V2S-Net)
8.5.1 Amount of Intraoperative Surface
8.5.2 Dependency on Initial Rigid Alignment
8.5.3 Registration Accuracy in Comparison to Surface Noise
8.5.4 Registration Accuracy in Comparison to Material Stiffness
8.5.5 Champfer-Distance vs. Mean Displacement Error
8.5.6 In-vivo, Human Breathing Motion
8.6 Full Intraoperative Pipeline
8.6.1 Intraoperative Reconstruction: SLAM and Intraoperative Map
8.6.2 Full Pipeline on Laparoscopic Human Data
8.7 Timing
9 Discussion
9.1 Intraoperative Model
9.2 Physical Accuracy
9.3 Limitations in Training Data
9.4 Limitations Caused by Difference in Pre- and Intraoperative Modalities
9.5 Ambiguity
9.6 Intraoperative Prototype
10 Conclusion
11 List of Publications
List of Figures
Bibliography
| Identifer | oai:union.ndltd.org:DRESDEN/oai:qucosa:de:qucosa:79370 |
| Date | 02 June 2022 |
| Creators | Pfeiffer, Micha |
| Contributors | Speidel, Stefanie, Gumhold, Stefan, Stoyanov, Danail, Technische Universität Dresden |
| Source Sets | Hochschulschriftenserver (HSSS) der SLUB Dresden |
| Language | English |
| Detected Language | English |
| Type | info:eu-repo/semantics/publishedVersion, doc-type:doctoralThesis, info:eu-repo/semantics/doctoralThesis, doc-type:Text |
| Rights | info:eu-repo/semantics/openAccess |
Page generated in 0.003 seconds