I. MARCO TEÓRICO
3. Sustento Teórico
3.4 Nivel A2 (Pre – Intermedio) según el Marco Común Europeo de
of 27 points randomly moved in a vicinity of 10 pixels, the interpolation was recovered using Thin Plate Splines (TPS) [Bookstein 1989]. Registration was performed between an undeformed CT image and a TPS deformed PET image. Two measures of the reg- istration error were considered. First, we show the mean of the absolute intensity error between the undeformed image and the image recovered through registration. Second, we warped randomly distributed points in the image with the TPS deformation, then applied the recovered transformation to these points, the second measure of the error is the mean distance between the undeformed points and the recovered points.
Figure 5.11 depicts the two registration error measures as function of the harmonic energy. Each curve represents a different method and consists of 15 points, averaged over the 4 patients (this yields 60 experiments for one single curve). Comparison is made with Normalized Mutual Information which is commonly used in the multi-modal alignment of PET and CT.
Figure5.12shows the fused images before and after registration, in an alignment made using multi modal boosted maximum margin (CMBMM).
5.7
Conclusion
In this Chapter, we presented two very novel approaches for metric learning towards multi- modal image fusion. The first approach, Cross Modality Similarity Sensitive Hashing is a generalization of similarity-sensitive hashing to multi-modal data. To the best of our knowledge, this is the first attempt to approach the challenging problem of cross-modality similarity learning as an embedding problem. We showed that using cross-modality sim- ilarity learning allows to efficiently perform alignment of medical images acquired with different modalities. While in retrieval applications the Hamming embedding is advanta- geous due to its low computational and storage complexity and easy integration into exist- ing database managements systems, the Hamming metric is discrete-valued and involves a non-differentiable non-linearity.
This is why we developed our second approach that extends large margin component analysis to deal with the multi-modal case and adopts boosted max margin concepts. The resulting metric is continuous, differentiable and is computationally efficient. Further- more, it seems to inherit strong discrimination power and outperforms other learning- based methods. Further improvement of the method such as the use of convex criteria to determine the embeddings and the metric could also add theoretical stability in the pro- cess. The use of non-linear data assumptions can be easily encoded in the process through kernel-based methods and is currently under investigation. Last but not least, the use of context through local interactions between observations could make the process more ro- bust and help differentiate between cases where similar features are observable in different
5.7. CONCLUSION 105
Figure 5.3: Creation of a common embedding space: features are extracted from a set of two perfectly aligned images, this feautres are each embeded in different spaces X and Y . Using similarity sensitive hashing we aim to lear two projection functions f and g that will map the elements from X and Y respectively into a common space H in which elements that were labeled as similar in the training set are embedded close to each other (red circle) while dissimlar pairs are embeded as far away as possible. The dimension of the embedding space H is a parameter of the algorithm
Figure 5.4: Distance map: plot of the learned distance taken between the feature vector extracted in the red square position on the left on the T1-MRI and all of the feature vectors extracted on the corresponding co-registered T2-MRI image. Far right is a profile extracted on the same line as the reference position. Bottom row presents the less distinctive case, notice the 15 voxels neighborhood around the extraction position.
1 0 1 0 1 0 1 0 1 0 1 0
Figure 5.5: Distance map: plot of the learned distance taken between the feature vector extracted in the red square position on the left on the T1-MRI and all of the feature vectors extracted on the corresponding co-registered T2-MRI image. Far right is a profile extracted on the same line as the reference position.
5.7. CONCLUSION 107 1 2 3 4 5 6 7 8 9 10 11 0.3 0.32 0.34 0.36 0.38 0.4 0.42 0.44 0.46 LMCA BoostedMetric
Figure 5.6: Evolution of the Equal Error Rate (EER) with the iterations of the alternate minimization for the PET CT dataset
0 2 4 6 8 10 12 14 16 0.2 0.21 0.22 0.23 0.24 0.25 0.26 LMCA BoostedMetric
Figure 5.7: Evolution of the Equal Error Rate (EER) with the iterations of the alternate minimization for the T1-MRI T2-MRI dataset
CM-SSH MOE-MRF Ma-Margin LMCA MaxMargin BMM NMI Unimodal CR
Figure 5.8: Evolution of the difference in dice coefficient as a function of the harmonic energy, each single curve factors in 100 experiments, the solid line curve represents the average dice coefficient increase while the whiskers ends represent the minimum and max- imum increase in the dice coefficient. Here is presented the case of T1 to PD MRI registra- tion, presented are the results with Normalized mutual information (NMI), Unimodal Cor- relation Ration (Unimodal CR), Mixture of experts with MRF (MOE-MRF), our Cross- modality similarity sensitive hashing (CM-SSH), our two adapted measures Corss Modal Max margin Boosted Max MArgin (Max-Margin BMM), and with LMCA (Max-Margin LMCA)
5.7. CONCLUSION 109
MOE MRF
Unimodal BMM
CM-SSH
CM-BMM
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.1 0.12 0.14 0.16 0.18 0.2 0.22 0.24 0.26 NMI 0.28 0.3CM-LMCA
Figure 5.9: Evolution of the difference in dice coefficient as a function of the harmonic energy, each single curve factors in 100 experiments, the solid line curve represents the average dice coefficient increase while the whiskers ends represent the minimum and max- imum increase in the dice coefficient. Here is presented the case of T1 to T2 MRI regis- tration, presented are the results with Normalized mutual information (NMI), Unimodal Bosted Max Margin (Unimodal BMM) which represents the Metric Learning ideal case, Mixture of experts with MRF (MOE-MRF), our Cross-modality similarity sensitive hash- ing (CM-SSH), our two adapted measures Corss Modal Max margin Boosted Max Margin (CM BMM), and with LMCA (CM-LMCA)
Figure 5.10: Sample of the registration results obtained for T1-T2 registration with Cross modality similarity sensitive hashing. Top row: Source Image T1-MRI image. Second Row: target T2-MRI image. Third Row: deformed image after multi-modal deformable registration. Bottom Row: left, deformation field of the registration, right, checker-board image between the target and the deformed source.
5.7. CONCLUSION 111 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 0.11 NMI CMSSH CMBMM CMLMCA 0.02 0.04 0.06 0.08 0.1 0.12 NMI CM SSH CM BMM CM LMCA 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0 0.5 1 1.5 2 2.5 3 3.5 NMI CMSSH CMBMM CMLMCA 0.5 1 1.5 2 2.5 3 3.5 NMI CM SSH CM BMM CM LMCA
Figure 5.11: Error measure as a function of the Harmonic Energy, our methods are de- noted as CM-BMM and CM-LMCA. Top row: mean absolute difference of the images. Bottom row: mean distance between undeformed points and points after transformation recovery
Figure 5.12: Sample from the PET CT registration data set. Registration was performed here with multi modal boosted maximum margin (CMBMM). Top row: fused images before registration, Bottom row: after registration
Chapter 6
Markov Random Field Training for
Image Registration
The previously discussed similarity metrics (Chapter 4and5) both suffer from the same drawback. In order to learn the similarity metric we need to have at our disposition a data set of perfectly registered images. This predicament is usually quite seldom in clinical cases, where simultaneous acquisition is usually infeasable.
This fact led us to try and relax this constraint and work by matching organs bound- aries. The registration process we have in mind involves detecting the boundaries of an organ and use this information as the driving force of the registration.
However boundary detection itself is already a challenge in the case of medical images. In this work we use a training data set of pairs of images that are not co-registered and we ask to also have manual segmentations of the organs of interest. Using these segmentations we learn probability distributions on the organs by boosting (section6.1), and use Markov Random Fields training (section6.3) to learn the correct amount of boundary smoothing to apply locally to get organ boundaries as close to the manual segmentation as possible.
In a sense the algorithm we present here performs concurrent segmentation and regis- tration of the organs. Concurrent segmentation and registration has first been investigated in the neuroimaging community [Ashburner 1997]. Recent work include [Xiaohua 2004] where a Maximum a Posteriori Model is computed to take into account both segmentation and registration, [Gooya 2011] where Expectation maximization is used to incorporate in- formation of tumor growth in the registration process, in [Lu 2011] where bayseian model of both registration and segmentation are learned and then assembled using a conditional modeling. All these approaches deal with inefficient bayesian modeling of the interactions which tend to be slow especially with large data such as medical images. Recently was proposed an interesting approach [Parisot 2012b] that uses a two leveled Markov Random Field to model the segmentation. One level controls the segmentation and the other con-
trols the registration. This method yields significantly faster performance. While being closer to this last work our method however lays the emphasis on the registration part, and even subpar segmentation results can lead to decent registration performances.