In the third and final approach, the most successful pre-processing and registration method, or combined methods, for the tested image datasets were connected and the 3D reconstruction final step was added to the workflow.
In the pre-processing stage the previously tested methods were subjected to an efficiency test, but to a different segmentation technique, the stain deconvolution method, explained in more detail further in this section. The method implemented for this first step was the CLAHE but with a difference in the algorithm presented in the section 3.2.1., since only the red channel was subjected to the histogram equalization. This was performed due to the high influence of the red channel in both Hematoxylin and Eosin stain, proving, through the segmentation, to be the best stain discriminant method. The image was also subjected to a FIR filter (filters with finite impulse response) convolution to remove line artifacts or deformations on the analyzed slice (section 3.1.).
Image Registration
Intensity- based registration Type: Rigid and Affine Feature- based registration Type: Similarity Demon algorithm Type: non- rigid B-spline registration Type: non- rigid Combination of two registration modelsPre-Processing
1. YCbCr transformation (with normalization) 2. CLAHE-redFigure 24. Schematic model representing the second workflow implemented in the study. In the first step,
the most successful pre-processing method was applied to the image dataset, followed by the image registration step, with several methods being tested.
The stain deconvolution technique developed for this study was based on the work carried out by (Ruifrok and Johnston 2001, Chan 2013) and also from (Unpublished 2015). The stain deconvolution is a segmentation method that is capable of deconvolve a stained image (for example with H&E) and generate separate images, each one with the distribution of a single stain (Hematoxylin, Eosin and Background, in the case of the tested datasets). This method assumes that the chemicals employed to stain the tissues slides follow the Beer-Lambert Law of absorption (11):
𝐼 = 𝐼𝑜𝑒−𝑆𝐷 (11)
where 𝐼 is the RGB image, 𝐼𝑜 is the intensity of the source, 𝐷 the stain density map and 𝑆 the
stain absorption matrix. This equation grants a logarithmic relation between the original RGB color channels and the stain intensities. The implemented algorithm starts by converting the RGB image 𝐼 into the optical density place (OD) – 𝐼𝑂𝐷 - through the previous equation in the
form (12):
𝐼𝑂𝐷= −log( 𝐼
𝐼𝑜) (12)
After the OD conversion, the image pixels are projected into the Maxwellian chromaticity plane. In the Maxwellian plane the distance between two points is proportional to their chromatic dissemblance (Maxwell and Zaidi 1993), or within this study, the pixels that belong to the same stain are projected closer than the pixels belonging to different stains. To calculate the pair of corresponding Maxwellian coordinates (𝑚1, 𝑚2) for each pixel 𝑝, the following equation (13) was implemented:
[𝑚1 𝑚2] = [ 0.7071 −0.7071 0 −0.4082 −0.4082 0.8165] 𝑝 ‖𝑝‖1 (13)
The described Maxwellian plane property and the pair of Maxwellian coordinates allows the implementation of an unsupervised classifier, for example kmeans, to assign a label for each pixel according to its stain. With the classification performed, the following step is the Stain Matrix estimation. For this study, it is considered the existence of three stains or classes, the Hematoxylin (class 1), Eosin (class 2) and the Background (class 3). The Maxwellian coordinates of the mean value in each of these classes - 𝜇𝑐= (𝜇𝑚1𝑐, 𝜇𝑚2𝑐), obtained through
the kmeans classification, is converted back to the OD space through the equations (14) exposed below:
𝑆𝑐,𝑟= 0.7071𝜇𝑚1𝑐− 0.4082𝜇𝑚2𝑐+ 0.33
𝑆𝑐,𝑔= −0.7071𝜇𝑚1𝑐− 0.4082𝜇𝑚2𝑐+ 0.33 (14)
𝑆𝑐,𝑏= 1 − 𝑆𝑐,𝑟− 𝑆𝑐,𝑔
where 𝑆𝑐,𝑟 is the red channel value for the stain vector of the class 𝑐, 𝑆𝑐,𝑔 the green channel
value and 𝑆𝑐,𝑏 the blue channel counterpart. The stain vector is generated for each class 𝑐 by
𝑆𝑐 = [𝑆𝑐,𝑟, 𝑆𝑐,𝑔, 𝑆𝑐,𝑏] 𝑇
and B stains, respectively. The stain density map 𝐷 is obtained through the inverse of the stain matrix as demonstrated in equation (15):
𝐷 = 𝑆−1𝐼
𝑂𝐷 (15)
The created stain matrix is then applied to the OD image to each pixel to obtain a final three channel image, composed by images from the three different stains, Hematoxylin, Eosin and Background. The stain images are separated and the first two, the relevant stains, suffer a contrast enhancement process. The contrast enhancement method embedded in the stain deconvolution framework performs a mapping of intensity values in grayscale images, so that the output data (image) presents a saturation of 1 % in high and low intensities, thus an increased contrast in intensities (MathWorks).
The registration method implemented was the automatic intensity-based registration
algorithm (section 3.2.2.) with transform type rigid and performed taking as constant fixed
image the reference slice of the image dataset (the central slice), and as moving images all the slices successively, from the first to the last in the dataset, aligning them to the reference and stacking them to form volumetric data.
To execute the 3D reconstruction and display were implemented using two different frameworks. The algorithms tested were the Marching Cubes algorithm (Lorensen and Cline 1987) implemented by (Hammer 2011) and the Isosurface framework from (MathWorks), adapted for the image datasets tested in this study.
The Marching Cubes algorithm creates polygonal surface representations of isosurfaces (Lorensen and Cline 1987) of 3D scalar fields, such as the slice stack formed after the registration complete process over all the dataset images. The implemented method consists in a vectorized version of the algorithm, which computes a triangulated mesh of the isosurface within a given 3D matrix of scalar values, according to the isosurface value (the constant value or level set from the 3D image represented in the isosurface), and specified in terms of a face and vertex list. The orientation of the triangles generated is selected according to the normal point from higher to lower values (Hammer 2011). This method enables the visualization of the generated 3D surface, with customizable colors and view.
The Isosurface framework from (MathWorks) also performs the extraction of the isosurface data from volumetric according to an isosurface value, although with a different process. This method performs first a volumetric interpolation using linear interpolation with (MathWorks), and then connects the points with equal isosurface value, forming a three dimensional surface. This method is combined with a 3D data smoothing technique (MathWorks) to smooth the edges of the reconstructed surface and a 2D polygon filling method (MathWorks) to enable its coloring and visualization.
3D Reconstruction
MarchingCubes Isosurface
Pre-Processing
CLAHE in red channelRegistration
Intensity-based rigid registration with reference sliceSegmentation
Stain deconvolution Reference Slice New image Registered Image is savedFigure 25. Schematic model representing the third and final workflow implemented in the study. The
approach starts by computing a loop for each slice in the dataset (minus the reference middle slice) in which the image suffers pre-processing, segmentation through stain deconvolution and registration with the reference slice as fixed image. All the registered images are stored and stacked to form the 3D data dataset, which will, posteriorly, serve as input of both 3D reconstruction frameworks.
Chapter 4
Results and Discussion
In this chapter, all the experimental results obtained with the three approaches described in the methodology (Chapter 3) and their respective discussion are presented. The pre- processing results will be the first to be analyzed, based on color contrast enhancement and segmentation results, followed by registration frameworks comparison and evaluation (second approach). Finally, the final workflow will be discussed, finishing with the 3D reconstruction results. All the selected parameters and decisions performed are properly indicated and justified, throughout the entire chapter. The workflows in this study were tested on a Windows 64bit Intel Core i7-4700HQ CPU at 2.40 GHz, with 8 GB of RAM computer system.