I close this chapter by presenting an application of WPC-based micropattern tracking. The micropattern is introduced to the optical train of the bright-field microscope to provide stage position information for non-motorized stages. Stage position information can be used to construct mosaic images from multiple frames of a microscopy video.
A complete tracking application would enable a microscopist to examine a specimen while a camera continuously acquires images. The stage position would be tracked, and
the acquired images would be assembled and presented to the microscopist with proper spatial context. The assembled images would enable the microscopist to examine regions larger than a single field of view at high magnification. This would provide a natural view of specimen structure, and may better enable a microscopist to understand the relationships between different regions of a specimen.
When moving a stage micrometer by hand, it is difficult to restrict motion to less than half the pattern wavelength in each direction (for the TEM grid micropattern, 12.5µm) for every frame of an image sequence. In these cases, WPC tracking is subject to an ambiguity of whole pattern distances. Image correlation can easily resolve this ambiguity for fixed specimens—image registration restricted to several whole-pattern displacements along each pattern axis provides a disambiguation of image pair alignments. WPC or MSC on the full-resolution image provides fine alignment.
Figure 5.22 shows an image mosaic assembled from 20 microscopy images using this approach. Images are formed on a Leica DM5000-B microscope with a 40X, 0.85NA ob- jective lens and a 1.2X tube lens. A SPOT Flex FX1520 camera acquires monochrome, 14-bit, 1024×1024 pixel images at 4 fps with 5µs exposures. The stage is moved contin- uously by hand during image acquisition. During tracking, normalized cross-correlation on images reduced to 256×256 pixel displaced by [−3. . .3] micropattern cycles dis- ambiguates among possible translations reported by pattern tracking. The microscopy mosaic image is constructed by merging all images into a common reference frame, over- writing data with each new frame (no blending is performed). The resulting mosaic is visually coherent, with some image seams visible upon close inspection. Improving the micropattern manufacturing process would improve the tracking accuracy, as discussed in Chapter 6, and would thereby improve the quality of microscopy mosaics assembled from WPC stage tracking information.
This example demonstrates that the WPC tracking technique provides reasonably reliable information in some situations. One would not usually use structured illumi-
Figure 5.22: A microscope mosaic image assembled from 20 images of octopus muscle tissue. The stage is moved by hand during image acquisition, and the micropattern is tracked using WPC and coarse image registration. Dotted lines show the size of a single image frame. Arrow points to a visible seam.
nation to create mosaics of the type seen in Figure 5.22. In fact, image registration on these images obtains a better alignment with fewer visible seams, and image blending techniques would improve the visual coherence of the mosaics even more [HPS08]. How- ever, WPC additionally handles sparse specimens, moving specimens, and z position tracking.
5.6
Discussion
In summary, this chapter has introduced a structured illumination approach for stage tracking in bright-field microscopy. A semitransparent pattern layer is introduced to the microscope’s light path, fixed to the specimen’s frame of reference. The pattern transforms the stage tracking problem into a multiple layer image analysis problem. A search for magnitude peaks at pattern frequencies in the frequency domain yields an estimate of pattern orientation that does not rely on knowing the pattern position. Two methods are proposed to track the pattern. Model-based spatial correlation (MSC) determines pattern translation using an image formation model to simulate multiple images of the pattern, and optimizing model parameters to find the best match to an observed image. Weighted phase correlation (WPC) computes a phase estimate for individual target frequencies in an image by first reweighting the image according to the target frequency’s wavelength, and then computing a single component of the discrete Fourier transform (DFT). Once the pattern’s position has been estimated, a model-based rotation optimization can provide an improved estimate of the pattern orientation.
The accuracy of the orientation and tracking methods were examined in detail. The frequency-based orientation method provides estimates with a 0.25◦ RMS error for an in focus square grid pattern. The model-based orientation method improves this estimate to 0.066◦ RMS error once the position of the pattern is determined.
accurate tracking for axis-aligned square grid patterns with no specimen present. Lateral tracking accuracy remains around 0.010 pixel (2.3 nm for the simulated microscope) RMS error in this case, falling to 0.057 pixel for arbitrary orientations. This accuracy is maintained down to approximately 18 dB SNR due to pattern transmission and below 10 dB due to defocus effects.
The WPC method relies on a phase estimate from individual pattern frequencies. Point sampling in the frequency domain requires infinite support in the spatial domain, and WPC attempts to obtain this by redistributing and reweighting the signal to analyze the target frequency. In the presence of uncorrelated noise, the WPC method maintains subpixel tracking accuracy. In the presence of correlated noise, however, the phase es- timate becomes unreliable, and tracking accuracy diminishes. Nevertheless, there are some practical imaging scenarios in which WPC may excel. For example, multimode microscopes capture fluorescence and bright-field images in quick succession [SSW+03]. For observations of fluorescent specimens that do not show up well in bright-field, one could use the bright-field channel to obtain tracking information from structured illu- mination. Because fluorescence microscopy uses epi-illumination (both the excitation and emission light travel through the objective lens), placing the micropattern below the specimen would cause no interference with the fluorescent imaging.
In comparison, MSC struggles to provide 0.25 pixel RMS error in the absence of rotation, though it seems to improve remarkably for many non-axis-aligned patterns. Uniform accuracy is maintained down to below 10 dB SNR due to pattern transmission and to 0 dB due to defocus effects.
The error obtained with both methods increases significantly on real microscopy images of the a micropattern grid. Tracking degrades even more in the presence of a specimen. However, the errors for experimental and simulated images with a specimen present agree quite well. This provides motivation to continue exploring pattern layer tracking in simulation.
For tracking contiguous, fixed specimens, one should use an image registration tech- nique; in the presence of sufficient specimen contrast, image registration fares far better than either WPC or MSC. However, these method provide something that image reg- istration cannot—they enable tracking with a fixed frame of reference in the presence of sparse specimen information and for moving specimens. Structured illumination pro- vides the additional ability to extend tracking into 3D. Throughout this chapter I have claimed that some of MSC’s accuracy issues are due to the pattern used for evaluation— so what is a better pattern? These issues are investigated in greater detail in Chapter 6.
Chapter 6
Pattern Design for Structured Illumination
Microscopy
In Chapter 5, I developed weighted phase correlation (WPC) and model-based spatial correlation (MSC) as techniques that provide lateral tracking of a semitransparent pat- tern layer in structured illumination microscopy. The pattern used in this evaluation was obtained from a TEM grid which produced dark square “holes” separated by thin bright “bars”. This pattern was chosen because the grids were readily available to purchase from a microscopy supply company and it was relatively simple for a materials scientist to deposit a metal image onto glass cover slips. This provided a working system with which to demonstrate the concept of structured illumination microscopy. The TEM grid pattern, however, is not the optimal pattern for structured illumination microscopy.
This chapter investigates pattern design for structured illumination microscopy, with the end goal being an understanding of how choices made in pattern design affect the ability to recover tracking information from the pattern layer. Specifically, I discuss how the choice of pattern influences the ability to track the stage independently from the specimen, determine focus position, and disambiguate among possible stage positions.
The following design goals are generally applicable to stage tracking in a broad range of microscopy experiments:
dent of specimen motion.
2. The pattern should enable tracking over large frame-to-frame displacements.
3. The pattern should enable determining the absolute stage position.
4. The pattern should be easily removable from the acquired images through image processing.
Note that there is an important distinction between goals 2 and 3. The former goal refers to tracking the stage relative to some fixed reference position. The latter goal refers to determining absolute stage position from a single image—a slide-based absolute positioning system.
Pattern design necessarily involves some trade-offs. Both WPC and MSC depend on finding the displacement of a regular pattern layer present in an image. The resolu- tion of these methods increases as that pattern matching is based on higher frequency components, as seen in Equation 5.9: Epx =
Eφ
360f. But, to enable tracking over large
frame-to-frame displacements the pattern should have a large wavelength, favoring low frequencies. Another consideration that favors low frequencies is that as the pattern moves farther out of focus, the high frequency components are attenuated more and therefore reach the noise floor first.
In order to track the stage independent of the specimen the pattern signal should dominate the specimen signal. This requires high-contrast, low transparency patterns. Patterns that are more transparent, however, enable removing the pattern from the specimen images with less information loss.
As I will show, to enable axial (focus) tracking, the pattern should contain multi- ple frequency components. The more frequency components that are present, the more terms can be used to determine focus. Choosing which frequencies to include, however, requires considering the frequencies present in the specimen. If the specimen has high
magnitude at many frequencies, it may be difficult to find an appropriate set of frequen- cies to use as a pattern. Additionally, the microscope point-spread function (PSF) may attenuate a large proportion of high frequencies too much to be useful.
The remainder of this chapter is organized as follows. Section 6.1 discusses pattern selection and design, including how to maximize pattern tracking accuracy in the pres- ence of an independently moving specimen. Section 6.2 discusses how to expand the frequency analysis of micropatterns to enable axial tracking of the microscope stage. Section 6.3 evaluates the approaches discussed here using simulated structured illumi- nation microscopy images. Section 6.4 summarizes results and discusses strengths and limitations of the methods presented here.