Images can be treated as two-dimensional data, and many of the signal process- ing approaches presented in the previous chapters are equally applicable to im- ages: some can be directly applied to image data while others require some modification to account for the two (or more) data dimensions. For example, both PCA and ICA have been applied to image data treating the two-dimen- sional image as a single extended waveform. Other signal processing methods including Fourier transformation, convolution, and digital filtering are applied to images using two-dimensional extensions. Two-dimensional images are usually represented by two-dimensional data arrays, and MATLAB follows this tradi- tion;* however, MATLAB offers a variety of data formats in addition to the standard format used by most MATLAB operations. Three-dimensional images can be constructed using multiple two-dimensional representations, but these multiple arrays are sometimes treated as a single volume image.
General Image Formats: Image Array Indexing
Irrespective of the image format or encoding scheme, an image is always repre- sented in one, or more, two dimensional arrays, I(m,n). Each element of the
*Actually, MATLAB considers image data arrays to be three-dimensional, as described later in this chapter.
variable,I, represents a single picture element, or pixel. (If the image is being treated as a volume, then the element, which now represents an elemental vol- ume, is termed a voxel.) The most convenient indexing protocol follows the traditional matrix notation, with the horizontal pixel locations indexed left to right by the second integer, n, and the vertical locations indexed top to bottom by the first integerm(Figure 10.1). This indexing protocol is termed pixel coor-
dinates by MATLAB. A possible source of confusion with this protocol is that
the vertical axis positions increase from top to bottom and also that the second integer references the horizontal axis, the opposite of conventional graphs.
MATLAB also offers another indexing protocol that accepts non-integer indexes. In this protocol, termed spatial coordinates, the pixel is considered to be a square patch, the center of which has an integer value. In the default coordi- nate system, the center of the upper left-hand pixel still has a reference of (1,1), but the upper left-hand corner of this pixel has coordinates of (0.5,0.5) (see
Figure 10.2). In this spatial coordinate system, the locations of image coordi- nates are positions on a (discrete) plane and are described by general variables
x and y. The are two sources of potential confusion with this system. As with
the pixel coordinate system, the vertical axis increases downward. In addition, the positions of the vertical and horizontal indexes (now better though of as coordinates) are switched: the horizontal index is first, followed by the vertical coordinate, as with conventional x,y coordinate references. In the default spatial coordinate system, integer coordinates correspond with their pixel coordinates, remembering the position swap, so thatI(5,4) in pixel coordinates references the same pixel as I(4.0,5.0) in spatial coordinates. Most routines expect a specific pixel coordinate system and produce outputs in that system. Examples of spatial coordinates are found primarily in the spatial transformation routines described in the next chapter.
It is possible to change the baseline reference in the spatial coordinate
FIGURE10.1 Indexing format for MATLAB images using the pixel coordinate sys-
FIGURE10.2 Indexing in the spatial coordinate system.
system as certain commands allow you to redefine the coordinates of the refer- ence corner. This option is described in context with related commands. Data Classes: Intensity Coding Schemes
There are four different data classes, or encoding schemes, used by MATLAB for image representation. Moreover, each of these data classes can store the data in a number of different formats. This variety reflects the variety in image types (color, grayscale, and black and white), and the desire to represent images as efficiently as possible in terms of memory storage. The efficient use of memory storage is motivated by the fact that images often require a large numbers of array locations: an image of 400 by 600 pixels will require 240,000 data points, each of which will need one or more bytes depending of the data format.
The four different image classes or encoding schemes are: indexed images,
RGB images, intensity images, and binary images. The first two classes are used
to store color images. In indexed images, the pixel values are, themselves, in- dexes to a table that maps the index value to a color value. While this is an efficient way to store color images, the data sets do not lend themselves to arithmetic operations (and, hence, most image processing operations) since the results do not always produce meaningful images. Indexed images also need an associated matrix variable that contains the colormap, and this map variable needs to accompany the image variable in many operations. Colormaps are N by 3 matrices that function as lookup tables. The indexed data variable points to a particular row in the map and the three columns associated with that row
contain the intensity of the colors red, green, and blue. The values of the three columns range between 0 and 1 where 0 is the absence of the related color and 1 is the strongest intensity of that color. MATLAB convention suggests that indexed arrays use variable names beginning inx..(or simplyx) and the sug- gested name for the colormap is map. While indexed variables are not very useful in image processing operations, they provide a compact method of storing color images, and can produce effective displays. They also provide a conve- nient and flexible method for colorizing grayscale data to produce a pseudocolor image.
The MATLAB Image Processing Toolbox provides a number of useful prepackaged colormaps. These colormaps can implemented with any number of rows, but the default is 64 rows. Hence, if any of these standard colormaps are used with the default value, the indexed data should be scaled to range between 0 and 64 to prevent saturation. An example of the application of a MATLAB colormap is given in Example 10.3. An extension of that example demonstrates methods for colorizing grayscale data using a colormap.
The other method for coding color image is the RGB coding scheme in which three different, but associated arrays are used to indicate the intensity of the three color components of the image: red, green, or blue. This coding scheme produces what is know as a truecolor image. As with the encoding used in indexed data, the larger the pixel value, the brighter the respective color. In this coding scheme, each of the color components can be operated on separately. Obviously, this color coding scheme will use more memory than indexed im- ages, but this may be unavoidable if extensive processing is to be done on a color image. By MATLAB convention the variable name RGB, or something similar, is used for variables of this data class. Note that these variables are actually three-dimensional arrays having dimensions N by M by 3. While we have not used such three dimensional arrays thus far, they are fully supported by MATLAB. These arrays are indexed asRGB(n,m,i)wherei= 1,2,3. In fact, all image variables are conceptualized in MATLAB as three-dimensional arrays, except that for non-RGB images the third dimension is simply 1.
Grayscale images are stored as intensity class images where the pixel value represents the brightness or grayscale value of the image at that point. MATLAB convention suggests variable names beginning with I for variables in class intensity. If an image is only black or white (not intermediate grays), then the binary coding scheme can be used where the representative array is a
logical array containing either 0’s or 1’s. MATLAB convention is to useBWfor variable names in the binary class. A common problem working with binary images is the failure to define the array as logical which would cause the image variable to be misinterpreted by the display routine. Binary class variables can be specified as logical (set the logical flag associated with the array) using the command BW = logical(A), assuming Aconsists of only zeros and ones. A logical array can be converted to a standard array using the unary plus operator:
A=ⴙBW. Since all binary images are of the form “logical,” it is possible to check if a variable is logical using the routine:isa(I, ’logical’); which will return a1 if true and zero otherwise.
Data Formats
In an effort to further reduce image storage requirements, MATLAB provides three different data formats for most of the classes mentioned above. The uint8 and uint16 data formats provide 1 or 2 bytes, respectively, for each array ele- ment. Binary images do not support the uint16 format. The third data format, the doubleformat, is the same as used in standard MATLAB operations and, hence, is the easiest to use. Image arrays that use the double format can be treated as regular MATLAB matrix variables subject to all the power of MATLAB and its many functions. The problem is that this format uses 8 bytes for each array element (i.e., pixel) which can lead to very large data storage requirements.
In all three data formats, a zero corresponds to the lowest intensity value, i.e., black. For the uint8 and uint16 formats, the brightest intensity value (i.e., white, or the brightest color) is taken as the largest possible number for that coding scheme: for uint8, 28-1, or 255; and for uint16, 216, or 65,535. For the double format, the brightest value corresponds to 1.0.
The isa routine can also be used to test the format of an image. The routine,isa(I,’type’)will return a 1 ifIis encoded in the formattype, and a zero otherwise. The variabletype can be:unit8,unit16, or double. There are a number of other assessments that can be made with the isaroutine that are described in the associated help file.
Multiple images can be grouped together as one variable by adding an- other dimension to the variable array. Since image arrays are already considered three-dimensional, the additional images are added to the fourth dimension. Multi-image variables are termed multiframe variables and each two-dimen- sional (or three-dimensional) image of a multiframe variable is termed a frame. Multiframe variables can be generated within MATLAB by incrementing along the fourth index as shown in Example 10.2, or by concatenating several images together using thecatfunction:
IMF = cat(4, I1, I2, I3,...);
The first argument, 4, indicates that the images are to concatenated along the fourth dimension, and the other arguments are the variable names of the images. All images in the list must be the same type and size.
Data Conversions
The variety of coding schemes and data formats complicates even the simplest of operations, but is necessary for efficient memory use. Certain operations
require a given data format and/or class. For example, standard MATLAB oper- ations require the data be in double format, and will not work correctly with Indexed images. Many MATLAB image processing functions also expect a spe- cific format and/or coding scheme, and generate an output usually, but not al- ways, in the same format as the input. Since there are so many combinations of coding and data type, there are a number of routines for converting between different types. For converting format types, the most straightforward procedure is to use theim2xxxroutines given below:
I_uint8 = im2uint8(I); % Convert to uint8 format I_uint16 = im2uint16(I); % Convert to uint16 format I_double = im2double(I); % Convert to double format
These routines accept any data class as input; however if the class is indexed, the input argument,I, must be followed by the termindexed. These routines also handle the necessary rescaling except for indexed images. When converting indexed images, variable range can be a concern: for example, to convert an indexed variable to uint8, the variable range must be between 0 and 255.
Converting between different image encoding schemes can sometimes be done by scaling. To convert a grayscale image in uint8, or uint16 format to an indexed image, select an appropriate grayscale colormap from the MATLAB’s established colormaps, then scale the image variable so the values lie within the range of the colormap; i.e., the data range should lie between 0 and N, where N is the depth of the colormap (MATLAB’s colormaps have a default depth of 64, but this can be modified). This approach is demonstrated in Example 10.3. However, an easier solution is simply to use MATLAB’s gray2ind function listed below. This function, as with all the conversion functions, will scale the input data appropriately, and in the case ofgray2indwill also supply an appro- priate grayscale colormap (although alternate colormaps of the same depth can be substituted). The routines that convert to indexed data are:
[x, map] = gray2ind(I, N); % Convert from grayscale to % indexed
% Convert from truecolor to indexed [x, map] = rgb2ind(RGB, N or map);
Both these routines accept data in any format, including logical, and pro- duce an output of type uint8 if the associated map length is less than or equal to 64, or uint16 if greater that 64. N specifies the colormap depth and must be less than 65,536. Forgray2indthe colormap isgray with a depth ofN, or the default value of 64 if N is omitted. For RGB conversion using rgb2ind, a colormap ofNlevels is generated to best match the RGB data. Alternatively, a
colormap can be provided as the second argument, in which casergb2indwill generate an output array,x, with values that best match the colors given inmap. Thergb2indfunction has a number of options that affect the image conversion, options that allow trade-offs between color accuracy and image resolution. (See the associated help file).
An alternative method for converting a grayscale image to indexed values is the routinegrayslicewhich converts using thresholding:
x = grayslice(I, N or V); % Convert grayscale to indexed using % thresholding
where any input format is acceptable. This function slices the image into N levels using a equal step thresholding process. Each slice is then assigned a specific level on whatever colormap is selected. This process allows some inter- esting color representations of grayscale images, as described in Example 10.4. If the second argument is a vector,V, then it contains the threshold levels (which can now be unequal) and the number of slices corresponds to the length of this vector. The output format is either uint8oruint16depending on the number of slices, similar to the two conversion routines above.
Two conversion routines convert from indexed images to other encoding schemes:
I = ind2gray(x, map); % Convert to grayscale intensity % encoding
RGB = ind2rgb(x, map); % Convert to RGB (“truecolor”) % encoding
Both functions accept any format and, in the case ofind2grayproduces outputs in the same format. Function ind2rgb produces outputs formatted as double. Function ind2gray removes the hue and saturation information while retaining the luminance, while function ind2rgb produces a truecolor RGB variable.
To convert an image to binary coding use:
BW = im2bw(I, Level); % Convert to binary logical encoding
where Levelspecifies the threshold that will be used to determine if a pixel is white (1) or black (0). The input image, I, can be either intensity, RGB, or indexed,* and in any format (uint8, uint16, or double). While most functions output binary images in uint8 format,im2bwoutputs the image in logical format.
*As with all conversion routines, and many other routines, when the input image is in indexed format it must be followed by the colormap variable.
In this format, the image values are either 0 or 1, but each element is the same size as the double format (8 bytes). This format can be used in standard MAT- LAB operations, but does use a great deal of memory. One of the applications of theditherfunction can also be used to generate binary images as described in the associated help file.
A final conversion routine does not really change the data class, but does scale the data and can be very useful. This routine converts general class double data to intensity data, scaled between 0 and 1:
I = mat2gray(A, [Anin Amax]); % Scale matrix to intensity % encoding, double format.
whereAis a matrix and the optional second term specifies the values of Ato be scaled to zero, or black (Amin), or 1, or white (Amin). Since a matrix is already in double format, this routine provides only scaling. If the second argument is missing, the matrix is scaled so that its highest value is 1 and its lowest value is zero. Using the default scaling can be a problem if the image contains a few irrelevant pixels having large values. This can occur after certain image process- ing operations due to border (or edge) effects. In such cases, other scaling must be imposed, usually determined empirically, to achieve a suitable range of im- age intensities.
The various data classes, their conversion routines, and the data formats they support are summarized in Table 1 below. The output format of the various conversion routines is indicated by the superscript: (1) uint8 or unit 16 depend- ing on the number of levels requested (N); (2) Double; (3) No format change (output format equals input format); and (4) Logical (size double).
Image Display
There are several options for displaying an image, but the most useful and easi- est to use is theimshowfunction. The basic calling format of this routine is:
TABLE10.1 Summary of Image Classes, Data Formats, and Conversion Routines
Class Formats supported Conversion routines
Indexed All gray2ind1, grayslice1, rgb2ind1
Intensity All ind2gray2
, mat2gray2,3
, rgb2gray3
RGB All ind2rgb2
Binary uint8, double im2bw4
imshow(I,arg)
where Iis the image array and argis an argument, usually optional, that de- pends on the data format. For indexed data, the variable name must be followed by the colormap, map. This holds for all display functions when indexed data are involved. For intensity class image variables,argcan be a scalar, in which case it specifies the number of levels to use in rendering the image, or, if arg is a vector, [low high], arg specifies the values to be taken to readjust the range limits of a specific data format.* If the empty matrix, [ ], is given asarg, or it is simply missing, the maximum and minimum values in arrayIare taken as thelowandhighvalues. Theimshowfunction has a number of other options that make it quite powerful. These options can be found with the help command. WhenIis an indexed variable, it should be followed by themapvariable.
There are two functions designed to display multiframe variables. The function montage (MFW) displays the various images in a gird-like pattern as shown in Example 10.2. Alternatively, multiframe variables can be displayed as a movie using theimmovieandmoviecommands:
mov = imovie(MFW); % Generate movie variable
movie(mov); % Display movie
Unfortunately the moviefunction cannot be displayed in a textbook, but is presented in one of the problems at the end of the chapter, and several amus- ing examples are presented in the problems at the end of the next chapter. The immovie function requires multiframe data to be in either Indexed or RGB