The pixels comprising a typical CMOS or CCD digital image sensor are physical devices which transduce the properties of the light they see into electrical signals. To analyze the image acquisition capability of a particular pixel device, we con- struct a model which describes its sensitivity to the specific property of interest. For example, the RGB pixels employed in color image sensors each have a distinct wavelength response profile. By measuring this wavelength dependent sensitivity, we obtain a wavelength transfer function h(λ) which models how the pixel responds to light of different monochromatic wavelengths.
Wavelength is just one of the possible properties of light that the pixels of an image sensor might measure. Recent work in our lab has demonstrated image sensors composed of pixels which are sensitive to the incident angle of light striking them [110]. In a similar manner to wavelength and color pixels, these angle-
sensitive pixels have a characteristic angular sensitivity. By measuring their angle- dependent response to light, we obtain the angular transfer function h(θ, φ). This function describes the detector response to plane wave incident light arriving from different incident angles.
We begin our discussion with the general implications of incident angle imaging. Before exploring this idea, we first introduce some notation which will be used throughout the remainder of the text. Let S(x, y, z) represent an arbitrary three- dimensional visual scene of interest, with (x, y, z) coordinates denoting spatial position. We assume Lambertian reflectance for the scene, such that illuminating light is reflected in an isotropic manner. We simplify the image-forming optical system to a single thin lens with focal length F and aperture D satisfying the paraxial small-angle approximation. For image capture, we assume that the image sensor is composed of a 2D array of pixels arranged in a regular grid and that its size is much smaller than D. The recorded image is finally represented by an array of voltages V (i, j), with (i, j) coordinates denoting pixel position and voltage V proportional to incident photon flux.
Pixel
1 2 3 4 4 3 2 1 θ1 θ3 θ4θ
10
θ
3θ
4Spatial position in visual scene
T rans fer h (θ ) Angle θ
Figure 6.1: The response of a pixel to a visual scene is the sum of the intensity of light arriving from different points, weighted by the angular transfer function h(θ, φ)
To start, we consider the simple imaging arrangement of Figure 6.1, with a 2D image S(x, y) placed a distance Z directly in front of a single pixel with angular transfer function h(θ, φ). We assume that the size of the pixel is small relative to the scene S(x, y) and therefore approximate it as a point at location (i, j). Light emanating from the scene at location (xk, yk) arrives at the pixel as a plane wave
with a characteristic solid angle of incidence (θk, φk), such that xk = Z tan θk and
yk = Z tan φk. Therefore, the contribution to the overall photon flux observed by the pixel from this kth location in the visual scene is vk = S(xk, yk)h(θk, φk). As this imaging arrangement uses no lens, all points of the scene are visible to the pixel. The pixel output will integrate the contributions from the entire visual scene, such that total pixel response V is given by the integral inner product
V = Z π/2 −π/2 Z π/2 −π/2 h(θ, φ)S (Z tan θZ tan φ) dθ dφ (6.1) In effect, the angular transfer function h(θ, φ) selectively strengthens or weakens the contribution of different scene locations to photon flux and therefore to the pixel response. Zo i θ D x ΔZ Image plane S’(i)
Zi Scene S(x)
Figure 6.2: Light arriving at a pixel placed behind a lens can be expressed in terms of the light reflected by different points in the visual scene.
The angular transfer function acts as a spatial weighting function whose spatial scale is determined by the distance Z from pixel to scene. Introducing a thin lens to the scene-pixel system does not fundamentally alter this behavior. A visual scene S(x, y) a distance Zo in front of a thin lens projects a real image at a distance Zi
behind the lens. The relationship between these two distances is given by the thin lens equation Z1
o + 1 Zi =
1
F, where F is the lens focal length. The projected image
is a scaled version of the original scene, scaled by the ratio M = Zo/Zi. We place a single pixel a distance Z behind the lens, approximated as a point at position (i, j). The geometry of the ray optics, illustrated in Fig. 6.1, imply that light at position (i, j) arriving from the scene S(x, y) is given as a function of location and incident angle by the expression
I(i, j, θ, φ) = S Zo Zi i +Zo Zi ∆Z tan θ,Zo Zi j + Zo Zi ∆Z tan φ ≈ S (Mi + M∆Z · θ, Mj + M∆Z · θ) (6.2) We have made a small angle approximation for the tangent function, and de- fined ∆Z as the distance between the image plane and the pixel: ∆Z = Z − Zi.
The ratio M = Zo/Zi sets the magnification of the system, and the distance ∆Z
sets the mapping between spatial and angular coordinates. Since the image may form either in front of or behind the pixel, the sign of ∆Z can be positive or negative.
The projected image does not have the isotropic emission properties of the original scene, as the finite aperture of the lens restricts the maximum observable incident angles θmax and φmax to ± arctan(D/2Z). Nevertheless, over the range
of incident angles passed by the lens, the pixel weights locations by its angular transfer function when it integrates light over the portion of the visual scene it sees. Assuming that the lens aperture transmittance is a 2D box function of
incident angle and that Z is large relative to lens aperture, the pixel output is therefore V = Z D/2Z −D/2Z Z D/2Z −D/2Z h(θ, φ)I(i, j, θ, φ) dθ dφ (6.3) Conventional image sensor pixels attempt to achieve an isotropic response to incident light and have angular transfer functions h(θ, φ) which are well approx- imated by simple symmetric concave functions. In contrast, recently developed angle-sensitive pixels have angular transfer functions which are windowed sinu- soids [28]. These pixels use micro-scale diffraction effects to achieve their periodic angular response. The principles of their operation are described in the next sec- tion.