Sparse 3-D Reconstruction From Multiple Views
This example shows you how to build a point cloud based on features matched across multiple pictures of an object.
Overview
This example shows how to reconstruct a 3-D scene from multiple 2-D images. This type of reconstruction can be used to develop 3-D models of objects or build 3-D world maps. The steps of a reconstruction process are shown and explained using a pair of images. The last step shows the same process using additional views of the object. This example uses a collection of images captured with a standard digital camera. The camera was calibrated using the cameraCalibrator
Read a Pair of Images
Load images into the workspace and save them to a structure. This structure will then be used to store other information associated with each of the images.
files = {'Globe01.jpg';'Globe02.jpg'}; for i = 1:numel(files)
files{i}=fullfile(matlabroot, 'toolbox', 'vision', 'visiondemos', ... 'sparseReconstructionImages', files{i});
images(i).image = imread(files{i}); end
figure;
montage(files); title('Pair of Original Images')
Load Camera Parameters
This example uses the camera parameters calculated by the cameraCalibrator app. The parameters are stored in thecameraParamsobject, and include the
camera intrinsics and lens distortion coefficients.
% Load precomputed camera parameters
load sparseReconstructionCameraParameters.mat
K = cameraParams.IntrinsicMatrix;
Remove Lens Distortion
Lens distortion can affect the accuracy of the final reconstruction. For this reason, the distortion is removed from each of the images using calibration information. This process straightens the lines that are bent by the radial distortion of the lens.
figure
distortedImage = images(1).image;
title('Globe01 Before and After Image Distortion is Removed') for i=1:numel(images) % remove lens distortion from both images
images(i).image = undistortImage(images(i).image, cameraParams); end
imshowpair(distortedImage, images(1).image,'montage');
Black regions appear on the edges of the image. The black regions are added by the undistortImage routine to keep the output size the same as the input size.
Calculate Extrinsics and Build Camera Matrices for Images
Each of the images contains a checkerboard, whose square size is known to be 22 millimeters. First you need to detect the checkerboard points in the image, and define the corresponding world points. Then use theextrinsics
function to compute the rotation and the translation of the camera relative to the checkerboard. This rotation and translation, together with the intrinsic parameters are stored in the Camera Matrix. The matrix relates 3-D world positions to 2-D pixel locations in the image [1].
squareSize = 22; % checkerboard square size in millimeters for i = 1:numel(images)
% Detect the checkerboard in the undistorted image
[imagePoints, boardSize] = detectCheckerboardPoints(images(i).image);
% Generate world points
Sparse 3-D Reconstruction From Multiple Views
% Extrinsic camera parameters
[Rotation, translation] = extrinsics(imagePoints, worldPoints, cameraPa
% Compute a projective transform for mapping 3-D position to 2-D pixel % locations [1].
images(i).CameraMatrix = [Rotation; translation] * K; end
Extract Feature Points
You need to find a set of points common to both images in order to calculate their corresponding 3-D locations. Detect and extract point feature
descriptors. In this example, SURF descriptors are used because they are robust with respect to both scale and rotation.
% Detect feature points and extract SURF descriptors in images for i = 1:numel(images)
%detect SURF feature points
images(i).points = detectSURFFeatures(rgb2gray(images(i).image),... 'MetricThreshold',600);
%extract SURF descriptors
[images(i).featureVectors,images(i).points] = ...
extractFeatures(rgb2gray(images(i).image),images(i).points); end
% Visualize several extracted SURF features from the Globe01 image figure; imshow(images(1).image);
title('1500 Strongest Feature Points from Globe01'); hold on;
plot(images(1).points.selectStrongest(1500));
Match Features Across Images
These features are matched between the images using the matchFeatures function.
matchFeatures(images(1).featureVectors, images(2).featureVectors,... 'Prenormalized', true,'MaxRatio',0.4) ; matchedPoints1 = images(1).points(indexPairs(:, 1)); matchedPoints2 = images(2).points(indexPairs(:, 2)); figure; % Visualize correspondences showMatchedFeatures(images(1).image,images(2).image,matchedPoints1,matchedP title('Original Matched Features from Globe01 and Globe02');
Eliminate Noisy Matches Based on Camera Locations
From the previous image, it is clear that not all of the matched features correspond across image views. Some of these noisy matches can be removed by utilizing the epipolar constraint. To satisfy the epipolar constraint, a point from one image must match a point on the corresponding epipolar line in the other image. If the matched point was not on the epipolar line, then it can be safely discarded as an outlier. The point falls on the epipolar line when the epipolar constraint equation is satisfied: where and are corresponding points in two images in homogeneous coordinates. is the fundamental matrix relating those images [1].
% Set a value near zero, It will be used to eliminate matches that % correspond to points that do not lie on an epipolar line.
epipolarThreshold = .05;
for k = 1:length(matchedPoints1)
% Compute the fundamental matrix using the example helper function % Evaluate the epipolar constraint
epipolarConstraint =[matchedPoints1.Location(k,:),1]...
*helperCameraMatricesToFMatrix(images(1).CameraMatrix,images(2).Cam *[matchedPoints2.Location(k,:),1]';
% Only consider feature matches where the absolute value of the % constraint expression is less than the threshold.
valid(k) = abs(epipolarConstraint) < epipolarThreshold; end
Sparse 3-D Reconstruction From Multiple Views
validpts2 = images(2).points(indexPairs(valid, 2)); figure;
showMatchedFeatures(images(1).image,images(2).image,validpts1,validpts2); title('Matched Features After Applying Epipolar Constraint');
Reconstruct the Object From Corresponding Features
A 3-D location of a feature can be estimated using linear triangulation. Use the point correspondences and the Direct Linear Transformation (DLT) algorithm to perform the linear triangulation. This algorithm requires creating an matrix that contains two equations from every image that includes the feature. The equations include the Camera Matrix, , and the image locations of the feature. is the row of for a given image and and are the image coordinates of a feature in that image [1] (p. 91).
Obtain the best estimate of the matched features position, , in world coordinates by minimizing the expression, . One way to perform this minimization is to use the Singular Value Decomposition of A [1].
% convert image to double format for plotting doubleimage = im2double(images(1).image);
points3D = ones(length(validpts1),4); % store homogeneous world coordinates color = ones(length(validpts1),3); % store color information
% For all point correspondences for i = 1:length(validpts1)
% For all image locations from a list of correspondences build an A pointInImage1 = validpts1(i).Location;
pointInImage2 = validpts2(i).Location;
P1 = images(1).CameraMatrix'; % Transpose to match the convention in P2 = images(2).CameraMatrix'; % in [1] A = [ pointInImage1(1)*P1(3,:) - P1(1,:);... pointInImage1(2)*P1(3,:) - P1(2,:);... pointInImage2(1)*P2(3,:) - P2(1,:);... pointInImage2(2)*P2(3,:) - P2(2,:)];
% Compute the 3-D location using the smallest singular value from the % singular value decomposition of the matrix A
[~,~,V]=svd(A);
X = V(:,end); X = X/X(end);
% Store location points3D(i,:) = X';
% Store pixel color for visualization y = round(pointInImage1(1));
x = round(pointInImage1(2));
color(i,:) = squeeze(doubleimage(x,y,:))'; end
Draw the 3-D Point Cloud
The 3-D points can now be plotted with the same units as the calibration. Using the scatter3 function, draw a point cloud representation of each feature location and its corresponding colors.
% add green point representing the origin points3D(end+1,:) = [0,0,0,1];
color(end+1,:) = [0,1,0];
% show images
figure('units','normalized','outerposition',[0 0 .5 .5])
subplot(1,2,1); montage(files,'Size',[1,2]); title('Original Images')
% plot point-cloud
hAxes = subplot(1,2,2); hold on; grid on;
scatter3(points3D(:,1),points3D(:,2),points3D(:,3),50,color,'fill') xlabel('x-axis (mm)');ylabel('y-axis (mm)');zlabel('z-axis (mm)') view(20,24);axis equal;axis vis3d
set(hAxes,'XAxisLocation','top','YAxisLocation','left',... 'ZDir','reverse','Ydir','reverse');
grid on
Sparse 3-D Reconstruction From Multiple Views
Reconstruct the Object From Multiple Views
A two image reconstruction only contains information covered by the two views. The other sides of the object are ignored. Features can be more accurately located in 3-D space, with multiple image views. Each additional view provides two constraints on the A matrix. Use the same process from the previous steps on a larger set of images. The helperReconstructFromImages helper function combines all the steps in this example . In order to match features across more than two images, the a KDTreeSearcher object from the Statistics Toolbox™ was used in place of the matchFeatures function [2]. This matching process is similar to the Image Search using Point Features Example, except that features are allowed to correspond to more than one image.
imageFileNames = {'Globe01.jpg';'Globe02.jpg';'Globe03.jpg';'Globe04.jpg';. 'Globe05.jpg';'Globe06.jpg';'Globe07.jpg';'Globe08.jpg'};
load sparseReconstructionCameraParameters.mat % Calibration results for i =1:numel(imageFileNames)
imageFileNames{i}=fullfile(matlabroot, 'toolbox', 'vision', 'visiondemo 'sparseReconstructionImages', imageFileNames{i});
end
% Perform reconstruction using all of these images
[points3D,color]=helperReconstructFromImages(imageFileNames, cameraParams, squareSize);
% add origin as green
points3D(end+1,:) = [0,0,0,1]; color(end+1,:) = [0,1,0];
figure('units','normalized','outerposition',[0 0 .5 .5])
% show images
subplot(1,2,1); montage(imageFileNames); title('Original Images')
% show pointcloud
hAxes = subplot(1,2,2); hold on; grid on;
xlabel('x-axis (mm)');ylabel('y-axis (mm)');zlabel('z-axis (mm)') view(26,22);axis equal;axis vis3d
set(hAxes,'XAxisLocation','top','YAxisLocation','left',... 'ZDir','reverse','Ydir','reverse');
title('Reconstructed Point Cloud')
References
[1] Hartley, Richard, and Andrew Zisserman. Multiple View Geometry in Computer Vision. Second Edition. Cambridge, 2000.
[2] Brown, Matthew, and David G. Lowe. "Unsupervised 3D Object
Recognition and Reconstruction in Unordered Datasets." 3-D Digital Imaging and Modeling, 2005. 3DIM 2005. Fifth International Conference on. IEEE, 2005.