对极几何和基本矩阵
对射
在
对
极
几
何
中
,
在
图
像
A
中
的
点
x
,
被
映
射
到
图
像
B
的
对
极
线
l
′
上
。
在对极几何中,在图像A中的点x,被映射到图像B的对极线l'上。
在对极几何中,在图像A中的点x,被映射到图像B的对极线l′上。
x
F
基
本
矩
阵
→
l
′
x\overrightarrow{\ \ \ \ \ \ F基本矩阵\ \ \ \ \ \ } l'
x F基本矩阵 l′
任
何
与
x
匹
配
的
点
x
’
必
然
在
对
极
线
l
′
上
,
x
F
x
′
=
0
。
任何与x匹配的点x’必然在对极线l'上,xFx'=0。
任何与x匹配的点x’必然在对极线l′上,xFx′=0。
F
=
e
×
′
H
π
,
其
中
d
e
t
(
e
×
′
)
=
2
,
d
e
t
(
H
π
)
=
3
,
d
e
t
(
F
)
≤
2
F=e'_{\times}H_{\pi},其中det(e'_{\times})=2,det(H_{\pi})=3,det(F)\leq2
F=e×′Hπ,其中det(e×′)=2,det(Hπ)=3,det(F)≤2
p ′ = R p + T p ′ , T , p 三 线 共 面 约 束 : ( R p ) T ∗ ( T × p ′ ) = 0 p T R T ∗ ( T × ∗ p ′ ) = 0 p T ( R T ∗ T × ) p ′ = 0 p T ( E ) p ′ = 0 又 p = K − 1 x p ’ = K ′ − 1 x ’ x T ( K − T E K ′ − 1 ) x ’ = 0 p'=Rp+T\\ p',T,p三线共面约束:(Rp)^T*(T×p')=0\\ p^TR^T*(T_×* p')=0\\ p^T(R^T*T_×) p'=0\\ p^T(E) p'=0 \\ 又p = K^{-1}x\\ p’= K'^{-1}x’ \\ x^T(K^{-T}E K'^{-1})x’=0 \\ p′=Rp+Tp′,T,p三线共面约束:(Rp)T∗(T×p′)=0pTRT∗(T×∗p′)=0pT(RT∗T×)p′=0pT(E)p′=0又p=K−1xp’=K′−1x’xT(K−TEK′−1)x’=0
基 本 矩 阵 计 算 基本矩阵计算 基本矩阵计算
归一化8点法:
step1:对于每一对对应点x与x’,有:
[ x y 1 ] [ f 11 f 12 f 13 f 21 f 22 f 23 f 31 f 32 f 33 ] [ x ′ y ′ 1 ] = 0 ⇓ x ′ x f 11 + x ′ y f 12 + x ′ f 13 + y ′ x f 21 + y ′ y f 22 + y ′ f 23 + x f 31 + y f 32 + f 33 = 0 ⇓ [ x ′ x x ′ y x ′ y ′ x y ′ y y ′ x y 1 ] f = 0 对 于 给 定 的 n 个 对 应 点 对 , 有 A n , 9 ∗ f = [ x 1 ′ x 1 x 1 ′ y 1 x 1 ′ y 1 ′ x 1 y 1 ′ y 1 y 1 ′ x 1 y 1 1 ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ x n ′ x n x n ′ y n x n ′ y n ′ x n y n ′ y n y n ′ x n y n 1 ] f = 0 \begin{bmatrix}x&y&1\end{bmatrix}\begin{bmatrix}f_{11}&f_{12}&f_{13}\\f_{21}&f_{22}&f_{23}\\f_{31}&f_{32}&f_{33}\end{bmatrix}\begin{bmatrix}x'\\y'\\1\end{bmatrix}=0 \\ \Downarrow \\ x'xf_{11}+x'yf_{12}+x'f_{13}+y'xf_{21}+y'yf_{22}+y'f_{23}+xf_{31}+yf_{32}+f_{33} = 0 \\ \Downarrow \\ \begin{bmatrix} x'x&x'y&x'&y'x&y'y&y'&x&y&1 \end{bmatrix}f=0 \\ 对于给定的n个对应点对,有 \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ A_{n,9}*f=\begin{bmatrix}x_1'x_1&x_1'y_1&x_1'&y_1'x_1&y_1'y_1&y_1'&x_1&y_1&1\\\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots\\x_n'x_n&x_n'y_n&x_n'&y_n'x_n&y_n'y_n&y_n'&x_n&y_n&1\end{bmatrix}f = 0 [xy1]⎣⎡f11f21f31f12f22f32f13f23f33⎦⎤⎣⎡x′y′1⎦⎤=0⇓x′xf11+x′yf12+x′f13+y′xf21+y′yf22+y′f23+xf31+yf32+f33=0⇓[x′xx′yx′y′xy′yy′xy1]f=0对于给定的n个对应点对,有 An,9∗f=⎣⎢⎡x1′x1⋮xn′xnx1′y1⋮xn′ynx1′⋮xn′y1′x1⋮yn′xny1′y1⋮yn′yny1′⋮yn′x1⋮xny1⋮yn1⋮1⎦⎥⎤f=0
如
果
A
的
秩
为
8
,
则
在
差
一
个
尺
度
因
子
的
情
况
下
,
解
是
唯
一
的
。
若
因
为
噪
声
等
原
因
,
A
的
秩
为
9
,
则
使
用
最
小
二
乘
解
法
,
m
i
n
‖
A
f
‖
如果A的秩为8,则在差一个尺度因子的情况下,解是唯一的。 若因为噪声等原因,A的秩为9,则使用最小二乘解法,min‖Af‖
如果A的秩为8,则在差一个尺度因子的情况下,解是唯一的。若因为噪声等原因,A的秩为9,则使用最小二乘解法,min‖Af‖
A
=
U
S
V
T
A=USV^T
A=USVT
f
的
解
就
是
系
数
矩
阵
A
最
小
奇
异
值
对
应
的
奇
异
向
量
(
V
最
后
一
列
)
f的解就是系数矩阵A最小奇异值对应的奇异向量(V最后一列)
f的解就是系数矩阵A最小奇异值对应的奇异向量(V最后一列)
step2:奇异性约束(F的秩为2):
如果基本矩阵是非奇异的,那么所计算的对极线将不重合
m
i
n
∣
∣
F
−
F
′
∣
∣
2
,
s
.
t
.
d
e
t
(
F
′
)
=
0
F
=
U
S
V
T
,
F
′
=
U
∗
d
i
a
g
(
s
1
,
s
2
,
0
)
∗
V
T
min ||F-F'||_2 \ , \ \ s.t. det(F')=0\\ F=USV^T,F'=U*diag(s_1,s_2,0) * V^T
min∣∣F−F′∣∣2 , s.t.det(F′)=0F=USVT,F′=U∗diag(s1,s2,0)∗VT
为了提高解的稳定性和精度,往往会对输入点集的坐标先进行归一化处理。
其 中 归 一 化 变 换 T : 1. 将 点 进 行 平 移 使 其 形 心 ( x , y 的 均 值 ) 位 于 原 点 。 2. 对 点 进 行 缩 放 使 特 征 点 到 原 点 的 距 离 为 2 3. 对 两 幅 图 进 行 独 立 的 上 述 变 换 其中归一化变换T:\\ 1.将点进行平移使其形心(x,y的均值)位于原点。\\2.对点进行缩放使特征点到原点的距离为\sqrt 2\\3.对两幅图进行独立的上述变换 其中归一化变换T:1.将点进行平移使其形心(x,y的均值)位于原点。2.对点进行缩放使特征点到原点的距离为23.对两幅图进行独立的上述变换
重投影误差
l
1
=
F
∗
x
2
=
[
a
1
b
1
c
1
]
e
1
=
x
1
T
l
1
a
1
2
+
b
1
2
l_1=F*x_2=\begin{bmatrix}a_1\\b_1\\c_1\end{bmatrix}\\ e_1= \frac{x_1^Tl_1}{\sqrt{a_1^2+b_1^2}}
l1=F∗x2=⎣⎡a1b1c1⎦⎤e1=a12+b12x1Tl1
添加链接描述
clear all;
clc;
bDrawDebug = true;
bShouldNormalizePts = true;
bAssumeTrueMatches = true; % Whether or not to assume all matches are true
% load images
I1 = imread('house1.jpg');
I2 = imread('house2.jpg');
%Load match files for the first example
% This is a N x 4 file where the first two numbers of each row
% are coordinates of corners in the first image and the last two
% are coordinates of corresponding corners in the second image:
% matches(i,1:2) is a point in the first image
% matches(i,3:4) is a corresponding point in the second image
matches = load('house_matches.txt');
%matches = generate_matches(I1,I2, 100, false);
%matches = load('library_matches.txt');
%matches = generate_matches(I1,I2, 200, bDrawDebug);
numMatches = size(matches,1);
if(bDrawDebug)
%display an overlay of the features ontop of the image
figure; imshow([I1 I2]); hold on; title('Overlay detected features (corners)');
hold on; plot(matches(:,1),matches(:,2),'ys'); plot(matches(:,3) + size(I1,2), matches(:,4), 'ys');
% display two images side-by-side with matches
% this code is to help you visualize the matches, you don't need
% to use it to produce the results for the assignment
figure; imshow([I1 I2]); hold on;
plot(matches(:,1), matches(:,2), '+r');
plot(matches(:,3)+size(I1,2), matches(:,4), '+r');
line([matches(:,1) matches(:,3) + size(I1,2)]', matches(:,[2 4])', 'Color', 'r');
end
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Fit a Fundamental Matrix using the matches
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
if (bAssumeTrueMatches)
display('Assuming all matches are true and fitting to all');
F = fit_fundamental(matches, bShouldNormalizePts);
% this is a function that you should write
% bShouldNormalizePts: 是否进行归一化
else
display('Estimating the fundamental matrix');
%If the matches cannot be assumed to be ground truth, then estimate the
%fundamental matrix using RANSAC
F = estimate_fundamental(matches, bShouldNormalizePts);
end
residuals = calc_residuals(F,matches);
display(['Mean residual is: ' , num2str(mean(residuals))]);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% display second image with epipolar lines reprojected
% from the first image
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% transform points from the 1st img to get epipolar lines in the 2nd image
L = (F * [matches(:,1:2) ones(numMatches,1)]')';
% find points on epipolar lines L closest to matches(:,3:4)
L = L ./ repmat(sqrt(L(:,1).^2 + L(:,2).^2), 1, 3); % rescale the line
pt_line_dist = sum(L .* [matches(:,3:4) ones(numMatches,1)],2);
closest_pt = matches(:,3:4) - L(:,1:2) .* repmat(pt_line_dist, 1, 2);
% find endpoints of segment on epipolar line (for display purposes)
pt1 = closest_pt - [L(:,2) -L(:,1)] * 100; % offset from the closest point is 10 pixels
pt2 = closest_pt + [L(:,2) -L(:,1)] * 100;
% display points and segments of corresponding epipolar lines
figure;
imshow(I2); hold on;
plot(matches(:,3), matches(:,4), '+r');
line([matches(:,3) closest_pt(:,1)]', [matches(:,4) closest_pt(:,2)]', 'Color', 'r');
line([pt1(:,1) pt2(:,1)]', [pt1(:,2) pt2(:,2)]', 'Color', 'b');
function [ F ] = fit_fundamental( matches, bShouldNormalize )
%FIT_FUNDAMENTAL Summary of this function goes here
%Output...
% F: Fundamental Matrix
%Inputs...
% matches:
% This is a N x 4 file where the first two numbers of each row
% are coordinates of corners in the first image and the last two
% are coordinates of corresponding corners in the second image:
% matches(i,1:2) is a point in the first image
% matches(i,3:4) is a corresponding point in the second image
% bShouldNormalize:
% Whether or not the points should be normalized prior to estimating
% the fundamental matrix
%homogenize the points
x1 = cart_2_homo( matches(:,1:2) );
x2 = cart_2_homo( matches(:,3:4) );
if bShouldNormalize
%The linear system to be solved will have terms that involve a
%product between feature coordinates. If the coordinates are large,
%then F could end up w/ values that are orders of magnitude
%different from one another. This could yield poor behavior in
%practice. So normalize the coordinates, compute F in the
%normalized coordinates and then adjust it back to the original
%coordinates later using the normalization transformations
%display('Normalizing Coordinates');
[transform_1, x1_norm] = normalize_coordinates(x1);
[transform_2, x2_norm] = normalize_coordinates(x2);
x1 = x1_norm;
x2 = x2_norm;
end
u1 = x1(:,1);
v1 = x1(:,2);
u2 = x2(:,1);
v2 = x2(:,2);
%group at least 8 known matches together in a useful form
temp = [ u2.*u1, u2.*v1, u2, v2.*u1, v2.*v1, v2, u1, v1, ones(size(matches,1), 1)];
%Obtain an estimate for F by solving the homogenous linear system using
%those matches
%display('Solving homogenous linear system');
[~,~,V] = svd(temp);
f_vec = V(:,9);
F = reshape(f_vec, 3,3); %reshape the 9x1 vec into the 3x3 fund matrix
F = rank_2_constraint(F); %enforce the rank 2 constraint on F
if bShouldNormalize
%if F was computed from normalized points, transform it back to
%the original coordinates
F = transform_2' * F * transform_1;
end
end
计算F时的秩约束:
function [ rank2_mat ] = rank_2_constraint( mat )
%RANK_2_CONSTRAINT Enforce a rank 2 constraint on 3x3 input matrix
% Detailed explanation goes here
%display('Enforcing Rank 2 Constraint');
%take the svd of the input matrix
[U, S, V] = svd(mat);
%throw out the smallest singular value
S(end) = 0;
%recalcualte the matrix to yield the closest rank 2 constraint of the
%original
rank2_mat = U*S*V';
end
function residuals = calc_residuals(F, matches)
%CALC_RESIDUALS Summary of this function goes here
% Detailed explanation goes here
numMatches = size(matches,1);
L = (F * [matches(:,1:2) ones(numMatches,1)]')'; % transform points from
% the first image to get epipolar lines in the second image
% find points on epipolar lines L closest to matches(:,3:4)
L = L ./ repmat(sqrt(L(:,1).^2 + L(:,2).^2), 1, 3); % rescale the line
distances = sum(L .* [matches(:,3:4) ones(numMatches,1)],2); %distances from each pt to its line
residuals = abs(distances);
end