Global Motion
1. What is global motion?
Estimate motion using all pixels in the image.
Parametric flow gives an equation, which describes optical flow for each pixel
Affine Flow
1. How to use Bergen et al. method to calculate affine motion?
u ⃗ = [ u ( x , y ) v ( x , y ) ] = [ x y 1 0 0 0 0 0 0 x y 1 ] [ a 1 a 2 b 1 a 3 a 4 b 2 ] u ⃗ ( x ⃗ ) = X x ⃗ a ⃗ \begin{aligned} \vec{u} &= \begin{bmatrix} u(x,y) \\ v(x,y) \end{bmatrix} = \begin{bmatrix} x & y & 1 & 0 & 0 &0 \\ 0 & 0 & 0 & x & y & 1 \\ \end{bmatrix} \begin{bmatrix} a_1 \\ a_2 \\ b_1 \\ a_3 \\ a_4 \\ b_2 \\ \end{bmatrix} \\ \vec{u}(\vec{x}) &= X_{\vec{x}}\vec{a} \end{aligned} uu(x)=[u(x,y)v(x,y)]=[x0y0100x0y01]⎣⎢⎢⎢⎢⎢⎢⎡a1a2b1a3a4b2⎦⎥⎥⎥⎥⎥⎥⎤=Xxa
So global motion is function w.r.t.
a
⃗
\vec{a}
a, and our target is to get the
min
E
(
u
⃗
)
=
∑
all pixel
(
I
t
+
I
x
u
+
I
y
v
)
2
=
∑
all pixel
(
I
t
+
I
⃗
T
u
⃗
)
2
min
E
(
a
⃗
)
=
∑
all pixel
(
I
t
+
I
⃗
T
X
x
⃗
a
⃗
)
2
⟶
∂
E
(
a
⃗
)
∂
a
⃗
=
∑
all pixel
2
(
I
t
+
I
⃗
T
X
x
⃗
a
⃗
)
I
⃗
T
X
x
⃗
=
0
∑
X
x
⃗
T
I
x
⃗
⃗
I
x
⃗
⃗
T
X
x
⃗
a
⃗
=
∑
X
x
⃗
T
I
x
⃗
⃗
I
t
\begin{aligned} \min E(\vec{u}) &= \sum_{\text{all pixel}} (I_t + I_x u+I_y v)^2 \\ &= \sum_{\text{all pixel}} (I_t + \vec{I}^T \vec{u})^2 \\ \min E(\vec{a})&=\sum_{\text{all pixel}} (I_t + \vec{I}^T X_{\vec{x}} \vec{a})^2 \\ \longrightarrow \frac{\partial E(\vec{a})}{\partial \vec{a}} &= \sum_{\text{all pixel}} 2 (I_t + \vec{I}^T X_{\vec{x}} \vec{a})\vec{I}^TX_{\vec{x}} = 0 \\ \sum X_{\vec{x}}^T\vec{I_{\vec{x}}}\vec{I_{\vec{x}}}^T X_{\vec{x}}\pmb{\vec{a}} &= \sum X_{\vec{x}}^T \vec{I_{\vec{x}}} I_t \end{aligned}
minE(u)minE(a)⟶∂a∂E(a)∑XxTIxIxTXxaaa=all pixel∑(It+Ixu+Iyv)2=all pixel∑(It+ITu)2=all pixel∑(It+ITXxa)2=all pixel∑2(It+ITXxa)ITXx=0=∑XxTIxIt
Remark: I t ∈ R I_t \in \mathbb{R} It∈R， I x ⃗ = [ I x I y ] I_{\vec{x}}= \begin{bmatrix} I_x \\ I_y \end{bmatrix} Ix=[IxIy]
Coarse-to-Fine (Pyramid) Global Flow
In hierarchical global flow estimation, we need
- start from top level, set a ⃗ 0 = 0 \vec{a}_0=0 a0=0
- warp the image by a ⃗ 0 \vec{a}_0 a0
- use formula we derived from min E ( u ⃗ ) \min E(\vec{u}) minE(u) to get δ a ⃗ \delta \vec{a} δa
- let a ⃗ 1 = a ⃗ 0 + δ a ⃗ \vec{a}_1 = \vec{a}_0 + \delta \vec{a} a1=a0+δa and move a layer down
- repeat step 1~4
Projective Flow
1. What are the defectives of affine flow?
Affine flow cannot capture camera pan and tilt. Any relative change of points along Z-Axis (the axis goes through camera) are beyond the capacity of affine transformation.
We can build the equation system as follows:
[
A
b
⃗
c
⃗
T
1
]
\begin{bmatrix} A & \vec{b} \\ \vec{c}^T & 1\\ \end{bmatrix}
[AcTb1]
{ u ⃗ T I x ⃗ + I t = 0 u ⃗ = x ⃗ ′ − x ⃗ = A x ⃗ + b ⃗ c ⃗ T x ⃗ + 1 − x ⃗ \begin{aligned} \begin{cases} \vec{u}^T I_{\vec{x}} + I_t = 0 \\ \vec{u} = \vec{x}' -\vec{x} = \frac{A\vec{x} + \vec{b}}{\vec{c}^T \vec{x} + 1} - \vec{x} \end{cases} \end{aligned} {uTIx+It=0u=x′−x=cTx+1Ax+b−x
So we have
min
ε
f
l
o
w
=
min
∑
(
u
⃗
T
I
x
⃗
+
I
t
)
2
=
min
∑
(
(
A
x
⃗
+
b
⃗
c
⃗
T
x
⃗
+
1
−
x
⃗
)
T
−
I
x
⃗
+
I
t
)
2
=
min
∑
(
(
A
x
⃗
+
b
⃗
−
(
c
⃗
T
x
⃗
+
1
)
x
⃗
)
T
I
x
⃗
+
(
c
⃗
T
x
⃗
+
1
)
I
t
)
2
Let
A
=
[
a
1
a
2
a
3
a
4
]
,
b
⃗
=
[
b
1
b
2
]
,
c
⃗
=
[
c
1
c
2
]
,
x
⃗
=
[
x
y
]
,
I
x
⃗
=
[
I
x
I
y
]
\begin{aligned} \min\varepsilon_{flow} &= \min \sum (\vec{u}^T I_{\vec{x}} + I_t)^2 \\ &= \min \sum ((\frac{A\vec{x} + \vec{b}}{\vec{c}^T \vec{x} + 1} - \vec{x})^T - I_{\vec{x}} + I_t)^2 \\ &= \min\sum ((A\vec{x} + \vec{b} - (\vec{c}^T \vec{x} + 1)\vec{x})^T I_{\vec{x}} + (\vec{c}^T \vec{x} + 1) I_t)^2 \\ \text{Let } A &= \begin{bmatrix} a_1 & a_2\\ a_3 & a_4\\ \end{bmatrix}, \vec{b} = \begin{bmatrix} b_1 \\ b_2 \\ \end{bmatrix}, \vec{c} = \begin{bmatrix} c_1 \\ c_2 \end{bmatrix}, \vec{x} = \begin{bmatrix} x \\ y \\ \end{bmatrix}, I_{\vec{x}}= \begin{bmatrix} I_x \\ I_y \\ \end{bmatrix} \\ \end{aligned}
minεflowLet A=min∑(uTIx+It)2=min∑((cTx+1Ax+b−x)T−Ix+It)2=min∑((Ax+b−(cTx+1)x)TIx+(cTx+1)It)2=[a1a3a2a4],b=[b1b2],c=[c1c2],x=[xy],Ix=[IxIy]
break it down, we can write it in the form:
min
ε
=
∑
[
ϕ
⃗
T
a
⃗
−
(
x
⃗
T
I
x
⃗
−
I
t
)
]
2
ϕ
⃗
=
[
I
x
x
I
x
y
I
x
I
y
x
I
y
y
I
y
x
(
I
t
−
x
I
x
−
y
I
y
)
y
(
I
t
−
x
I
x
−
y
I
y
)
]
,
a
⃗
=
[
a
1
a
2
b
1
a
3
a
4
b
2
c
1
c
2
]
\begin{aligned} \min\varepsilon &= \sum [\vec{\phi}^T \vec{a} - (\vec{x}^TI_{\vec{x}} - I_t)]^2 \\ \vec{\phi} =\begin{bmatrix} I_x x\\ I_x y\\ I_x \\ I_y x \\ I_y y\\ I_y \\ x(I_t - x I_x - yI_y) \\ y(I_t - x I_x - yI_y) \end{bmatrix} &, \vec{a} = \begin{bmatrix} a_1 \\ a_2 \\ b_1 \\ a_3 \\ a_4 \\ b_2 \\ c_1 \\ c_2 \\ \end{bmatrix} \\ \end{aligned}
minεϕ=⎣⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎡IxxIxyIxIyxIyyIyx(It−xIx−yIy)y(It−xIx−yIy)⎦⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎤=∑[ϕTa−(xTIx−It)]2,a=⎣⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎡a1a2b1a3a4b2c1c2⎦⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎤
and take the derivative, we then have:
∑ ∂ ε ∂ a ⃗ = 0 ∑ 2 ϕ ⃗ ( ϕ ⃗ T a ⃗ − ( x ⃗ T I x ⃗ − I t ) ) = 0 ( ∑ ϕ ⃗ ϕ ⃗ T ) a ⃗ = ∑ ( x ⃗ T I x ⃗ − I t ) ϕ ⃗ \begin{aligned} \sum \frac{\partial \varepsilon}{\partial \vec{a}} &= 0 \\ \sum 2\vec{\phi}(\vec{\phi}^T\vec{a} - (\vec{x}^TI_{\vec{x}} - I_t)) &= 0 \\ (\sum \vec{\phi}\vec{\phi}^T) \vec{a} &= \sum (\vec{x}^TI_{\vec{x}} - I_t) \vec{\phi} \end{aligned} ∑∂a∂ε∑2ϕ(ϕTa−(xTIx−It))(∑ϕϕT)a=0=0=∑(xTIx−It)ϕ
Pseudo Perspective
We can simplify the perspective transformation by linear approximation like pseudo perspective and bilinear. Then repeat the processes above.
扫一扫
284
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
