PatchMatchNet单应变换公式推导

Camera file cam.txt stores the camera parameters, which includes extrinsic, intrinsic, minimum depth and maximum depth:

extrinsic
E00 E01 E02 E03
E10 E11 E12 E13
E20 E21 E22 E23
E30 E31 E32 E33

intrinsic
K00 K01 K02
K10 K11 K12
K20 K21 K22

DEPTH_MIN DEPTH_MAX 

$$\begin{gathered} extrinsic=\mathbf{T}=\left[\begin{array}{cc}\mathbf{R}& \mathbf{t}\\ 0^T& 1\end{array}\right] \\ intrinsic=\mathbf{K}=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right] \end{gathered}$$

$$d\mathbf{p}=d\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\mathbf{K}({\mathbf{R}\mathbf{P}}_w+\mathbf{t})=({\mathbf{K}\mathbf{R}\mathbf{P}}_w+\mathbf{K}\mathbf{t})=\left[\begin{array}{cc}\mathbf{K}\mathbf{R}& \mathbf{K}\mathbf{t}\end{array}\right]\left[\begin{array}{c}{\mathbf{P}}_w\\ 1\end{array}\right]$$
$${\mathbf{P}}_w={\mathbf{R}}^{-1}({\mathbf{K}}^{-1}\mathbf{p}d-\mathbf{t})$$
${\mathbf{P}}_w$为世界坐标系下的坐标。

考虑参考图中的某个像素$\mathbf{p}$，以及其深度$d_j$，计算这个点在源图$i$中的坐标：
$$\begin{array}{rl}{\mathbf{p}}_{i,j}& ={\mathbf{p}}_i(d_j)\\ & ={\mathbf{K}}_i({\mathbf{R}}_i{\mathbf{P}}_w+{\mathbf{t}}_i)\\ & ={\mathbf{K}}_i({\mathbf{R}}_i{\mathbf{R}}_0^{-1}({\mathbf{K}}_0^{-1}\mathbf{p}{d}_j-{\mathbf{t}}_0)+{\mathbf{t}}_i)\\ & ={\mathbf{K}}_i{\mathbf{R}}_i{\mathbf{R}}_0^{-1}{\mathbf{K}}_0^{-1}\mathbf{p}{d}_j-{\mathbf{K}}_i{\mathbf{R}}_i{\mathbf{R}}_0^{-1}{\mathbf{t}}_0+{\mathbf{K}}_i{\mathbf{t}}_i\\ & =\hat{\mathbf{R}}\mathbf{p}{d}_j+\hat{\mathbf{t}}\end{array}$$

proj_mat = extrinsics.copy()
proj_mat[:3, :4] = np.matmul(intrinsics, proj_mat[:3, :4])

有下式：
$$\mathbf{p}\mathbf{r}\mathbf{o}\mathbf{j}\_\mathbf{m}\mathbf{a}\mathbf{t}=\left[\begin{array}{cc}\mathbf{K}\mathbf{R}& \mathbf{K}\mathbf{t}\\ 0^T& 1\end{array}\right]$$
$${\mathbf{p}\mathbf{r}\mathbf{o}\mathbf{j}\_\mathbf{m}\mathbf{a}\mathbf{t}}^{-1}=\left[\begin{array}{cc}{\mathbf{R}}^{-1}{\mathbf{K}}^{-1}& -{\mathbf{R}}^{-1}\mathbf{t}\\ 0^T& 1\end{array}\right]$$

proj = torch.matmul(src_proj, torch.inverse(ref_proj))
rot = proj[:, :3, :3]  # [B,3,3]
trans = proj[:, :3, 3:4]  # [B,3,1]

$$\begin{array}{rl}\mathbf{p}\mathbf{r}\mathbf{o}\mathbf{j}& ={\mathbf{p}\mathbf{r}\mathbf{o}\mathbf{j}\_\mathbf{m}\mathbf{a}\mathbf{t}}_i\times {\mathbf{p}\mathbf{r}\mathbf{o}\mathbf{j}\_\mathbf{m}\mathbf{a}\mathbf{t}}_0^{-1}\\ & =\left[\begin{array}{cc}{\mathbf{K}}_i{\mathbf{R}}_i& {\mathbf{K}}_i{\mathbf{t}}_i\\ 0^T& 1\end{array}\right]\left[\begin{array}{cc}{\mathbf{R}}_0^{-1}{\mathbf{K}}_0^{-1}& -{\mathbf{R}}_0^{-1}{\mathbf{t}}_0\\ 0^T& 1\end{array}\right]\\ & =\left[\begin{array}{cc}{\mathbf{K}}_i{\mathbf{R}}_i{\mathbf{R}}_0^{-1}{\mathbf{K}}_0^{-1}& -{\mathbf{K}}_i{\mathbf{R}}_i{\mathbf{R}}_0^{-1}{\mathbf{t}}_0+{\mathbf{K}}_i{\mathbf{t}}_i\\ 0^T& 1\end{array}\right]\\ & =\left[\begin{array}{cc}\hat{\mathbf{R}}& \hat{\mathbf{t}}\\ 0^T& 1\end{array}\right]\\ & =\left[\begin{array}{cc}\mathbf{r}\mathbf{o}\mathbf{t}& \mathbf{t}\mathbf{r}\mathbf{a}\mathbf{n}\mathbf{s}\\ 0^T& 1\end{array}\right]\end{array}$$