Image Inpainting for Irregular Holes Using Partial Convolutions 论文笔记

损失函数

由6部分组成，太复杂了。
$${\mathcal{L}}_{total}={\mathcal{L}}_{valid}+6{\mathcal{L}}_{hole}+0.05{\mathcal{L}}_{perceptual}+120({\mathcal{L}}_{styleout}+{\mathcal{L}}_{stylecomp})+0.1{\mathcal{L}}_{tv}$$

pixel loss

两部分组成，分别是空洞部分和非空洞部分。
$${\mathcal{L}}_{\text{hole }}=\frac{1}{N_{{\mathbf{I}}_{gt}}}{\Vert (1-M)\odot \left({\mathbf{I}}_{out}-{\mathbf{I}}_{gt}\right)\Vert }_1$$
$${\mathcal{L}}_{\text{valid }}=\frac{1}{N_{{\mathbf{I}}_{gt}}}{\Vert M\odot \left({\mathbf{I}}_{\text{out }}-{\mathbf{I}}_{gt}\right)\Vert }_1$$

perceptual loss

$${\mathcal{L}}_{\text{perceptual }}=\sum _{p=0}^{P-1}\frac{{\Vert {\Psi }_p^{{\mathbf{I}}_{out}}-{\Psi }_p^{{\mathbf{I}}_{gt}}\Vert }_1}{N_{{\Psi }_p}{\mathbf{I}}_{gt}}+\sum _{p=0}^{P-1}\frac{{\Vert {\Psi }_p^{{\mathbf{I}}_{\text{comp }}}-{\Psi }_p^{{\mathbf{I}}_{gt}}\Vert }_1}{N_{{\Psi }_p}{\mathbf{I}}_{gt}}$$

style loss

comp表示用真值填充非空洞部分，输出填充空洞部分，style loss要分别计算comp和out的损失值。
$$\begin{array}{rl}& {\mathcal{L}}_{{\text{style }}_{\text{out }}}=\sum _{p=0}^{P-1}\frac{1}{C_p{C}_p}{\Vert K_p\left({\left({\Psi }_p^{{\mathbf{I}}_{out}}\right)}^{\top }\left({\Psi }_p^{{\mathbf{I}}_{out}}\right)-{\left({\Psi }_p^{{\mathbf{I}}_{gt}}\right)}^{\top }\left({\Psi }_p^{{\mathbf{I}}_{gt}}\right)\right)\Vert }_1\\ & {\mathcal{L}}_{styl{e}_{comp}}=\sum _{p=0}^{P-1}\frac{1}{C_p{C}_p}{\Vert K_p\left({\left({\Psi }_p^{{\mathbf{I}}_{comp}}\right)}^{\top }\left({\Psi }_p^{{\mathbf{I}}_{comp}}\right)-{\left({\Psi }_p^{{\mathbf{I}}_{gt}}\right)}^{\top }\left({\Psi }_p^{{\mathbf{I}}_{gt}}\right)\right)\Vert }_1\end{array}$$

total variation —— tv

为R上的平滑惩罚，其中R为空洞区域1像素膨胀的区域。
$${\mathcal{L}}_{tv}=\sum _{(i,j)\in R,(i,j+1)\in R}\frac{{\Vert {\mathbf{I}}_{\text{comp }}^{i,j+1}-{\mathbf{I}}_{\text{comp }}^{i,j}\Vert }_1}{N_{{\mathbf{I}}_{\text{comp }}}}+\sum _{(i,j)\in R,(i+1,j)\in R}\frac{{\Vert {\mathbf{I}}_{\text{comp }}^{i+1,j}-{\mathbf{I}}_{\text{comp }}^{i,j}\Vert }_1}{N_{{\mathbf{I}}_{\text{comp }}}}$$