Image Inpainting with Cascaded Modulation GAN and Object-Aware Training

Haitian Zheng1,2, Zhe Lin2, Jingwan Lu2, Scott Cohen2, Eli Shechtman2, Connelly Barnes2, Jianming Zhang2, Ning Xu2, Sohrab Amirghodsi2, and Jiebo Luo1

Recent image inpainting methods have made great progress but often struggle to generate plausible image structures when dealing with large holes in complex images. This is partially due to the lack of effective network structures that can capture both the long-range dependency and high-level semantics of an image. We propose cascaded modulation GAN (CM-GAN), a new network design consisting of an encoder with Fourier convolution blocks that extract multi-scale feature representations from the input image with holes and a dual-stream decoder with a novel cascaded global-spatial modulation block at each scale level. In each decoder block, global modulation is first applied to perform coarse and semantic-aware structure synthesis, followed by spatial modulation to further adjust the feature map in a spatially adaptive fashion. In