Diffusers代码学习-SDXL Inpainting

参考之前的文章（见这里Diffusers代码学习-Inpainting），Inpaiting的思路是一致的，需要原始图像和要在原始图像中替换的内容的蒙版图片。然后创建一个提示词来描述要用什么来替换蒙版区域。

# 以下代码为程序运行进行设置

import os
os.environ["HF_ENDPOINT"] = "https://hf-mirror.com"
rom diffusers import AutoPipelineForText2Image
import torch

pipeline_text2image = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, variant="fp16", use_safetensors=True
).to("cuda")

prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipeline_text2image(prompt=prompt).images[0]

from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid

# 这里使用from_pipe方法加载checkpoint,避免额外消耗内存

pipeline = AutoPipelineForInpainting.from_pipe(pipeline_text2image).to("cuda")

# 加载原始图片和蒙版图片

img_url = "https://hf-mirror.com/datasets/huggingface/documentation-images/resolve/main/diffusers/sdxl-text2img.png"
mask_url = "https://hf-mirror.com/datasets/huggingface/documentation-images/resolve/main/diffusers/sdxl-inpaint-mask.png"

init_image = load_image(img_url)
mask_image = load_image(mask_url)

# 通过提示词，原始图片和蒙版图片生成新图片

prompt = "A deep sea diver floating"
image = pipeline(prompt=prompt, image=init_image, mask_image=mask_image, strength=0.85, guidance_scale=12.5).images[0]
image2 = make_image_grid([init_image, mask_image, image], rows=1, cols=3)

image2.show()

以下为原始图片，蒙版图片和生成结果的对比图。