stable diffusion XL 和 t2i-adapter-sketch-sdxl的模型都可以从modelscope下载
import torch
from diffusers import T2IAdapter, StableDiffusionXLAdapterPipeline, DDPMScheduler
from diffusers.utils import load_image
from PIL import Image
sketch_image = load_image("https://www.modelscope.cn/api/v1/models/AI-ModelScope/t2iadapter_sketch_sd14v1/repo?Revision=master&FilePath=.%2Fimages%2Fsketch.png&View=true")
model_id = "./modelscope/hub/AI-ModelScope/stable-diffusion-xl-base-1___0"
adapter = T2IAdapter.from_pretrained(
"./modelscope/hub/AI-ModelScope/t2i-adapter-sketch-sdxl-1___0",
# subfolder="models_XL",
torch_dtype=torch.float16,
adapter_type="full_adapter_xl",
)
scheduler = DDPMScheduler.from_pretrained(model_id, subfolder="scheduler")
pipe = StableDiffusionXLAdapterPipeline.from_pretrained(
model_id, adapter=adapter, torch_dtype=torch.float16, variant="fp16", scheduler=scheduler
).to("cuda")
generator = torch.manual_seed(42)
sketch_image_out = pipe(
prompt="a photo of a room in real world, high quality",
negative_prompt="extra digit, fewer digits, cropped, worst quality, low quality",
image=sketch_image,
generator=generator,
guidance_scale=7.5,
).images[0]
sketch_image_out.save("sketch.jpg")
网络结构图
T2IAdapter 使用了AdapterBlock,总共四层,每层包含两个AdapterResnetBlock。可以得到每层的输出共计四个结果,和stable diffusion unet的encoder每层输出的结果相加,然后再一起进入下一个encoder参与计算。
controlnet是直接和encoder每层输出的结果相加,不再进入下一层encoder,而是skip connection进入到decoder部分。
controlnet代码中的一些细节
(代码用的是stable diffusion XL,和图片数值有出入)
网络结构代码
ModuleList(
(0): AdapterBlock(
(resnets): Sequential(
(0): AdapterResnetBlock( #(1,320,64,64)
(block1): Conv2d(320, 320, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(act): ReLU()
(block2): Conv2d(320, 320, kernel_size=(1, 1), stride=(1, 1))
)
(1): AdapterResnetBlock( )
)
)
(1): AdapterBlock(
(in_conv): Conv2d(320, 640, kernel_size=(1, 1), stride=(1, 1)) #(1,640,64,64)
(resnets): Sequential(
(0): AdapterResnetBlock( )
(1): AdapterResnetBlock( )
)
)
(2): AdapterBlock(
(downsample): AvgPool2d(kernel_size=2, stride=2, padding=0) #(1,640,32,32)
(in_conv): Conv2d(640, 1280, kernel_size=(1, 1), stride=(1, 1)) #(1,1280,32,32)
(resnets): Sequential(
(0): AdapterResnetBlock( )
(1): AdapterResnetBlock( )
)
)
(3): AdapterBlock(
(resnets): Sequential(
(0): AdapterResnetBlock() #(1,1280,32,32)
(1): AdapterResnetBlock()
)
)
)