论文链接: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
代码链接
作为backbone,用于各种下游任务,提供预训练模型,对标resnet。
swin transformer block:W-MSA+SW-MSA
W-MSA:Window Multi-head Self-Attention,窗口注意力机制
SW-MSA:Shifted Window Multi-head Self-Attention,滑动窗口注意力机制
分层特征图(hierarchical feature maps)
分层特征图是通过无卷积下采样来构建的,称为 Patch Merging。
转移窗口注意力(shifted window attention)
论文和程序中都要注意三个不同的概念,分别是resolution/ patches/ windows
resolution: 输入图片的分辨率是像素分辨率,程序Part 1 输入图片是像素分辨率,但是Part 2程序中对应的H/W是patches 分辨率,不是像素分辨率。
patches: 图像4*4像素区域称为一个patch,分类任务输入图像像素分辨率是224*224,patch_size = 4,所以patches__resolution = 56*56
windows: 窗口大小由patches定义的,不是像素定义的,论文及程序中window_size = 7,说明一个window有7*7=49个patches
Part 1 :图像处理
Part 1包含两部分,Patch Partition 将像素分辨率图像转换为patches分辨率的图像,每个patch视为一个token,特征就是patch范围内的RGB值的展开,token_feature = 48;Linear Embedding 将token_feature转换为需要的维度(Swin_T/C=96) ,以上两个部分在程序中通过PatchEmbed类同时实现:
class PatchEmbed(nn.Module):
r""" Image to Patch Embedding
Args:
img_size (int): Image size. Default: 224.
patch_size (int): Patch token size. Default: 4.
in_chans (int): Number of input image channels. Default: 3.
embed_dim (int): Number of linear projection output channels. Default: 96. Swin_T.C = Swin_S.C = 96 Swin_B.C = 128 Swin_L.C = 192
norm_layer (nn.Module, optional): Normalization layer. Default: None
return:
(3136,96):序列长度是3136((224/4)*(224/4))个,每个向量是96维特征
"""
def __init__(self, img_size=224, patch_size=4, in_chans=3, embed_dim=96, norm_layer=None):
super().__init__()
img_size = to_2tuple(img_size) # img_size = (img_size, img_size)
patch_size = to_2tuple(patch_size) # patch_size = (patch_size, patch_size)
patches_resolution = [img_size[0] // patch_size[0], img_size[1] // patch_size[1]]
self.img_size = img_size # self.img_size = (224, 224)
self.patch_size = patch_size # self.patch_size = (4, 4)
self.patches_resolution = patches_resolution # self.patches_resolution = (56, 56)
self.num_patches = patches_resolution[0] * patches_resolution[1]
self.in_chans = in_chans # 3
self.embed_dim = embed_dim # 96
# 3 96 (4,4) (4,4)
self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_size, stride=patch_size)
if norm_layer is not None:
self.norm = norm_layer(embed_dim)
else:
self.norm = None
def forward(self, x):
B, C, H, W = x.shape
'''
输入图像分辨率必须为(224,224)
'''
# FIXME look at relaxing size constraints
assert H == self.img_size[0] and W == self.img_size[1], \
f"Input image size ({H}*{W}) doesn't match model ({self.img_size[0]}*{self.img_size[1]})."
'''
x.shape = (B, 3, H, W)
self.proj(x).shape = (B, 96, H//4, W//4)
self.proj(x).flatten(2).shape = (B, 96, H//4 * W//4)
self.proj(x).flatten(2).transpose(1,2).shape = (B, H//4 * W//4, 96)
'''
x = self.proj(x).flatten(2).transpose(1, 2) # B Ph*Pw C
if self.norm is not None:
x = self.norm(x)
return x # x.shape = (B, L, C) L = H*W 这里的H,W是patches的分辨率,不是输入x的像素分辨率 C = 96
def flops(self):
Ho, Wo = self.patches_resolution
flops = Ho * Wo * self.embed_dim * self.in_chans * (self.patch_size[0] * self.patch_size[1])
if self.norm is not None:
flops += Ho * Wo * self.embed_dim
return flops
Part 2 : Swin Transformer Stage
Swin Transformer Stage 包括两个部分:Patch Merging 和 Swin Transformer Block
1. Patch Merging
Patch Merging的作用是分辨率减半,通道数加倍,类似于CNN的作用,在Transformer中实现Hierarchical
class PatchMerging(nn.Module):
# PatchMerging作用: 输入分辨率减半,通道数翻倍
# 这里的分辨率是转换成patches的分辨率,不是原图像素的分辨率
r""" Patch Merging Layer.
Args:
input_resolution (tuple[int]): Resolution of input feature.
dim (int): Number of input channels.
norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm
"""
def __init__(self, input_resolution, dim, norm_layer=nn.LayerNorm):
super().__init__()
self.input_resolution = input_resolution
self.dim = dim
self.reduction = nn.Linear(4 * dim, 2 * dim, bias=False)
self.norm = norm_layer(4 * dim)
def forward(self, x):
"""
x: B, H*W, C
"""
H, W = self.input_resolution
B, L, C = x.shape
assert L == H * W, "input feature has wrong size"
assert H % 2 == 0 and W % 2 == 0, f"x size ({H}*{W}) are not even."
x = x.view(B, H, W, C)
# 0::2 偶数
# 1::2 奇数
x0 = x[:, 0::2, 0::2, :] # B H/2 W/2 C
x1 = x[:, 1::2, 0::2, :] # B H/2 W/2 C
x2 = x[:, 0::2, 1::2, :] # B H/2 W/2 C
x3 = x[:, 1::2, 1::2, :] # B H/2 W/2 C
x = torch.cat([x0, x1, x2, x3], -1) # B H/2 W/2 4*C
x = x.view(B, -1, 4 * C) # B H/2*W/2 4*C
x = self.norm(x)
x = self.reduction(x) # 本来是C 后来变为4C,通过self.reduction(x)变回2C,最终由 C-->2C, 达到分辨率减半,通道数加倍的结果
return x
# 输入x.shape = (B, H*W, C),输出x.shape = (B, H/2*W/2, 2C) 分辨率减半,通道数翻倍
def extra_repr(self) -> str:
return f"input_resolution={self.input_resolution}, dim={self.dim}"
def flops(self):
H, W = self.input_resolution
flops = H * W * self.dim
flops += (H // 2) * (W // 2) * 4 * self.dim * 2 * self.dim
return flops
参考链接:
Swin-Transformer 详解_swin transformer-CSDN博客
Swin Transformer 论文详解及程序解读 - 知乎
https://github.com/SwinTransformer/Swin-Transformer-Object-Detection