0、论文
Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image. ICCV 19.arxiv.org1、项目地址
mks0601/3DMPPE_POSENET_RELEASEgithub.com![1f8635fa740db3bf6ec4abe9c09cdfea.png](https://i-blog.csdnimg.cn/blog_migrate/a3c319da7a92ad674028a44001b63b34.jpeg)
PoseNet部分是生成root-relative pose的部分。
2、my notes on the paper
banana16314:10. 3DMPPE: multi-person 3D pose estimationzhuanlan.zhihu.com3、The Code
3.1 指定backbone残差网络类型和结构
作为残差网络的backbone网络的结构图:
![a1b882d3e5e2b41443a822370a80884b.png](https://i-blog.csdnimg.cn/blog_migrate/59bc2efb3cb36b980d63966552783e86.png)
class ResNetBackbone(nn.Module):
def __init__(self, resnet_type):
resnet_spec = {18: (BasicBlock, [2, 2, 2, 2], [64, 64, 128, 256, 512], 'resnet18'),
34: (BasicBlock, [3, 4, 6, 3], [64, 64, 128, 256, 512], 'resnet34'),
50: (Bottleneck, [3, 4, 6, 3], [64, 256, 512, 1024, 2048], 'resnet50'),
101: (Bottleneck, [3, 4, 23, 3], [64, 256, 512, 1024, 2048], 'resnet101'),
152: (Bottleneck, [3, 8, 36, 3], [64, 256, 512, 1024, 2048], 'resnet152')}
block, layers, channels, name = resnet_spec[resnet_type]
self.name = name
self.inplanes = 64
super(ResNetBackbone, self).__init__()
self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3,
bias=False)
self.bn1 = nn.BatchNorm2d(64)
self.relu = nn.ReLU(inplace=True)
self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
self.layer1 = self._make_layer(block, 64, layers[0])
self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
for m in self.modules():
if isinstance(m, nn.Conv2d):
# nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
nn.init.normal_(m.weight, mean=0, std=0.001)
elif isinstance(m, nn.BatchNorm2d):
nn.init.constant_(m.weight, 1)
nn.init.constant_(m.bias, 0)
实验部分采用ResNet 50. batch_size为32.
Conv1:输入3通道的256x256大小的RGB图像,输出64通道,特征图尺寸64x64.
之后的每层,都是通过_make_layer()函数实现:
def _make_layer(self, block, planes, blocks, stride=1):
downsample = None
if stride != 1 or self.inplanes != planes * block.expansion:
downsample = nn.Sequential(
nn.Conv2d(self.inplanes, planes * block.expansion,
kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(planes * block.expansion),
)
layers = []
layers.append(block(self.inplanes, planes, stride, downsample))
self.inplanes = planes * block.expansion
for i in range(1, blocks):
layers.append(block(self.inplanes, planes))
return nn.Sequential(*layers)
向前传播,提取深层特征,即下采样层和各层的输出尺寸:
def forward(self, x):
# 32 3 256 256
x = self.conv1(x)
x = self.bn1(x)
x = self.relu(x)
x0 = self.maxpool(x)
#print(x.size())# 32 64 64 64
x1 = self.layer1(x0)
#print(x.size())# 32 256 64 64
x2 = self.layer2(x1)
#print(x.size()) # 32 512 32 32
x3 = self.layer3(x2)
#print(x.size())# 32 1024 16 16
x4 = self.layer4(x3)
# print(x.size())# 32 2048 8 8
return x4
最深层的特征图尺寸8x8,包含深层语义特征。
网络权重的初始化,从Model_zoo里面获取:
def init_weights(self):
org_resnet = torch.utils.model_zoo.load_url(url = model_urls[self.name])
# drop orginal resnet fc layer, add 'None' in case of no fc layer, that will raise error
org_resnet.pop('fc.weight', None)
org_resnet.pop('fc.bias', None)
self.load_state_dict(org_resnet)
print("Initialize resnet from model zoo")
整个backbone残差网络,80行代码搞定!
3.2 构建头部网络-headnet
头部网络headnet是对于backbone的特征图进行处理,获取各个3d坐标。
首先,8x8的特征图进行上采样恢复到64x64,这里采用的是 nn.ConvTranspose2d()转置卷积操作,相比nn.Upsample(),转置卷积需要学习参数;
最后,从特征图生成关节热力图,实现是由1x1卷积;
再从热力图获取坐标。
特征图的上采样:
self.deconv_layers = self._make_deconv_layer(3)
...
def _make_deconv_layer(self, num_layers):
layers = []
for i in range(num_layers):
layers.append(
nn.ConvTranspose2d(
in_channels=self.inplanes,
out_channels=self.outplanes,
kernel_size=4,
stride=2,
padding=1,
output_padding=0,
bias=False))
layers.append(nn.BatchNorm2d(self.outplanes))
layers.append(nn.ReLU(inplace=True))
self.inplanes = self.outplanes
return nn.Sequential(*layers)
8x8的特征图,经过3次转置卷积,尺寸恢复为64x64.
然后,1x1卷积生成热力图:
self.final_layer = nn.Conv2d(
in_channels=self.inplanes,
out_channels=joint_num * cfg.depth_dim,
kernel_size=1,
stride=1,
padding=0
)
定义头部网络headnet的forward:
def forward(self, x):
x = self.deconv_layers(x)
x = self.final_layer(x)
return x
3.3 backbone和head结合为PoseNet
class ResPoseNet(nn.Module):
def __init__(self, backbone, head, joint_num):
super(ResPoseNet, self).__init__()
self.backbone = backbone
self.head = head
self.joint_num = joint_num
def forward(self, input_img, target=None):
fm = self.backbone(input_img)
hm = self.head(fm)
coord = soft_argmax(hm, self.joint_num)
if target is None:
return coord
else:
target_coord = target['coord']
target_vis = target['vis']
target_have_depth = target['have_depth']
## coordinate loss
loss_coord = torch.abs(coord - target_coord) * target_vis
loss_coord = (loss_coord[:,:,0] + loss_coord[:,:,1] + loss_coord[:,:,2] * target_have_depth)/3.
return loss_coord
def get_pose_net(cfg, is_train, joint_num):
backbone = ResNetBackbone(cfg.resnet_type)
head_net = HeadNet(joint_num)
if is_train:
backbone.init_weights()
head_net.init_weights()
model = ResPoseNet(backbone, head_net, joint_num)
return model
其中,从热力图获取坐标,采取的操作是soft-argmax/integral operation. 原理可以参考:https://zhuanlan.zhihu.com/p/92417329
soft-argmax:
def soft_argmax(heatmaps, joint_num):
heatmaps = heatmaps.reshape((-1, joint_num, cfg.depth_dim*cfg.output_shape[0]*cfg.output_shape[1]))
heatmaps = F.softmax(heatmaps, 2)
heatmaps = heatmaps.reshape((-1, joint_num, cfg.depth_dim, cfg.output_shape[0], cfg.output_shape[1]))
accu_x = heatmaps.sum(dim=(2,3))
accu_y = heatmaps.sum(dim=(2,4))
accu_z = heatmaps.sum(dim=(3,4))
accu_x = accu_x * torch.cuda.comm.broadcast(torch.arange(1,cfg.output_shape[1]+1).type(torch.cuda.FloatTensor), devices=[accu_x.device.index])[0]
accu_y = accu_y * torch.cuda.comm.broadcast(torch.arange(1,cfg.output_shape[0]+1).type(torch.cuda.FloatTensor), devices=[accu_y.device.index])[0]
accu_z = accu_z * torch.cuda.comm.broadcast(torch.arange(1,cfg.depth_dim+1).type(torch.cuda.FloatTensor), devices=[accu_z.device.index])[0]
accu_x = accu_x.sum(dim=2, keepdim=True) -1
accu_y = accu_y.sum(dim=2, keepdim=True) -1
accu_z = accu_z.sum(dim=2, keepdim=True) -1
coord_out = torch.cat((accu_x, accu_y, accu_z), dim=2)
return coord_out
整个模型的构建117行代码。