深度学习系列15：人群热力图的制作

1. 问题介绍

通过深度学习的方法，可以识别出区域人流的密度，如下图：

再通过摄像头坐标投影转换到地图上，就可以绘制人群热力图了。资料集合见：https://github.com/gjy3035/Awesome-Crowd-Counting
https://zhuanlan.zhihu.com/p/266434424
技术路线大致有3个方向：
1）行人检测，包括传统cv方法和深度学习模型方法，但是这个方法对遮挡非常敏感。
2）机器学习回归法，直接回归行人数。
3）密度图法，当前主流。密度图的制作方法：使用高斯核模拟人头，然后做一下normalization。这里有几个分类：① 固定尺寸密度图，使用同样的高斯核，不受perspective变换影响；② 透视密度图，不同位置的人头用不同的高斯核。③ KNN密度图，使用k个最近的人头决定当前人头的高斯核。
至于人数，密度图的积分就是人数。详细可参考这篇:https://www.zhihu.com/question/406153313/answer/1471711287

2. 问题背景

2.1 训练数据

《Learning from Synthetic Data for Crowd Counting in the Wild》由西北工业大学的学者提出，使用计算机图形工具创建拥挤人群数据集，并开源了他们创建的大型数据集，在此数据集上训练的算法精度获得了大幅提升，超越了之前的state-of-the-art。作者已经公开了该数据集，并开源了数据创建标注工具：https://github.com/gjy3035/GCC-CL

项目主页：https://gjy3035.github.io/GCC-CL/

论文地址：https://arxiv.org/pdf/1903.03303.pdf

其他数据集还包括：
UCF_CC_50（UCF50），Shanghai Tech Part A/B（SHT A/B），WorldExpo’10（WE），UCF-QNRF（QNRF），GCC、UCSD、MALL

2.2 模型

下面是从图片来进行分析的模型清单：
[SKT] Efficient Crowd Counting via Structured Knowledge Transfer (ACM MM(oral)) [paper][code]
[C-CNN] A Real-Time Deep Network for Crowd Counting (ICASSP) [paper]
[MobileCount] MobileCount: An Efficient Encoder-Decoder Framework for Real-Time Crowd Counting [conference paper] [journal paper] [code]
[PCC-Net] PCC Net: Perspective Crowd Counting via Spatial Convolutional Network (T-CSVT) [paper] [code]
[SRN+PS] Scale-Recursive Network with point supervision for crowd scene analysis (Neurocomputing) [paper]
[SANet] Scale Aggregation Network for Accurate and Efficient Crowd Counting (ECCV) [paper]
[ACSCP] Crowd Counting via Adversarial Cross-Scale Consistency Pursuit (CVPR) [paper] [unofficial code: PyTorch]
[CMTL] CNN-based Cascaded Multi-task Learning of High-level Prior and Density Estimation for Crowd Counting (AVSS) [paper] [code]
[MCNN] Single-Image Crowd Counting via Multi-Column Convolutional Neural Network (CVPR) [paper] [unofficial code: TensorFlow PyTorch]
下面是从视频流来分析的模型清单：
[STDNet] Spatiotemporal Dilated Convolution with Uncertain Matching for Video-based Crowd Estimation (TMM) [paper]
[EPF] Estimating People Flows to Better Count them in Crowded Scenes (ECCV) [paper]
[MLSTN] Multi-level feature fusion based Locality-Constrained Spatial Transformer network for video crowd counting (Neurocomputing) [paper](extension of LSTN)
[LSTN] Locality-Constrained Spatial Transformer Network for Video Crowd Counting (ICME(oral)) [paper]
Fast Video Crowd Counting with a Temporal Aware Network [paper]
[ConvLSTM] Spatiotemporal Modeling for Crowd Counting in Videos (ICCV) [paper]

3. 上手实践

3.1 NWPU库

这里不涉及训练，仅进行test。使用如下的git库：
git clone https://github.com/gjy3035/NWPU-Crowd-Sample-Code.git
pip install -r requirements.txt安装必须的库
下载预训练模型：http://share.crowdbenchmark.com:2443/home/Pre-trained_Models_NWPU-Crowd，我这里用的是SCAR。模型有如下选择：

然后运行下面的代码：注意修改net使用的模型参数，以及模型地址的位置。

from matplotlib import pyplot as plt

import matplotlib
import os
import random
import torch
from torch.autograd import Variable
import torchvision.transforms as standard_transforms
import misc.transforms as own_transforms
import pandas as pd

from models.CC import CrowdCounter
from config import cfg
from misc.utils import *
import scipy.io as sio
from PIL import Image, ImageOps

torch.cuda.set_device(0)
torch.backends.cudnn.benchmark = True

mean_std = ([0.446139603853, 0.409515678883, 0.395083993673], [0.288205742836, 0.278144598007, 0.283502370119])
img_transform = standard_transforms.Compose([
        standard_transforms.ToTensor(),
        standard_transforms.Normalize(*mean_std)
    ])
restore = standard_transforms.Compose([
        own_transforms.DeNormalize(*mean_std),
        standard_transforms.ToPILImage()
    ])
pil_to_tensor = standard_transforms.ToTensor()
LOG_PARA = 100.0
net = CrowdCounter(cfg.GPU_ID, 'SCAR')
net.cuda()
model_path = 'saved_exp_para/SCAR/SCAR-latest.pth'
net.load_state_dict(torch.load(model_path))
net.eval()
imgname = '2.jpeg'
img = Image.open(imgname)
if img.mode == 'L':
    img = img.convert('RGB')
img = img_transform(img)[None, :, :, :]
with torch.no_grad():
    img = Variable(img).cuda()
    crop_imgs, crop_masks = [], []
    b, c, h, w = img.shape
    rh, rw = 576, 768
    for i in range(0, h, rh):
        gis, gie = max(min(h-rh, i), 0), min(h, i+rh)
        for j in range(0, w, rw):
            gjs, gje = max(min(w-rw, j), 0), min(w, j+rw)
            crop_imgs.append(img[:, :, gis:gie, gjs:gje])
            mask = torch.zeros(b, 1, h, w).cuda()
            mask[:, :, gis:gie, gjs:gje].fill_(1.0)
            crop_masks.append(mask)
    crop_imgs, crop_masks = map(lambda x: torch.cat(x, dim=0), (crop_imgs, crop_masks))

    # forward may need repeatng
    crop_preds = []
    nz, bz = crop_imgs.size(0), 1
    for i in range(0, nz, bz):
        gs, gt = i, min(nz, i+bz)
        crop_pred = net.test_forward(crop_imgs[gs:gt])
        crop_preds.append(crop_pred)
    crop_preds = torch.cat(crop_preds, dim=0)

    # splice them to the original size
    idx = 0
    pred_map = torch.zeros(b, 1, h, w).cuda()
    for i in range(0, h, rh):
        gis, gie = max(min(h-rh, i), 0), min(h, i+rh)
        for j in range(0, w, rw):
            gjs, gje = max(min(w-rw, j), 0), min(w, j+rw)
            pred_map[:, :, gis:gie, gjs:gje] += crop_preds[idx]
            idx += 1

    # for the overlapping area, compute average value
    mask = crop_masks.sum(dim=0).unsqueeze(0)
    pred_map = pred_map / mask

pred_map = pred_map.cpu().data.numpy()[0,0,:,:]
#pred = np.sum(pred_map) / LOG_PARA
from skimage import io
io.imshow(1-pred_map/np.max(pred_map))

如果用Res101_SFCN，需要做一下如下修改：

from collections import OrderedDict
net = CrowdCounter(cfg.GPU_ID, 'Res101_SFCN')
net.cuda()
model_path = 'saved_exp_para/Res_SFCN/SFCN+-all_ep_321_mae_90.7_mse_487.2_nae_0.375.pth'
modeldata = torch.load(model_path)
modeldata2 = OrderedDict()
for k,v in modeldata.items():
    modeldata2[k.replace('module.','')] = v
net.load_state_dict(modeldata2)
net.eval()

3.2 轻量化模型

https://github.com/HCPLab-SYSU/SKT。本文提出了一个简单而有效的结构化知识迁移框架， 把现有人群计数模型的结构化知识（层内知识+层间知识）充分地迁移至轻量化模型。 最后生成的轻量化模型， 参数量和计算量只有原来的6%，在GPU上至少有6.5× 倍的加速， 效果跟原模型差不多、甚至更好，可以真正地运用到实际场景。