G-TAD（G-TAD: Sub-Graph Localization for Temporal Action Detection）论文代码01

G-TAD

感觉作者构思的时序和语义妙的~

摘要

时间动作检测是视频理解中一个基本而又具有挑战性的任务。视频上下文是有效检测动作的重要线索，但目前的研究主要集中在时间上下文上，而忽略了语义上下文等重要上下文属性。在此工作中，我们提出了一个图卷积网络(GCN)模型，自适应地将多层次语义上下文纳入视频特征，并将时间动作检测作为子图定位问题。具体地说，我们将视频片段表示为图节点，将片段-片段关联表示为边，将与上下文相关的动作表示为目标子图。以图卷积为基本运算，设计了一个名为GCNeXt的GCN块，通过聚合节点上下文来学习每个节点的特征，并动态更新图中的边。为了定位每个子图，我们还设计了一个结构层来嵌入每个子图到欧几里得空间。大量的实验表明，G-TAD能够在不需要额外监督的情况下找到有效的视频上下文，并且在两个检测基准上都达到了最新的性能。在ActivityNet-1.3上，平均得到34.09%的地图;在THUMOS14上，在IoU@0.5与提议处理方法结合时，它达到了51.6%。
代码网址是:https://github.com/frostinassiky/gtad

数据输入

1）视频处理
采用了TSN时序分段网络进行双流的特征提取。

class VideoDataSet(data.Dataset):  # thumos
    def __init__(self, opt, subset="train", mode="train"):
        self.temporal_scale = opt["temporal_scale"]  # 128
        self.temporal_gap = 1. / self.temporal_scale # 1/128
        self.subset = subset
        self.mode = mode
        self.feature_path = opt["feature_path"]
        self.video_info_path = opt["video_info"]
        self.video_anno_path = opt["video_anno"]
        self.feat_dim = opt['feat_dim']
        # Assuming someone wont outsmart this by mutating the dict 😝.
        # Consider to use YACS and FB code structure in the future.
        self.cfg = opt

        #### THUMOS
        self.skip_videoframes = opt['skip_videoframes']
        self.num_videoframes = opt['temporal_scale']
        self.max_duration = opt['max_duration']
        self.min_duration = opt['min_duration']
        if self.feature_path[-3:]=='200':
            self.feature_dirs = [self.feature_path + "/flow/csv", self.feature_path + "/rgb/csv"]
        else:
            self.feature_dirs = [self.feature_path]
        self._get_data()
        self.video_list = self.data['video_names']
        # self._getDatasetDict()
        self._get_match_map()

    def _getDatasetDict(self):
        anno_df = pd.read_csv(self.video_info_path)
        anno_database = load_json(self.video_anno_path)
        self.video_dict = {}
        for i in range(len(anno_df)):
            video_name = anno_df.video.values[i]
            video_info = anno_database[video_name]
            video_subset = anno_df.subset.values[i]
            if self.subset in video_subset:
                self.video_dict[video_name] = video_info
        self.video_list = list(self.video_dict.keys())
        print("%s subset video numbers: %d" % (self.subset, len(self.video_list)))

在dataset.py的

		self.durations = {}
        #h5py.File() to read data
        self.flow_val = h5py.File(self.feature_path+'/flow_val.h5', 'r')
        self.rgb_val = h5py.File(self.feature_path+'/rgb_val.h5', 'r')
        self.flow_test = h5py.File(self.feature_path+'/flow_test.h5', 'r')
        self.rgb_test = h5py.File(self.feature_path+'/rgb_test.h5', 'r')
        #
        for num_video, video_name in enumerate(video_name_list):
            print('Getting video %d / %d' % (num_video, len(video_name_list)), flush=True)
            anno_df_video = anno_df[anno_df.video == video_name]
            if self.mode == 'train':
                gt_xmins = anno_df_video.startFrame.values[:]
                gt_xmaxs = anno_df_video.endFrame.values[:]

            if 'val' in video_name:
                feature_h5s = [
                    self.flow_val[video_name][::self.skip_videoframes,...],
                    self.rgb_val[video_name][::self.skip_videoframes,...]
                ]

            elif 'test' in video_name:
                feature_h5s = [
                    self.flow_test[video_name][::self.skip_videoframes,...],
                    self.rgb_test[video_name][::self.skip_videoframes,...]
                ]

            num_snippet = min([h5.shape[0] for h5 in feature_h5s])
            df_data = np.concatenate([h5[:num_snippet, :]
                                      for h5 in feature_h5s],
                                     axis=1)

            # df_snippet = [start_snippet + skip_videoframes * i for i in range(num_snippet)] 
            df_snippet = [skip_videoframes * i for i in range(num_snippet)] 
            num_windows = int((num_snippet + stride - num_videoframes) / stride)
            windows_start = [i * stride for i in range(num_windows)]
            if num_snippet < num_videoframes:
                windows_start = [0]
                # Add on a bunch of zero data if there aren't enough windows.
                tmp_data = np.zeros((num_videoframes - num_snippet, self.feat_dim))
                df_data = np.concatenate((df_data, tmp_data), axis=0)
                df_snippet.extend([
                    df_snippet[-1] + skip_videoframes * (i + 1)
                    for i in range(num_videoframes - num_snippet)
                ])
            elif num_snippet - windows_start[-1] - num_videoframes > int(num_videoframes / skip_videoframes):
                windows_start.append(num_snippet - num_videoframes)

            for start in windows_start:
                tmp_data = df_data[start:start + num_videoframes, :]

                tmp_snippets = np.array(df_snippet[start:start + num_videoframes])
                if self.mode == 'train':
                    tmp_anchor_xmins = tmp_snippets - skip_videoframes / 2.
                    tmp_anchor_xmaxs = tmp_snippets + skip_videoframes / 2.
                    tmp_gt_bbox = []
                    tmp_ioa_list = []
                    for idx in range(len(gt_xmins)):
                        tmp_ioa = ioa_with_anchors(gt_xmins[idx], gt_xmaxs[idx],
                                                   tmp_anchor_xmins[0],
                                                   tmp_anchor_xmaxs[-1])
                        tmp_ioa_list.append(tmp_ioa)
                        if tmp_ioa > 0:
                            tmp_gt_bbox.append([gt_xmins[idx], gt_xmaxs[idx]])

                    if len(tmp_gt_bbox) > 0 and max(tmp_ioa_list) > 0.9:
                        list_gt_bbox.append(tmp_gt_bbox)
                        list_anchor_xmins.append(tmp_anchor_xmins)
                        list_anchor_xmaxs.append(tmp_anchor_xmaxs)
                        list_videos.append(video_name)
                        list_indices.append(tmp_snippets)
                        if self.feature_dirs:
                            list_data.append(np.array(tmp_data).astype(np.float32))
                elif "infer" in self.mode:
                    list_videos.append(video_name)
                    list_indices.append(tmp_snippets)
                    list_data.append(np.array(tmp_data).astype(np.float32))

模型框架

gtad-postprocess.py

进行
python gtad_postprocess.py

动作分类

采用ActivityNet进行动作分类
https://github.com/activitynet/ActivityNet

实现细节

培训和推理：我们使用PyTorch 1.1、Python 3.7和CUDA 10.0实现和编译我们的框架。我们使用 GCNeXt块对我们的模型进行端到端训练，批大小为16。前5个epoch的学习率是ActivityNet-1.3上的和THUMOS-14上的，之后5个epoch的学习率降低了10。在推理中，在《Learning video representations from correspondence proposals》之后利用全局视频上下文，我们从《Untrimmednets for weakly supervised action recognition and detection》（弱监督的时序边界）和《Cuhk & ethz & siat submission to activitynet challenge 2016》获得视频分类分数，并将其乘以融合的置信分数进行评估。对于后处理，我们使用Soft-NMS，其中阈值为0.84，并选择top-M预测进行最终评估，其中M为100 (ActivityNet1.3)， 200 (THUMOS-14)。更多细节可以在补充材料中找到。

实验对比

GTAD论文实验

AGT论文实验

AGT论文中的state-of-the-art做的实验：
Transformer NB！
本来自己的实验想要和AGT的对比一下，不过AGT用的是I3D得到的双流特征作为输入来做的。

P-GCN论文实验

在G-TAD的论文中，联合了P-GCN一起进行了判断

运行

直接运行论文里面的sh就可以，注意windows格式错误，然后在linux下重新自己手敲一遍就可以了。

set -ex
echo "$(date "+%Y.%m.%d-%H.%M.%S")"
python gtad_train.py
python gtad_inference.py
python gtad_postprocess.py
echo "$(date "+%Y.%m.%d-%H.%M.%S")"