【C4-2022】奶牛异常行为监测系统

★★★ 本文源自AI Studio社区精品项目，【点击此处】查看更多精品内容 >>>

---

前言

1 项目概述

本项目采用轻量级的目标检测算法、关键点检测算法以及基于骨骼点的动作识别算法搭建奶牛异常行为监测系统。基于PaddleDetection和PaddleVideo完成算法开发，使用AI Studio开放平台进行模型训练以及借助PyQt5完成系统界面设计。
模型开发与训练所使用的数据集来源于开源数据集与自制数据集。其中奶牛目标检测的数据集来源于COCO数据集、奶牛关键点检测的数据集来源于Animal-Pose数据集以及基于奶牛骨骼点的动作识别的数据集来源于各大视频网站，经标注后用于模型开发。
奶牛异常行为监测系统下图所示，将视频数据输入至该系统，该系统首先通过目标检测得到目标框，然后将目标框输入至关键点检测模型得到奶牛骨骼点数据。通过关联的奶牛，获得奶牛连续时间骨骼点数据作为奶牛动作识别的输入，通过奶牛动作识别模型获得奶牛的行为识别结果，最终判断奶牛是否具有异常行为，是否需要进行专家诊断，最终通过桌面端以及移动端软件进行展示。

2 项目视频展示

一、训练环境准备

!git clone https://gitee.com/paddlepaddle/PaddleDetection.git

%cd /home/aistudio/PaddleDetection/
!pip install --upgrade -r requirements.txt -i https://mirror.baidu.com/pypi/simple

二、数据集准备

%cd /home/aistudio/data/
!unzip -oq /home/aistudio/data/data151339/Animal_Pose.zip

/home/aistudio/data

二、奶牛骨骼点检测模型训练

2.1 奶牛骨骼点数据准备

%cd /home/aistudio/data/Keypoints/
import json

json_data = open('keypoints.json')
keypoints = json.load(json_data)
print(len(keypoints['images']))
print(len(keypoints['categories']))
print(len(keypoints['annotations']))
names = []
for item in keypoints['categories']:
    names.append([item['name'], item['id']])
for ann_item in keypoints['annotations']:
    if ann_item['category_id'] == 5:
        print(ann_item)
        break

/home/aistudio/data/Keypoints
4608
5
6117
{'image_id': 5, 'bbox': [71, 252, 216, 314], 'keypoints': [[103, 269, 1], [86, 269, 1], [92, 284, 1], [108, 260, 1], [83, 261, 1], [110, 303, 1], [0, 0, 0], [0, 0, 0], [0, 0, 0], [108, 307, 1], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [105, 289, 1], [121, 263, 1], [211, 272, 1]], 'num_keypoints': 20, 'category_id': 5}

%cd /home/aistudio/data/
!unzip -oq /home/aistudio/data/annot.zip

/home/aistudio/data

import json

json_data = open('annot/train.json')
keypoints = json.load(json_data)

print(len(keypoints))
for item in keypoints:
    print(item)
    break

22246
{'joints_vis': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'joints': [[620.0, 394.0], [616.0, 269.0], [573.0, 185.0], [647.0, 188.0], [661.0, 221.0], [656.0, 231.0], [610.0, 187.0], [647.0, 176.0], [637.0201, 189.8183], [695.9799, 108.1817], [606.0, 217.0], [553.0, 161.0], [601.0, 167.0], [692.0, 185.0], [693.0, 240.0], [688.0, 313.0]], 'image': '015601864.jpg', 'scale': 3.021046, 'center': [594.0, 257.0]}

#将Animal-Pose数据集中的标注格式转换成MPII格式标注
%cd /home/aistudio/data/Keypoints/
import json

animal_pose = open('keypoints.json', 'r')
keypoints = json.load(animal_pose)
MPII_type = {'root':[]}
pose_new_item = {'image_id':0, 'joints_vis':[], 'joints':[], 'image':0, 'scale':0, 'center':0}

for ann_item in keypoints['annotations']:
    if ann_item['category_id'] == 5:
        for points in ann_item['keypoints']:
            pose_new_item['joints_vis'].append(points[2])
            pose_new_item['joints'].append([points[0], points[1]])
        pose_new_item['image'] = keypoints['images'][str(ann_item['image_id'])]
        x1, y1, x2, y2 = ann_item['bbox']
        pose_new_item['scale'] = (y2-y1)/200
        pose_new_item['center'] = [(x1+x2)/2, (y1+y2)/2]
        pose_new_item['image_id'] = int(ann_item['image_id'])
        MPII_type['root'].append(pose_new_item)
        pose_new_item = {'image_id':0, 'joints_vis':[], 'joints':[], 'image':0, 'scale':0, 'center':0}
print(len(MPII_type['root']))
f = open('mpii_type.json', 'w')
json.dump(MPII_type, f)

/home/aistudio/data/Keypoints
842

2.2 骨骼点检测模型训练

2.2.1 修改训练配置文件

2.2.2 Training

%cd /home/aistudio/PaddleDetection/
!python3 tools/train.py -c configs/keypoint/hrnet/animal_pose.yml

2.3 骨骼点检测模型导出

!python tools/export_model.py -c configs/keypoint/hrnet/animal_pose.yml -o weights=output/animal_pose/model_final.pdparams

Warning: import ppdet from source directory without installing, run 'python setup.py install' to install ppdet firstly
[06/12 19:41:51] ppdet.utils.checkpoint INFO: Finish loading model weights: output/animal_pose/model_final.pdparams
[06/12 19:41:51] ppdet.engine INFO: Export inference config file to output_inference/animal_pose/infer_cfg.yml
W0612 19:42:02.842363 22728 gpu_context.cc:278] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 10.1
W0612 19:42:02.842419 22728 gpu_context.cc:306] device: 0, cuDNN Version: 7.6.

# 保留final_model
%cd /home/aistudio/PaddleDetection/output/
!mkdir animal_key_point
%cd /home/aistudio/PaddleDetection/output/animal_pose/
!mv model_final.pdparams /home/aistudio/PaddleDetection/output/animal_key_point/
!mv model_final.pdopt /home/aistudio/PaddleDetection/output/animal_key_point/
%cd /home/aistudio/PaddleDetection/output/
!rm -rf animal_pose

/home/aistudio/PaddleDetection/output/animal_pose

三、奶牛检测模型训练

3.1 奶牛数据集准备

#从coco数据集中提取奶牛数据
%cd /home/aistudio/data/data105593/
!unzip -oq /home/aistudio/data/data105593/train2017.zip
!unzip -oq /home/aistudio/data/data105593/annotations_trainval2017.zip
!unzip -oq /home/aistudio/data/data105593/val2017.zip

/home/aistudio/data/data105593

%cd /home/aistudio/data/data105593/annotations/
import json

################################################################
########################写train coco json#######################
################################################################
isinstance_train = open("instances_train2017.json", "r")
data = json.load(isinstance_train)

cow_coco_dict = {"info":[], "licenses":0, "images":[], "annotations":[], "categories":[]}
cow_coco_dict["info"].append(data["info"])
cow_coco_dict["licenses"] = data["licenses"]

for item_cate in data["categories"]:
    if item_cate["name"] == "cow":
        item_cate["id"] == 0
        cow_coco_dict["categories"] = item_cate
image_id_list = []
for item_ann in data["annotations"]:
    if item_ann["category_id"] == 21:
        item_ann["category_id"] == 0
        cow_coco_dict["annotations"].append(item_ann)
        if item_ann["image_id"] not in image_id_list:
            image_id_list.append(item_ann["image_id"])
for item_image in data["images"]:
    if item_image["id"] in image_id_list:
        cow_coco_dict["images"].append(item_image)
print("train images:{}".format(len(cow_coco_dict["images"])))
f = open("cow_coco_train.json", "w")
json.dump(cow_coco_dict, f)
print("-------------train json file have been writen------------")
################################################################
#####################验证 json格式是否能解析正确##################
################################################################
isinstance_train = open("cow_coco_train.json", "r")
data = json.load(isinstance_train)

/home/aistudio/data/data105593/annotations
train images:1968
-------------train json file have been writen------------

%cd /home/aistudio/data/data105593/annotations/
import json

################################################################
########################写val coco json#######################
################################################################
isinstance_val = open("instances_val2017.json", "r")
data_val = json.load(isinstance_val)


cow_coco_dict_val = {"info":[], "licenses":0, "images":[], "annotations":[], "categories":[]}
cow_coco_dict_val["info"].append(data_val["info"])
cow_coco_dict_val["licenses"] = data_val["licenses"]
for item_cate in data_val["categories"]:
    if item_cate["name"] == "cow":
        item_cate["id"] == 0
        cow_coco_dict_val["categories"].append(item_cate)
image_id_list = []
for item_ann in data_val["annotations"]:
    if item_ann["category_id"] == 21:
        item_ann["category_id"] == 0
        cow_coco_dict_val["annotations"].append(item_ann)
        if item_ann["image_id"] not in image_id_list:
            image_id_list.append(item_ann["image_id"])
for item_image in data_val["images"]:
    if item_image["id"] in image_id_list:
        cow_coco_dict_val["images"].append(item_image)
print("val images:{}".format(len(cow_coco_dict_val["images"])))
#print(cow_coco_dict_val)
f = open("cow_coco_val.json", "w")
json.dump(cow_coco_dict_val, f)
print("-------------val json file have been writen------------")
################################################################
#####################验证 json格式是否能解析正确##################
################################################################
isinstance_val = open("cow_coco_val.json", "r")
data_val = json.load(isinstance_val)

/home/aistudio/data/data105593/annotations
val images:87
-------------val json file have been writen------------

分割线

###########################################
###########################################
%cd /home/aistudio/data
!mkdir cow

from pycocotools.coco import COCO
import os
import shutil
from tqdm import tqdm
import matplotlib.pyplot as plt
import cv2
from PIL import Image, ImageDraw
 
# 需要设置的路径

savepath = "/home/aistudio/data/cow/"  
img_dir = savepath + 'images/'
anno_dir = savepath + 'annotations/'
datasets_list = ['train2017', 'val2017']
 
# coco有80类，这里写要提取类的名字，以person为例
classes_names = ['cow']
 
# 包含所有类别的原coco数据集路径
'''
目录格式如下：
$COCO_PATH
----|annotations
----|train2017
----|val2017
----|test2017
'''
dataDir = '/home/aistudio/data/data105593/'
 
headstr = """\
<annotation>
    <folder>VOC</folder>
    <filename>%s</filename>
    <source>
        <database>My Database</database>
        <annotation>COCO</annotation>
        <image>flickr</image>
        <flickrid>NULL</flickrid>
    </source>
    <owner>
        <flickrid>NULL</flickrid>
        <name>company</name>
    </owner>
    <size>
        <width>%d</width>
        <height>%d</height>
        <depth>%d</depth>
    </size>
    
"""
objstr = """\
    <object>
        <name>%s</name>
        <bndbox>
            <cx>%s</cx>
            <cy>%s</cy>
            <w>%s</w>
            <h>%s</h>
        </bndbox>
    </object>
"""
tailstr = '''\
</annotation>
'''
 
 
# 检查目录是否存在，如果存在，先删除再创建，否则，直接创建
def mkr(path):
    if not os.path.exists(path):
        os.makedirs(path)  # 可以创建多级目录
 
 
def id2name(coco):
    classes = dict()
    for cls in coco.dataset['categories']:
        classes[cls['id']] = cls['name']
    return classes
 
 
def write_xml(anno_path, head, objs, tail):
    f = open(anno_path, "w")
    f.write(head)
    for obj in objs:
        f.write(objstr % (obj[0], obj[1], obj[2], obj[3], obj[4]))
    f.write(tail)
 
 
def save_annotations_and_imgs(coco, dataset, filename, objs):
    # 将图片转为xml，例:COCO_train2017_000000196610.jpg-->COCO_train2017_000000196610.xml
    dst_anno_dir = os.path.join(anno_dir, dataset)
    mkr(dst_anno_dir)
    anno_path = dst_anno_dir + '/' + filename[:-3] + 'xml'
    img_path = dataDir + dataset + '/' + filename
    print("img_path: ", img_path)
    dst_img_dir = os.path.join(img_dir, dataset)
    mkr(dst_img_dir)
    dst_imgpath = dst_img_dir + '/' + filename
    print("dst_imgpath: ", dst_imgpath)
    img = cv2.imread(img_path)
    # if (img.shape[2] == 1):
    #    print(filename + " not a RGB image")
    #   return
    shutil.copy(img_path, dst_imgpath)
 
    head = headstr % (filename, img.shape[1], img.shape[0], img.shape[2])
    tail = tailstr
    write_xml(anno_path, head, objs, tail)
 
 
def showimg(coco, dataset, img, classes, cls_id, show=True):
    global dataDir
    I = Image.open('%s/%s/%s' % (dataDir, dataset, img['file_name']))
    # 通过id，得到注释的信息
    annIds = coco.getAnnIds(imgIds=img['id'], catIds=cls_id, iscrowd=None)
    # print(annIds)
    anns = coco.loadAnns(annIds)
    # print(anns)
    # coco.showAnns(anns)
    objs = []
    for ann in anns:
        class_name = classes[ann['category_id']]
        if class_name in classes_names:
            print(class_name)
            if 'bbox' in ann:
                bbox = ann['bbox']
                xmin = float(bbox[0])
                ymin = float(bbox[1])
                xmax = float(bbox[2])  #bbox[2] + bbox[0]
                ymax = float(bbox[3])  #bbox[3] + bbox[1]
                obj = [class_name, xmin, ymin, xmax, ymax]
                objs.append(obj)
                draw = ImageDraw.Draw(I)
                draw.rectangle([xmin, ymin, xmax, ymax])
    if show:
        plt.figure()
        plt.axis('off')
        plt.imshow(I)
        plt.show()
 
    return objs
 
 
for dataset in datasets_list:
    # ./COCO/annotations/instances_train2017.json
    annFile = '{}/annotations/instances_{}.json'.format(dataDir, dataset)
 
    # 使用COCO API用来初始化注释数据
    coco = COCO(annFile)
 
    # 获取COCO数据集中的所有类别
    classes = id2name(coco)
    print(classes)
    # [1, 2, 3, 4, 6, 8]
    classes_ids = coco.getCatIds(catNms=classes_names)
    print(classes_ids)
    for cls in classes_names:
        # 获取该类的id
        cls_id = coco.getCatIds(catNms=[cls])
        img_ids = coco.getImgIds(catIds=cls_id)
        print(cls, len(img_ids))
        # imgIds=img_ids[0:10]
        for imgId in tqdm(img_ids):
            img = coco.loadImgs(imgId)[0]
            filename = img['file_name']
            # print(filename)
            objs = showimg(coco, dataset, img, classes, classes_ids, show=False)
            print(objs)
            save_annotations_and_imgs(coco, dataset, filename, objs)
 

import xml.etree.ElementTree as ET
import os
import json
 
coco = dict()
coco['images'] = []
coco['type'] = 'instances'
coco['annotations'] = []
coco['categories'] = []
 
category_set = dict()
image_set = set()
 
category_item_id = 0
image_id = 20210000000
annotation_id = 0
 
 
def addCatItem(name):
    global category_item_id
    category_item = dict()
    category_item['supercategory'] = 'none'
    category_item_id += 1
    category_item['id'] = category_item_id
    category_item['name'] = name
    coco['categories'].append(category_item)
    category_set[name] = category_item_id
    return category_item_id
 
 
def addImgItem(file_name, size):
    global image_id
    if file_name is None:
        raise Exception('Could not find filename tag in xml file.')
    if size['width'] is None:
        raise Exception('Could not find width tag in xml file.')
    if size['height'] is None:
        raise Exception('Could not find height tag in xml file.')
    image_id += 1
    image_item = dict()
    image_item['id'] = image_id
    image_item['file_name'] = file_name
    image_item['width'] = size['width']
    image_item['height'] = size['height']
    coco['images'].append(image_item)
    image_set.add(file_name)
    return image_id
 
 
def addAnnoItem(object_name, image_id, category_id, bbox):
    global annotation_id
    annotation_item = dict()
    annotation_item['segmentation'] = []
    seg = []
    # bbox[] is x,y,w,h
    # left_top
    seg.append(bbox[0])
    seg.append(bbox[1])
    # left_bottom
    seg.append(bbox[0])
    seg.append(bbox[1] + bbox[3])
    # right_bottom
    seg.append(bbox[0] + bbox[2])
    seg.append(bbox[1] + bbox[3])
    # right_top
    seg.append(bbox[0] + bbox[2])
    seg.append(bbox[1])
 
    annotation_item['segmentation'].append(seg)
 
    annotation_item['area'] = bbox[2] * bbox[3]
    annotation_item['iscrowd'] = 0
    annotation_item['ignore'] = 0
    annotation_item['image_id'] = image_id
    annotation_item['bbox'] = bbox
    annotation_item['category_id'] = 1  #category_id
    annotation_id += 1
    annotation_item['id'] = annotation_id
    coco['annotations'].append(annotation_item)
 
 
def parseXmlFiles(xml_path):
    for f in os.listdir(xml_path):
        if not f.endswith('.xml'):
            continue
 
        bndbox = dict()
        size = dict()
        current_image_id = None
        current_category_id = None
        file_name = None
        size['width'] = None
        size['height'] = None
        size['depth'] = None
 
        xml_file = os.path.join(xml_path, f)
        print(xml_file)
 
        tree = ET.parse(xml_file)
        root = tree.getroot()
        if root.tag != 'annotation':
            raise Exception('pascal voc xml root element should be annotation, rather than {}'.format(root.tag))
 
        # elem is <folder>, <filename>, <size>, <object>
        for elem in root:
            current_parent = elem.tag
            current_sub = None
            object_name = None
 
            if elem.tag == 'folder':
                continue
 
            if elem.tag == 'filename':
                file_name = elem.text
                if file_name in category_set:
                    raise Exception('file_name duplicated')
 
            # add img item only after parse <size> tag
            elif current_image_id is None and file_name is not None and size['width'] is not None:
                if file_name not in image_set:
                    current_image_id = addImgItem(file_name, size)
                    print('add image with {} and {}'.format(file_name, size))
                else:
                    raise Exception('duplicated image: {}'.format(file_name))
                    # subelem is <width>, <height>, <depth>, <name>, <bndbox>
            for subelem in elem:
                bndbox['cx'] = None
                bndbox['cy'] = None
                bndbox['w'] = None
                bndbox['h'] = None
 
                current_sub = subelem.tag
                if current_parent == 'object' and subelem.tag == 'name':
                    object_name = subelem.text
                    if object_name not in category_set:
                        current_category_id = addCatItem(object_name)
                    else:
                        current_category_id = category_set[object_name]
 
                elif current_parent == 'size':
                    if size[subelem.tag] is not None:
                        raise Exception('xml structure broken at size tag.')
                    size[subelem.tag] = int(subelem.text)
 
                # option is <xmin>, <ymin>, <xmax>, <ymax>, when subelem is <bndbox>
                for option in subelem:
                    if current_sub == 'bndbox':
                        if bndbox[option.tag] is not None:
                            raise Exception('xml structure corrupted at bndbox tag.')
                        bndbox[option.tag] = float(option.text)
 
                # only after parse the <object> tag
                if bndbox['cx'] is not None:#'xmin'
                    if object_name is None:
                        raise Exception('xml structure broken at bndbox tag')
                    if current_image_id is None:
                        raise Exception('xml structure broken at bndbox tag')
                    if current_category_id is None:
                        raise Exception('xml structure broken at bndbox tag')
                    bbox = []
                    # x
                    bbox.append(bndbox['cx'])
                    # y
                    bbox.append(bndbox['cy'])
                    # w
                    bbox.append(bndbox['w'])
                    # h
                    bbox.append(bndbox['h'])
                    print('add annotation with {},{},{},{}'.format(object_name, current_image_id, current_category_id,
                                                                   bbox))
                    addAnnoItem(object_name, current_image_id, current_category_id, bbox)
 
 
if __name__ == '__main__':
    # 需要自己设定的地址，一个是已生成的是xml文件的父目录；一个是要生成的json文件的目录
    xml_dir = r'/home/aistudio/data/cow/annotations'
    json_dir = r'/home/aistudio/data/cow/annotations'
    dataset_lists = ['val2017']
    for dataset in dataset_lists:
        xml_path = os.path.join(xml_dir, dataset)
        json_file = json_dir + '/{}.json'.format(dataset)
        parseXmlFiles(xml_path)
        json.dump(coco, open(json_file, 'w'))
    
    xml_dir = r'/home/aistudio/data/cow/annotations'
    json_dir = r'/home/aistudio/data/cow/annotations'
    dataset_lists = ['train2017']
    for dataset in dataset_lists:
        xml_path = os.path.join(xml_dir, dataset)
        json_file = json_dir + '/{}.json'.format(dataset)
        parseXmlFiles(xml_path)
        json.dump(coco, open(json_file, 'w'))

3.2 检测模型训练

3.2.1 修改配置文件

%cd /home/aistudio/PaddleDetection/
!python tools/train.py -c configs/ppyoloe/ppyoloe_cow.yml --eval

3.3 模型导出

%cd /home/aistudio/PaddleDetection/
!python tools/export_model.py -c configs/ppyoloe/ppyoloe_cow.yml -o weights=output/ppyoloe_cow/model_final.pdparams

%cd /home/aistudio/PaddleDetection/output/
!mkdir cow_detection
%cd /home/aistudio/PaddleDetection/output/ppyoloe_cow/
!mv best_model.pdparams /home/aistudio/PaddleDetection/output/cow_detection
!mv best_model.pdopt /home/aistudio/PaddleDetection/output/cow_detection

%cd /home/aistudio/PaddleDetection/output/
!rm -rf ppyoloe_cow

四、动作异常识别模型训练

4.1 骨骼点数据准备

%cd /home/aistudio
!unzip -oq /home/aistudio/data/data151339/Final_data.zip

/home/aistudio

%cd /home/aistudio/
import numpy as np
import os

training = []
label_list = []

for item in os.listdir("/home/aistudio/Final_data"):
    training.append(np.load("/home/aistudio/Final_data/{}".format(item)))
np.save("training.npy", training)
for i in range(len(os.listdir("/home/aistudio/Final_data"))):
    label_list.append(0)

np.save("labels.npy", label_list)

/home/aistudio

%cd /home/aistudio/
print(len(os.listdir("Final_data")))
print(np.load("labels.npy").shape)

/home/aistudio
135
(135,)

4.2 定义数据集Reader

%cd /home/aistudio/
from paddle.io import Dataset
import numpy as np


class Reader(Dataset):
    def __init__(self, data_path, label_path):
        super().__init__()
        self.data_list = np.load(data_path)
        self.label_list = np.load(label_path)
        self.index_list = list()
        for index in range(len(self.label_list)):
            self.index_list.append(index)

    def __getitem__(self, idx):
        idx = self.index_list[idx]
        x = self.data_list[idx]
        x = x.astype("float32")
        return x, self.label_list[idx]

    def __len__(self):
        return len(self.index_list)

/home/aistudio

4.3 AGCN Model搭建

import paddle as pp
import paddle.nn as nn
import paddle.nn.functional as F
import paddle


class GCN(nn.Layer):
    def __init__(self, in_channels, out_channels, vertex_nums=20, stride=1):
        super(GCN, self).__init__()
        self.conv1 = nn.Conv2D(in_channels=in_channels,
                               out_channels=3 * out_channels,
                               kernel_size=1,
                               stride=1)
        self.conv2 = nn.Conv2D(in_channels=vertex_nums * 3,
                               out_channels=vertex_nums,
                               kernel_size=1)

    def forward(self, x):
        # x --- N,C,T,V
        x = self.conv1(x)  # N,3C,T,V
        N, C, T, V = x.shape
        x = paddle.reshape(x, [N, C // 3, 3, T, V])  # N,C,3,T,V
        x = paddle.transpose(x, perm=[0, 1, 2, 4, 3])  # N,C,3,V,T
        x = paddle.reshape(x, [N, C // 3, 3 * V, T])  # N,C,3V,T
        x = paddle.transpose(x, perm=[0, 2, 1, 3])  # N,3V,C,T
        x = self.conv2(x)  # N,V,C,T
        x = paddle.transpose(x, perm=[0, 2, 3, 1])  # N,C,T,V
        return x


class Block(paddle.nn.Layer):
    def __init__(self,
                 in_channels,
                 out_channels,
                 vertex_nums=20,
                 temporal_size=9,
                 stride=1,
                 residual=True):
        super(Block, self).__init__()
        self.residual = residual
        self.out_channels = out_channels

        self.bn_res = nn.BatchNorm2D(out_channels)
        self.conv_res = nn.Conv2D(in_channels=in_channels,
                                  out_channels=out_channels,
                                  kernel_size=1,
                                  stride=(stride, 1))
        self.gcn = GCN(in_channels=in_channels,
                       out_channels=out_channels,
                       vertex_nums=vertex_nums)
        self.tcn = nn.Sequential(
            nn.BatchNorm2D(out_channels),
            nn.ReLU(),
            nn.Conv2D(in_channels=out_channels,
                      out_channels=out_channels,
                      kernel_size=(temporal_size, 1),
                      padding=((temporal_size - 1) // 2, 0),
                      stride=(stride, 1)),
            nn.BatchNorm2D(out_channels),
        )

    def forward(self, x):
        if self.residual:
            y = self.conv_res(x)
            y = self.bn_res(y)
        x = self.gcn(x)
        x = self.tcn(x)
        out = x + y if self.residual else x
        out = F.relu(out)
        return out


class AGCN(nn.Layer):
    """
    AGCN model improves the performance of ST-GCN using
    Adaptive Graph Convolutional Networks.
    Args:
        in_channels: int, channels of vertex coordinate. 2 for (x,y), 3 for (x,y,z). Default 2.
    """
    def __init__(self, in_channels=2, **kwargs):
        super(AGCN, self).__init__()

        self.data_bn = nn.BatchNorm1D(20 * 2)
        self.agcn = nn.Sequential(
            Block(in_channels=in_channels,
                  out_channels=64,
                  residual=False,
                  **kwargs), Block(in_channels=64, out_channels=64, **kwargs),
            Block(in_channels=64, out_channels=64, **kwargs),
            Block(in_channels=64, out_channels=64, **kwargs),
            Block(in_channels=64, out_channels=128, stride=2, **kwargs),
            Block(in_channels=128, out_channels=128, **kwargs),
            Block(in_channels=128, out_channels=128, **kwargs),
            Block(in_channels=128, out_channels=256, stride=2, **kwargs),
            Block(in_channels=256, out_channels=256, **kwargs),
            Block(in_channels=256, out_channels=256, **kwargs))

        self.pool = nn.AdaptiveAvgPool2D(output_size=(1, 1))
        
        self.out = pp.nn.Linear(in_features=256, out_features=2)
        
    def forward(self, x):
        # data normalization
        print(x.shape)
        N, C, T, V, M = x.shape

        x = x.transpose((0, 4, 1, 2, 3))  # N, M, C, T, V
        x = x.reshape((N * M, C, T, V))

        x = self.agcn(x)

        x = self.pool(x)  # NM,C,T,V --> NM,C,1,1
        C = x.shape[1]
        x = paddle.reshape(x, (N, M, C)).mean(axis=1)  # N,C,1,1
        x = self.out(x)
        
        return x

4.4 AGCN模型训练

data_path = 'training.npy'
label_path = 'labels.npy'

input_define = pp.static.InputSpec(shape=[-1, 3, 30, 20, 1], dtype="float32", name='data')
# 定义label输入层，用于计算损失和准确率
label_define = pp.static.InputSpec(shape=[-1, 1], dtype="int64", name="label")
# 实例化网络对象并定义优化器等训练逻辑
model = pp.Model(AGCN(in_channels=3), inputs=input_define, labels=label_define)
# 此处lr使用CosineAnnealingDecay对学习率进行调整
lr = paddle.optimizer.lr.CosineAnnealingDecay(learning_rate=0.00625 , T_max=100, last_epoch=-1, verbose=False)
# 此处使用Momentum优化器
optimizer = pp.optimizer.Momentum(learning_rate=lr, parameters=model.parameters())
# 损失函数使用交叉熵，评价指标使用准确率
# 其中Top-k表示推理得到的概率分布中，概率最高的前k个推理结果中是否包含正确标签，如果包含则视为正确，这里的1，2，3分别计算k为1~3的情况
model.prepare(optimizer=optimizer,
              loss=pp.nn.CrossEntropyLoss(),
              metrics=pp.metric.Accuracy(topk=(1, 5)))
train_set = Reader(data_path=data_path,
                            label_path=label_path)

# print(train_set[0][0].shape)

model.fit(train_data=train_set,
          batch_size=32,
          epochs=100,
          save_dir="output/",
          save_freq=5,
          eval_freq=5,
          log_freq=20)
print("训练完毕")

4.5 AGCN模型推理

import paddle as pp
import numpy as np

def softmax(x):
    
    f_x = np.exp(x) / np.sum(np.exp(x))
    return f_x

input_define = pp.static.InputSpec(shape=[-1, 3, 30, 20, 1], dtype="float32", name='data')

model_infer = pp.Model(AGCN(in_channels=3), inputs=input_define)
model_infer.load("output/final.pdparams")
model_infer.prepare()

data = np.load("training.npy")
data_ = np.reshape(data[0], (1, 1, 3, 30, 20, 1))
data_ = data_.astype("float32")
print(data_.shape)
result = model_infer.predict(data_, batch_size=1)
print(result)
print(softmax(result))
tput/",
          save_freq=5,
          eval_freq=5,
          log_freq=20)
print("训练完毕")

4.5 AGCN模型推理

import paddle as pp
import numpy as np

def softmax(x):
    
    f_x = np.exp(x) / np.sum(np.exp(x))
    return f_x

input_define = pp.static.InputSpec(shape=[-1, 3, 30, 20, 1], dtype="float32", name='data')

model_infer = pp.Model(AGCN(in_channels=3), inputs=input_define)
model_infer.load("output/final.pdparams")
model_infer.prepare()

data = np.load("training.npy")
data_ = np.reshape(data[0], (1, 1, 3, 30, 20, 1))
data_ = data_.astype("float32")
print(data_.shape)
result = model_infer.predict(data_, batch_size=1)
print(result)
print(softmax(result))