关于NTU-RGB+D数据集skeleton数据

关于NTU-RGB+D数据集skeleton数据

本人最近在做相关的研究，分享一下关于此数据集的经验。
这个数据集是来源于南洋理工大学的ROSE实验室，16年出了60种动作，后来又新增了60种，成为NTU RGB+D 120。随数据集发表的还有他们的文章——NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis 文章是在CVPR上发表的，算是权威的数据集了。数据集提供了60种动作，3种拍摄角度，多种互动场景和多种演员设置（年龄、性别覆盖比较全面），同一动作可能重复表演（站姿，坐姿）。拍摄设备来自于kinect 2.0。
数据集的获取方法是需要申请账号的，在表明研究者的身份后官方会邮件回复的。

数据集官网：
http://rose1.ntu.edu.sg/Datasets/actionRecognition.asp

该数据集包含每个样本的RGB视频，深度图序列，3D骨架数据和红外视频。这里我们只聊skeleton文件，根据官方在GitHub上公布的matlab源码可以获取整个skeleton的信息：

function []=show_skeleton_on_RGB_frames(...
    skeletonfilename,rgbfilename,outputvideofilename)
% Draws the skeleton data on RGB frames.
%
% Argrument:
%   skeletonfilename: full adress and filename of the .skeleton file.
%   rgbfilename: corresponding RGB video file
%   outputvideofilename (optional): the filename for output video file.
%
% For further information please refer to:
%   NTU RGB+D dataset's webpage:
%       http://rose1.ntu.edu.sg/Datasets/actionRecognition.asp
%   NTU RGB+D dataset's github page:
%        https://github.com/shahroudy/NTURGB-D
%   CVPR 2016 paper:
%       Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang,
%       "NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis",
%       in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

bodyinfo = read_skeleton_file(skeletonfilename);

videofile = VideoReader(rgbfilename);
rgbvid = read(videofile);

if nargin>2 % if the output video file is given
    writerObj = VideoWriter(outputvideofilename);
    open(writerObj);
end

% in the skeleton structure, each joint is connected to some other joint:
connecting_joint = ...
    [2 1 21 3 21 5 6 7 21 9 10 11 1 13 14 15 1 17 18 19 2 8 8 12 12];

% reapeat this for every frame
% for f=1:numel(bodyinfo)
for f=1:numel(bodyinfo)
    try
        imrgb = rgbvid(:,:,:,f);
        
        % for all the detected skeletons in the current frame:
        for b=1:numel(bodyinfo(f).bodies)
            % for all the 25 joints within each skeleton:
            for j=1:25
                try
                    % use red color for drawing joint connections
                    rv=255;
                    gv=0;
                    bv=0;
                    
                    k = connecting_joint(j);
                    
                    joint = bodyinfo(f).bodies(b).joints(j);
                    dx = joint.colorX;
                    dy = joint.colorY;
                    joint2 = bodyinfo(f).bodies(b).joints(k);
                    dx2 = joint2.colorX;
                    dy2 = joint2.colorY;
                    
                    xdist=abs(dx-dx2);
                    ydist=abs(dy-dy2);
                    
                    % locate the pixels of the connecting line between the
                    % two joints
                    if xdist>ydist
                        xrange = [dx:sign(dx2-dx):dx2];
                        yrange = [dy:sign(dy2-dy)*abs((dy2-dy)/(dx2-dx)):dy2];
                    else
                        yrange = [dy:sign(dy2-dy):dy2];
                        xrange = [dx:sign(dx2-dx)*abs((dx2-dx)/(dy2-dy)):dx2];
                    end
                    % draw the line!
                    for i=1:numel(xrange)
                        dx = int32(round(xrange(i)));
                        dy = int32(round(yrange(i)));
                        imrgb(dy-3:dy+3,dx-3:dx+3,1)=rv;
                        imrgb(dy-3:dy+3,dx-3:dx+3,2)=gv;
                        imrgb(dy-3:dy+3,dx-3:dx+3,3)=bv;
                    end
                    
                    joint = bodyinfo(f).bodies(b).joints(j);
                    dx = int32(round(joint.colorX));
                    dy = int32(round(joint.colorY));
                    
                    % use green color to draw joints
                    rv=0;
                    gv=255;
                    bv=0;
                    imrgb(dy-7:dy+7,dx-7:dx+7,1)=rv;
                    imrgb(dy-7:dy+7,dx-7:dx+7,2)=gv;
                    imrgb(dy-7:dy+7,dx-7:dx+7,3)=bv;
                catch err1
                    disp(err1);
                end
            end
        end
        imrgb = imrgb(1:1080,1:1920,:);
        %imshow(imrgb);
        writeVideo(writerObj,imrgb);
        pause(0.001);
    catch err2
        disp(err2);
    end
end
close(writerObj);
end

给一些我自己的注释方便理解：
% framecount当前文件一个有的帧数（1位）
% bodycount 当前帧内存在的人个数（1位）
% body.bodyID 当前人的ID（1位_17）
% arrayint 记录当前的身体信息（6位）
% lean 向X，Y方向偏移程度（2位）
% body.trackingState 身体追踪状态（1位）
% body.jointCount 骨骼点计数
% jointinfo 骨骼点信息（11位*25）
% joint.x,joint.y,joint.z 骨骼点的位置
% joint.depthX，joint.depthY 骨骼点红外位置
% joint.colorX，joint.colorY 骨骼点RGB位置

文件种可能会出现一人或者两人场景，也存在单人误判为两人的情况。也有因为损坏或者采集数据出现特别小的文件，官方给的建议是弃用。

关于数据处理，因为自己也是菜就先导成CSV格式再做其可视化和转成其他array格式训练，发一下批量导CSV的matlab：

% 通过skeleton生成csv文件，按照C和P分成两种数据集
clc;
clear all;
close all;

skeleton_path='F:\NTU_RGB_D_DATABASE\nturgbd_skeletons_s001_to_s017\nturgb+d_skeletons';
% skeleton_path='C:\Users\DaiMoYU\Desktop\skeleton_test';
C_path='F:\skeleton2csv_stand\C';
P_path='F:\skeleton_to_csv\P';
flag_30frames=1;% 是否标准化为30帧
max_stand=0;%归一化信息：最大值
min_stand=0;%归一化信息：最小值
%==========================================================================
%创建文件夹
dir_C_path=dir(C_path);
[size_dir_C_path_a,size_dir_C_path_b]=size(dir_C_path);
for ii=3:size_dir_C_path_a
   Cn_path=[dir_C_path(ii).folder,'\',dir_C_path(ii).name];
   create_filesbin(Cn_path);
end
%==========================================================================
%读取skeleton文件
dir_skeleton_path=dir(skeleton_path);
[size_dir_skeleton_path_a,size_dir_skeleton_path_b]=size(dir_skeleton_path);
n_skeleton=size_dir_skeleton_path_a-3;
for ii=3:size_dir_skeleton_path_a
    skeleton_file_path=[dir_skeleton_path(ii).folder,'\',dir_skeleton_path(ii).name];
    skeletonfilename=skeleton_file_path;
    bodyinfo = read_skeleton_file(skeletonfilename);
    %读取每一帧
    frames=zeros(numel(bodyinfo),150);
    for f=1:numel(bodyinfo)
        % 读取每个人
        for b=1:numel(bodyinfo(f).bodies)
            for j=1:25
                joint_x(b,j) = bodyinfo(f).bodies(b).joints(j).x;
                joint_y(b,j) = bodyinfo(f).bodies(b).joints(j).y;
                joint_z(b,j) = bodyinfo(f).bodies(b).joints(j).z;
            end
        end
        %只有一个人
        if b==1
            for j=1:25
                frames(f,(j-1)*3+1)=joint_x(1,j);
                frames(f,(j-1)*3+2)=joint_y(1,j);
                frames(f,(j-1)*3+3)=joint_z(1,j);
            end
        end
        %只有两个人
        if b==2
            for j=1:25
                frames(f,(j-1)*3+1)=joint_x(1,j);
                frames(f,(j-1)*3+2)=joint_y(1,j);
                frames(f,(j-1)*3+3)=joint_z(1,j);
                
                frames(f,75+(j-1)*3+1)=joint_x(2,j);
                frames(f,75+(j-1)*3+2)=joint_y(2,j);
                frames(f,75+(j-1)*3+3)=joint_z(2,j);
            end
        end
    end
    % 此时的frames为[frames，joints]排列，joints=75*2为[body1，body2]排列
    % body1和body2都为X-Y-Z排列
    if flag_30frames==1%是否30帧标准化
        frames_sorce=numel(bodyinfo);
        distance_frames = fix(frames_sorce/30);
        for jj=1:30
            frames_30(jj,:)=frames((jj-1)*distance_frames+1,:);
            max_stand=max(max(max(frames_30)),max_stand);
            min_stand=min(min(min(frames_30)),min_stand);
        end
    else
        disp('未30帧标准化。')
        frames_30=frames;
        max_stand=max(max(max(frames_30)),max_stand);
        min_stand=min(min(min(frames_30)),min_stand);
    end
    %  分析文件名称
    file_name=dir_skeleton_path(ii).name(1:20);
    action = file_name(17:20);
    c_n=file_name(5:8);
    csv_name=[C_path,'\',c_n,'\',action,'\',file_name,'.csv'];
    %   存储csv
    if max(max(frames_30))==min(min(frames_30))
        disp('文件无效');
    else
        csvwrite(csv_name,frames_30);
    end
    disp(csv_name);

end
%==========================================================================

以及create_filesbin函数：

function create_filesbin(savepath)
% savepath='F:\train_all\';
    % 创建于此目录下
    savepath=[savepath,'\'];
    for i=1:60
        file_name = sprintf('%s',num2str(i));
        if length(file_name)==1
            file_name = ['A00',file_name];
        else
            file_name = ['A0',file_name];
        end
        file_path_name = strcat(savepath,file_name);
        file_path_name_ = strcat(file_path_name,'\');

        if exist(file_path_name_)==0   %该文件夹不存在，则直接创建
            mkdir(file_path_name_);
        else                            %该文件夹存在，则先删除再创建
            rmdir(file_path_name_, 's'); %该文件夹中有没有文件均可
            mkdir(file_path_name_);
        end
    end
end

这里再给出我用前12帧在lstm网络对第13帧的预测的可视化代码，都是转成CSV格式做的读取：

% 绘出前12帧在LSTM输入和预测的结果;
clc;close all;clear all; 
path_12 = 'F:\train_pre\testA001\TEST_7_12\S010C001P007R001A001.csv';
path_1_pre  = 'F:\train_pre\pre_S010C001P007R001A001_10_12_13.csv';
%load data
data1_12 = load(path_12);
data1_12_sorce=data1_12;
data_pre_13 = load(path_1_pre);
[size_data1_12_a,size_data1_12_b] = size(data1_12);
[size_data_1_pre_a,size_data_1_pre_b] = size(data_pre_13);
%处理归一化
distance = 4.720282+2.259431;
data1_12=data1_12+2.259431;
data1_12=data1_12/distance;
figure(1)
subplot(1,2,1)
for time_point=1:3
    %骨骼点赋值与绘制
    for joint=1:25
       point_x(joint)= data1_12((joint-1)*3+1,time_point);
       point_z(joint)= data1_12((joint-1)*3+2,time_point);
       point_y(joint)= data1_12((joint-1)*3+3,time_point);
       plot3(point_x(joint),point_y(joint),point_z(joint));
       
       point_pre_x(joint) = data_pre_13(1,(joint-1)*3+1);
       point_pre_z(joint) = data_pre_13(1,(joint-1)*3+2);
       point_pre_y(joint) = data_pre_13(1,(joint-1)*3+3);
       
       plot3(point_pre_x(joint),point_pre_y(joint),point_pre_z(joint));
       hold on;
    end
   %调色板
   temp1=1; 
   temp2=0+time_point*1/3;
   temp3=0;
%    temp1=1; 
%    temp2=0;
%    temp3=0;
   %躯干
   plot3([point_x(1),point_x(2)],[point_y(1),point_y(2)],[point_z(1),point_z(2)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(1),point_x(17)],[point_y(1),point_y(17)],[point_z(1),point_z(17)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(1),point_x(13)],[point_y(1),point_y(13)],[point_z(1),point_z(13)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(21),point_x(2)],[point_y(21),point_y(2)],[point_z(21),point_z(2)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(21),point_x(5)],[point_y(21),point_y(5)],[point_z(21),point_z(5)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(21),point_x(9)],[point_y(21),point_y(9)],[point_z(21),point_z(9)],'Color',[temp1 temp2 temp3]);
   %头部
   plot3([point_x(21),point_x(3)],[point_y(21),point_y(3)],[point_z(21),point_z(3)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(4),point_x(3)],[point_y(4),point_y(3)],[point_z(4),point_z(3)],'Color',[temp1 temp2 temp3]);
    %左臂
   plot3([point_x(9),point_x(10)],[point_y(9),point_y(10)],[point_z(9),point_z(10)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(10),point_x(11)],[point_y(10),point_y(11)],[point_z(10),point_z(11)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(11),point_x(12)],[point_y(11),point_y(12)],[point_z(11),point_z(12)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(12),point_x(24)],[point_y(12),point_y(24)],[point_z(12),point_z(24)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(12),point_x(25)],[point_y(12),point_y(25)],[point_z(12),point_z(25)],'Color',[temp1 temp2 temp3]);
    %右臂
   plot3([point_x(5),point_x(6)],[point_y(5),point_y(6)],[point_z(5),point_z(6)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(6),point_x(7)],[point_y(6),point_y(7)],[point_z(6),point_z(7)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(7),point_x(8)],[point_y(7),point_y(8)],[point_z(7),point_z(8)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(8),point_x(22)],[point_y(8),point_y(22)],[point_z(8),point_z(22)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(8),point_x(23)],[point_y(8),point_y(23)],[point_z(8),point_z(23)],'Color',[temp1 temp2 temp3]);
   %左腿
   plot3([point_x(17),point_x(18)],[point_y(17),point_y(18)],[point_z(17),point_z(18)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(18),point_x(19)],[point_y(18),point_y(19)],[point_z(18),point_z(19)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(20),point_x(19)],[point_y(20),point_y(19)],[point_z(20),point_z(19)],'Color',[temp1 temp2 temp3]);
   %右腿
   plot3([point_x(13),point_x(14)],[point_y(13),point_y(14)],[point_z(13),point_z(14)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(14),point_x(15)],[point_y(14),point_y(15)],[point_z(14),point_z(15)],'Color',[temp1 temp2 temp3]);
   plot3([point_x(15),point_x(16)],[point_y(15),point_y(16)],[point_z(15),point_z(16)],'Color',[temp1 temp2 temp3]);
end

temp1=0; 
temp2=0;
temp3=1;
%躯干
plot3([point_pre_x(1),point_pre_x(2)],[point_pre_y(1),point_pre_y(2)],[point_pre_z(1),point_pre_z(2)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(1),point_pre_x(17)],[point_pre_y(1),point_pre_y(17)],[point_pre_z(1),point_pre_z(17)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(1),point_pre_x(13)],[point_pre_y(1),point_pre_y(13)],[point_pre_z(1),point_pre_z(13)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(21),point_pre_x(2)],[point_pre_y(21),point_pre_y(2)],[point_pre_z(21),point_pre_z(2)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(21),point_pre_x(5)],[point_pre_y(21),point_pre_y(5)],[point_pre_z(21),point_pre_z(5)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(21),point_pre_x(9)],[point_pre_y(21),point_pre_y(9)],[point_pre_z(21),point_pre_z(9)],'Color',[temp1 temp2 temp3]);
%头部
plot3([point_pre_x(21),point_pre_x(3)],[point_pre_y(21),point_pre_y(3)],[point_pre_z(21),point_pre_z(3)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(4),point_pre_x(3)],[point_pre_y(4),point_pre_y(3)],[point_pre_z(4),point_pre_z(3)],'Color',[temp1 temp2 temp3]);
%左臂
plot3([point_pre_x(9),point_pre_x(10)],[point_pre_y(9),point_pre_y(10)],[point_pre_z(9),point_pre_z(10)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(10),point_pre_x(11)],[point_pre_y(10),point_pre_y(11)],[point_pre_z(10),point_pre_z(11)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(11),point_pre_x(12)],[point_pre_y(11),point_pre_y(12)],[point_pre_z(11),point_pre_z(12)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(12),point_pre_x(24)],[point_pre_y(12),point_pre_y(24)],[point_pre_z(12),point_pre_z(24)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(12),point_pre_x(25)],[point_pre_y(12),point_pre_y(25)],[point_pre_z(12),point_pre_z(25)],'Color',[temp1 temp2 temp3]);
%右臂
plot3([point_pre_x(5),point_pre_x(6)],[point_pre_y(5),point_pre_y(6)],[point_pre_z(5),point_pre_z(6)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(6),point_pre_x(7)],[point_pre_y(6),point_pre_y(7)],[point_pre_z(6),point_pre_z(7)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(7),point_pre_x(8)],[point_pre_y(7),point_pre_y(8)],[point_pre_z(7),point_pre_z(8)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(8),point_pre_x(22)],[point_pre_y(8),point_pre_y(22)],[point_pre_z(8),point_pre_z(22)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(8),point_pre_x(23)],[point_pre_y(8),point_pre_y(23)],[point_pre_z(8),point_pre_z(23)],'Color',[temp1 temp2 temp3]);
%左腿
plot3([point_pre_x(17),point_pre_x(18)],[point_pre_y(17),point_pre_y(18)],[point_pre_z(17),point_pre_z(18)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(18),point_pre_x(19)],[point_pre_y(18),point_pre_y(19)],[point_pre_z(18),point_pre_z(19)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(20),point_pre_x(19)],[point_pre_y(20),point_pre_y(19)],[point_pre_z(20),point_pre_z(19)],'Color',[temp1 temp2 temp3]);
%右腿
plot3([point_pre_x(13),point_pre_x(14)],[point_pre_y(13),point_pre_y(14)],[point_pre_z(13),point_pre_z(14)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(14),point_pre_x(15)],[point_pre_y(14),point_pre_y(15)],[point_pre_z(14),point_pre_z(15)],'Color',[temp1 temp2 temp3]);
plot3([point_pre_x(15),point_pre_x(16)],[point_pre_y(15),point_pre_y(16)],[point_pre_z(15),point_pre_z(16)],'Color',[temp1 temp2 temp3]);
grid on;


%绘制帧间差图
mat_data=xlsread(path_12);
[a,b]=size(mat_data);

for i=1:(b-1)
    D(i)=sum(sum(abs(mat_data(:,i)-mat_data(:,i+1))));
    maxD=max(D);
end
subplot(1,2,2)
title(['单位时刻动作位移程度']);
plot(1:b-1,D);
disp(path_12);
disp('frame_distance_max:')
disp(maxD);

暂时分享到这里，希望有大佬带带路。