【数据集】Kinetics-600 dataset介绍

---

最近在做行为识别，然后介绍一下目前种类多且数据量很大的行为数据集 Kinetics-600 dataset。

参考链接：

activityNet：http://activity-net.org/challenges/2018/index.html

GitHub：https://github.com/activitynet/ActivityNet

kinetics：https://deepmind.com/research/open-source/open-source-datasets/kinetics/

paper：https://arxiv.org/abs/1705.06950

---

1、背景与来源

目前activitynet第三届比赛已经开始了，这项比赛始于2016CVPR，是与ImageNet齐名的在视频理解方面最重要的比赛。

在这个比赛下的Task A – Trimmed Action Recognition比赛是一个视频分类比赛，数据集就是kinetics-600数据集。数据集有Google的deepmind团队提供，2017年是第一届比赛，当时有400个类别，20多万数据，今年又对数据集进行了扩增，现在有600个类别，共50万左右的视频。

2、介绍

视频来源于YouTube，一共有600个类别，每个类别至少600个视频以上，每段视频持续10秒左右。类别主要分为三大类：人与物互动，比如演奏乐器；人人互动，比如握手、拥抱；运动等。即person、person-person、person-object。

3、论文介绍

3.1 数据集类别介绍

下面这些介绍主要是参考于17年deepmind发的论文”The Kinetics Human Action Video Dataset”，当时还是400类的数据集。论文请见。

数据集主要关注人类行为，action类的列表包括：

单人行为，例如绘画，喝酒，大笑，抽拳;

人人行为，例如拥抱，亲吻，握手；

人物行为，例如打开礼物，修剪草坪，洗碗。

一些行动是比较细粒度的，需要时序推理来区分，例如，不同类型的游泳。其他动作类别需要强调区分对象，例如演奏不同类型的乐器。

这些类别并没有严格的层级划分，但是还是有些的，类似父类子类关系，例如音乐类（打鼓、长号、小提琴、……），个人卫生类（刷牙、剪指甲、洗手、……），跳舞类（芭蕾、macarena、tap、……），烹饪(切割、煎、脱皮、……）等。

数据集有400个类别，每个动作都有400-1150个视频片段，每段视频的时长都在10秒左右。目前的版本有306245视频，分为三个部分，训练时每个类为250-1000个视频，验证时每个类50个视频，测试时每个类100个视频。

每个类都包含了一种行动。但是，一个特定的剪辑可以包含
几种动作。例如，开车”时“发短信”;“弹奏尤克里里”时“跳草裙舞”;“跳舞”时“刷牙”。这种情况下，这个视频只会标记一个标签，并不会同时存在于两个类种。因此，取top-5的准确率更为合适。

3.2 Benchmark

一共验证了三种处理行为识别的主流模型，分别是LSTM、two stream和3dcnn。

放上准确率，可以看出two-stream还是占据主导优势的。在kinetics数据集上，top-1是61.0，top-5是81.3。

3.3 分类结果分析

上图是那些类分的比较好，那些类分的比较差。

还有一点，由于很多动作比较细粒度，类别之间容易造成混淆，论文中也给出了最容易混淆的几个类别，比如，跳远和三级跳远，吃汉堡和吃甜甜圈。swing跳舞和跳萨尔萨舞等都会混淆。

如下图：

当然，由于使用two stream模型，光流模型和RGB模型可能对不同的动作有不同的准确度，对于这些特定类别，可以在融合时对光流和RGB设定不同的权重。

上图中，对于机械舞，光流效果就会比RGB好很多，而对于做蛋糕，则是RGB效果更好。

3.4 粒度划分

最后给出这些类别的一个粒度划分，可能有多个类别都会属于同一大类。即父类子类关系。

首先列出有哪些父类，然后再给出每个父类下的子类。

3.4.1父类：

共38大类，每个类后面的数字代表有几个子类。

arts and crafts (12)
athletics – jumping (6)
athletics – throwing + launching (9)
auto maintenance (4)
ball sports (25)
body motions (16)
cleaning (13)
cloths (8)
communication (11)
cooking (22)
dancing (18)
eating + drinking (17)
electronics (5)
garden + plants (10)
golf (3)
gymnastics (5)
hair (14)
hands (9)
head + mouth (17)
heights (15)
interacting with animals (19)
juggling (6)
makeup (5)
martial arts (10)
miscellaneous (9)
mobility – land (20)
mobility – water (10)
music (29)
paper (12)
personal hygiene (6)
playing games (13)
racquet + bat sports (8)
snow + ice (18)
swimming (3)
touching person (11)
using tools (13)
water sports (8)
waxing (4)

3.4.2父类与子类

arts and crafts (12)
arranging flowers
blowing glass
brush painting
carving pumpkin
clay pottery making
decorating the christmas tree
drawing
getting a tattoo
knitting
making jewelry
spray painting
weaving basket

athletics – jumping (6)
high jump
hurdling
long jump
parkour
pole vault
triple jump

athletics – throwing + launching (9)
archery
catching or throwing frisbee
disc golfing
hammer throw
javelin throw
shot put
throwing axe
throwing ball
throwing discus

auto maintenance (4)
changing oil
changing wheel
checking tires
pumping gas

ball sports (25)
bowling
catching or throwing baseball
catching or throwing softball
dodgeball
dribbling basketball
dunking basketball
golf chipping
golf driving
golf putting
hitting baseball
hurling (sport)
juggling soccer ball
kicking field goal
kicking soccer ball
passing American football (in game)
passing American football (not in game)
playing basketball
playing cricket
playing kickball
playing squash or racquetball
playing tennis
playing volleyball
shooting basketball
shooting goal (soccer)
shot put

body motions (16)
air drumming
applauding
baby waking up
bending back
clapping
cracking neck
drumming fingers
finger snapping
headbanging
headbutting
pumping fist
shaking head
stretching arm
stretching leg
swinging legs

cleaning (13)
cleaning floor
cleaning gutters
cleaning pool
cleaning shoes
cleaning toilet
cleaning windows
doing laundry
making bed
mopping floor
setting table
shining shoes
sweeping floor
washing dishes

cloths (8)
bandaging
doing laundry
folding clothes
folding napkins
ironing
making bed
tying bow tie
tying knot (not on a tie)
tying tie

communication (11)
answering questions
auctioning
bartending
celebrating
crying
giving or receiving award
laughing
news anchoring
presenting weather forecast
sign language interpreting
testifying

cooking (22)
baking cookies
barbequing
breading or breadcrumbing
cooking chicken
cooking egg
cooking on campfire
cooking sausages
cutting pineapple
cutting watermelon
flipping pancake
frying vegetables
grinding meat
making a cake
making a sandwich
making pizza
making sushi
making tea
peeling apples
peeling potatoes
picking fruit
scrambling eggs
tossing salad

dancing (18)
belly dancing
breakdancing
capoeira
cheerleading
country line dancing
dancing ballet
dancing charleston
dancing gangnam style
dancing macarena
jumpstyle dancing
krumping
marching
robot dancing
salsa dancing
swing dancing
tango dancing
tap dancing
zumba

eating + drinking (17)
bartending
dining
drinking
drinking beer
drinking shots
eating burger
eating cake
eating carrots
eating chips
eating doughnuts
eating hotdog
eating ice cream
eating spaghetti
eating watermelon
opening bottle
tasting beer
tasting food

electronics (5)
assembling computer
playing controller
texting
using computer
using remote controller (not gaming)

garden + plants (10)
blowing leaves
carving pumpkin
chopping wood
climbing tree
decorating the christmas tree
egg hunting
mowing lawn
planting trees
trimming trees
watering plants

golf (3)
golf chipping
golf driving
golf putting

gymnastics (5)
bouncing on trampoline
cartwheeling
gymnastics tumbling
somersaulting
vault

hair (14)
braiding hair
brushing hair
curling hair
dying hair
fixing hair
getting a haircut
shaving head
shaving legs
trimming or shaving beard
washing hair
waxing back
waxing chest
waxing eyebrows
waxing legs

hands (9)
air drumming
applauding
clapping
cutting nails
doing nails
drumming fingers
finger snapping
pumping fist
washing hands

head + mouth (17)
balloon blowing
beatboxing
blowing nose
blowing out candles
brushing teeth
gargling
headbanging
headbutting
shaking head
singing
smoking
smoking hookah
sneezing
sniffing
sticking tongue out
whistling
yawning

heights (15)
abseiling
bungee jumping
climbing a rope
climbing ladder
climbing tree
diving cliff
ice climbing
jumping into pool
paragliding
rock climbing
skydiving
slacklining
springboard diving
swinging on something
trapezing

interacting with animals (19)
bee keeping
catching fish
feeding birds
feeding fish
feeding goats
grooming dog
grooming horse
holding snake
ice fishing
milking cow
petting animal (not cat)
petting cat
riding camel
riding elephant
riding mule
riding or walking with horse
shearing sheep
training dog
walking the dog

juggling (6)
contact juggling
hula hooping
juggling balls
juggling fire
juggling soccer ball
spinning poi

makeup (5)
applying cream
doing nails
dying hair
filling eyebrows
getting a tattoo

martial arts (10)
arm wrestling
capoeira
drop kicking
high kick
punching bag
punching person
side kick
sword fighting
tai chi
wrestling

miscellaneous (9)
digging
extinguishing fire
garbage collecting
laying bricks
moving furniture
spraying
stomping grapes
tapping pen
unloading truck

mobility – land (20)
crawling baby
driving car
driving tractor
faceplanting
hoverboarding
jogging
motorcycling
parkour
pushing car
pushing cart
pushing wheelchair
riding a bike
riding mountain bike
riding scooter
riding unicycle
roller skating
running on treadmill
skateboarding
surfing crowd
using segway
waiting in line

mobility – water (10)
crossing river
diving cliff
jumping into pool
scuba diving
snorkeling
springboard diving
swimming backstroke
swimming breast stroke
swimming butterfly stroke
water sliding

music (29)
beatboxing
busking
playing accordion
playing bagpipes
playing bass guitar
playing cello
playing clarinet
playing cymbals
playing didgeridoo
playing drums
playing flute
playing guitar
playing harmonica
playing harp
playing keyboard
playing organ
playing piano
playing recorder
playing saxophone
playing trombone
playing trumpet
playing ukulele
playing violin
playing xylophone
recording music
singing
strumming guitar
tapping guitar
whistling

paper (12)
bookbinding
counting money
folding napkins
folding paper
opening present
reading book
reading newspaper
ripping paper
shredding paper
unboxing
wrapping present
writing

personal hygiene (6)
brushing teeth
taking a shower
trimming or shaving beard
washing feet
washing hair
washing hands

playing games (13)
egg hunting
flying kite
hopscotch
playing cards
playing chess
playing monopoly
playing paintball
playing poker
riding mechanical bull
rock scissors paper
shuffling cards
skipping rope
tossing coin

racquet + bat sports (8)
catching or throwing baseball
catching or throwing softball
hitting baseball
hurling (sport)
playing badminton
playing cricket
playing squash or racquetball
playing tennis

snow + ice (18)
biking through snow
bobsledding
hockey stop
ice climbing
ice fishing
ice skating
making snowman
playing ice hockey
shoveling snow
ski jumping
skiing (not slalom or crosscountry)
skiing crosscountry
skiing slalom
sled dog racing
snowboarding
snowkiting
snowmobiling
tobogganing

swimming (3)
swimming backstroke
swimming breast stroke
swimming butterfly stroke

touching person (11)
carrying baby
hugging
kissing
massaging back
massaging feet
massaging legs
massaging person’s head
shaking hands
slapping
tickling

using tools (13)
bending metal
blasting sand
building cabinet
building shed
changing oil
changing wheel
checking tires
plastering
pumping gas
sanding floor
sharpening knives
sharpening pencil
welding

water sports (8)
canoeing or kayaking
jetskiing
kitesurfing
parasailing
sailing
surfing water
water skiing
windsurfing

waxing (4)
waxing back
waxing chest
waxing eyebrows
waxing legs