理解基于SVM进行图像动作识别中的一些问题

背景

需要从一组动作中识别这是什么动作？

理解

假设有两个投球和拍球的动作视频。我们会从每个视频中提取特征，首先是将每个视频转换为一定帧数的图像，如每个视频都以一定的频率转换为20frame的图像。以投球视频来说，将投球视频转换为20frame的图像，然后利用图像处理技术以及SVM "kernel trick"方法提取每一frame的特征值。然后将这20frame的特征值都打上“0”的标签；同样，对于拍球的动作，也这样做，然后将拍球的这20frame提取出来的特征向量打上“1”的标签。有了特征，有了标签就可以进行训练模型了。
然后现在有一个新的动作视频，将这个新的动作视频转换为20frame图片，然后提取每一个frame里面的特征，带入到训练好的模型中，得到一组标签，选择众数最多的标签作为识别的动作。
以下的代码可以自己看一下，捋一下逻辑

__author__ = 'somnath'


import numpy as np
import cv2
import sys
import os
import glob
from sklearn import svm
from scipy.stats import mode

'''
Program: Sports Action Recognition

Description:
    This program perform the sports action recognition task. First it processes the input videos from UCF sports action
    data set.The data set contains 13 different sports action which individually contains multiple videos. A video
    directory contain a video file and corresponding frames. I iterate over the different sports actions and read video
    frames from each video directory to extract features. I take equal number of videos from each categories.
    Further, to optimize the process, I sorted features with highest gradient for HOG. I have used SVM classifier and
    cross validation for classification and evaluation respectively.

    I have used provided image frame for each video as I found issue to process *.avi file in Mac.



 Feature Extraction: I have used Histogram of Oriented Gradient ( HOG ) method to extract features vector.

                     HOG:  It is constructed by dividing the image into cells and for each cell computing the
                     distribution of intensity gradients or edge directions. The concatenating each of these gradient
                     orientation histograms yields the HOG.

                     hogDescriptor = cv2.HOGDescriptor()
                     hist          = hogDescriptor.compute(gray)

                     I use above two functions to create HOG Descriptor and histogram.
                     Further, I sort the histogram values and take max 15000 values from each frame for evaluation.

 Classifier: I have used Support Vector Machine (SVM) classifier. The classifier parameters are set based on best result
     achieved from different runs. Following are the parameters that has been decided based on the multiple executions.

     Parameters:
      gamma=0.01   Lowering the gamma value gives better result, but takes more time. Optimum value has been chosen.
      C=13
      kernel_type = rbf ( default )
      degree = 3 ( default )

 Evaluation: It is based on K-Fold cross validation mechanism.
               First, I shuffle the feature list which contains features as well as label at the very first element of
               the feature vector to obtain better result. The complete set of shuffled features are divided equally
               into k=13 sub parts. k-1 subset is used for training and one subset is used for validation. I iterate the
               process for k=13 times with different subset combinations for training and validation.

               Evaluation Metrics:
                At each iteration, evaluation metrics sensitivity, specificity and accuracy are calculated
                based on True Positive (TP), False Positive (FP), False Negative (FN) and True Negative (TN) rates.

                Sensitivity = ( True Positive Rate) = TP / ( TP + FN )
                Specificity = ( True Negative Rate) = TN / ( TN + FP )
                Accuracy    = ( TP + TN ) / ( TP + FN + FP + TN )

               At the end of all iterations of cross validation, I average them all to get average rate.


 Testing: I also have tested my model to check if that works with unseen data or videos.
          For that, I have taken one video from "Diving-Side/014" which has been correctly predicted by my model.
          Result is given below.
'''

sportsActionPath = "D:\\gitWorkingRepos\\Sports-Action-Recognition-master\\Train\\"
#sportsActionPath = "/Users/somnath/MY_PROG/ComputerVision/pa3/Training"


# Sports Action Tag
sportsActionTag = {
    'VolleyballSpiking': 0,
    'WallPushups':1,
    'Golf-Swing-Front':2,
    'Golf-Swing-Side':3,
    'Kicking-Front':4,
    'Kicking-Side':5,
    'Lifting':6,
    'Run-Side':7,
    'SkateBoarding-Front':8,
    'Swing-SideAngle':9,
    'Walk-Front':10,
    'Swing-Bench':11,
    'Riding-Horse':12
}



# Distinct Sports Action Number
sportsActionNumber = len(sportsActionTag)

featuresLimit = 15000


'''
Function Name: featureExtraction()
Input Args   : <Sports Action Path>, <Action name>, <Training/ Validation>,
Returns      : <Array: Feature List>
Description  : This function extract features from each frames of a video and consolidated them.
               While it extract features, it add label to feature at the beginning of feature vector based on Sports
               Action Type. It helps to keep tack of feature and corresponding label while shuffle the features during
               cross validation.

               - I have used histogram of oriented gradient (HOG) method to extract the features.
                 Following methods from cv2 have been used.
                 hogDescriptor = cv2.HOGDescriptor()
                  - It takes default parameter values as Window Size= 64 x 128, block size= 16x16,
                    block stride= 8x8, cell size= 8x8, bins= 9
                 hist = hogDescriptor.compute(gray)
                  - Returns the list of histogram

               - Sorted the Histogram and taken top 15000 for evaluation.
               - I take equal number of image frame from all the videos.
'''
def featureExtraction( videoPath, actionName, type):


    # Set frame path, if jpeg directory doesn't exist , take images from video dir
    framePath = videoPath
    if os.path.exists( framePath + "/jpeg") :
        framePath += "/jpeg/"

    # Extract feature
    imageFrames = getListOfDir(framePath)
    #print "DEBUG: Image Frames - ", imageFrames
    frameCount = 0
    frameIndex = 0

    # Feature List for a video
    videoFeatures  = []

    for iFrame in imageFrames:

        frameIndex += 1
        iFrame = framePath+"\\"+iFrame
        # Read Frame
        frame = cv2.imread(iFrame)
        gray=cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)

        # HOG Descriptor , default value it takes window size= 64x128, block size= 16x16, block stride= 8x8, cell size= 8x8, bins= 9
        hogDescriptor = cv2.HOGDescriptor()

        # Returns histogram
        hist = hogDescriptor.compute(gray)

        #sortedHogDescriptor = hogDescriptor
        sortedHogHist = np.sort(hist, axis=None)

        keyFeatures = sortedHogHist[- featuresLimit : ]

        if type == "Trng":
            keyFeatures = np.insert(keyFeatures, 0, sportsActionTag[actionName])

        videoFeatures.append(keyFeatures)

        # Lowest number of frame available in a video
        if frameCount >= 3:
            break

        frameCount += 1


    return videoFeatures


'''
Function Name: getImageList()
Input Args   : <Image Directory>
Return       : <Array:List of Images>
Description  : This function returns list of images.
'''
def getImageList(imageDirectory):

    # Find different type of images
    rImages = glob.glob(imageDirectory + "/*.jpg")
    rImages +=  glob.glob(imageDirectory + "/*.jpeg")
    rImages +=  glob.glob(imageDirectory + "/*.png")

    return rImages


'''
Function Name: getListOfDir()
Input Args   : < Path >
Return       : <Array: List of Directory >
Description  : This function returns all the directories under the specified paths
'''
def getListOfDir(path):
    # Read each sport action directory
    dirs  = os.listdir(path)

    sportsActionsCount = 0
    filtered_dir  = []
    # Remove . .. and hidden directory
    for dir in dirs:
        if not dir.startswith("."):
            filtered_dir.append(dir)

    return filtered_dir

'''
Function Name: getSportsActionName()
Input Args   : < Sports Action Index>
Return       : <Sports Action Name>
Description  : This function returns the name of Sports Action based on index value

'''
def getSportsActionName(saIndex):

    keys   = sportsActionTag.keys()

    for key in keys:
        if saIndex == sportsActionTag[key]:
            return key

'''
Function Name: evaluation()
Input Args   : < 1D Array: Truth>, <1D Array: Predicted>, < Sports Action Index>
Return       : <Accuracy>,<Sensitivity>,<Specificity>
Description  :  This function calculate evaluation metrics sensitivity, specificity and accuracy
               based on True Positive (TP), False Positive (FP), False Negative (FN) and True Negative (TN) rate.

               Sensitivity = ( True Positive Rate) = TP / ( TP + FN )
               Specificity = ( True Negative Rate) = TN / ( TN + FP )
               Accuracy    = ( TP + TN ) / ( TP + FN + FP + TN )

'''

def evaluation( truth, predicted, categoryIndex ):

    # TP,FP,FN,TN indicate True Positive, False Positive, False Negative, True Negative respectively
    TP = 1
    FP = 1
    FN = 1
    TN = 1

    # Categories are Sports Action 1=>0, Sports Action 2=> 1, Sports Action 3=>2  etc..
    for fIndex in range(len(truth)):

         # Positive prediction for each feature
        if ( int(predicted[fIndex]) == categoryIndex):
            # TP=> when P[i] = T[i] = Ci
            if (int(truth[fIndex]) == int (predicted[fIndex])):
                TP += 1
            else:
                FP += 1
        else: # Negative Prediction
            if ( int ( truth[fIndex]) == categoryIndex ):
                FN += 1
            else:
                TN += 1


    # Calculate Sensitivity - True Positive Rate - Recall
    sensitivity = TP / float ( TP + FN )

    # Specificity - True Negative Rate
    specificity = TN / float ( TN + FP )

    #Calculate accuracy
    accuracy =  ( TP + TN ) / float ( TP + FP + FN + TN )


    return sensitivity, specificity, accuracy

'''
Function Name: crossValidation()
Input Args   : < Array: Feature and Label List - Fits element of vector indicates action label and rest are for features>
Retrun       : None
Description  : It perform K-Fold cross validation.
               First, I shuffle the feature list which contains features as well as label at the very first element of
               the feature vector to obtain better result. The complete set of shuffled features are divided equally
               into k=13 sub parts. k-1 subset is used for training and one subset is used for validation. I iterate the
               process for k=13 times with different subset combinations for training and validation.

               Evaluation Metrics:
                At each iteration, evaluation metrics sensitivity, specificity and accuracy are calculated
                based on True Positive (TP), False Positive (FP), False Negative (FN) and True Negative (TN) rates.

                Sensitivity = ( True Positive Rate) = TP / ( TP + FN )
                Specificity = ( True Negative Rate) = TN / ( TN + FP )
                Accuracy    = ( TP + TN ) / ( TP + FN + FP + TN )

               At the end of all iterations of cross validation, I average them all to get average rate.
'''
def crossValidation( featureAndLabelList):

    # Randomize the sample
    np.random.shuffle(featureAndLabelList)


    # Evaluation Metrics
    sensitivity = 0.0
    specificity = 0.0
    accuracy    = 0.0


    # split feature set in equal subsets same as number of sports actions for cross validation
    subsetLength =  len(featureAndLabelList) / sportsActionNumber
    for rIndex in range(sportsActionNumber):

        print ("INFO: Cross Validation Iteration - ", rIndex)
        trainigSet   = []
        valdationSet = []
        feature = []
        label   = []


        if ( rIndex == 0 ):
            trainigSet = featureAndLabelList[1*subsetLength:]
            valdationSet = featureAndLabelList[0: subsetLength]
        elif ( rIndex == (sportsActionNumber -1) ):
            trainigSet = featureAndLabelList[:(sportsActionNumber -1)*subsetLength]
            valdationSet = featureAndLabelList[(sportsActionNumber -1)*subsetLength : ]
        else:
            trainigSet = np.concatenate ((featureAndLabelList[:rIndex * subsetLength] , featureAndLabelList[(rIndex + 1) * subsetLength: ]), axis=0 )
            valdationSet = featureAndLabelList[rIndex * subsetLength : (rIndex + 1 ) * subsetLength]

        # Get all features in a array
        for featureAndLabel in trainigSet:
            label.append(int(featureAndLabel[0]))
            feature.append((np.delete(featureAndLabel, 0)).tolist())


        # Train model
        print ("INFO: Training ... ")
        clf = svm.SVC(gamma=0.01, C=13)
        clf.fit(feature,label)

        # Prepare validation feature and label to be predicted
        print ("INFO: Prediction for ", getSportsActionName(rIndex))
        vFeatureList = []
        vLabelList   = [] # Ground Truth
        for featureAndLabel in valdationSet:
            vFeatureList.append(featureAndLabel[1:].tolist())
            vLabelList.append(featureAndLabel[0])

        # Predict the class label for Validation Feature List
        predictedLabel = clf.predict(vFeatureList)

        # predict validation set and calculate accuracy
        print ("INFO: Evaluating ... ")
        #print "\t Truth - ", vLabelList
        #print "\t Predicted - ", str(predictedLabel.tolist())

        # Evaluation < Truth>, <Predicted>, <Sports Action Index>
        (sen, spec , accu ) = evaluation(vLabelList , predictedLabel.tolist() , rIndex)

        sensitivity += sen
        specificity += spec
        accuracy    += accu

        print ("\t   Sensitivity : ", sen)
        print ("\t   Specificity : ", spec)
        print ("\t   Accuracy    : ", accu)


    # Average evaluation metrics
    avgSensitivity = sensitivity / sportsActionNumber
    avgSpecificity = specificity / sportsActionNumber
    avgAccuracy = accuracy / sportsActionNumber


    print ("  *** Overall Evaluation ***")
    print ("    Average Sensitivity: ", avgSensitivity)
    print ("    Average Specificity: ", avgSpecificity)
    print ("    Average Accuracy   : ", avgAccuracy)



def main():
    print ("INFO: Action Recognition")

    sportsActionList = getListOfDir( sportsActionPath )
    print ("INFO: Sports Action - ",sportsActionList)

    sportsActionFeatures = []

    firstActionFlag = 0
    for sportsActionName in sportsActionList:
        sportsActionDir = sportsActionPath + sportsActionName
        # Get list of videos from each sports action
        videoList = getListOfDir(sportsActionDir)

        print ("INFO: Video List:", videoList)

        videoCount = 1
        videoFeatures = []
        # For all video in each action category
        for video  in videoList:

            # For good result decided to use same number of videos from Action Sports. And same number of frame from each frame
            if videoCount > 2:
                break
            # complete path of video containing jpeg images
            videoPath = sportsActionDir + "\\" + video
            #print ("\tVideo Path:", videoPath)

            # Extract Features
            videoFeatures = featureExtraction(videoPath , sportsActionName, 'Trng')

            # Put together all the videos
            if firstActionFlag == 0:
                sportsActionFeatures = videoFeatures
                firstActionFlag = 1
            else:
                sportsActionFeatures = np.concatenate( (sportsActionFeatures, videoFeatures), axis=0)

            videoCount += 1

    ## K-Fold Cross Validation method
    #crossValidation(sportsActionFeatures)

    ## ****  Testing with unseen data **** ##

    np.random.shuffle(sportsActionFeatures)
    label = []
    feature = []
    # Get all features in a array
    for featureAndLabel in sportsActionFeatures:
        label.append(int(featureAndLabel[0]))
        feature.append((np.delete(featureAndLabel, 0)).tolist())



    # Train model
    print ("INFO: Training ... ")
    clf = svm.SVC(gamma=0.01, C=13)
    clf.fit(feature,label)

    # Test Path
    tPath = "D:\\gitWorkingRepos\\Sports-Action-Recognition-master\\Test\\"
    vFeatures = featureExtraction(tPath , sportsActionName, 'Test')
    predictedLabels = clf.predict(vFeatures)

    #print "Predicted Labels:", predictedLabels
    predictedLabelMode = (mode(predictedLabels))[0]
    print ("\t Predicted Sports Action:{0} - {1}".format(predictedLabelMode,getSportsActionName(predictedLabelMode) ))


if __name__ == "__main__":
    main()




'''
RESULT:
INFO: Cross Validation Iteration -  0
INFO: Training ...
INFO: Prediction for  Diving-Side
INFO: Evaluating ...
	   Sensitivity :  0.692307692308
	   Specificity :  0.963636363636
	   Accuracy    :  0.934959349593
INFO: Cross Validation Iteration -  1
INFO: Training ...
INFO: Prediction for  Golf-Swing-Back
INFO: Evaluating ...
	   Sensitivity :  0.272727272727
	   Specificity :  0.910714285714
	   Accuracy    :  0.853658536585
INFO: Cross Validation Iteration -  2
INFO: Training ...
INFO: Prediction for  Golf-Swing-Front
INFO: Evaluating ...
	   Sensitivity :  0.5
	   Specificity :  0.965811965812
	   Accuracy    :  0.943089430894
INFO: Cross Validation Iteration -  3
INFO: Training ...
INFO: Prediction for  Golf-Swing-Side
INFO: Evaluating ...
	   Sensitivity :  0.9
	   Specificity :  0.946902654867
	   Accuracy    :  0.943089430894
INFO: Cross Validation Iteration -  4
INFO: Training ...
INFO: Prediction for  Kicking-Front
INFO: Evaluating ...
	   Sensitivity :  0.2
	   Specificity :  0.982300884956
	   Accuracy    :  0.918699186992
INFO: Cross Validation Iteration -  5
INFO: Training ...
INFO: Prediction for  Kicking-Side
INFO: Evaluating ...
	   Sensitivity :  0.1
	   Specificity :  0.982300884956
	   Accuracy    :  0.910569105691
INFO: Cross Validation Iteration -  6
INFO: Training ...
INFO: Prediction for  Lifting
INFO: Evaluating ...
	   Sensitivity :  0.888888888889
	   Specificity :  0.973684210526
	   Accuracy    :  0.967479674797
INFO: Cross Validation Iteration -  7
INFO: Training ...
INFO: Prediction for  Run-Side
INFO: Evaluating ...
	   Sensitivity :  0.583333333333
	   Specificity :  0.90990990991
	   Accuracy    :  0.878048780488
INFO: Cross Validation Iteration -  8
INFO: Training ...
INFO: Prediction for  SkateBoarding-Front
INFO: Evaluating ...
	   Sensitivity :  0.3
	   Specificity :  0.955752212389
	   Accuracy    :  0.90243902439
INFO: Cross Validation Iteration -  9
INFO: Training ...
INFO: Prediction for  Swing-SideAngle
INFO: Evaluating ...
	   Sensitivity :  0.46511627907
	   Specificity :  0.934090909091
	   Accuracy    :  0.892339544513
INFO: Cross Validation Iteration -  10
INFO: Training ...
INFO: Prediction for  Walk-Front
INFO: Evaluating ...
	   Sensitivity :  0.363636363636
	   Specificity :  0.955357142857
	   Accuracy    :  0.90243902439
INFO: Cross Validation Iteration -  11
INFO: Training ...
INFO: Prediction for  Swing-Bench
INFO: Evaluating ...
	   Sensitivity :  0.8
	   Specificity :  0.940677966102
	   Accuracy    :  0.934959349593
INFO: Cross Validation Iteration -  12
INFO: Training ...
INFO: Prediction for  Riding-Horse
INFO: Evaluating ...
	   Sensitivity :  0.9
	   Specificity :  0.902654867257
	   Accuracy    :  0.90243902439
  *** Overall Evaluation ***
    Average Sensitivity:  0.535846909997
    Average Specificity:  0.947984173698
    Average Accuracy   :  0.914169958709


### Testing with unseen data or video which has not been used for training
Test Video: /Users/somnath/MY_PROG/ComputerVision/PA3/ucf_sports_actions/ucf_action/Diving-Side/014

INFO: Training ...
	 Predicted Sports Action:[0] - Diving-Side

'''