PCA SVM KFold face detection

PCA,SVM,KFold face detection

Abstract

​ In this paper, we concentrate on PCA, SVM, and KFold with face detection. In the present research, face detection often uses deep learning networks to train the face detection model. However, it only sets parameters again on the other train and doesn’t break through. Traditional machine learning has become a basis for a breakthrough. The PCA can reduce the dimension of data to decrease the number of data to calculate and reduce the time of the training model. The SVM can try its best to distinguish between two kinds of data to classify two different types. SVM is a small to medium-sized data sample, non-linear and multi-dimensional. It can get a line to distinguish the different data types and get classified by the line.

Keywords: PCA, SVM, KFold, face detection

Introduction

​ In this research, we used PCA, SVM, and KFold for face detection.

​ Face detection often uses deep learning. Due to getting mass face data, deep learning can get good portability, strong adaptability, and strong learning ability. However, deep learning has a problem of a large amount of calculation, poor portability, and complex model design. Therefore, we adopt PCA, SVM, and KFold rather than deep learning.
​ PCA(Principal Component Analysis) is used to detection on multivariable data. In general, multivariable data can enhance the effort of data sampling and increase the correlation between variables. PCA can keep the completeness of the information and decrease the dimension of data.
​ SVM is based on the VC dimension theory and the principle of minimum structural risk. It can get the best compromise between model complexity and learning ability to get the best generalization ability. It has advantages in small samples and nonlinear and high-dimensional pattern recognition.
​ KFold is used for model tuning to find the parameter’s value that optimizes the model generalization performance. When finding the parameter, the model is retained on all the training data sets, and the independent test sets are used to make the final evaluation of the model performance. The advantage of KFold is that each sample point has only one chance to be included in the training set or test set during each iteration.

Related work

PCA

​ PCA is a dimension reduction method that can map multidimensional features to low-dimensional features, which is conducive to data dimension reduction and convenient operation.

Yang, Jian, et al. proposed a two-dimensional principal component analysis for image representation. Based on a two-dimensional image matrix, the original image matrix can be directly used to construct the image covariance matrix, and the feature vector can be derived for image feature extraction.

Yang, Jian, et al. “Two-dimensional PCA: a new approach to appearance-based face representation and recognition.” IEEE transactions on pattern analysis and machine intelligence 26.1 (2004): 131-137.

---

Maghari, Ashraf YA. applied PCA and independent component analysis (ICA) to baseline face recognition, demonstrated the relative performance of PCA and ICA, respectively, and concluded that PCA performed well.

Maghari, Ashraf YA. “Recognition of partially occluded faces using regularized ICA.” Inverse Problems in Science and Engineering 29.8 (2021): 1158-1177.

---

Daffertshofer, Andreas, et al. uses principal component analysis (PCA) to reduce high-dimensional data sets to a small number of patterns or structures and applies it to simulated and accurate data, obtaining that PCA allows detection of patterns in kinematic and EMG data sets and can separate invariant structures and variances in data.

Daffertshofer, Andreas, et al. “PCA in studying coordination and variability: a tutorial.” Clinical biomechanics 19.4 (2004): 415-428.

SVM

A support vector machine is a dichotomous model that can best classify data using a single line.
Huang, Shujun, et al. use support vector machines for cancer genes and subtypes to further explore the tumor genome.

Huang, Shujun, et al. “Applications of support vector machine (SVM) learning in cancer genomics.” Cancer genomics & proteomics 15.1 (2018): 41-51.

---

Gu, Jie, and Shan Lu. adopt an effective intrusion detection framework based on Naive Bayesian feature embedding support vector machines. Naive Bayes feature transformation is used to obtain high-quality new data on the original data. The SVM classifier is trained using the transformed data to establish an intrusion detection model.

Gu, Jie, and Shan Lu. “An effective intrusion detection approach using SVM with naïve Bayes feature embedding.” Computers & Security 103 (2021): 102158.

---

KFold

​ The cross-validation method is used for model tuning, finding the hyperparameters that make the model generalization performance optimal, retraining the model, and making the final evaluation of the model performance using independent test sets.

​ Tamilarasi, P. and R. Uma Rani adopt a KFold as a kind of resampling method, using it to compute machine learning models on small samples of data. The small deviation and negative value are very beneficial to the familiarity of model training.

Tamilarasi, P., and R. Uma Rani. “Diagnosis of crime rate against women using k-fold cross validation through machine learning.” 2020 Fourth International Conference on Computing Methodologies and Communication (ICCMC). IEEE, 2020.

---

Jung, Yoonsuh proposed a new cross-validation method called k-1 fold for model data validation based on the KFold framework and hold out fold for model construction. The model selection based on the average predicted value can reduce the estimation bias caused by the standard.

Jung, Yoonsuh. “Multiple predicting K-fold cross-validation for model selection.” Journal of Nonparametric Statistics 30.1 (2018): 197-215.

---

Face detection

Face detection can be considered as a specific object detection.

Face detection can be considered as a specific object detection, and face detection can be understood as face location in general mode. Early face detection is mainly based on extracting manual features from local areas of images, but high precision images can be obtained only under the premise of control. The emergence of deep learning has significantly improved the accuracy of face detection.

Minaee, Shervin, et al. “Going deeper into face detection: A survey.” arXiv preprint arXiv:2103.14983 (2021).

---

Hosni Mahmoud et al. proposed a hypothesis that challenges the hidden face based on skin color detection. When the head and shoulder contours are detected, the face is clustered, and the standardized mixing method is used to detect the face by combining the standardized RGB and YCbCr spatial colors.

Hosni Mahmoud, Hanan A., and Hanan Abdullah Mengash. “A novel technique for automated concealed face detection in surveillance videos.” Personal and Ubiquitous Computing 25.1 (2021): 129-140.

---

Soleymani et al. proposed a differential deformation attack detection framework, which uses a deeply connected network. This is the first work using the connected network architecture for deformation attack detection, which compares this model with other deep learning models using two different deformation datasets, VISAPP17 and MorGAN.

Soleymani, Sobhan, et al. “Differential morphed face detection using deep siamese networks.” International Conference on Pattern Recognition. Springer, Cham, 2021.

---

System

Preprocessing

We make code read all images and convert them to one-dimensional, we define label(label $\le$ 120), and define aub_label(sub_label $\le$ 26). We define all data sets as a two-dimension matrix n$\times$m(n is a sample, m is an attribute)

def get_picture():
    label = 1
    while (label <= 120):
        sub_label = 1
        while(sub_label <= 26):
            file_name = PICTURE_PATH + "\\AR" + str(label).zfill(3) + "-" + str(sub_label) + ".tif"
            for name in glob.glob(file_name):
                img = Image.open(name)
                all_data_set.append(list(img.getdata()))
                all_data_label.append(label)
                sub_label += 1
        label += 1

PCA

​ Principal component analysis (PCA), which is a kind of dimension reduction method, when can be affected by multiple factors, some factors that influence events in basically the same, so these factors can replace with a factor, then it can be a total factor to decrease, but achieve dimension reduction process, make the data handling easier, faster, in high-dimensional data, The structural relationship between data is impossible, so dimensionality reduction of original features is carried out under the condition that information is not lost as much as possible, so as to achieve new variables that become as little as possible to the collected data.
​ PCA is used to map n-dimensional features on k-dimension, with k-dimension as the main component, and all points are mapped to a fitted oblique line to reduce data and form higher reliability while retaining part of the spatial information before mapping.

Figure1. The top figure shows the original data distribution. The below figure shows the dimension reduction data distribution after PCA.

If you subtract the mean of your sample and return it to 0, the distribution of your sample hasn’t changed, but the axes have shifted. Using the variance formula$var(x)=\frac{1}{m}{\sum }_{i=1}^m(x_i-\overline{x}{)}^2$, $\overline{x}=0$, At this point, the maximum unit vector $\omega$ of variance after sample point mapping is sought, Define the direction of an axis $w$, Map the sample to $w$，It maximizes the variance of the mapping to $w$，When $var(X_{project})=\frac{1}{m}{\sum }_{i=1}^m(X^{(i)}\cdot \omega {)}^2$ is the most largest，getting value $\omega$.

for n_components in range(80,81):
    # PCA dimensional reduction
    pca = PCA(n_components=n_components, svd_solver='auto',
              whiten=True).fit(all_data_set)
    # Total data set after PCA dimensionality reduction
    all_data_pca = pca.transform(all_data_set)
    eigenfaces = pca.components_.reshape((n_components, 100, 80))
    # X is the data after dimension reduction, and y is the corresponding class label
    X = np.array(all_data_pca)
    y = np.array(all_data_label)
    # output Eigenfaces
    plt.figure("Eigenfaces")
    for i in range(1, 81):
        plt.subplot(8, 10, i).imshow(eigenfaces[i-1], cmap="gray")
        plt.xticks(())
        plt.yticks(())
    plt.show()

SVM

​ Support vector machine (SVM) is a kind of binary classification model, which maps the instance feature vector to some points in the space, and uses a line to distinguish these two kinds of points with the partition hyperplane with the largest interval. When encountering new points, the line can also make a good classification. This hyperplane is less disturbed by local samples, so it produces the most robust classification results and the strongest partition generalization ability for unknown samples.

​ To adopt $w^{T}X+b=0$ to partition hyperplane. w = { w 1 ; w 2 ; ⋅ ⋅ ⋅ ; w d } w=\left\{w_1;w_2; \cdot \cdot \cdot;w_d \right\} w={w1​;w2​;⋅⋅⋅;wd​}, $X$ is the training sample, $b$ is the displacement term. $\omega$ is the normal vector that determines the direction of the hyperplane. and $d$ is the number of eigenvalues. $b$ determines the distance between the hyperplane and the origin as shown in Fig 3.

Figure2. This is SVM divides the data.

A partition hyperplane is determined by determining the normal vector $w$ and the displacement $b$, and the distance between this hyperplane and any point on the marginal hyperplane on both sides is $\frac{1}{||w||}$

def SVM(kernel_name, param):#Enter the kernel function name and parameter gamma value to return the accuracy of the SVM training cross-validation 
    # The average accuracy was calculated by ten-fold cross-validation
    # n_splitsCross validation, random selection
    kf = KFold(n_splits=10, shuffle=True)
    precision_average = 0.0
    param_grid = {'C': [1e3, 5e3, 1e4, 5e4, 1e5]}  #Automatically enumerate the optimal C parameter
    clf = GridSearchCV(SVC(kernel=kernel_name, class_weight='balanced', gamma=param),
                       param_grid)#Automatically adjust parameters to obtain the optimal results and parameters
    for train, test in kf.split(X):
        clf = clf.fit(X[train], y[train])
        test_pred = clf.predict(X[test])
        # Calculated average accuracy
        precision = 0
        for i in range(0, len(y[test])):
            if (y[test][i] == test_pred[i]):
                precision = precision + 1
        precision_average = precision_average + float(precision) / len(y[test])
    precision_average = precision_average / 10
    return precision_average

KFold

​ KFold uses the no-repeat sampling technique, in which each sample point can be divided into data sets or test sets only once in each iteration. The KFold divides the data set into K pieces in equal proportions, one of which is used as the test set, and the remaining k-1 pieces are used as the training data set. This process is treated as an experiment, and a KFold can only be completed after K experiments. Finally, the K experimental test results were averaged to obtain the final result, as show in Fig 3.

Figure3. This is progress of KFold.

def K_fold(clf, fold_size):
    precision_average = 0.0
    kf = KFold(n_splits=fold_size, shuffle=True)
    test_k = [], train_k = []
    #It is used to display misidentified pictures.
    Divide the data into K groups
    Each time a subset of the K group is treated as a validation set, k experiments are performed.
    Get average value

Experiment

Dataset

We adopt 3120 AR face data, as show in Fig 4.

Figure 4, Example of partial data.

Eigenfaces

​ The Eigenfaces adopt PAC. Eigenfaces processes high-dimensional face data into low-dimensional data and then performs data analysis and processing to obtain recognition results. It shows a good result of the outline, as show in Fig 5.

The figure shows Eigenfaces of AR data.

Gamma

We use three different kernels contrust of rbf, poly, and sigmoid to process gamma after the experiment. , as show in Fig 6.

The accuracy curve varies with different gamma and kernel functions. The blue line is rbf. It is as the increase of gamma and precision get smaller observably. The yellow line is poly, and the green line is sigmoid. They are as the increase of gamma and keep slight wobble but flat overall.

Fold size

​ We adopt different fold size to process experiment, , as show in Fig 7.

Figure 8, The influence of the k-value of k-repeat cross-validation on accuracy.

After the experiment, we find the precision enhances observably in 2 to 7, but the line is nearly flat overall in seven to ten, then the precision keeps high accuracy.

Component

After the experiment, the precision enhances observably from 10 to 25, but the line is nearly flat overall from 25 to 125, then the precision keeps high accuracy. However, the line begins to decrease from 125 to 200.

Figure 9, The influence of n_components retained by PCA on accuracy.

Misidentification

These are the comparisons of misidentified face images. The left image is the test image, and the right image is the error-identified image. The appearance of each pair of images is similar.

Figure 10, A comparison of misidentified face images

Conclusion

​ The paper proposed a robust approach for face detection. PCA reduces the dimension of data to prevent calculated amounts. SVM can generally divide different data into two types. Kfold can reduce the amount of data used for training and testing. The method performs better than the traditional deep learning method, Although the method has misidentification, it has the high precision in general.

ounts. SVM can generally divide different data into two types. Kfold can reduce the amount of data used for training and testing. The method performs better than the traditional deep learning method, Although the method has misidentification, it has the high precision in general.