计算机视觉 ， 深度学习 (Computer Vision, Deep Learning)
Are you one of those who are excited about deep learning? Have you thought about creating something which can detect human emotions?Well, if that’s so then you are in the right place.
您是对深度学习感到兴奋的人之一吗？ 您是否考虑过要创建一种可以检测人类情感的东西？如果是这样，那么您来对地方了。
In this article, I am going to walk you through my emotion recognition model step by step, along with the insights I discover.
在本文中，我将逐步引导您完成我的情绪识别模型，以及发现的见解。
To make it easy for you to navigate across my blog I am going to list down points which this article is going to talk about.
为了使您轻松浏览我的博客，我将列出本文要讨论的要点。
- Understanding Dataset 了解数据集
- Creating Helper Functions 创建助手功能
- Converting data into the required format 将数据转换为所需格式
- Getting your Base Model 获取基本模型
- Preparing the Fine Tuned Model 准备微调模型
- Drawing Insights 绘图见解
- Re-constructing fine-tuned model 重建微调模型
- Making Predictions from images taken from Google 根据从Google拍摄的图像做出预测
So, if you are excited about it, then let’s begin!
因此，如果您对此感到兴奋，那就开始吧！
了解数据集 (Understanding Data set)
The data set of facial emotion recognition can be downloaded here from Kaggle. In case the data couldn’t be downloaded from there, you can use this drive link to download the dataset.
面部表情识别的数据集可以在此处从Kaggle下载。 如果无法从那里下载数据，则可以使用此驱动器链接下载数据集。
Now that you have the data set you are good to follow along.So, without further ado, let's start loading our data set. Since the processing of images takes a lot of time and resources on the local machine, hence I’ll be using colab’s GPU to train my model. So, in case if you don’t have your own GPU’s make sure you switched to colab to follow along.
现在您已经拥有了很好的数据集，接下来就不用多说了，让我们开始加载我们的数据集。 由于图像处理在本地计算机上花费大量时间和资源，因此我将使用colab的GPU来训练我的模型。 因此，如果您没有自己的GPU，请确保切换到colab进行后续操作。
导入必要的库 (Importing necessary Libraries)
#Importing Libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# importing tesnsorflow model libraries
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, AveragePooling2D
from tensorflow.keras.layers import Dense, Activation, Dropout, Flatten, BatchNormalization
from tensorflow.keras.preprocessing import image
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.metrics import categorical_accuracy
from tensorflow.keras.models import model_from_json,load_model
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras.optimizers import *# Loading dataset
data = pd.read_csv('fer2013.csv')After loading the data set, let’s try to explore our data. The data set consists of 3 columns namely emotion (target variable), pixels, and Usage.From data[‘emotion’].unique(), we can find how many types of emotions are present in our data set.
加载数据集后，让我们尝试探索我们的数据。 数据集由3列组成，即情感 (目标变量)， 像素和用法 。从data ['emotion']。unique()中，我们可以找到数据集中存在多少种情感。
In this case, here are 6 types of emotions which gets mapped as following: 0 -> Anger 1 -> Disgust 2 -> Fear 3 -> Happy 4 -> Sad 5 -> Surprise 6 -> Neutral
在这种情况下，以下是6种类型的情绪： 0->愤怒1->厌恶2->恐惧3->快乐4->悲伤5->惊喜6->中立
探索“使用情况”列 (Exploring ‘Usage’ Column)
# exploring values in Usage column
data['Usage].value_counts//========= Output ================//
Training 28709
PublicTest 3589
PrivateTest 3589
Name: Usage, dtype: int64The above code tells us that the data set consists of “Training”, “PublicTest”, and “PrivateTest”. Now we have to decide which type of data to be used for training, validation, and testing.
上面的代码告诉我们，数据集由“ Training”，“ PublicTest”和“ PrivateTest”组成。 现在，我们必须确定用于训练，验证和测试的数据类型。
I have decided to use the entire data set which is marked for ‘Training’ to be used as train-set, PublicTest as validation/dev-set, and PrivateTest as test-set. I have already separated out dataset into training, validation, and testing and saved it as separate CSV files.
我决定将标记为“培训”的整个数据集用作训练集，将PublicTest用作验证/开发集，并将PrivateTest用作测试集。 我已经将数据集分为训练，验证和测试，并将其保存为单独的CSV文件。
The reason for my choice is we want our model at the end to perform well on TestData, therefore PublicTest Data seems to be the best choice for me, as it would depict our PrivateTest. So, if we generalize well on validation set we will be doing well on test-data as well. You can download CSV files from the link below.Training-Data, Validation-Data, and Testing-Data
我之所以选择，是因为我们希望我们的模型最终在TestData上表现良好，因此PublicTest Data对我来说似乎是最佳选择，因为它可以描述我们的PrivateTest。 因此，如果我们对验证集进行良好的概括，那么在测试数据上也将表现出色。 您可以从下面的链接下载CSV文件。 培训数据 ， 验证数据和测试数据
By now we have loaded and understood a bit of what our data is about and what we are going to predict. Now let's dive into creating Helper functions for our model.
到现在为止，我们已经加载并了解了一些关于数据的内容以及我们将要预测的内容。 现在让我们潜入我们的模型创建辅助功能。
辅助功能 (Helper Functions)
Now, we will define some functions which will help us extract features and targets from our data set and we will store them in separate lists for later use.
现在，我们将定义一些函数，这些函数将帮助我们从数据集中提取特征和目标，并将它们存储在单独的列表中以备后用。
提取功能 (Extract Features)
# defining a function to extract features
def extract_features(filename):
X = []
Y = []
headers = True
for line in open(filename):
if headers:
headers = False
else:
row = line.split(',')
Y.append(int(row[0]))
X.append([int(feature) for feature in row[1].split()])
X,Y = np.array(X)/255.0, np.array(Y)
return (X,Y)The function above parses the entire file and separate out features and targets. Since our target emotion is found in the first column we slice it using row[0] and appends it into our list ‘Y’ while the rest features are appended in list ‘X’.
上面的函数解析整个文件，并分离出功能和目标。 因为我们的目标情感在第一列中找到，所以我们使用row [0]对其进行切片，并将其附加到列表“ Y”中，其余特征附加在列表“ X”中。
# variables to hold our files
train_file_name = 'Training_Data.csv'
val_file_name = 'Validation_Data.csv'
test_file_name = 'Testing_Data.csv'
# calling our helper function and storing fetaures in variables
train_X, train_Y = extract_features(train_file_name)
val_X, val_Y = extract_features(val_file_name)
test_X, test_Y = extract_features(test_file_name)
# converting into numpy array and casting it to float32
train_X = np.array(train_X,'float32')
train_Y = np.array(train_Y,'float32')
val_X = np.array(val_X,'float32')
val_Y = np.array(val_Y,'float32')
test_X = np.array(test_X,'float32')
test_Y = np.array(test_Y,'float32')Let’s check the shape of our data.
让我们检查数据的形状。
# Checking shape of our dataprint(train_X.shape)
print(train_Y.shape)
print(val_X.shape)
print(val_Y.shape)
print(test_X.shape)
print(test_Y.shape)############# Output ############
(28709, 2304) // training-data
(28709,)
(3589, 2304) //validtaion-data
(3589,)
(3589, 2304) //testing-data
(3589,)绘图功能 (Plotting Function)
import matplotlib.pyplot as plt
def plot(history):
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1, len(acc) + 1)
plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel('Epochs')
plt.ylabel('Acc')
plt.legend()
plt.figure()
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.show()The above plotting function is used to analyze how our model performs as the number of epochs increases.In order to make use of the same function, I have introduced a history parameter. Hence, given a history of any model, it will display its accuracy and loss graphs with the number of epochs along the x-axis.
上面的绘图函数用于分析我们的模型随着历元数的增加如何执行。为了使用相同的函数，我引入了一个历史参数。 因此，给定任何模型的历史记录，它将显示其准确性和损失图以及沿x轴的历元数。
情绪分析 (Emotion Analysis)
Our next helper function is drawing bar plots to analyze predicted emotion. This function will construct a bar graph and will display the confidence level of each emotion predicted by our model. This way we can keep track of how our model is performing and what is confusing our model.
我们的下一个辅助功能是绘制条形图以分析预测的情绪。 此功能将构建一个条形图，并显示由我们的模型预测的每种情绪的置信度。 这样，我们可以跟踪模型的性能以及模型的混乱之处。
def emotion_analysis(emotions):
objects = ['angry', 'disgust', 'fear', 'happy', 'sad', 'surprise', 'neutral']
y_pos = np.arange(len(objects))
plt.bar(y_pos, emotions, align='center', alpha=0.9)
plt.tick_params(axis='x', which='both', pad=10,width=4,length=10)
plt.xticks(y_pos, objects)
plt.ylabel('percentage')
plt.title('emotion')
plt.show()显示预测 (Display Predictions)
Our final helper function is used to draw predictions on our own image. In the function below, we simply make use of the TensorFlow image preprocessing library and just convert it into the required format before making predictions.
我们的最终辅助函数用于在我们自己的图像上绘制预测。 在下面的函数中，我们仅使用TensorFlow图像预处理库，并在进行预测之前将其转换为所需的格式。
from skimage import io
from tensorflow.keras.preprocessing import image
import matplotlib.pyplot as plt
def display_predictions(img_file,model):
img = image.load_img(img_file, grayscale=True, target_size=(48, 48))
show_img=image.load_img(img_file, grayscale=False, target_size=(200, 200))
x = image.img_to_array(img)
x = np.expand_dims(x, axis = 0)
x /= 255
custom = model.predict(x)
#print(custom[0])
emotion_analysis(custom[0])
x = np.array(x, 'float32')
x = x.reshape([48, 48]);
plt.gray()
plt.imshow(show_img)
plt.show()
m=0.000000000000000000001
a=custom[0]
for i in range(0,len(a)):
if a[i]>m:
m=a[i]
ind=i
print('Expression Prediction:',objects[ind])将数据转换为所需格式 (Converting Data into the required format)
After getting our helper functions in check, now we can call our extract_features function and convert the data in the required format.Our model expects data of shape(N,d,d,1), where N equals the number of training examples, d equals dimensions. I have already saved my data in different files, so I’ll just be loading data and converting it into the format expected by our model.
检查好辅助函数后，现在我们可以调用我们的extract_features函数并以所需的格式转换数据。我们的模型期望shape(N，d，d，1)的数据，其中N等于训练示例的数量d等于尺寸。 我已经将数据保存在其他文件中，因此我将仅加载数据并将其转换为模型所期望的格式。
# Storing data in different variables
train_file_name = 'Training_Data.csv'
val_file_name = 'Validation_Data.csv'
test_file_name = 'Testing_Data.csv'
# Extracting features and labels
train_X, train_Y = extract_features(train_file_name)
val_X, val_Y = extract_features(val_file_name)
test_X, test_Y = extract_features(test_file_name)
# casting to float
train_X = np.array(train_X,'float32')
train_Y = np.array(train_Y,'float32')
val_X = np.array(val_X,'float32')
val_Y = np.array(val_Y,'float32')
test_X = np.array(test_X,'float32')
test_Y = np.array(test_Y,'float32')
# converting data into (N,d,d,1)
N, D = train_X.shape
val_n,val_d = val_X.shape
test_n, test_d = test_X.shape
train_X = train_X.reshape(N, 48, 48, 1)
val_X = val_X.reshape(val_n, 48, 48, 1)
test_X = test_X.reshape(test_n,48,48,1)定义基本模型 (Defining Base Model)
Finally, after all the pre-processing stuff we can now construct our base model.I chose to go with a traditional base model of having multiple convolution layers and after two convolution layers there follows the max-pooling layer.However, the results of this architecture turn out to be relatively poor.For 30 epochs, the validation accuracy was stuck around 0.25 something and there was no improvement at all.Below is the model architecture I used for my base model.
最后，在完成所有预处理工作之后，我们现在可以构建基本模型了。我选择使用具有多个卷积层的传统基本模型，然后在两个卷积层之后跟随最大池化层。架构相对较差。在30个时期内，验证准确性停留在0.25左右，根本没有改善。以下是我用于基础模型的模型架构。
def base_model():#1st convolution layer
model = Sequential()
model.add(Conv2D(64, kernel_size=(3, 3), activation='relu', input_shape=(X_train.shape[1:])))
model.add(Conv2D(64,kernel_size= (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2), strides=(2, 2)))
model.add(Dropout(0.5))#2nd convolution layer
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2), strides=(2, 2)))
model.add(Dropout(0.5))#3rd convolution layer
model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2), strides=(2, 2)))model.add(Flatten())#fully connected neural networks
model.add(Dense(1024, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(1024, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(num_labels, activation='softmax'))return modelThe output accuracy was as follows after 30 epochs.
30个纪元后，输出精度如下。
Epoch 30/30449/449 [==============================] - 8s 17ms/step - loss: 1.8105 - accuracy: 0.2513 - val_loss: 1.8126 - val_accuracy: 0.2494As you can see that the results from our previous model were pretty poor as our training and validation accuracy were not really improving.
如您所见，由于我们的训练和验证准确性并未真正提高，因此先前模型的结果非常差。
The question arises here as to what changes could we bring in our next Model architecture?
这里的问题是，我们可以在下一个模型体系结构中带来哪些变化？
I planned to make the following changes in my next model architecture which include:- Addition of a batch normalization layer.- Using Dense layer of 128 nodes,- making use of regularizers,- learning rate 0.001
我计划在我的下一个模型体系结构中进行以下更改，包括：-添加批标准化层。-使用128个节点的密集层，-使用正则化器，-学习率0.001
调整模型架构 (Tuned Model Architecture)
# tuned_model
def tuned_model():
model = Sequential()
input_shape = (48,48,1)
#1st convolution layer
model.add(Conv2D(64, (5, 5), input_shape=input_shape,activation='relu', padding='same'))
model.add(Conv2D(64, (5, 5), activation='relu', padding='same'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.5))
#2nd convolution layer
model.add(Conv2D(128, (5, 5),activation='relu',padding='same'))
model.add(Conv2D(128, (5, 5),activation='relu',padding='same'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.5))
#3rd convolution layer
model.add(Conv2D(256, (3, 3),activation='relu',padding='same'))
model.add(Conv2D(256, (3, 3),activation='relu',padding='same'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dense(128))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(7))
model.add(Activation('softmax'))
my_optimiser = tf.keras.optimizers.Adam(
learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, amsgrad=False,
name='Adam')
model.compile(loss='categorical_crossentropy', metrics=['accuracy'],optimizer=my_optimiser)
return modelI defined another model architecture based on the considerations I took into account.You can find about how BatchNormalization works and improves our model from here.
我根据考虑的因素定义了另一种模型体系结构。您可以从此处找到BatchNormalization的工作原理并改进我们的模型。
过度拟合模型 (Overfitting the model)
Let’s take our model towards overfitting and then make some analysis as to what is the best set of hyperparameters in this case.
让我们将模型用于过度拟合，然后对这种情况下最佳的超参数集进行一些分析。
history = model_1.fit(train_X,
Y_train,
batch_size=64,
epochs=80,
verbose=1,
validation_data=(val_X,Y_val),
shuffle=True,
)At the end of 80 epochs, we can clearly see that our model has overfitted as our training accuracy keeps on increasing while validation accuracy becomes stagnant at around 0.65.
在80个时代结束时，我们可以清楚地看到我们的模型已经过拟合，因为我们的训练准确性不断提高，而验证准确性却停滞在0.65左右。
Epoch 80/80 449/449 [==============================] - 33s 73ms/step - loss: 0.2126 - accuracy: 0.9244 - val_loss: 1.6008 - val_accuracy: 0.6503# Let's save this model for further testing
model_1.save('base_model(fer).h5')It’s now time to call one of our Helper function to plot the graph to see the progress of our model over 80 epochs.
现在是时候调用我们的辅助函数之一来绘制图形了，以查看我们的模型在80个时期内的进度。
# Lets plot the graph and see what it tell us
plot(history) # This will display the accuracy and loss graphs.
绘图见解 (Drawing Insights)
The graphs clearly confirm that we should stop the training around 10 epochs as after 10 epochs there is no real improvement in our model.So, major takeaways from the overfitted model are to train around for 10–12 epochs and evaluate the result.That’s exactly what we are going to do next!But before running another model, let us make some predictions from our overfitted model to compare it against the model we will create later.
这些图清楚地表明，我们应该停止训练约10个纪元，因为在10个纪元后我们的模型没有真正的改善。因此，过度拟合模型的主要收获是训练10–12个纪元并评估结果。但是，在运行另一个模型之前，让我们根据过度拟合的模型进行一些预测，以将其与稍后创建的模型进行比较。
根据过拟合模型进行预测 (Making Predictions from overfitted Model)
I randomly downloaded images from google of different emotions and then tested them out using my overfitted model.Below shows the picture of the output I got on random images from google.
我从谷歌随机下载了不同情绪的图像，然后使用过拟合模型对其进行了测试。下面显示了我从谷歌随机图像中获得的输出图片。
As you can see that our model didn’t predict all emotions accurately.In the first and third images, it wrongly classified as fear. I have tried many images of disgust but it fails to classify any image as one. I think the data is not sufficient enough to classify, so we can just drop those rows for a later experiment, but for now, we’ll not remove it.The bar graph also helps us in analyzing where our model gets confused most. According to my findings, it has difficulty in detecting angry, surprise, and fear emotions as it is not able to distinguish it well due to the same landmarks.
如您所见，我们的模型无法准确预测所有情绪，在第一张和第三张图像中，它被错误地归类为恐惧。 我尝试过许多令人反感的图像，但无法将任何图像归类为一个图像。 我认为数据不足以进行分类，因此我们可以删除这些行以进行以后的实验，但是目前我们不会将其删除。条形图还可以帮助我们分析模型最容易混淆的地方。 根据我的发现，由于具有相同的地标性，它无法很好地区分它，因此很难检测到愤怒 ， 惊讶和恐惧的情绪。
Let’s now run the model for 12 epochs and make predictions from that model to further strengthen our analysis.
现在，我们将模型运行12个时期，并根据该模型做出预测，以进一步加强我们的分析。
重新运行12个时期的调整模型 (Re-running Tuned Model for 12 epochs)
# Running model for 12 epochs
model_2 = base_model()
hist_2 = model_2.fit(train_X, Y_train,
batch_size=64,
epochs=12,
verbose=1,
validation_data=(val_X, Y_val),
shuffle=True,)This is what we got at the end of 12 epochs.
这是我们在12个阶段结束时得到的。
Epoch 12/12 449/449 [==============================] - 33s 73ms/step - loss: 0.9368 - accuracy: 0.6493 - val_loss: 1.0876 - val_accuracy: 0.5932Now Lets Plot it
现在开始绘制
So, now we have got rid of overfitting. The validation accuracy is around 0.59, which is less in normal circumstances but on this dataset, I think is par enough accuracy.
因此，现在我们摆脱了过度拟合的问题。 验证准确度约为0.59，在正常情况下会稍差一些，但在此数据集上，我认为这已经足够了。
Let's save the model and draw predictions on the same images we downloaded from google.
让我们保存模型并在从Google下载的同一张图片上绘制预测。
从模型中得出预测 (Drawing Predictions from the Model)
Bingo! Our final model seems to distinguish well between different emotions.As you can see in the above prediction it is now able to recognize anger emotion, which previously it recognize as fear. Moreover, it was also able to predict surprise emotion in the last picture.
宾果 ！ 我们的最终模型似乎可以很好地区分不同的情绪，正如您在上述预测中所看到的那样，它现在能够识别愤怒情绪，而以前它被视为恐惧。 此外，它还能够预测最后一张照片中的惊喜情绪。
Despite the improvement in model accuracy, it’s still not quite accurate and better model structure could certainly help.
尽管模型准确性有所提高，但仍然不够准确，更好的模型结构肯定会有所帮助。
外卖和未来的改进 (Takeaways and Future Improvements)
If we closely analyze the prediction confidence, we can clearly see that our model still seems to be confused between anger, surprise, and fear.It should be also noted that disgust emotion never got predicted, so removing the rows will do no harm.
如果我们仔细分析预测的置信度，我们可以清楚地看到我们的模型似乎仍然被愤怒，惊讶和恐惧所混淆，还应该指出， 厌恶情绪从未得到预测，因此删除行不会造成任何伤害。
Another suggestion for future work can be to collect images of different human emotions from the internet and make different directories of those emotions. We can make use of the Image generator library of TensorFlow to Augment images and create our own dataset.Lastly, we can use transfer learning or pre-trained network and attach our convolution and classifier at the bottom and make predictions.
未来工作的另一个建议是从互联网上收集不同人类情感的图像，并为这些情感建立不同的目录。 我们可以使用TensorFlow的图像生成器库来增强图像并创建自己的数据集。最后，我们可以使用转移学习或预训练网络并将卷积和分类器附加在底部并进行预测。
结论 (Conclusion)
Thank you so much for sticking too long with me and this post.I know it got too long, however, I thought sharing my valuable insights might help some Deep Learning enthusiasts create a better intuition of the task at hand.
非常感谢您坚持我和这篇文章太久了。我知道这篇文章太长了，但是，我认为分享我的宝贵见解可能会帮助一些深度学习爱好者更好地理解手头的任务。
The entire code to this post can be found here.
这篇文章的完整代码可以在这里找到。
Finally, I’ll be looking forward to your feedback on the article.
最后，我们期待您对本文的反馈。
扫一扫
1211
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
