Sentiment analysis of Weibo short text based on attention mechanism and BERT model

Sentiment analysis of Weibo short text

based on attention mechanism and

BERT model

Chunyu Jiao 

CCNU

Artificial Intelligence Academy

Wuhan, China

1060023370@qq.com

Abstract—With the gradual development of deep learning techniques, scholars have made great progress in using well-performing neural networks for sentiment analysis of text. However, this technique then became difficult to break the upper limit, and previous techniques were unable to handle texts with complex sentiments, and more fine-grained sentiment analysis was the key to solve the current problem. As a result, multi-categorization text analysis at the comment text level is shifted to aspect-level judgment of statements in comments. In this project, aspect-level sentiment analysis is improved for the short text of microblog comments based on the attention mechanism. The model proposed in this paper uses the attention mechanism to analyze the weights of short texts and precisely weight the summation, and finally the category output of short texts can be obtained. For the characteristics of microblog comments such as few words, short sentences, frequent network phrases and double-entendres, this paper achieves good results by improving the lexical model, and the algorithm in this paper conducts comparative experiments on the Chinese dataset of microblog users' comments, and the experimental The results show that the algorithm can achieve higher classification accuracy and F1 value in aspect-level sentiment classification task than other algorithms.

Keywords—Natural Language Processing, Short text, Attention mechanism, Word vector 

1. Introduction

In the 21st century, people's communication is very fast-paced, and people are more willing to express their moods at the moment through short messages. Therefore, the analysis of short texts becomes more and more important. Microblog comment data are studied using natural language processing techniques. The purpose of this paper is to further improve the progress of short textbook analysis module in the field of natural language processing by proposing a new deep neural network to process short textbook data, which is used to change the difficult problems of existing research on short textbook sentiment analysis and further achieve optimization to achieve more accurate short textbook sentiment analysis in practice. The starting point of this study is to count the large number of comments made on weibo, analyze their sentiment tendencies, and thus mine the sentiment tendencies of the majority of user comments. Sentiment analysis is generally classified according to the size of the processed text granularity, and can be divided into three approaches based on document-level sentiment analysis, sentence-level sentiment analysis, and aspect-level sentiment analysis. The aspect-level sentiment analysis based on the bert model that we will study in this paper. As one of the tasks of sentiment analysis task, it is a more fine-grained sentiment analysis method compared with the other two sentiment analysis methods, which is aimed at identifying the sentiment polarity of different aspects in the comment text. Currently, sentiment analysis methods in aspect-level based sentiment analysis tasks are divided into the following three main categories: lexicon and rule-based sentiment analysis methods, traditional machine learning-based sentiment analysis methods and deep neural network-based sentiment analysis methods. In this paper, the deep learning approach is mainly used to build the model[1].

1. Related work
   1. Sentiment classification

The traditional sentiment analysis is to first construct a sentiment dictionary or sentiment collocation template by the lexicon and rule-based sentiment analysis method, then use it as a benchmark to compare the sentiment words or fixed collocations contained in the review text, and finally calculate the sentiment tendency of the text. Later scholars improved it by adopting a machine learning-based approach to classify texts, mainly by extracting features and modeling the training data set with polarity labels, and then learning them by machine learning algorithms to finally achieve the judgment of sentiment polarity[2]. The former provides the feature set needed by the classification model to fit the data to the training text, while the latter affects the classification model's ability to The latter affects the ability of the classification model to learn the feature set. In this paper, we adopt a deep learning approach for text sentiment analysis, where deep neural networks can better capture the dependencies between aspects and contextual content, better learn deep features, and break the limitations of machine learning. The application of deep learning methods in aspect-level sentiment analysis tasks consists of two main parts, one is the neural network language model, and the other is the neural network model to extract deep features[4]. A distributed representation approach is proposed, which is to first map words to a continuous low-dimensional vector space using a neural network language model, and then calculate the cosine similarity between each word to capture the correlation between individual words. The word vectors are obtained by constructing a co-occurrence matrix of word-context combination training, which not only considers local information, but also utilizes global corpus information. The target words are used as input features, stitched together with contextual features, and then input to the classification layer for sentiment polarity analysis[4]. Meanwhile, in the sentiment analysis task, different words in the context are associated with the target word to different degrees, so the weights of different words, or even different sentences, need to be calculated and analyzed using the attention mechanism[5]. Deep learning-based sentiment analysis algorithms can be better applied to big data, and large-scale neural network models are highly capable of understanding the deep semantics of text, so a large number of researchers expect to use deep learning methods to build neural network models to automatically extract potential semantic features of text and apply them directly to the sentiment classification task, so as to achieve a better The goal is to achieve better classification results without human involvement[6].

1. 1.  Attention mechanism networks

The attention model was first used in machine translation, where the attention mechanism is the attention assigned to the input weights. The first use of the attention mechanism was in the encoder-decoder, where the attention mechanism obtains the input variables at the next level by doing a weighted average of the hidden states of the encoder over all time steps: the generic expression for the attention mechanism can be written as O=softmax(QKt)VO=softmax(QKt)V. The attention mechanism can be broadly understood as a one-layer perceptron (softmax summation) consisting of a matrix of query terms and the corresponding key terms and the value terms to be weighted averaged.

The attention module in the network structure is responsible for automatically learning the attention weights αij which automatically capture the correlation between hi (encoder hidden states, which we call candidate states) and sj (decoder hidden states, which we call query states). These attention weights are then used to construct the content vector C, which is passed to the decoder as input. At each decoding position j, the content vector cj is a weighted sum of all hidden states of the encoder and their corresponding attention weights.

 

Fig.1. Encode-Decoder framework diagram

1. Short text feature extraction model

A total of four classification categories emotions in microblogs are analyzed and studied for feature engineering, and a text feature-based emotion classification algorithm is proposed for short textbook genre classif