A simple method for gait emotion recognition

Title: A simple method for gait emotion recognition

Author: Dong Qiang, San Liu

Abstract

The automatic recognition of human emotion from visual features is an essential function for many intelligent appli cations. In recent years, gait-based emotion recognition, especially gait skeletons-based feature, has drawn a lot of attention, and many methods have been developed gradu ally. The common approach is to first extract affective fea tures from joint skeletons, and then to combine the skeleton joints and affective features as the feature vector for emo tion classification. However, the combination process of these methods may be inflexible, leading to inadequate use of the complementary relationship between skeleton joints and affective features. Moreover, the long range depen dencies in both spatial and temporal domains of the gait  sequence are rarely considered. To solve these problems, we propose a novel two-stream network with transformer  based complementarity, called TNTC. Skeleton joints and affective features are encoded into two separate images as the inputs of two streams, respectively. A new transformer  based complementarity module (TCM) is proposed to con nect the complementarity between two streams hierarchi cally by capturing long range dependencies. Experimental results show that TNTC surpasses state-of-the-art methods on the latest dataset in terms of accuracy.

References

[1] Aniket Bera, Tanmay Randhavane, and Dinesh Manocha. Modelling multi-channel emotions using facial expression and trajectory cues for improving socially-aware robot navigation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 257– 266, 2019. 1

[2] Uttaran Bhattacharya, Trisha Mittal, Rohan Chandra, Tanmay Randhavane, Aniket Bera, and Dinesh Manocha. Step: Spatial temporal graph convolutional networks for emotion perception from gaits. Proceedings of the AAAI Conference on Artificial Intelligence , 34(02):1342–1350, Apr. 2020. 1 , 2 , 3 , 4

[3] Uttaran Bhattacharya, Christian Roncal, Trisha Mittal, Rohan Chandra, Kyra Kapsaskis, Kurt Gray, Aniket Bera, and Dinesh Manocha. Take an emotion walk: Perceiving emotions from gaits using hierarchical attention pooling and affective mapping. In Computer Vision – ECCV 2020, pages 145–163, 2020. 1 , 3 , 4

[4] J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248–255, June 2009. 2

[5] Yilin Dong, Xinde Li, Jean Dezert, Mohammad Omar Khyam, Md Noor-A-Rahim, and Shuzhi Sam Ge. Dezertsmarandache theory-based fusion for human activity recognition in body sensor networks. IEEE Transactions on In dustrial Informatics , 16(11):7138–7149, 2020. 1

[6] Yilin Dong, Xinde Li, Jean Dezert, Rigui Zhou, Changming Zhu, Lei Cao, Mohammad Omar Khyam, and Shuzhi Sam Ge. Multisource weighted domain adaptation with evidential reasoning for activity recognition. IEEE Transactions on Industrial Informatics , 19(4):5530–5542, 2022. 1

[7] Yilin Dong, Xinde Li, Jean Dezert, Rigui Zhou, Changming Zhu, and Shuzhi Sam Ge. Multi-criteria analysis of sensor reliability for wearable human activity recognition. IEEE Sensors Journal , 21(17):19144–19156, 2021. 1

[8] Yilin Dong, Xinde Li, Jean Dezert, Rigui Zhou, Changming Zhu, Lai Wei, and Shuzhi Sam Ge. Evidential reasoning with hesitant fuzzy belief structures for human activity recognition. IEEE Transactions on Fuzzy Systems, 29(12):3607– 3619, 2021. 1

[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Con

ference on Learning Representations , 2020. 1

[10] Sylmarie D´avila-Montero, Jocelyn Alisa Dana-Lˆe, Gary Bente, Angela T. Hall, and Andrew J. Mason. Review and challenges of technologies for real-time human behavior monitoring. IEEE Transactions on Biomedical Circuits

and Systems , 15(1):2–28, 2021. 1