Spatiotemporal emotion recognition based on 3D time-frequency domain feature matrix

doi:10.19682/j.cnki.1005-8885.2022.0017

中国邮电高校学报(英文版) ›› 2022, Vol. 29 ›› Issue (5): 62-72.doi: 10.19682/j.cnki.1005-8885.2022.0017

• Artificial Intelligence • 上一篇下一篇

Spatiotemporal emotion recognition based on 3D time-frequency domain feature matrix

晁浩¹,连卫芳²,刘永利¹

1. 河南理工大学
2. 河南理工大学计算机科学与技术学院

收稿日期:2021-05-06 修回日期:2022-04-11 出版日期:2022-10-31 发布日期:2022-10-28
通讯作者: 连卫芳 E-mail:1137844972@qq.com
基金资助:
国家自然科学科学基金;河南省高等学校重点科研计划项目;河南省高校基本科研业务费专项资金项目

Spatiotemporal emotion recognition based on 3D time-frequency domain feature matrix

Chao Hao¹, Lian Weifang¹,², Liu Yongli¹

Received:2021-05-06 Revised:2022-04-11 Online:2022-10-31 Published:2022-10-28
Contact: Wei-Fang LIAN E-mail:1137844972@qq.com

摘要/Abstract

摘要：

The research of emotion recognition based on electroencephalogram (EEG) signals often ignores the related information between the brain electrode channels and the contextual emotional information existing in EEG signals, which may contain important characteristics related to emotional states. Aiming at the above defects, a spatiotemporal emotion recognition method based on a 3-dimensional (3D) time-frequency domain feature matrix was proposed. Specifically, the extracted time-frequency domain EEG features are first expressed as a 3D matrix format according to the actual position of the cerebral cortex. Then, the input 3D matrix is processed successively by multivariate convolutional neural network (MVCNN) and long short-term memory (LSTM) to classify the emotional state. Spatiotemporal emotion recognition method is evaluated on the DEAP data set, and achieved accuracy of 87.58% and 88.50% on arousal and valence dimensions respectively in binary classification tasks, as well as obtained accuracy of 84.58% in four class classification tasks. The experimental results show that 3D matrix representation can represent emotional information more reasonably than two-dimensional (2D). In addition, MVCNN and LSTM can utilize the spatial information of the electrode channels and the temporal context information of the EEG signal respectively.

关键词: spatiotemporal emotion recognition model 3-dimensinal (3D) feature matrix| time-frequency features| multivariate convolutional neural network (MVCNN)| long short-term memory (LSTM)

Abstract:

Key words: spatiotemporal emotion recognition model 3-dimensinal (3D) feature matrix| time-frequency features| multivariate convolutional neural network (MVCNN)| long short-term memory (LSTM)

Chao Hao Lian Weifang Liu Yongli. Spatiotemporal emotion recognition based on 3D time-frequency domain feature matrix[J]. The Journal of China Universities of Posts and Telecommunications, 2022, 29(5): 62-72.

参考文献

[1] YIN Z, WANG Y X, LIU L, et al. Cross-subject EEG feature selection for emotion recognition using transfer recursive feature elimination. Frontiers in Neurorobotics, 2017, 11: Article 19.

[2] GUI Q, JIN Z P, XU W Y, et al. Multichannel EEG-based biometric using improved RBF neural networks. Proceedings of the 2015 IEEE Signal Processing in Medicine and Biology Symposium ( SPMB’15 ), 2015, Dec 12, Philadelphia, PA, USA. Piscataway, NJ, USA: IEEE, 2015: 1 -6.

[3] FRANTZIDIS C A, BRATSAS C, PAPADELIS C L, et al. Toward emotion aware computing: an integrated approach using multichannel neurophysiological recordings and affective visual stimuli. IEEE Transactions on Information Technology in Biomedicine, 2010, 14(3): 589 -597.

[4] ZHENG W L, ZHU J Y, LU B L. Identifying stable patterns over time for emotion recognition from EEG. IEEE Transactions on Affective Computing, 2019, 10(3): 417 -429.

[5] CHEN X, XU X Y, LIU A P, et al. The use of multivariate EMD and CCA for denoising muscle artifacts from few-channel EEG recordings. IEEE Transactions on Instrumentation and Measurement, 2018, 67(2): 359 -370.

[6] KHOSROWABADI R, QUEK H C, WAHAB A, et al. EEG- based emotion recognition using self-organizing map for boundary detection. Proceedings of the 20th International Conference on Pattern Recognition, 2010, Aug 23 - 26, Istanbul, Turkey. Piscataway, NJ, USA: IEEE, 2010: 4242 -4245.

[7] RAHMAN M A, HOSSAIN M F, HOSSAIN M, et al. Employing PCA and t-statistical approach for feature extraction and classification of emotion from multichannel EEG signal. Egyptian Informatics Journal, 2020, 21(1): 23 -35.

[8] TIBDEWAL M N, TALE S A. Multichannel detection of epilepsy using SVM classifier on EEG signal. Proceedings of the 2016 International Conference on Computing Communication Control and Automation ( ICCUBEA’17), 2016, Aug 12 - 13, Pune, India. Piscataway, NJ, USA: IEEE, 2016: 1 -6.

[9] LIN W Q, LI C, SUN S Q. Deep convolutional neural network for emotion recognition using EEG and peripheral physiological signal. Proceedings of the International Conference on Image and Graphics ( ICIG’17), 2017, Sep 13 - 15, Shanghai, China. LNIP 10667. Berlin, Germany: Springer, 2017: 385 -394.

[10] LIU N J, FANG Y C, LI L, et al. Multiple feature fusion for automatic emotion recognition using EEG signals. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’18), 2018, Apr 15 - 20, Calgary, Canada. Piscataway, NJ, USA: IEEE, 2018: 896 -900.

[11] KHOSROWABADI R, QUEK C, ANG K K, et al. ERNN: a biologically inspired feedforward neural network to discriminate emotion from EEG signal. IEEE Transactions on on Neural Networks and Learning Systems, 2014, 25(3): 609 -620.

[12] HASSAN M M, ALAM M G R, UDDIN M Z, et al. Human emotion recognition using deep belief network architecture. Information Fusion, 2019, 51: 10 -18.

[13] LI Y J, HUANG J J, ZHOU H Y, et al. Human emotion recognition with electroencephalographic multidimensional features by hybrid deep neural networks. Applied Sciences, 2017, 7(10): Article 1060.

[14] Li X, SONG D W, ZHANG P, et al. Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. Proceedings of the 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM’16), 2016, Dec 15 - 18, Shenzhen, China. Piscataway, NJ, USA: IEEE, 2017: 352 -359.

[15] SONG T F, ZHENG W M, SONG P, et al. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing, 2020, 11(3): 532 - 541.

[16] MOON S E, JANG S, LEE J S. Convolutional neural network approach for eeg-based emotion recognition using brain connectivity and its spatial information. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’18), 2018, Apr 15 -20, Calgary, Canada. Piscataway, NJ, USA: IEEE, 2018: 2556 -2560.

[17] KOELSTRA S, YAZDANI A, SOLEYMANI M, et al. Single trial classification of EEG and peripheral physiological signals for recognition of emotions induced by music videos. Proceedings of the International Conference on Brain Informatics (BI’10), 2010, Aug 28 - 30, Toronto, Canada. LNAI 6334. Berlin, Germany: Springer, 2010: 89 -100.

[18] RUSSELL J A, LEWICKA M, NIIT T. A cross-cultural study of a circumplex model of affect. Journal of Personality and Social Psychology, 1989, 57(5): 848 -856.

[19] IWAKI T, NOSHIRO M. EEG activity over frontal regions during positive and negative emotional experience. Proceedings of the 2012 ICME International Conference on Complex Medical Engineering? ( CME’12 ), 2012, Jul 1 - 4, Kobe, Japan. Piscataway, NJ, USA: IEEE, 2012: 418 -422.

[20] LI X, ZHANG P, SONG D W, et al. EEG based emotion identification using unsupervised deep feature learning. Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ‘15), 2015, Aug 9 - 13, Santiago, Chile. New York, NY, USA: ACM, 2015: 1 -3.

[21] ATKINSON J, CAMPOS D. Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers. Expert Systems with Applications, 2016, 47: 35 -41.

[22] TANG H, LIU W, ZHENG W L, et al. Multimodal emotion recognition using deep neural networks. Proceedings of the 24th International Conference on Neural Information Processing (CONIP’17), 2017, Oct 14 - 18, Guangzhou, China. LNTCS 10637. Berlin, Germany: Springer, 2017: 811 -819.

[23] XING X F, LI Z Q, XU T Y, et al. SAE + LSTM: a new framework for emotion recognition from multi-channel EEG. Frontiers in Neurorobotics, 2019, 13: Article 37.

[24] CHEN J X, JIANG D M, ZHANG Y N, et al. Emotion recognition from spatiotemporal EEG representations with hybrid convolutional recurrent neural networks via wearable multi-channel headset. Computer Communications, 2020, 154: 58 -65.

[25] JADHAV N, MANTHALKAR R, JOSHI Y. Electroencephalo-graphy-based emotion recognition using gray-level co-occurrence matrix features. Proceedings of the 2016 International Conference on Computer Vision and Image Processing ( CVIP’16): Vol 1, 2016, Feb 26 - 28, Roorkee, India. AISC 459. Berlin, Germany: Springer, 2016: 335 -343.

[26] MEI H, XU X M. EEG-based emotion classification using convolutional neural network. Proceedings of the 2017 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC’17), 2017, Dec 15 - 17, Shenzhen, China. Piscataway, NJ, USA: IEEE, 2017: 130 -135.

[27] KWON Y H, SHIN S B, KIM S D. Electroencephalography based fusion two-dimensional (2D)-convolution neural networks (CNN) model for emotion recognition system. Sensors, 2018, 18 (5): Article 1383.

[28] CIMTAY Y, EKMEKCIOGLU E. Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition. Sensors, 2020, 20(7): Article 2034.

[29] CHO J C, HWANG H. Spatio-temporal representation of an electoencephalogram for emotion recognition using a three-dimensional convolutional neural network. Sensors, 2020, 20(12): Article 3491.

Spatiotemporal emotion recognition based on 3D time-frequency domain feature matrix

Spatiotemporal emotion recognition based on 3D time-frequency domain feature matrix

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 0

编辑推荐

Metrics

本文评价