The Journal of China Universities of Posts and Telecommunications ›› 2022, Vol. 29 ›› Issue (5): 62-72.doi: 10.19682/j.cnki.1005-8885.2022.0017

Previous Articles     Next Articles

Spatiotemporal emotion recognition based on 3D time-frequency domain feature matrix

Chao Hao1, Lian Weifang1,2, Liu Yongli1   

  • Received:2021-05-06 Revised:2022-04-11 Online:2022-10-31 Published:2022-10-28
  • Contact: Wei-Fang LIAN E-mail:1137844972@qq.com

Abstract:

The research of emotion recognition based on electroencephalogram (EEG) signals often ignores the related information between the brain electrode channels and the contextual emotional information existing in EEG signals, which may contain important characteristics related to emotional states. Aiming at the above defects, a spatiotemporal emotion recognition method based on a 3-dimensional (3D) time-frequency domain feature matrix was proposed. Specifically, the extracted time-frequency domain EEG features are first expressed as a 3D matrix format according to the actual position of the cerebral cortex. Then, the input 3D matrix is processed successively by multivariate convolutional neural network (MVCNN) and long short-term memory (LSTM) to classify the emotional state. Spatiotemporal emotion recognition method is evaluated on the DEAP data set, and achieved accuracy of 87.58% and 88.50% on arousal and valence dimensions respectively in binary classification tasks, as well as obtained accuracy of 84.58% in four class classification tasks. The experimental results show that 3D matrix representation can represent emotional information more reasonably than two-dimensional (2D). In addition, MVCNN and LSTM can utilize the spatial information of the electrode channels and the temporal context information of the EEG signal respectively.

Key words: spatiotemporal emotion recognition model 3-dimensinal (3D) feature matrix| time-frequency features| multivariate convolutional neural network (MVCNN)| long short-term memory (LSTM)