中国邮电高校学报(英文) ›› 2022, Vol. 29 ›› Issue (5): 62-72.doi: 10.19682/j.cnki.1005-8885.2022.0017

• Artificial Intelligence • 上一篇    下一篇

Spatiotemporal emotion recognition based on 3D time-frequency domain feature matrix

晁浩1,连卫芳2,刘永利1   

  1. 1. 河南理工大学
    2. 河南理工大学计算机科学与技术学院
  • 收稿日期:2021-05-06 修回日期:2022-04-11 出版日期:2022-10-31 发布日期:2022-10-28
  • 通讯作者: 连卫芳 E-mail:1137844972@qq.com
  • 基金资助:
    国家自然科学科学基金;河南省高等学校重点科研计划项目;河南省高校基本科研业务费专项资金项目

Spatiotemporal emotion recognition based on 3D time-frequency domain feature matrix

Chao Hao1, Lian Weifang1,2, Liu Yongli1   

  • Received:2021-05-06 Revised:2022-04-11 Online:2022-10-31 Published:2022-10-28
  • Contact: Wei-Fang LIAN E-mail:1137844972@qq.com

摘要:

The research of emotion recognition based on electroencephalogram (EEG) signals often ignores the related information between the brain electrode channels and the contextual emotional information existing in EEG signals, which may contain important characteristics related to emotional states. Aiming at the above defects, a spatiotemporal emotion recognition method based on a 3-dimensional (3D) time-frequency domain feature matrix was proposed. Specifically, the extracted time-frequency domain EEG features are first expressed as a 3D matrix format according to the actual position of the cerebral cortex. Then, the input 3D matrix is processed successively by multivariate convolutional neural network (MVCNN) and long short-term memory (LSTM) to classify the emotional state. Spatiotemporal emotion recognition method is evaluated on the DEAP data set, and achieved accuracy of 87.58% and 88.50% on arousal and valence dimensions respectively in binary classification tasks, as well as obtained accuracy of 84.58% in four class classification tasks. The experimental results show that 3D matrix representation can represent emotional information more reasonably than two-dimensional (2D). In addition, MVCNN and LSTM can utilize the spatial information of the electrode channels and the temporal context information of the EEG signal respectively.


关键词: spatiotemporal emotion recognition model 3-dimensinal (3D) feature matrix| time-frequency features| multivariate convolutional neural network (MVCNN)| long short-term memory (LSTM)

Abstract:

The research of emotion recognition based on electroencephalogram (EEG) signals often ignores the related information between the brain electrode channels and the contextual emotional information existing in EEG signals, which may contain important characteristics related to emotional states. Aiming at the above defects, a spatiotemporal emotion recognition method based on a 3-dimensional (3D) time-frequency domain feature matrix was proposed. Specifically, the extracted time-frequency domain EEG features are first expressed as a 3D matrix format according to the actual position of the cerebral cortex. Then, the input 3D matrix is processed successively by multivariate convolutional neural network (MVCNN) and long short-term memory (LSTM) to classify the emotional state. Spatiotemporal emotion recognition method is evaluated on the DEAP data set, and achieved accuracy of 87.58% and 88.50% on arousal and valence dimensions respectively in binary classification tasks, as well as obtained accuracy of 84.58% in four class classification tasks. The experimental results show that 3D matrix representation can represent emotional information more reasonably than two-dimensional (2D). In addition, MVCNN and LSTM can utilize the spatial information of the electrode channels and the temporal context information of the EEG signal respectively.

Key words: spatiotemporal emotion recognition model 3-dimensinal (3D) feature matrix| time-frequency features| multivariate convolutional neural network (MVCNN)| long short-term memory (LSTM)