Facial expression recognition based on improved ResNet

doi:10.19682/j.cnki.1005-8885.2023.2003

中国邮电高校学报(英文) ›› 2023, Vol. 30 ›› Issue (1): 28-38.doi: 10.19682/j.cnki.1005-8885.2023.2003

• Artificial Intelligence • 上一篇下一篇

Facial expression recognition based on improved ResNet

Wang Xianlun, Wang Guangyu, Cui Yuxia

College of Electromechanical Engineering, Qingdao University of Science and Technology, Qingdao 266061, China

收稿日期:2021-08-10 修回日期:2022-06-30 接受日期:2023-02-13 出版日期:2023-02-28 发布日期:2023-02-28
通讯作者: Wang Guangyu, E-mail: 904887985@qq.com E-mail:904887985@qq.com
基金资助:
This work was supported by the National Natural Science Foundation of China (51105213).

Facial expression recognition based on improved ResNet

Wang Xianlun, Wang Guangyu, Cui Yuxia

College of Electromechanical Engineering, Qingdao University of Science and Technology, Qingdao 266061, China

Received:2021-08-10 Revised:2022-06-30 Accepted:2023-02-13 Online:2023-02-28 Published:2023-02-28
Contact: Wang Guangyu, E-mail: 904887985@qq.com E-mail:904887985@qq.com
Supported by:
This work was supported by the National Natural Science Foundation of China (51105213).

摘要/Abstract

摘要： Facial expression recognition (FER) is a vital application of image processing technology. In this paper, a FER model based on the residual network is proposed. The proposed model introduces the idea of the DenseNet, in which the outputs of the residual blocks are not simply added but are linked to the channel dimension. In addition, transfer learning is used to reduce training costs and accelerate training speed. The accuracy and robustness of the proposed FER model were tested by K-fold cross-validation. Experimental results show that the proposed method has competitive performances on FER2013, FER plus (FERPlus), and the real-world affective faces database (RAF-DB).

关键词: facial expression recognition, convolutional neural networks, ResNet, transfer learning, K-fold cross-validation

Abstract: Facial expression recognition (FER) is a vital application of image processing technology. In this paper, a FER model based on the residual network is proposed. The proposed model introduces the idea of the DenseNet, in which the outputs of the residual blocks are not simply added but are linked to the channel dimension. In addition, transfer learning is used to reduce training costs and accelerate training speed. The accuracy and robustness of the proposed FER model were tested by K-fold cross-validation. Experimental results show that the proposed method has competitive performances on FER2013, FER plus (FERPlus), and the real-world affective faces database (RAF-DB).

Key words: facial expression recognition, convolutional neural networks, ResNet, transfer learning, K-fold cross-validation

中图分类号:

TP391.4

Wang Xianlun, Wang Guangyu, Cui Yuxia. Facial expression recognition based on improved ResNet[J]. The Journal of China Universities of Posts and Telecommunications, 2023, 30(1): 28-38.

参考文献

References
[1] BRUCE V, YOUNG A. Chapter 4: Messages from facial movements. Face Perception. London, UK: Psychology Press, 2011: 1 -56.
[2] MARTIN C. The philosophy of deception. Oxford, UK: Oxford University Press, 2011.
[3] ZHAO G Y, PIETIKAINEN M. Dynamic texture recognition using local binary patterns with an application to facial expressions.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 915 -928.

[4] LIONG S T, WONG K S. Micro-expression recognition using apex frame with phase information. Proceedings of the 2017 Asia-Pacific Signal and Information Processing Association Summit and Conferencen (APSIPA ASC'17), 2017, Dec 12 - 15, Kuala Lumpur, Malaysia. Piscataway, NJ, USA: IEEE, 2017: 534 -537.
[5] WANG Y, SEE J, PHAN R C W, et al. LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recognition. Computer Vision-ACCV 2014: Proceedings of the 12th Asian Conference on Computer Vision (ACCV'14),
2014, Nov 1 - 5, Singapore. LNIP 9003. Berlin, Germany: Springer, 2015: 525 -537.
[6] HUANG X H, WANG S J, ZHAO G Y, et al. Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. Proceedings of the 2015 IEEE International Conference on Computer Vision Workshop (ICCVW'15), 2015, Dec 7 - 13, Santiago, Chile. Piscataway, NJ, USA: IEEE, 2015: 1 -9.
[7] HUANG X H, ZHAO G Y, HONG X P, et al. Spontaneous facial micro-expression analysis using spatiotemporal completed local
quantized patterns. Neurocomputing, 2016, 175(Part A): 564 -578.
[8] LI Y T, HUANG X H, ZHAO G Y. Can micro-expression be recognized based on single apex frame? Proceedings of the 25th
IEEE International Conference on Image Processing (ICIP'18), 2018, Oct 7 - 10, Athens, Greece. Piscataway, NJ, USA: IEEE, 2018: 3094 -3098.
[9] GAN Y S, LIONG S T, YAU W C, et al. OFF-ApexNet on micro-expression recognition system. Signal Processing: Image Communication, 2019, 74: 129 -13.
[10] LIONG S T, GAN Y S, SEE J, et al. Shallow triple stream three-dimensional CNN ( STSTNet ) for micro-expression recognition system. Proceedings of the 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG'19), 2019, May 14 - 18, Lille, France. Piscataway, NJ, USA: IEEE, 2019: 1 -5.
[11] WANG M, DENG W H. Deep visual domain adaptation: A survey. Neurocomputing, 2018, 312: 135 -153.
[12] KAHOU S E, BOUTHILLIER X, LAMBLIN P, et al. EmoNets: Multimodal deep learning approaches for emotion recognition in
video. Journal on Multimodal User Interfaces, 2016, 10: 99 -111.
[13] LIU A B, YANG Y Q, SUN Q Y, et al. A deep fully convolution neural network for semantic segmentation based on adaptive
feature fusion. Proceedings of the 5th International Conference on Information Science and Control Engineering ( ICISCE'18 ),
2018, Jul 20 - 22, Zhengzhou, China. Piscataway, NJ, USA: IEEE, 2018: 1 -5.
[14] LIU P, HAN S Z, MENG Z B, et al. Facial expression recognition via a boosted deep belief network. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, Jun 23 - 28, Columbus, OH, USA. Piscataway, NJ, USA: IEEE, 2014: 1805 -1812.
[15] KIM Y, LEE H, PROVOST E M. Deep learning for robust feature generation in audiovisual emotion recognition. Proceedings
of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing ( ICASSP'13 ), 2013, May 26 - 31, Vancouver, Canada. Piscataway, NJ, USA: IEEE, 2013: 3687 -3691.
[16] PATEL D, HONG X P, ZHAO G Y. Selective deep features for micro-expression recognition. Proceedings of the 23rd International Conference on Pattern Recognition ( ICPR'16 ), 2016, Dec 4 - 8, Cancun, Mexico. Piscataway, NJ, USA: IEEE, 2016: 2259 -2264.
[17] KIM D H, BADDAR W J, JANG J, et al. Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Transactions on Affective Computing, 2019, 10 ( 2 ):
223 -236.
[18] SHAO J, QIAN Y S. Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing,
2019, 355: 82 -92.
[19] SANG D V, DAT N V, THUAN D P. Facial expression recognition using deep convolutional neural networks. Proceedings of the 9th International Conference on Knowledge and Systems Engineering (KSE'17), 2017, Oct 19 - 21, Hue, Vietnam. Piscataway, NJ, USA: IEEE, 2017: 130 -135.
[20] LI S, DENG W D. Deep facial expression recognition: A survey. IEEE Transactions on Affective Computing, 2022, 13(3): 1195 -
1215.
[21] PHAM L, VU T H, TRAN T A. Facial expression recognition using residual masking network. Proceedings of the 25th International Conference on Pattern Recognition ( ICPR'20 ), 2021, Jan 10 -15, Milan, Italy. Piscataway, NJ, USA: IEEE, 2021: 4513 -4519.
[22] CUI R K, PLESTED J, LIU J X. Declarative residual network for robust facial expression recognition. Neural Information
Processing: Proceedings of the 27th International Conference on Neural Information Processing (ICONIP'20), 2020, Nov 18 -
22, Bangkok, Thailand. CCIS 1332. Berlin, Germany: Springer, 2020: 345 -352.
[23] YASMIN S, PATHAN R K, BISWAS M, et al. Development of a robust multi-scale featured local binary pattern for improved
facial expression recognition. Sensors, 2020, 20(18): Article 5391.
[24] ZAFEIRIOU S, ZHANG C, ZHANG Z Y, et al. A survey on face detection in the wild: Past, present and future. Computer
Vision and Image Understanding, 2015, 138: 1 -24.
[25] XU Y Z, YU G Z, WANG Y P, et al. A hybrid vehicle detection method based on Viola-Jones and HOG + SVM from UAV
images. Sensors, 2016, 16(8): Article 1325.
[26] KAMAROL S K A, JAWARD M H, PARKKINEN J, et al. Spatiotemporal feature extraction for facial expression recognition.
IET Image Processing, 2016, 10(7): 534 -541.
[27] LIDONG H, WEI Z, JUN W, et al. Combination of contrast limited adaptive histogram equalisation and discrete wavelet
transform for image enhancement. IET Image Processing, 2015, 9(10): 908 -915.
[28] ZHOU M, JIN K, WANG S Z, et al. Color retinal image enhancement based on luminosity and contrast adjustment. IEEE
Transactions on on Biomedical Engineering, 2018: 65 ( 3 ): 521 -527.
[29] MIZUTANI Y, KURIKI H, KODAMA Y, et al. Enhanced universal filtered-DFTs-OFDM for long-delay multipath environment. IEICE Transactions on Communications, 2019, E103 - B(4): 467 -475.
[30] ZHANG N, QIN Q, YUAN H, et al. NTTU: An area-efficient low-power NTT-uncoupled architecture for NTT-based multiplication. IEEE Transactions on Computers, 2020, 69(4): 520 -533.
[31] RANJAN R, PATEL V M, CHELLAPPA R. HyperFace: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(1): 121 -135.
[32] FUENTES A, YOON S, KIM S C, et al. A robust deep-learning-based detector for real-time tomato plant diseases and pests recognition. Sensors, 2017, 17(9): Article 2022.
[33] HAN S S, KIM M S, LIM W, et al. Classification of the clinical images for benign and malignant cutaneous tumors using a deep
learning algorithm. Journal of Investigative Dermatology, 2018, 138(7): 1529 -1538.
[34] WU Z F, SHEN C H, VAN DEN HENGEL A. Wider or deeper: Revisiting the ResNet model for visual recognition. Pattern Recognition, 2019, 90: 119 -133.
[35] ZHANG K P, ZHANG Z P, LI Z F, et al. Joint face detection and alignment using multitask cascaded convolutional networks.
IEEE Signal Processing Letters, 2016, 23(10): 1499 -1503.
[36] YANG J C, WANG X J, HAN S J, et al. Improved real-time facial expression recognition based on a novel balanced and
symmetric local gradient coding. Sensors, 2019, 19(8): Article 1899.
[37] LI H F, LI Q. End-to-end training for compound expression recognition. Sensors, 2020, 20(17): Article 4727.
[38] VO T H, LEE G S, YANG H J, et al. Pyramid with super resolution for in-the-wild facial expression recognition. IEEE Access, 2020, 8: 131988 -132001.
[39] LIAN Z, LI Y, TAO J H, et al. Expression analysis based on face regions in real-world conditions. International Journal of
Automation and Computing, 2020, 17(1): 96 -107.
[40] PRAMERDORFER C, KAMPEL M. Facial expression recognition using convolutional neural networks: State of the art. arXiv Preprint, arXiv:1612. 02903, 2016.
[41] TANG Y C. Deep learning using linear support vector machines. Proceedings of the 30th International Conference on Machine
Learning (ICML'13), 2013, Jun 16 - 21, Atlanta, GA, USA. Cambridge, MA, USA: JMLR (Journal of Machine Learning Research). org, 2013: 1 -5.
[42] KIM B K, DONG S Y, ROH J, et al. Fusing aligned and non-aligned face information for automatic affect recognition in the wild: A deep learning approach. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW'16), 2016, Jun 26-Jul 1, Las Vegas, NV, USA. Piscataway, NJ, USA: IEEE, 2016: 48 -57.
[43] MINAEE S, MINAEI M, ABDOLRASHIDI A. Deep-emotion: Facial expression recognition using attentional convolutional network. Sensors, 2021, 21(9): Article 3046.
[44] HUA W T, DAI F, HUANG L Y, et al. HERO: Human emotions recognition for realizing intelligent Internet of things. IEEE Access, 2019, 7: 24321 -24332.
[45] CONNIE T, AL-SHABI M, CHEAH W P, et al. Facial expression recognition using a hybrid CNN-SIFT aggregator. Multi-disciplinary Trends in Artificial Intelligence: Proceedings of the 11th International Workshop on Multi-disciplinary Trends in
Artificial Intelligence ( MIWAI'17 ), 2017, Nov 20 - 22, Gadong, Brunei. LNAI 10607. Berlin, Germany: Springer, 2017: 139 -149.
[46] BARSOUM E, ZHANG C, FERRER C C, et al. Training deep networks for facial expression recognition with crowd-sourced
label distribution. Proceedings of the 18th ACM International Conference on Multimodal Interaction (ICMI'16), 2016, Nov
12 -16, Tokyo, Japan. New York, NY, USA: ACM, 2016: 279 -283.
[47] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition. arXiv Preprint, arXiv:1409. 1556, 2014.
[48] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference
on Computer Vision and Pattern Recognition (CVPR'16), 2016, Jun 27 - 30, Las Vegas, NV, USA. Piscataway, NJ, USA: IEEE, 2016: 770 -778.
[49] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition ( CVPR'17 ), 2017, Jul 21 - 26, Honolulu, HI, USA. Piscataway, NJ, USA: IEEE, 2017: 4700 -4708.
[50] BURKERT P, TRIER F, AFZAL M Z, et al. DeXpression: Deep convolutional neural network for expression recognition. arXiv Preprint, arXiv. 1509. 05371, 2016.
[51] LI S, DENG W H. Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Transactions on Image Processing, 2019, 28(1): 356 -370.
[52] FAN Y R, LAM J C K, LI V O K. Multi-region ensemble convolutional neural network for facial expression recognition. Artificial Neural Networks and Machine Learning: Proceedings of the 27th International Conference on Artificial Neural Networks (ICANN'18): Part I, 2018, Oct 4 - 7, Rhodes, Greece, LNTCS 11139. Berlin, Germany: Springer, 2018: 84 -94.

Facial expression recognition based on improved ResNet

Facial expression recognition based on improved ResNet

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 12

编辑推荐

Metrics

本文评价

[1]	Kong Chao, Ou Weihua, Gong Xiaofeng, Li Weian, Han Jie, Yao Yi, Xiong Jiahao. Face anti-spoofing based on multi-modal and multi-scale features fusion [J]. 中国邮电高校学报(英文版), 2022, 29(6): 73-82.
[2]	Jia Wei, Gong Chao. Precise and efficient Chinese license plate recognition in the real monitoring scene of intelligent transportation system[J]. 中国邮电高校学报(英文版), 2022, 29(3): 1-14.
[3]	Song Yue, Wu Chengmao, Tian Xiaoping, Song Qiuyu. Enhanced kernel-based fuzzy local information clustering integrating neighborhood membership [J]. 中国邮电高校学报(英文版), 2021, 28(6): 65-81.
[4]	Xue Chenzi, Wei Yifei, Zhang Yong. Performance optimization for smart grid blockchain integrated with fog computing using DDQN[J]. 中国邮电高校学报(英文版), 2021, 28(2): 68-78.
[5]	汪昭颖, 周军华, 廖中华, 翟翔, 张连平. Semantic segmentation of track image based on deep neural network[J]. 中国邮电高校学报(英文版), 2020, 27(5): 23-33.
[6]	陈法权樊军. Real-time prediction of the motion tendency of human lower limbs during gait [J]. 中国邮电高校学报(英文版), 2020, 27(4): 1-7.
[7]	李端张洪欣 Muhammad Saad Khan 米芳. Recognition of motor imagery tasks for BCI using CSP and chaotic PSO twin SVM[J]. 中国邮电高校学报(英文版), 2017, 24(3): 83-90.
[8]	张捷范旭慧班登科. Smooth support vector machine based on circular tangent function[J]. 中国邮电高校学报(英文版), 2016, 23(1): 68-72.
[9]	王亮刘贵喜段红岩. Dynamic and combined gestures recognition based on multi-feature fusion in a complex environment [J]. Acta Metallurgica Sinica(English letters), 2015, 22(2): 81-88.
[10]	李岩周亚建袁开国郭玉翠钮心忻. Exposing photo manipulation with inconsistent perspective geometry[J]. Acta Metallurgica Sinica(English letters), 2014, 21(4): 83-91.
[11]	刘敬徐国胜郑世慧肖达谷利泽. Data streams classification with ensemble model based on decision-feedback[J]. Acta Metallurgica Sinica(English letters), 2014, 21(1): 79-85.
[12]	卢官明左加阔. Orthogonal isometric projection for face recognition[J]. Acta Metallurgica Sinica(English letters), 2011, 18(1): 91-97.