Fine-grained emotion prediction for movie and television scene images

doi:10.19682/j.cnki.1005-8885.2024.1007

中国邮电高校学报(英文) ›› 2024, Vol. 31 ›› Issue (3): 43-55.doi: 10.19682/j.cnki.1005-8885.2024.1007

• Artificial Intelligence • 上一篇下一篇

Fine-grained emotion prediction for movie and television scene images

苏志斌¹,周璇烨¹,刘冰²,任慧¹

1. State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China
2. Key Laboratory of Acoustic Visual Technology and Intelligent Control System, Communication University of China, Beijing 100024, China
3. School of Information and Communication Engineering, Communication University of China, Beijing 100024, China

收稿日期:2022-12-27 修回日期:2023-06-29 出版日期:2024-06-30 发布日期:2024-06-30
通讯作者: 任慧 E-mail:renhui@cuc.edu.cn
基金资助:
the Open Project of Key Laboratory of Audio and Video Restoration and Evaluation (2021KFKT005).

Fine-Grained Emotion Prediction for Movie and Television scene images

Su Zhibin, Zhou Xuanye, Liu Bing, Ren Hui

1. State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China
2. Key Laboratory of Acoustic Visual Technology and Intelligent Control System, Communication University of China, Beijing 100024, China
3. School of Information and Communication Engineering, Communication University of China, Beijing 100024, China

Received:2022-12-27 Revised:2023-06-29 Online:2024-06-30 Published:2024-06-30
Contact: Ren Hui E-mail:renhui@cuc.edu.cn
Supported by:
the Open Project of Key Laboratory of Audio and Video Restoration and Evaluation (2021KFKT005).

摘要/Abstract

摘要： For the task of content retrieval, analysis and generation of film and television scene images in the field of
intelligent editing, fine-grained emotion recognition and prediction of images is of great significance. In this paper,
the fusion of traditional perceptual features, art features and multi-channel deep learning features are used to reflect
the emotion expression of different levels of the image. In addition, the integrated learning model with stacking
architecture based on linear regression coefficient and sentiment correlations, which is called the LS-stacking
model, is proposed according to the factor association between multi-dimensional emotions. The experimental
results prove that the mixed feature and LS-stacking model can predict well on the 16 emotion categories of the self-
built image dataset. This study improves the fine-grained recognition ability of image emotion by computers, which
helps to increase the intelligence and automation degree of visual retrieval and post-production system.

关键词: fine-grained emotion prediction, movie and television scene images, stacking model, linear regression

Abstract: For the task of content retrieval, analysis and generation of film and television scene images in the field of
intelligent editing, fine-grained emotion recognition and prediction of images is of great significance. In this paper,
the fusion of traditional perceptual features, art features and multi-channel deep learning features are used to reflect
the emotion expression of different levels of the image. In addition, the integrated learning model with stacking
architecture based on linear regression coefficient and sentiment correlations, which is called the LS-stacking
model, is proposed according to the factor association between multi-dimensional emotions. The experimental
results prove that the mixed feature and LS-stacking model can predict well on the 16 emotion categories of the self-
built image dataset. This study improves the fine-grained recognition ability of image emotion by computers, which
helps to increase the intelligence and automation degree of visual retrieval and post-production system.

Key words: fine-grained emotion prediction, movie and television scene images, stacking model, linear regression

Su Zhibin, Zhou Xuanye, Liu Bing, Ren Hui. Fine-Grained Emotion Prediction for Movie and Television scene images[J]. The Journal of China Universities of Posts and Telecommunications, 2024, 31(3): 43-55.

参考文献

[1] FONG B, WESTERINK J. Affective computing in consumer electronics. IEEE Transactions on Affective Computing, 2012, 3(2): 129 -131.

[2] MELINTE D O, VLADAREANU L. Facial expressions recognition for human-robot interaction using deep convolutional neural networks with rectified adam optimizer. Sensors, 2020, 20(8): Article 2393.

[3] ROSA R L, RODRIGUEZ D Z, BRESSAN G. Music recommendation system based on user's sentiments extracted from social networks. IEEE Transactions on Consumer Electronics, 2015, 61(3): 359 -367.

[4] ALAMEDA-PINEDA X, RICCI E, YAN Y, et al. Recognizing emotions from abstract paintings using non-linear matrix completion. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'16), 2016, Jun 27 - 30, Las Vegas, NV, USA. Piscataway, NJ, USA: IEEE, 2016: 5240 -5248.

[5] MACHAJDIK J, HANBURY A. Affective image classification using features inspired by psychology and art theory. Proceedings of the 18th ACM international conference on Multimedia (MM'10), 2010, Oct 25 -29, Florence, Italy. New York, NY, USA: ACM, 2010: 83 -92.

[6] SHINDE S R, SABALE S, KULKARNI S, et al. Experiments on content based image classification using color feature extraction. Proceedings of the 2015 International Conference on Communication, Information & Computing Technology (ICCICT'15), 2015, Jan 15 - 17, Mumbai, India. Piscataway, NJ, USA: IEEE, 2015: 1 -6.

[7] YANULEVSKAYA V, VAN GEMERT J C, ROTH K, et al. Emotional valence categorization using holistic image features. Proceedings of the 15th IEEE International Conference on Image Processing, 2008, Oct 12 - 15, San Diego, CA, USA. Piscataway, NJ, USA: IEEE, 2008: 101 -104.

[8] ZHAO S C, GAO Y, JIANG X L, et al. Exploring principles-of-art features for image emotion recognition. Proceedings of the 22nd ACM International Conference on Multimedia (MM'14), 2014, Nov 3 - 7, Orlando, FL, USA. New York, NY, USA: ACM, 2014: 47 -56.

[9] BORTH D, JI R R, CHEN T, et al. Large-scale visual sentiment ontology and detectors using adjective noun pairs. Proceedings of the 21st ACM International Conference on Multimedia (MM'13), 2013, Oct 21 - 25, Barcelona, Spain. New York, NY, USA: ACM, 2013: 223 -232.

[10] ZHANG J, CHEN M, SUN H, et al. Object semantics sentiment correlation analysis enhanced image sentiment classification. Knowledge-Based Systems, 2020, 191(C): Article 105245.

[11] NAG P, SANGSKRITI S, JANNAT M E. A closer look into Paintings'style using convolutional neural network with transfer learning. Proceedings of the 11th International Joint Conference on computational Intelligence ( IJCCI'19), 2019, Sept 17 - 19, Vienna, Austria. Lisbon, Portugal: SciTePress ( Science and Technology Publications, LDA), 2020: 317 -328.

[12] KARNADE A M, DALVI P, KALBANDE D R. Emotion identification using CNN-based transfer learning. Advanced Computing Technologies and Applications: Proceedings of the 2nd International Conference on Advanced Computing Technologies and Applications (ICACTA´20), 2020, Feb 28 -29, Mumbai, India. Singapore: Springer, 2020: 337 -344.

[13] JAQUETTI P F, PILLA V J, BORBA G B, et al. VGG FACE Fine-tuning for classification of facial expression images of emotion. Proceedings of the XXVII Brazilian Congress in Biomedica Engineering (CBEB'20), 2020, Oct 26 - 30, VitÓria, Brazil. 2022: 1539 -1546.

[14] ZHANG B, QUAN C Q, REN F J. Study on CNN in the recognition of emotion in audio and images. Proceedings of the IEEE/ ACIS 15th International Conference on Computer and Information Science (ICIS'16), 2016, Jun 26 - 29, Okayama, Japan. Piscataway, NJ, USA: IEEE, 2016: 1 -5.

[15] RAO T R, XU M, XU D. Learning multi-level deep representations for image emotion classification. arXiv Preprint, arXiv: 1611. 07145, 2016.

[16] ZHAO S C, LIN C, XU P F, et al. CycleEmotionGAN: Emotional semantic consistency preserved cyclegan for adapting image emotions. Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI'19), 2019, Jan 27 - Feb 1, Honolulu, HI, USA. Menlo Park, CA, USA: AAAI, 2019: 2620 -2627.

[17] ZHANG W, HE X Y, LU W Z. Exploring discriminative representations for image emotion recognition with CNNs. IEEE Transactions on Multimedia, 2020, 22(2): 515 -523.

[18] YANG J F, SHE D Y, LAI Y K, et al. Retrieving and classifying affective images via deep metric learning. Proceedings of the 32nd AAAI Conference on Artificial Intelligence ( AAAI'18), 2018, Feb 2 - 7, New Orleans, LA, USA. Menlo Park, CA, USA: AAAI, 2018: 491 -498.

[19] ZHU X G, LI L, ZHANG W G, et al. Dependency exploitation: a unified CNN-RNN approach for visual emotion recognition. Proceedings of the 26th International Joint Conference on Artificial Intelligence ( IJCAI'17 ), 2017, Aug 19 - 25, Melbourne, Australia. San Francisco, CA, USA: Morgan Kaufmann, 2017: 3595 -3601.

[20] BARRIONUEVO C, IERACHE J, SATTOLO I. Emotion recognition through facial expressions using supervised learning with logistic regression. Computer Science: Proceedings of the 26th Argentine Congress ( CACIC'20), 2020, Oct 5 - 9, San Justo, Buenos Aires, Argentina. CCIS 1409. 2021: 233 -246.

[21] KUMAR C M, SHIVAKUMAR G. A thermal imaging based classification of affective states using multiclass SVM. Intelligent Systems Design and Applications: Proceedings of the 18th International Conference on Intelligent Systems Design and Applications (ISDA'18), 2018, Dec 6 -8, Vellore, India. AISC 940. 2020: 53 -63.

[22] PATIL S, PATIL Y M. Face expression recognition using SVM and KNN classifier with HOG features. Applied Computational Technologies: Proceedings of the 2022 International Conference on Computing in Engineering & Technology (ICCET'22), 2022, Feb 12 - 13, Lonere, India. SIST 303. Singapore: Springer, 2022: 416 -424.

[23] TIWARI K, PATEL M. Facial expression recognition using random forest classifier. Proceedings of the International Conference on Artificial Intelligence: Advances and Applications 2019 ( ICAIAA'19 ), 2019, Jun 28 - 29, Jaipur, India. Singapore: Springer, 2020: 121 -130.

[24] HUANG R L, DONG H B, YIN G S, et al. Ensembling 3D CNN framework for video recognition. Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN'19), 2019, Jul 14 - 19, Budapest, Hungary. Piscataway, NJ, USA: IEEE, 2019: 1 -7.

[25] VINAY A, SAMPAT P R, BELAVADI S V, et al. Face recognition using interest points and ensemble of classifiers. Proceedings of the 4th International Conference on Recent Advances in Information Technology (RAIT'18), 2018, Mar 15 -17, Dhanbad, India. Piscataway, NJ, USA: IEEE, 2018: 1 -8.

[26] OLUWAFEMI A G, WANG Z H. Multi-Class weather classification from still image using said ensemble method. Proceedings of the 2019 Southern African Universities Power Engineering Conference/ Robotics and Mechatronics/ Pattern Recognition Association of South Africa ( SAUPEC/ RobMech/PRASA'19), 2019, Jan 28 - 30, Bloemfontein, South Africa. Piscataway, NJ, USA: IEEE, 2019: 135 -140.

[27] YANG H, ZHANG Y S, DING W Z. Multiple heterogeneous P-DCNNs ensemble with stacking algorithm: A novel recognition method of space target ISAR images under the condition of small sample set. IEEE Access, 2020, 8: 75543 -75570.

[28] SU Z B, LIU B, ZHOU X Y, et al. Multidimensional sentiment recognition of film and television scene images. Journal of Electronic Imaging. 2021, 30(6): Article 063014.

[29] SU Z B, QIAN Y H, GU Y, et al. Research on emotion space of film and television scene images based on subjective perception. The Journal of China Universities of Posts and Telecommunications, 2019, 26(1): 75 -81.

[30] MACHAJDIK J, HANBURY A. Affective image classification using features inspired by psychology and art theory. Proceedings of the 18th ACM International Conference on Multimedia (MM'10), 2010, Oct 25 -29, Florence, Italy. New York, NY, USA: ACM, 2010: 83 -92.

[31] ACHANTA R, HEMAMI S, ESTRADA F, et al. Frequency tuned salient region detection. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'09), 2009, Jun 20 - 25, Miami, FL, USA. Piscataway, NJ, USA: IEEE, 2009: 1597 -1604.

[32] LIU Z, MAO H Z, WU C Y, et al. A convnet for the 2020s. Proceedings of the 2022 IEEE/ CVF Conference on Computer Vision and Pattern Recognition (CVPR'22), 2022, Jun 18 -24, New Orleans, LA, USA. Piscataway, NJ, USA: IEEE, 2022: 11976 -11986.

[33] TAN M X, LE Q V. Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the 36th International Conference on Machine Learning (ICML'19), 2019, Jun 9 -15, Long Beach, CA, USA. PMLR 97. Red Hook, NY, USA: Curran Associates, 2019: 6105 -6114.

[34] FEI X H, ZHANG Q, LING Q. Vehicle exhaust concentration estimation based on an improved stacking model. IEEE Access, 2019, 7: 179454 -179463.

[35] YADAV A, VISHWAKARMA D K. A deep learning architecture of RA-DLNet for visual sentiment analysis. Multimedia Systems, 2020, 26: 431 -451.

[36] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: Transformers for image recognition at scale. Proceedings of the 9th International Conference on Learning Representations (ICLR'21), 2021, May 3 -7, Vienna, Austria. 2021: 1 -21.

Fine-grained emotion prediction for movie and television scene images

Fine-Grained Emotion Prediction for Movie and Television scene images

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 0

编辑推荐

Metrics

本文评价