Fine-grained emotion prediction for movie and television scene images

doi:10.19682/j.cnki.1005-8885.2024.1007

中国邮电高校学报(英文) ›› 2024, Vol. 31 ›› Issue (3): 43-55.doi: 10.19682/j.cnki.1005-8885.2024.1007

• Artificial Intelligence • 上一篇下一篇

Fine-grained emotion prediction for movie and television scene images

苏志斌¹,周璇烨¹,刘冰²,任慧¹

1. State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China
2. Key Laboratory of Acoustic Visual Technology and Intelligent Control System, Communication University of China, Beijing 100024, China
3. School of Information and Communication Engineering, Communication University of China, Beijing 100024, China

收稿日期:2022-12-27 修回日期:2023-06-29 出版日期:2024-06-30 发布日期:2024-06-30
通讯作者: 任慧 E-mail:renhui@cuc.edu.cn
基金资助:
the Open Project of Key Laboratory of Audio and Video Restoration and Evaluation (2021KFKT005).

Fine-Grained Emotion Prediction for Movie and Television scene images

Su Zhibin, Zhou Xuanye, Liu Bing, Ren Hui

1. State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China
2. Key Laboratory of Acoustic Visual Technology and Intelligent Control System, Communication University of China, Beijing 100024, China
3. School of Information and Communication Engineering, Communication University of China, Beijing 100024, China

Received:2022-12-27 Revised:2023-06-29 Online:2024-06-30 Published:2024-06-30
Contact: Ren Hui E-mail:renhui@cuc.edu.cn
Supported by:
the Open Project of Key Laboratory of Audio and Video Restoration and Evaluation (2021KFKT005).

摘要/Abstract

摘要： For the task of content retrieval, analysis and generation of film and television scene images in the field of
intelligent editing, fine-grained emotion recognition and prediction of images is of great significance. In this paper,
the fusion of traditional perceptual features, art features and multi-channel deep learning features are used to reflect
the emotion expression of different levels of the image. In addition, the integrated learning model with stacking
architecture based on linear regression coefficient and sentiment correlations, which is called the LS-stacking
model, is proposed according to the factor association between multi-dimensional emotions. The experimental
results prove that the mixed feature and LS-stacking model can predict well on the 16 emotion categories of the self-
built image dataset. This study improves the fine-grained recognition ability of image emotion by computers, which
helps to increase the intelligence and automation degree of visual retrieval and post-production system.

关键词: fine-grained emotion prediction, movie and television scene images, stacking model, linear regression

Abstract: For the task of content retrieval, analysis and generation of film and television scene images in the field of
intelligent editing, fine-grained emotion recognition and prediction of images is of great significance. In this paper,
the fusion of traditional perceptual features, art features and multi-channel deep learning features are used to reflect
the emotion expression of different levels of the image. In addition, the integrated learning model with stacking
architecture based on linear regression coefficient and sentiment correlations, which is called the LS-stacking
model, is proposed according to the factor association between multi-dimensional emotions. The experimental
results prove that the mixed feature and LS-stacking model can predict well on the 16 emotion categories of the self-
built image dataset. This study improves the fine-grained recognition ability of image emotion by computers, which
helps to increase the intelligence and automation degree of visual retrieval and post-production system.

Key words: fine-grained emotion prediction, movie and television scene images, stacking model, linear regression

Su Zhibin, Zhou Xuanye, Liu Bing, Ren Hui. Fine-Grained Emotion Prediction for Movie and Television scene images[J]. The Journal of China Universities of Posts and Telecommunications, 2024, 31(3): 43-55.

参考文献

[1] FONG B, WESTERINK J. Affective computing in consumer

electronics. IEEE Transactions on Affective Computing, 2012,

3(2): 129 -131.

[2] MELINTE D O, VLADAREANU L. Facial expressions recognition

for human-robot interaction using deep convolutional neural

networks with rectified adam optimizer. Sensors, 2020, 20(8):

Article 2393.

[3] ROSA R L, RODRIGUEZ D Z, BRESSAN G. Music

recommendation system based on user's sentiments extracted from

social networks. IEEE Transactions on Consumer Electronics,

2015, 61(3): 359 -367.

[4] ALAMEDA-PINEDA X, RICCI E, YAN Y, et al. Recognizing

emotions from abstract paintings using non-linear matrix

completion. Proceedings of the 2016 IEEE Conference on

Computer Vision and Pattern Recognition (CVPR'16), 2016, Jun

27 - 30, Las Vegas, NV, USA. Piscataway, NJ, USA: IEEE,

2016: 5240 -5248.

[5] MACHAJDIK J, HANBURY A. Affective image classification

using features inspired by psychology and art theory. Proceedings

of the 18th ACM international conference on Multimedia

(MM'10), 2010, Oct 25 -29, Florence, Italy. New York, NY,

USA: ACM, 2010: 83 -92.

[6] SHINDE S R, SABALE S, KULKARNI S, et al. Experiments on

content based image classification using color feature extraction.

Proceedings of the 2015 International Conference on

Communication, Information & Computing Technology

(ICCICT'15), 2015, Jan 15 - 17, Mumbai, India. Piscataway,

NJ, USA: IEEE, 2015: 1 -6.

[7] YANULEVSKAYA V, VAN GEMERT J C, ROTH K, et al.

Emotional valence categorization using holistic image features.

Proceedings of the 15th IEEE International Conference on Image

Processing, 2008, Oct 12 - 15, San Diego, CA, USA.

Piscataway, NJ, USA: IEEE, 2008: 101 -104.

[8] ZHAO S C, GAO Y, JIANG X L, et al. Exploring principles-of-

art features for image emotion recognition. Proceedings of the 22nd

ACM International Conference on Multimedia (MM'14), 2014,

Nov 3 - 7, Orlando, FL, USA. New York, NY, USA: ACM,

2014: 47 -56.

[9] BORTH D, JI R R, CHEN T, et al. Large-scale visual sentiment

ontology and detectors using adjective noun pairs. Proceedings of

the 21st ACM International Conference on Multimedia (MM'13),

2013, Oct 21 - 25, Barcelona, Spain. New York, NY, USA:

ACM, 2013: 223 -232.

[10] ZHANG J, CHEN M, SUN H, et al. Object semantics sentiment

correlation analysis enhanced image sentiment classification.

Knowledge-Based Systems, 2020, 191(C): Article 105245.

[11] NAG P, SANGSKRITI S, JANNAT M E. A closer look into

Paintings'style using convolutional neural network with transfer

learning. Proceedings of the 11th International Joint Conference on

Computational Intelligence ( IJCCI'19), 2019, Sept 17 - 19,

Vienna, Austria. Lisbon, Portugal: SciTePress ( Science and

Technology Publications, LDA), 2020: 317 -328.

[12] KARNADE A M, DALVI P, KALBANDE D R. Emotion

identification using CNN-based transfer learning. Advanced

Computing Technologies and Applications: Proceedings of the 2nd

International Conference on Advanced Computing Technologies and

Applications (ICACTA´20), 2020, Feb 28 -29, Mumbai, India.

Singapore: Springer, 2020: 337 -344.

[13] JAQUETTI P F, PILLA V J, BORBA G B, et al. VGG FACE

Fine-tuning for classification of facial expression images of emotion.

Proceedings of the XXVII Brazilian Congress in Biomedical

Engineering (CBEB'20), 2020, Oct 26 - 30, VitÓria, Brazil.

2022: 1539 -1546.

[14] ZHANG B, QUAN C Q, REN F J. Study on CNN in the

recognition of emotion in audio and images. Proceedings of the

IEEE/ ACIS 15th International Conference on Computer and

Information Science (ICIS'16), 2016, Jun 26 - 29, Okayama,

Japan. Piscataway, NJ, USA: IEEE, 2016: 1 -5.

[15] RAO T R, XU M, XU D. Learning multi-level deep

representations for image emotion classification. arXiv Preprint,

arXiv: 1611. 07145, 2016.

[16] ZHAO S C, LIN C, XU P F, et al. CycleEmotionGAN: Emotional

semantic consistency preserved cyclegan for adapting image

emotions. Proceedings of the 33rd AAAI Conference on Artificial

Intelligence (AAAI'19), 2019, Jan 27 - Feb 1, Honolulu, HI,

USA. Menlo Park, CA, USA: AAAI, 2019: 2620 -2627.

[17] ZHANG W, HE X Y, LU W Z. Exploring discriminative

representations for image emotion recognition with CNNs. IEEE

Transactions on Multimedia, 2020, 22(2): 515 -523.

[18] YANG J F, SHE D Y, LAI Y K, et al. Retrieving and classifying

affective images via deep metric learning. Proceedings of the 32nd

AAAI Conference on Artificial Intelligence ( AAAI'18), 2018,

Feb 2 - 7, New Orleans, LA, USA. Menlo Park, CA, USA:

AAAI, 2018: 491 -498.

[19] ZHU X G, LI L, ZHANG W G, et al. Dependency exploitation: a

unified CNN-RNN approach for visual emotion recognition.

Proceedings of the 26th International Joint Conference on Artificial

Intelligence ( IJCAI'17 ), 2017, Aug 19 - 25, Melbourne,

Australia. San Francisco, CA, USA: Morgan Kaufmann, 2017:

3595 -3601.

[20] BARRIONUEVO C, IERACHE J, SATTOLO I. Emotion

recognition through facial expressions using supervised learning

with logistic regression. Computer Science: Proceedings of the

26th Argentine Congress ( CACIC'20), 2020, Oct 5 - 9, San

Justo, Buenos Aires, Argentina. CCIS 1409. 2021: 233 -246.

[21] KUMAR C M, SHIVAKUMAR G. A thermal imaging based

classification of affective states using multiclass SVM. Intelligent

Systems Design and Applications: Proceedings of the 18th

International Conference on Intelligent Systems Design and

Applications (ISDA'18), 2018, Dec 6 -8, Vellore, India. AISC

940. 2020: 53 -63.

[22] PATIL S, PATIL Y M. Face expression recognition using SVM and

KNN classifier with HOG features. Applied Computational

Technologies: Proceedings of the 2022 International Conference on

Computing in Engineering & Technology (ICCET'22), 2022, Feb

12 - 13, Lonere, India. SIST 303. Singapore: Springer, 2022:

416 -424.

[23] TIWARI K, PATEL M. Facial expression recognition using

random forest classifier. Proceedings of the International

Conference on Artificial Intelligence: Advances and Applications

2019 ( ICAIAA'19 ), 2019, Jun 28 - 29, Jaipur, India.

Singapore: Springer, 2020: 121 -130.

[24] HUANG R L, DONG H B, YIN G S, et al. Ensembling 3D CNN

framework for video recognition. Proceedings of the 2019

International Joint Conference on Neural Networks (IJCNN'19),

2019, Jul 14 - 19, Budapest, Hungary. Piscataway, NJ, USA:

IEEE, 2019: 1 -7.

[25] VINAY A, SAMPAT P R, BELAVADI S V, et al. Face

recognition using interest points and ensemble of classifiers.

Proceedings of the 4th International Conference on Recent

Advances in Information Technology (RAIT'18), 2018, Mar 15 -

17, Dhanbad, India. Piscataway, NJ, USA: IEEE, 2018: 1 -8.

[26] OLUWAFEMI A G, WANG Z H. Multi-Class weather

classification from still image using said ensemble method.

Proceedings of the 2019 Southern African Universities Power

Engineering Conference/ Robotics and Mechatronics/ Pattern

Recognition Association of South Africa ( SAUPEC/ RobMech/

PRASA'19), 2019, Jan 28 - 30, Bloemfontein, South Africa.

Piscataway, NJ, USA: IEEE, 2019: 135 -140.

[27] YANG H, ZHANG Y S, DING W Z. Multiple heterogeneous P-

DCNNs ensemble with stacking algorithm: A novel recognition

method of space target ISAR images under the condition of small

sample set. IEEE Access, 2020, 8: 75543 -75570.

[28] SU Z B, LIU B, ZHOU X Y, et al. Multidimensional sentiment

recognition of film and television scene images. Journal of

Electronic Imaging. 2021, 30(6): Article 063014.

[29] SU Z B, QIAN Y H, GU Y, et al. Research on emotion space of

film and television scene images based on subjective perception.

The Journal of China Universities of Posts and

Telecommunications, 2019, 26(1): 75 -81.

[30] MACHAJDIK J, HANBURY A. Affective image classification

using features inspired by psychology and art theory. Proceedings

of the 18th ACM International Conference on Multimedia

(MM'10), 2010, Oct 25 -29, Florence, Italy. New York, NY,

USA: ACM, 2010: 83 -92.

[31] ACHANTA R, HEMAMI S, ESTRADA F, et al. Frequency tuned

salient region detection. Proceedings of the 2009 IEEE Conference

on Computer Vision and Pattern Recognition (CVPR'09), 2009,

Jun 20 - 25, Miami, FL, USA. Piscataway, NJ, USA: IEEE,

2009: 1597 -1604.

[32] LIU Z, MAO H Z, WU C Y, et al. A convnet for the 2020s.

Proceedings of the 2022 IEEE/ CVF Conference on Computer

Vision and Pattern Recognition (CVPR'22), 2022, Jun 18 -24,

New Orleans, LA, USA. Piscataway, NJ, USA: IEEE, 2022:

11976 -11986.

[33] TAN M X, LE Q V. Efficientnet: Rethinking model scaling for

convolutional neural networks. Proceedings of the 36th

International Conference on Machine Learning (ICML'19), 2019,

Jun 9 -15, Long Beach, CA, USA. PMLR 97. Red Hook, NY,

USA: Curran Associates, 2019: 6105 -6114.

[34] FEI X H, ZHANG Q, LING Q. Vehicle exhaust concentration

estimation based on an improved stacking model. IEEE Access,

2019, 7: 179454 -179463.

[35] YADAV A, VISHWAKARMA D K. A deep learning architecture

of RA-DLNet for visual sentiment analysis. Multimedia Systems,

2020, 26: 431 -451.

[36] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image

is worth 16x16 words: Transformers for image recognition at scale.

Proceedings of the 9th International Conference on Learning

Representations (ICLR'21), 2021, May 3 -7, Vienna, Austria.

2021: 1 -21.

Fine-grained emotion prediction for movie and television scene images

Fine-Grained Emotion Prediction for Movie and Television scene images

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 0

编辑推荐

Metrics

本文评价