Face anti-spoofing based on multi-modal and multi-scale features fusion

doi:10.19682/j.cnki.1005-8885.2022.1004

中国邮电高校学报(英文) ›› 2022, Vol. 29 ›› Issue (6): 73-82.doi: 10.19682/j.cnki.1005-8885.2022.1004

Face anti-spoofing based on multi-modal and multi-scale features fusion

Kong Chao, Ou Weihua, Gong Xiaofeng, Li Weian, Han Jie, Yao Yi, Xiong Jiahao

1. School of Big Data and Computer Science, Guizhou Normal University, Guiyang 550025, China 2. Special Key Laboratory of Artificial Intelligence and Intelligent Control of Guizhou Province, Guizhou Institute of Technology, Guiyang 550003, China 3. Guizhou Science and Technology Information Center, Department of Science and Technology of Guizhou Province, Guiyang 550002, China

收稿日期:2021-02-19 修回日期:2021-07-10 出版日期:2022-12-30 发布日期:2022-12-30
通讯作者: Ou Weihua E-mail:1305532268@qq.com
基金资助:
This work was supported by the National Natural Science Foundation of China (61962010, 62262005), and the Natural

Science Foundation of Guizhou Priovince ( QianKeHeJichu [2019]1425).

Face anti-spoofing based on multi-modal and multi-scale features fusion

Kong Chao, Ou Weihua, Gong Xiaofeng, Li Weian, Han Jie, Yao Yi, Xiong Jiahao

1. School of Big Data and Computer Science, Guizhou Normal University, Guiyang 550025, China 2. Special Key Laboratory of Artificial Intelligence and Intelligent Control of Guizhou Province, Guizhou Institute of Technology, Guiyang 550003, China 3. Guizhou Science and Technology Information Center, Department of Science and Technology of Guizhou Province, Guiyang 550002, China

Received:2021-02-19 Revised:2021-07-10 Online:2022-12-30 Published:2022-12-30
Contact: Ou Weihua E-mail:1305532268@qq.com
Supported by:
This work was supported by the National Natural Science Foundation of China (61962010, 62262005), and the Natural

Science Foundation of Guizhou Priovince ( QianKeHeJichu [2019]1425).

摘要/Abstract

摘要：

Face anti-spoofing is used to assist face recognition system to judge whether the detected face is real face or fake face. In the traditional face anti-spoofing methods, features extracted by hand are used to describe the difference between living face and fraudulent face. But these handmade features do not apply to different variations in an unconstrained environment. The convolutional neural network (CNN) for face deceptions achieves considerable results. However, most existing neural network-based methods simply use neural networks to extract single-scale features from single-modal data, while ignoring multi-scale and multi-modal information. To address this problem, a novel face anti-spoofing method based on multi-modal and multi-scale features fusion ( MMFF) is proposed. Specifically, first residual network ( Resnet )-34 is adopted to extract features of different scales from each modality, then these features of different scales are fused by feature pyramid network (FPN), finally squeeze-and-excitation fusion ( SEF) module and self-attention network ( SAN) are combined to fuse features from different modalities for classification. Experiments on the CASIA-SURF dataset show that the new method based on MMFF achieves better performance compared with most existing methods.

关键词:

face anti-spoofing, multi-modal fusion, multi-scale fusion, self-attention network (SAN), feature pyramid network (FPN)

Abstract: Face anti-spoofing is used to assist face recognition system to judge whether the detected face is real face or fake face. In the traditional face anti-spoofing methods, features extracted by hand are used to describe the difference between living face and fraudulent face. But these handmade features do not apply to different variations in an unconstrained environment. The convolutional neural network (CNN) for face deceptions achieves considerable results. However, most existing neural network-based methods simply use neural networks to extract single-scale features from single-modal data, while ignoring multi-scale and multi-modal information. To address this problem, a novel face anti-spoofing method based on multi-modal and multi-scale features fusion ( MMFF) is proposed. Specifically, first residual network ( Resnet )-34 is adopted to extract features of different scales from each modality, then these features of different scales are fused by feature pyramid network (FPN), finally squeeze-and-excitation fusion ( SEF) module and self-attention network ( SAN) are combined to fuse features from different modalities for classification. Experiments on the CASIA-SURF dataset show that the new method based on MMFF achieves better performance compared with most existing methods.

Key words: face anti-spoofing, multi-modal fusion, multi-scale fusion, self-attention network (SAN), feature pyramid network (FPN)

中图分类号:

TP391.4

Kong Chao, Ou Weihua, Gong Xiaofeng, Li Weian, Han Jie, Yao Yi, Xiong Jiahao. Face anti-spoofing based on multi-modal and multi-scale features fusion[J]. The Journal of China Universities of Posts and Telecommunications, 2022, 29(6): 73-82.

参考文献

1. ZHANG S F, ZHU X Y, LEI Z, et al. Detecting face with densely connected face proposal network. Neurocomputing, 2018, 284: 119-127

2. ZHANG S F, WANG X B, LEI Z, et al. Faceboxes: A CPU real-time and accurate unconstrained face detector. Neurocomputing, 2019, 364: 297-309

3. WANG X B, ZHANG S F, WANG S, et al. Mis-classified vector guided softmax loss for face recognition. Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI’20), 2020, Feb 7-12, Honolulu, HI, USA. Menlo Park, CA, USA: American Association for Artificial Intelligence, 2020: 12241-12248

4. ZHANG S F, WEN L Y, SHI H L, et al. Single-shot scale-aware network for real-time face detection. International Journal of Computer Vision, 2019, 127(6): 537-559

5. CHI C, ZHANG S, XING J, et al. Selective refinement network for high performance face detection. Proceedings of the 33th AAAI Conference on Artificial Intelligence (AAAI’19), 2019, Jan 27-Feb 1, New York, NY, USA. Menlo Park, CA, USA: American Association for Artificial Intelligence, 2019: 8231-8238

6. YANG X, LUO W H, BAO L C, et al. Face anti-spoofing: Model matters, so does data. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'19), 2019, Jun 15-20, Long Beach, CA, USA. Piscataway, NJ, USA: IEEE, 2019: 3507-3516

7. ALOTAIBI A, MAHMOOD A. Deep face liveness detection based on nonlinear diffusion using convolution neural network. Signal, Image and Video Processing, 2017, 11(4): 713-720

8. PINTO A, SCHWARTZ W R, PEDRINI H, et al. Using visual rhythms for detecting video-based facial spoof attacks. IEEE Transactions on Information Forensics and Security, 2015, 10(5): 1025-1038

9. BOULKENAFET Z, KOMULAINEN J, HADID A. Face antispoofing using speeded-up robust features and fisher vector encoding. IEEE Signal Processing Letters, 2016, 24(2): 141-145

10. CHINGOVSKA I, ANJOS A, MARCEL S. On the effectiveness of local binary patterns in face anti-spoofing. Proceedings of the 2012 International Conference of Biometrics Special Interest Group (BIOSIG’12), 2012, Sept 6-7, Darmstadt, Germany. Piscataway, NJ, USA: IEEE, 2012: 7p

11. PAN G, SUN L, WU Z H, et al. Eyeblink-based anti-spoofing in face recognition from a generic webcamera. Proceedings of the IEEE 11th International Conference on Computer Vision, 2007, Oct 14-21, Rio de Janeiro, Brazil. Piscataway, NJ, USA: IEEE, 2007: 8p

12. TAN X Y, LI Y, LIU J, et al. Face liveness detection from a single image with sparse low rank bilinear discriminative model. Computer Vision: Proceedings of the 11th European Conference on Computer Vision (ECCV’10): Part VI, 2010, Sept 5-11, Heraklion, Greece. LNCS 6316. Berlin, Germany: Springer, 2010: 504-517

13. MÄÄTTÄ J, HADID A, PIETIKÄINEN M. Face spoofing detection from single images using texture and local shape analysis. IET Biometrics, 2012, 1(1): 3-10

14. OJALA T, PIETIKAINEN M, MAENPAA T. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(7): 971-987

15. YANG J W, LEI Z, LIAO S C, et al. Face liveness detection with component dependent descriptor. Proceedings of the 2013 International Conference on Biometrics (ICB’13), 2013, Jun 4-7, Madrid, Spain. Piscataway, NJ, USA: IEEE, 2013: 6p

16. SCHWARTZ W R, ROCHA A, PEDRINI H. Face spoofing detection through partial least squares and low-level descriptors. Proceedings of the 2011 IEEE International Joint Conference on Biometrics (IJCB’11), 2011, Oct 11-13, Washington, DC, USA. Piscataway, NJ, USA: IEEE, 2011: 8p

17. KIM W, SUH S, HAN J J. Face liveness detection from a single image via diffusion speed model. IEEE Transactions on Image Processing, 2015, 24(8): 2456-2465

18. BOULKENAFET Z, KOMULAINEN J, HADID A. Face anti-spoofing based on color texture analysis. Proceedings of the 2015 International Conference on Image Processing (ICIP’15), 2015, Sept 27-30, Quebec City, Canada. Piscataway, NJ, USA: IEEE, 2015: 2636-2640

19. PATEL K, HAN H, JAIN A K. Secure face unlock: Spoof detection on smartphones. IEEE Transactions on Information Forensics and Security, 2016, 11(10): 2268-2283

20. WEN D, HAN H, JAIN A K. Face spoof detection with image distortion analysis. IEEE Transactions on Information Forensics and Security, 2015, 10(4): 746-761

21. XU Z Q, LI S, DENG W H. Learning temporal features using LSTM-CNN architecture for face anti-spoofing. Proceedings of the 3rd IAPR Asian Conference on Pattern Recognition (ACPR’15), 2015, Nov 3-6, Kuala Lumpur, Malaysia. Piscataway, NJ, USA: IEEE, 2015: 141-145

22. LI L, FENG X Y, BOULKENAFET Z, et al. An original face anti-spoofing approach using partial convolutional neural network. Proceedings of the 6th International Conference on Image Processing Theory, Tools and Applications (IPTA’15), 2016, Dec 12-15, Oulu, Finland. Piscataway, NJ, USA: IEEE, 2016: 6p

23. YANG J W, LEI Z , LI S Z . Learn convolutional neural network for face anti-spoofing. arXiv Preprint, arXiv:1408.5601, 2014

24. ATOUM Y, LIU Y J, JOURABLOO A, et al. Face anti-spoofing using patch and depth-based CNNs. Proceedings of the 2017 IEEE International Joint Conference on Biometrics (IJCB’17), 2017, Oct 1-4, Denver, CO, USA. Piscataway, NJ, USA: IEEE, 2017: 319-328

25. PATEL K, HAN H, JAIN A K. Cross-database face antispoofing with robust feature representation. Proceedings of the 11th Chinese Conference on Biometric Recognition (CCBR’16), 2016, Oct 14-16, Chengdu, China. LNIP 9967. Berlin, Germany: Springer, 2016: 611-619

26. LIU Y J, JOURABLOO A, LIU X M. Learning deep models for face anti-spoofing: Binary or auxiliary supervision. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, Jun 18-23, Salt Lake City, UT, USA. Piscataway, NJ, USA: IEEE, 2018: 389-398

27. JOURABLOO A, LIU Y J, LIU X M. Face de-spoofing: Anti-spoofing via noise modeling. Proceedings of the 15th European Conference on Computer Vision (ECCV’18): Part XIII, 2018, Sept 8-14, Munich, Germany. LNCS 11217. Berlin, Germany: Springer, 2018: 290-306

28. FENG L T, PO L M, LI Y M, et al. Integration of image quality and motion cues for face anti-spoofing: A neural network approach. Journal of Visual Communication and Image Representation, 2016, 38(C): 451-460

29. LIU Y J, STEHOUWER J, JOURABLOO A, et al. Deep tree learning for zero-shot face anti-spoofing. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'19), 2019, Jun 15-20, Long Beach, CA, USA. Piscataway, NJ, USA: IEEE, 2019: 4680-4689

30. ZHANG S F, LIU A J, WAN J, et al. Casia-surf: A large-scale multi-modal benchmark for face anti-spoofing. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2020, 2(2): 182-193

31. LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'17), 2017, Jul 21-26, Honolulu, HI, USA. Piscataway, NJ, USA: IEEE, 2017: 2117-2125

32. WANG X L, GIRSHICK R, GUPTA A, et al. Non-local neural networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'18), 2018, Jun 18-23, Salt Lake City, UT, USA. Piscataway, NJ, USA: IEEE, 2018: 7794-7803

33. PARKIN A, GRINCHUK O. Recognizing multi-modal face spoofing with face recognition networks. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW'19), 2019, Jun 16-17, Long Beach, CA, USA. Piscataway, NJ, USA: IEEE, 2019: 1617-1623

34. YU Z T, QIN Y X, LI X B, et al. Multi-modal face anti-spoofing based on central difference networks. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW'20), 2020, Jun 14-19, Seattle, WA, USA . Piscataway, NJ, USA: IEEE, 2020: 650-651

35. LIU A J, TAN Z C, LI X, et al. Static and dynamic fusion for multi-modal cross-ethnicity face anti-spoofing. arXiv Preprint, arXiv:1912.02340, 2019

36. HE K, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'16), 2016, Jun 27-30, Las Vegas, NV, USA. Piscataway, NJ, USA: IEEE, 2016: 770-778

37. HU J, SHEN L, SUN G. Squeeze-and-excitation networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'18), 2018, Jun 18-23, Salt Lake City, UT, USA. Piscataway, NJ, USA: IEEE, 2018: 7132-7141

38. ZHANG S F, WANG X B, LIU A J, et al. A dataset and benchmark for large-scale multi-modal face anti-spoofing. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'19) , 2019, Jun 15-20, Long Beach, CA, USA. Piscataway, NJ, USA: IEEE, 2019: 919-928

39. IU X L, LU R G, LIU W. Face liveness detection based on enhanced local binary patterns. Proceedings of the 2017 Chinese Automation Congress (CAC’17), 2017, Oct 20-22, Jinan, China. Piscataway, NJ, USA: IEEE, 2017: 6301-6305

40. PARVEEN S, AHMAD S M S, ABBAS N H, et al. Face liveness detection using dynamic local ternary pattern (DLTP). Computers, 2016, 5(2): 10-11

41. KIM W, SUH S, HAN J J. Face liveness detection from a single imagevia diffusion speed model. IEEE Transactions on Image Processing. 2015, 24(8): 2456-2465.

42. MÄÄTTÄ J, HADID A, PIETIKÄINEN M. Face spoofing detection from single images using micro-texture analysis. Proceedings of the 2011 IEEE International Joint Conference on Biometrics (IJCB’11), 2011, Oct 11-13, Washington, DC, USA. Piscataway, NJ, USA: IEEE, 2011

[1]	Wang Xianlun, Wang Guangyu, Cui Yuxia. Facial expression recognition based on improved ResNet[J]. 中国邮电高校学报(英文版), 2023, 30(1): 28-38.
[2]	Jia Wei, Gong Chao. Precise and efficient Chinese license plate recognition in the real monitoring scene of intelligent transportation system[J]. 中国邮电高校学报(英文), 2022, 29(3): 1-14.
[3]	Song Yue, Wu Chengmao, Tian Xiaoping, Song Qiuyu. Enhanced kernel-based fuzzy local information clustering integrating neighborhood membership [J]. 中国邮电高校学报(英文版), 2021, 28(6): 65-81.
[4]	Xue Chenzi, Wei Yifei, Zhang Yong. Performance optimization for smart grid blockchain integrated with fog computing using DDQN[J]. 中国邮电高校学报(英文版), 2021, 28(2): 68-78.
[5]	汪昭颖, 周军华, 廖中华, 翟翔, 张连平. Semantic segmentation of track image based on deep neural network[J]. 中国邮电高校学报(英文版), 2020, 27(5): 23-33.
[6]	陈法权樊军. Real-time prediction of the motion tendency of human lower limbs during gait [J]. 中国邮电高校学报(英文版), 2020, 27(4): 1-7.
[7]	李端张洪欣 Muhammad Saad Khan 米芳. Recognition of motor imagery tasks for BCI using CSP and chaotic PSO twin SVM[J]. 中国邮电高校学报(英文版), 2017, 24(3): 83-90.
[8]	张捷范旭慧班登科. Smooth support vector machine based on circular tangent function[J]. 中国邮电高校学报(英文版), 2016, 23(1): 68-72.
[9]	王亮刘贵喜段红岩. Dynamic and combined gestures recognition based on multi-feature fusion in a complex environment [J]. Acta Metallurgica Sinica(English letters), 2015, 22(2): 81-88.
[10]	李岩周亚建袁开国郭玉翠钮心忻. Exposing photo manipulation with inconsistent perspective geometry[J]. Acta Metallurgica Sinica(English letters), 2014, 21(4): 83-91.
[11]	刘敬徐国胜郑世慧肖达谷利泽. Data streams classification with ensemble model based on decision-feedback[J]. Acta Metallurgica Sinica(English letters), 2014, 21(1): 79-85.
[12]	卢官明左加阔. Orthogonal isometric projection for face recognition[J]. Acta Metallurgica Sinica(English letters), 2011, 18(1): 91-97.

Face anti-spoofing based on multi-modal and multi-scale features fusion

Face anti-spoofing based on multi-modal and multi-scale features fusion

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 12

编辑推荐

Metrics

本文评价