Research on unified recognition model and algorithm for multi-modal gestures

doi:10.19682/j.cnki.1005-8885.2019.1004

中国邮电高校学报(英文) ›› 2019, Vol. 26 ›› Issue (2): 30-42.doi: 10.19682/j.cnki.1005-8885.2019.1004

• Artificial Intelligence • 上一篇下一篇

Research on unified recognition model and algorithm for multi-modal gestures

Guo Xiaopei, Feng Zhiquan, Sun Kaiyun, Liu Hong, Xie Wei, Bi Jianping

1. School of Information Science and Engineering, University of Jinan, Jinan 250022, China
2. Shandong Provincial Key Laboratory of Network-based Intelligent Computing, Jinan 250022, China
3. School of Information Science and Engineering, Shandong Normal University, Jinan 250014, China
4. School of Information and Electrical Engineering, Harbin Institute of Technology at Weihai, Weihai 264209, China
5. Affiliated Hospital of Shandong University of Traditional Chinese Medicine, Jinan 250022, China

收稿日期:2018-08-07 修回日期:2019-04-04 出版日期:2019-04-30 发布日期:2019-06-14
通讯作者: Corresponding author: Feng Zhiquan, E-mail: ise_fengzq@ujn.edu.cn E-mail:ise_fengzq@ujn.edu.cn
作者简介:Corresponding author: Feng Zhiquan, E-mail: ise_fengzq@ujn.edu.cn
基金资助:
This work was supported by the National Key R&D Program of China (2018YFB1004901) and the National Natural Science Foundation of China (61472163, 61472232).

Research on unified recognition model and algorithm for multi-modal gestures

Guo Xiaopei, Feng Zhiquan, Sun Kaiyun, Liu Hong, Xie Wei, Bi Jianping

1. School of Information Science and Engineering, University of Jinan, Jinan 250022, China
2. Shandong Provincial Key Laboratory of Network-based Intelligent Computing, Jinan 250022, China
3. School of Information Science and Engineering, Shandong Normal University, Jinan 250014, China
4. School of Information and Electrical Engineering, Harbin Institute of Technology at Weihai, Weihai 264209, China
5. Affiliated Hospital of Shandong University of Traditional Chinese Medicine, Jinan 250022, China

Received:2018-08-07 Revised:2019-04-04 Online:2019-04-30 Published:2019-06-14
Contact: Corresponding author: Feng Zhiquan, E-mail: ise_fengzq@ujn.edu.cn E-mail:ise_fengzq@ujn.edu.cn
About author:Corresponding author: Feng Zhiquan, E-mail: ise_fengzq@ujn.edu.cn
Supported by:
This work was supported by the National Key R&D Program of China (2018YFB1004901) and the National Natural Science Foundation of China (61472163, 61472232).

摘要/Abstract

摘要： In gesture recognition, static gestures, dynamic gestures and trajectory gestures are collectively known as multi-modal gestures. To solve the existing problem in different recognition methods for different modal gestures, a unified recognition algorithm is proposed. The angle change data of the finger joints and the movement of the centroid of the hand were acquired respectively by data glove and Kinect. Through the preprocessing of the multi-source heterogeneous data, all hand gestures were considered as curves while solving hand shaking, and a uniform hand gesture recognition algorithm was established to calculate the Pearson correlation coefficient between hand gestures for gesture recognition. In this way, complex gesture recognition was transformed into the problem of a simple comparison of curves similarities. The main innovations: 1) Aiming at solving the problem of multi-modal gesture recognition, an unified recognition model and a new algorithm is proposed; 2) The Pearson correlation coefficient for the first time to construct the gesture similarity operator is improved. By testing 50 kinds of gestures, the experimental results showed that the method presented could cope with intricate gesture interaction with the 97.7% recognition rate.

关键词: Kinect, data glove, multi-modal gesture, gesture interaction

Abstract: In gesture recognition, static gestures, dynamic gestures and trajectory gestures are collectively known as multi-modal gestures. To solve the existing problem in different recognition methods for different modal gestures, a unified recognition algorithm is proposed. The angle change data of the finger joints and the movement of the centroid of the hand were acquired respectively by data glove and Kinect. Through the preprocessing of the multi-source heterogeneous data, all hand gestures were considered as curves while solving hand shaking, and a uniform hand gesture recognition algorithm was established to calculate the Pearson correlation coefficient between hand gestures for gesture recognition. In this way, complex gesture recognition was transformed into the problem of a simple comparison of curves similarities. The main innovations: 1) Aiming at solving the problem of multi-modal gesture recognition, an unified recognition model and a new algorithm is proposed; 2) The Pearson correlation coefficient for the first time to construct the gesture similarity operator is improved. By testing 50 kinds of gestures, the experimental results showed that the method presented could cope with intricate gesture interaction with the 97.7% recognition rate.

Key words: Kinect, data glove, multi-modal gesture, gesture interaction

中图分类号:

TP391

Guo Xiaopei, Feng Zhiquan, Sun Kaiyun, Liu Hong, Xie Wei, Bi Jianping. Research on unified recognition model and algorithm for multi-modal gestures[J]. The Journal of China Universities of Posts and Telecommunications, 2019, 26(2): 30-42.

参考文献

References
1. Liu X. Research of dynamic gesture recognition based on the kinect depth image. Shenyang, LN, China: Northeastern University, 2014: 77p (in Chinese)
2. Lee C, Xu Y. Online, interactive learning of gestures for human/robot interfaces. Proceedings of IEEE International Conference on Robotics and Automation, Apr 22 -28, 1996, Minneapolis, MN, USA: IEEE, 2002(4): 2982 -2987
3. Nayan M K, Manalee D S. Recognition of sign language alphabets and numbers based on hand kinematics using a data glove. Procedia Computer Science, 2018(133): 55 -62
4. Kazuma I, Masato S, Hirokazu T. Development of finger motion skill learning support system based on data gloves. Procedia Computer Science, 2014, 35(6): 1307 -1314
5. Dutta D, Modak S, Kumar A, et al. Bayesian network aided grasp and grip efficiency estimation using a smart data glove for post-stroke diagnosis. Biocybernetics and Biomedical Engineering, 2017, 37(1): 44 -58
6. Yao Y, Zhang L J, Qiao W B. Hand part labeling and gesture recognition from RGB-D data. Journal of Computer-Aided Design and Computer Graphics, 2013, 25(12): 1810 -1816 (in
Chinese)
7. Li H B, Ding L J, et al. Static three-dimensional gesture recognition based on kinect skeleton data. Computer Applications and Software, 2015, 32(9): 161 -165 (in Chinese)
8. Batabyal T, Chattopadhyay T, Mukherjee D P. Action recognition using joint coordinates of 3D skeleton data. IEEE International Conference on Image Processing, Sep 27 -30, 2015, Quebec City, Canada: IEEE, 2015: 4107 -4111
9. Zhu H M, Pun C M. Human action recognition with skeletal information from depth camera. IEEE International Conference on Information and Automation, July 26 -31, 2014, Hulun Buir,
NM, China: IEEE, 2014: 1082 -1085
10. Ding I J, Chang C W. Feature design scheme for kinect-based DTW human gesture recognition. Multimedia Tools and Applications, 2016, 75(16): 9669 -9684
11. Wu D, Pigou L, Kindermans P J, et al. Deep dynamic neural networks for multimodal gesture segmentation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(8): 1583 -1597
12. Yang X W, Feng Z Q, Huang Z Z, et al. A gesture recognition algorithm using hausdorff-like distance template matching based on the main direction of gesture. Applied Mechanics and Materials, 2015, 713 -715(3): 2156 -2159
13. Singha J, Laskar R H. Hand gesture recognition using two-level speed normalization, feature selection and classifier fusion. Multimedia Systems, 2017, 23(4): 499 -514
14. Lee W P, Kaoli C, Huang J Y. A smart TV system with body-gesture control, tag-based rating and context-aware recommendation. Knowledge-Based Systems, 2014, 56(3): 167 -178
15. Ghaleb F F M, Youness E A, Elmezain M, et al. Vision-based hand gesture spotting and recognition using CRF and SVM. Journal of Software Engineering and Applications, 2015, 8(7): 313 -323
16. Dias T, Variz M, Jorge P, et al. Gesture interaction system for social web applications on smart TVs. Proceedings of the 10th Conference on Open Research Areas in Information Retrieval, May 15 -17, 2013, Lisbon, Portugal, 2013: 225 -226
17. Kim Y C, Jiang J S. Remote gesture control using HMM classifier based on electric potential sensors. International Journal of Multimedia and Ubiquitous Engineering, 2015, 10(2): 413 -420
18. Wang L, Liu G X, Duan H Y. Dynamic and combined gestures recognition based on multi-feature fusion in a complex environment. The Journal of China Universities of Posts and
Telecommunications, 2015, 22(2): 81 -88
19. Shehu V, Dika A. Curve similarity measurement algorithms for automatic gesture detection systems. Proceedings of IEEE 35th International Convention MIPRO, May 21 -25, 2012, Opatija, Croatia: IEEE, 2012: 973 -976
20. Shen L, Huo S C, Chen M S, et al. Multi-touch gesture recognition algorithm of vehicle electronic devices-based on Bezier curve optimization strategy. 2017 32nd Youth Academic Annual Conference of Chinese Association of Automation (YAC), May 19 -21, 2017, Hefei, AH, China: IEEE, 2017: 720 -723
21. Wu J X, Cheng J. Bayesian co-boosting for multi-modal gesture recognition. The Journal of Machine Learning Research, 2017, 15(1): 3013 -3036
22. Liang B, Zheng L. Multi-modal gesture recognition using skeletal joints and motion trail model. Workshop at the European Conference on Computer Vision, Sep 6 -12, 2014, Zurich,
Switzerland, 2014: 623 -628
23. Jia X Y, Xu C S, Bai X. The invention and way of thinking on least squares. Journal of Northwest University (Natural Science Edition), 2006, 36(3): 507 -511 (in Chinese)
24. Yang F, Feng X, Ruan L, et al. Correlation study of water tree and VLF tan based on pearson correlation coefficient. High Voltage Apparatus, 2014, 50(6): 21 -31 (in Chinese)

[1]	Guo Xiangbo, Wang Jian, Huang Mengjie, Wang Minghui, Yang Jian, Yu Yongtao. Deep knowledge tracking algorithm based on forgetting law[J]. 中国邮电高校学报(英文版), 2023, 30(1): 17-27.
[2]	Wang Xianlun, Wang Guangyu, Cui Yuxia. Facial expression recognition based on improved ResNet[J]. 中国邮电高校学报(英文版), 2023, 30(1): 28-38.
[3]	Wu Hongxin, Lin Zhijian, Chen Pingping, Chen Feng. Joint partial computation offloading and resource allocation in MEC-enable networks[J]. 中国邮电高校学报(英文版), 2023, 30(1): 80-86.
[4]	Kong Chao, Ou Weihua, Gong Xiaofeng, Li Weian, Han Jie, Yao Yi, Xiong Jiahao. Face anti-spoofing based on multi-modal and multi-scale features fusion [J]. 中国邮电高校学报(英文版), 2022, 29(6): 73-82.
[5]	张函井音吉赵永利. FS-LSTM: sales forecasting in e-commerce on feature selection [J]. 中国邮电高校学报(英文版), 2022, 29(5): 92-98.
[6]	Shi Jinjing, Wang Wenxuan, Xiao Zimeng, Mu Shuai, Li Qin. Quantum classifier with parameterized quantum circuit based on the isolated quantum system[J]. 中国邮电高校学报(英文版), 2022, 29(4): 21-31.
[7]	Liu Hailing, Zhang Jie, Qin Sujuan, Gao Fei. Quantum algorithm for soft margin support vector machine with hinge loss function[J]. 中国邮电高校学报(英文版), 2022, 29(4): 32-41.
[8]	Wang Jian, Qiao Kuoyuan, Yuan Yanlei, Liu Xiaole, Yang Jian. Adaptive learning path recommendation model for examination-oriented education[J]. 中国邮电高校学报(英文版), 2022, 29(4): 77-88.
[9]	Meng Wei, Wang Liting, Lu Meng. Summary of research on recommendation system based on serendipity[J]. 中国邮电高校学报(英文版), 2022, 29(4): 89-105.
[10]	Jia Wei, Gong Chao. Precise and efficient Chinese license plate recognition in the real monitoring scene of intelligent transportation system[J]. 中国邮电高校学报(英文版), 2022, 29(3): 1-14.
[11]	Song Yue, Wu Chengmao, Tian Xiaoping, Song Qiuyu. Enhanced kernel-based fuzzy local information clustering integrating neighborhood membership [J]. 中国邮电高校学报(英文版), 2021, 28(6): 65-81.
[12]	Xue Chenzi, Wei Yifei, Zhang Yong. Performance optimization for smart grid blockchain integrated with fog computing using DDQN[J]. 中国邮电高校学报(英文版), 2021, 28(2): 68-78.
[13]	Guo Hairu, Meng Xueyao, Liu Yongli, Liu Shen. Improved HHO algorithm based on good point set and nonlinear convergence formula[J]. 中国邮电高校学报(英文版), 2021, 28(2): 48-67.
[14]	吴成茂曹卓. Entropy-like distance driven fuzzy clustering with local information constraints for image segmentation [J]. 中国邮电高校学报(英文版), 2021, 28(1): 24-40.
[15]	山蕊蒋林邓军勇崔朋飞张玉婷吴皓月谢晓燕. Parallel design of convolutional neural networks for remote sensing images object recognition based on data-driven array processor [J]. 中国邮电高校学报(英文版), 2020, 27(6): 87-100.

Research on unified recognition model and algorithm for multi-modal gestures

Research on unified recognition model and algorithm for multi-modal gestures

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价