Video super-resolution reconstruction based on deep convolutional neural network and spatio-temporal similarity

doi:10.1016/S1005-8885(16)60060-2

中国邮电高校学报(英文) ›› 2016, Vol. 23 ›› Issue (5): 68-81.doi: 10.1016/S1005-8885(16)60060-2

Video super-resolution reconstruction based on deep convolutional neural network and spatio-temporal similarity

李玲慧¹,杜军平²,梁美玉¹,任楠¹,范聃¹

1. 北京邮电大学计算机学院，智能通信软件与多媒体北京市重点实验室
2. 北京邮电大学

收稿日期:2016-03-21 修回日期:2016-09-29 出版日期:2016-10-30 发布日期:2016-10-26
通讯作者: 杜军平 E-mail:junpingdu@126.com
基金资助:
the National Natural Science Foundation of China

Video super-resolution reconstruction based on deep convolutional neural network and spatio-temporal similarity

Li Linghui1, Du Junping 2, Liang Meiyu1, Ren Nan1, Fan Dan1

Received:2016-03-21 Revised:2016-09-29 Online:2016-10-30 Published:2016-10-26
Supported by:
the National Natural Science Foundation of China

摘要/Abstract

摘要： Existing learning-based super-resolution (SR) reconstruction algorithms are mainly designed for single image, which ignore the spatio-temporal relationship between video frames. Aiming at applying the advantages of learning-based algorithms to video SR field, a novel video SR reconstruction algorithm based on deep convolutional neural network (CNN) and spatio-temporal similarity (STCNN-SR) was proposed in this paper. It is a deep learning method for video SR reconstruction, which considers not only the mapping relationship among associated low-resolution (LR) and high-resolution (HR) image blocks, but also the spatio-temporal non-local complementary and redundant information between adjacent low-resolution video frames. The reconstruction speed can be improved obviously with the pre-trained end-to-end reconstructed coefficients. Moreover, the performance of video SR will be further improved by the optimization process with spatio-temporal similarity. Experimental results demonstrated that the proposed algorithm achieves a competitive SR quality on both subjective and objective evaluations, when compared to other state-of-the-art algorithms.

关键词: video SR reconstruction, deep convolutional neural network, spatio-temporal similarity, Zernike moment feature

Abstract: Existing learning-based super-resolution (SR) reconstruction algorithms are mainly designed for single image, which ignore the spatio-temporal relationship between video frames. Aiming at applying the advantages of learning-based algorithms to video SR field, a novel video SR reconstruction algorithm based on deep convolutional neural network (CNN) and spatio-temporal similarity (STCNN-SR) was proposed in this paper. It is a deep learning method for video SR reconstruction, which considers not only the mapping relationship among associated low-resolution (LR) and high-resolution (HR) image blocks, but also the spatio-temporal non-local complementary and redundant information between adjacent low-resolution video frames. The reconstruction speed can be improved obviously with the pre-trained end-to-end reconstructed coefficients. Moreover, the performance of video SR will be further improved by the optimization process with spatio-temporal similarity. Experimental results demonstrated that the proposed algorithm achieves a competitive SR quality on both subjective and objective evaluations, when compared to other state-of-the-art algorithms.

Key words: video SR reconstruction, deep convolutional neural network, spatio-temporal similarity, Zernike moment feature

中图分类号:

TP37

Li Linghui, Du Junping,Liang Meiyu, Ren Nan, Fan Dan. Video super-resolution reconstruction based on deep convolutional neural network and spatio-temporal similarity[J]. JOURNAL OF CHINA UNIVERSITIES OF POSTS AND TELECOM, 2016, 23(5): 68-81.

参考文献

1. Yang C Y, Ma C, Yang M H. Single-image super-resolution: a benchmark. Proceedings of the 13th European Conference on Computer Vision (ECCV’14): Part IV, Sept 6-12, 2014, Zurich, Switzerland. LNCS 8692. Berlin, Germany: Springer-Verlag, 2014: 372-386

2. Sun J, Xu Z B, Shum H Y, et al. Image super-resolution using gradient profile prior. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08), Jun 24-26, 2008, Anchorage, AK, USA. Piscataway, NJ, USA: IEEE, 2008: 8p

3. Kim K I, Kwon Y. Single-image super-resolution using sparse regression and natural image prior. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2010, 32(6): 1127-1133

4. Zhang H C, Yang J C, Zhang Y N, et al. Non-local kernel regression for image and video restoration. Proceedings of the 11th European Conference on Computer Vision (ECCV ’10): Part III, Sept 5-11, 2010, Crete, Greece. LNCS 6313. Berlin, Germany: Springer-Verlag , 2010: 566-579

5. Su H, Tang L, Wu Y, et al. Spatially adaptive block-based super-resolution. IEEE Transactions on Image Processing, 2012, 21(3): 1031-1045

6. He L, Qi H R, Zaretzki R. Beta process joint dictionary learning for coupled feature spaces with application to single image super-resolution. Proceedings of the 2013 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’13), Jun 23-28, 2013, Portland, OR, USA. Piscataway, NJ, USA: IEEE, 2013: 345-352

7. Purkait P, Chanda B. Image upscaling using multiple dictionaries of natural image patches. Proceedings of the 11th Asian Conference on Computer Vision (ACCV’13): Part III, Nov 5-9, 2012, Daejeon, Republic of Korea. LNCS 7726. Berlin, Germany: Springer-Verlag , 2013: 284-295

8. Dong W S, Zhang L, Lukac R, et al. Sparse representation based image interpolation with nonlocal autoregressive modeling. IEEE Transactions on Image Processing, 2013, 22(4): 1382-1394

9. Protter M, Elad M, Takeda H, et al. Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Transactions on Image Processing, 2009, 18(1): 36-51

10. Gao X B, Wang Q, Li X L, et al. Zernike-moment-based image super resolution. IEEE Transactions on Image Processing, 2011, 20(10): 2738-2747

11. Liu C, Sun D Q. On Bayesian adaptive video super resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(2): 346-360

12. Yang J C, Wright J, Huang T, et al. Image super-resolution as sparse representation of raw image patches. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08), Jun 23-28, 2008, Anchorage, AK, USA. Piscataway, NJ, USA: IEEE, 2008: 8p

13. Wang S L, Zhang L, Liang Y, et al. Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12), Jun 16-21, 2012, Providence, RI, USA. Piscataway, NJ, USA: IEEE, 2012: 2216-2223

14. Timofte R, De V, Gool L V. Anchored neighborhood regression for fast example-based super-resolution. Proceedings of the 2013 IEEEE International Conference on Computer Vision (ICCV’13), Dec 2-8, 2013, Sydney, Australia. Piscataway, NJ, USA: IEEE, 2013: 1920-1927

15. Dong C, Loy C C, He K M, et al. Learning a deep convolutional network for image super-resolution. Proceedings of the European Conference on Computer Vision (ECCV ’14), Sept 6-12, 2014, Zurich, Switzerland. LNCS 8692. Berlin, Germany: Springer-Verlag, 2014: 184-199

16. Dong C, Loy C C, He K M, et al. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307

17. Liang M Y, Du J P, Liu H G. Spatiotemporal super-resolution reconstruction based on robust optical flow and Zernike moment for video sequences. Mathematical Problems in Engineering, 2013: 745752/1-14

18. Liang M Y, Du J P, Lee J M, et al. Spatio-temporal adaptive super-resolution reconstruction model based on Zernike moment for spatial video sequences. China Communications, 2012, 9(12): 93-107

19. He K M, Sun J. Convolutional neural networks at constrained time cost, Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15), Jun 7-12, 2015, Boston, MA, USA. Piscataway, NJ, USA: IEEE, 2015: 5353-5360

20. Shan Q, Li Z R, Jia J Y, et al. Fast image/video upsampling. ACM Transactions on Graphics, 2008, 27(5): 32-39

21. Zhu Y, Zhang Y N, Yuille A L. Single image super-resolution using deformable patches. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14), Jun 23-28, 2014, Columbus, OH, USA. Piscataway, NJ, USA: IEEE, 2014: 2917-2924

22. Kwon Y M, Kim M G, Kim J L. Multiscale rank-based ordered dither algorithm for digital halftoning. Information Systems, 2015, 48: 241-247

23. Wang Z, Simoncelli E P, Bovik A C. Multi-scale structural similarity for image quality assessment. Proceedings of the 37th IEEE Asilomar Conference on Signals, Systems, and Computers (ACSSC’03): Vol 2, Nov. 9-12, 2003, Pacific Grove, CA, USA. Alamitos, CA, USA: IEEE Computer Society, 2003: 1398-1402

24. Zhang L, Zhang L, Mou X Q, et al. FSIM: a feature similarity index for image quality assessment. IEEE Transactions on Image Processing, 2011, 20(8): 2378-2386

Video super-resolution reconstruction based on deep convolutional neural network and spatio-temporal similarity

Video super-resolution reconstruction based on deep convolutional neural network and spatio-temporal similarity

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 1

编辑推荐

Metrics

本文评价