Multi-scale fusion residual encoder-decoder approach for low illumination image enhancement

doi:10.19682/j.cnki.1005-8885.2022.0004

The Journal of China Universities of Posts and Telecommunications ›› 2022, Vol. 29 ›› Issue (2): 63-72.doi: 10.19682/j.cnki.1005-8885.2022.0004

Previous Articles Next Articles

Multi-scale fusion residual encoder-decoder approach for low illumination image enhancement

Received:2021-01-13 Revised:2021-07-09 Online:2022-04-26 Published:2022-04-26
Supported by:
This work was supported by the National Key Research and Development Project ( 2019YFC0121502 ), the Graduate Innovation Fund Project of Xi'an University of Posts and Telecommunications ( CXJJLY2019062 ), and the Key Laboratory of Network Data Analysis and Intelligent Processing of Shaanxi Province.

Abstract

Abstract:

The sensing light source of the line scan camera cannot be fully exposed in a low light environment due to the extremely small number of photons and high noise, which leads to a reduction in image quality. A multi-scale fusion residual encoder-decoder (FRED) was proposed to solve the problem. By directly learning the end-to-end mapping between light and dark images, FRED can enhance the image's brightness with the details and colors of the original image fully restored. A residual block (RB) was added to the network structure to increase feature diversity and speed up network training. Moreover, the addition of a dense context feature aggregation module (DCFAM) made up for the deficiency of spatial information in the deep network by aggregating the context's global multi-scale features. The experimental results show that the FRED is superior to most other algorithms in visual effect and quantitative evaluation of peak signa-to-noise ratio (PSNR) and structural similarity index measure (SSIM). For the factor that FRED can restore the brightness of images while representing the edge and color of the image effectively, a satisfactory visual quality is obtained under the enhancement of low-light.

Key words: image enhancement| low illumination|feature fusion| residual network

References

[1] LUO H L, AO Y, YUAN P. Image inpainting using generative adversarial networks. Acta Electronica Sinica, 2020, 48 (10):1891 -1898 (in Chinese).

[2] XIONG W, YU J H, LIN Z, et al. Foreground-aware image inpainting. Proceedings of the 2019 IEEE/ CVF Conference on Computer Vision and Pattern Recognition (CVPR’19), 2019, Jun 15 -20, Long Beach, CA, USA. Piscataway, NJ, USA: IEEE, 2019: 5840 -5848.

[3] ZHAO L, MO Q H, LIN S H, et al. UCTGAN: diverse image inpainting based on unsupervised cross-space translation. Proceedings of the 2020 IEEE/ CVF Conference on Computer Vision and Pattern Recognition (CVPR’20), 2020, Jun 13 -19, Seattle, WA, USA. Piscataway, NJ, USA: IEEE, 2020: 5741 -5750.

[4] LIANG J W, QIU T R, ZHOU A Y, et al. Ensemble of multiscale fine-tuning convolutional neural network for recognition of benign and malignant thyroid nodules. Journal of Computer-Aided Design

& Computer Graphics, 2021, 33(1): 81 -91 (in Chinese).

[5] ANCHA S, NAN J, HELD D. Combining deep learning and verification for precise object instance detection. Proceedings of the 3rd Conference on Robot Learning ( CoRL’19), 2019, Oct 30 - Nov 1, Osaka, Japan. Cambridge, MA,USA: MIT, 2020: 122 -141.

[6] LIANG X Y, LIN X K, QUAN J C, et al. Research on the progress of image instance segmentation technology based on deep learning. Acta Electronica Sinica, 2020, 48(12): 2476 – 2486 (in Chinese).

[7] LIN Z, ZHANG Z, CHEN L Z, et al. Interactive image segmentation with first click attention. Proceedings of the 2020

IEEE/ CVF Conference on Computer Vision and Pattern Recognition ( CVPR’20), 2020, Jun 13 - 19, Seattle, WA,

USA. Piscataway, NJ, USA: IEEE, 2020: 13339 -13348.

[8] LÜZ W, WANG K J, ZOU G F, et al. Illumination compensation method for face image based on improved gamma correction. Proceedings of the 32nd Chinese Control Conference, 2013, Jul 26 -28, Xi’an, China. Piscataway, NJ, USA: IEEE, 2013: 3733 -3737.

[9] CHENG H D, SHI X J. A simple and effective histogram equalization approach to image enhancement. Digital Signal

Processing, 2004, 14(2): 158 -170.

[10] ABDULLAH-AL-WADUD M, KABIR M H, DEWAN M A A, et al. A dynamic histogram equalization for image contrast enhancement. IEEE Transactions on Consumer Electronics, 2007, 53(2): 593 -600.

[11] CELIK T, TJAHJADI T. Contextual and variational contrast enhancement. IEEE Transactions on Image Processing, 2011, 20(12): 3431 -3441.

[12] LEE C W, LEE C, KIM C S. Contrast enhancement based on layered difference representation of 2D histograms. IEEE Transactions on Image Processing, 2013, 22(12): 5372 -5384.

[13] LAND E H, MCCANN J J. Lightness and Retinex theory. Journal of the Optical Society of America, 1971, 61(1): 1 -11.

[14] RAHMAN Z, JOBSON D J, WOODELL G A. Multi-scale Retinex for color image enhancement. Proceedings of 3rd IEEE International Conference on Image Processing (ICIP’96): Vol 3, 1996, Sept 16 - 19, Lausanne, Switzerland. Piscataway, NJ, USA: IEEE, 1996: 1003 -1006.

[15] FU X Y, ZENG D L, HUANG Y, et al. A weighted variational model for simultaneous reflectance and illumination estimation. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16), 2016, Jun 27 -30, Las Vegas, NV, USA. Piscataway, NJ, USA: IEEE, 2016: 2782 -2790.

[16] PARK S, YU S, MOON B, et al. Low-light image enhancement using variational optimization-based retinex model. IEEE Transactions on Consumer Electronics, 2017, 63(2): 178 -184. [17] WEI C, WANG W J, YANG W H, et al. Deep Retinex decomposition for low-light enhancement. arXiv Pxeprint, arXiv: 1808. 04560, 2018.

[18] LI M D, LIU J Y, YANG W H, et al. Structure-revealing low-light image enhancement via robust Retinex model. IEEE

Transactions on Image Processing, 2018, 27(6): 2828 -2841.

[19] REN X T, YANG W H, CHENG W H, et al. LR3M: robust low-light enhancement via low-rank regularized Retinex model. IEEE Transactions on Image Processing, 2020, 29: 5862 -5876.

[20] LORE K G, AKINTAYO A, SARKAR S. LLNet: a deep autoencoder approach to natural low-light image enhancement. Pattern Recognition, 2017, 61: 650 -662.

[21] CHEN C, CHEN Q F, XU J, et al. Learning to see in the dark. Proceedings of the 2018 IEEE/ CVF Conference on Computer Vision and Pattern Recognition (CVPR’18), 2018, Jun 18 - 23, Salt Lake City, UT, USA. Piscataway, NJ, USA: IEEE, 2018: 3291 -3300.

[22] MAHARJAN P, LI L, LI Z, et al. Improving extreme low-light image denoising via residual learning. Proceedings of the 2019 IEEE International Conference on Multimedia and Expo (ICME’19), 2019, Jul 8 - 12, Shanghai, China. Piscataway, NJ, USA: IEEE, 2019: 916 -921.

[23] ZHANG C, YAN Q S, ZHU Y, et al. Attention-based network for low-light image enhancement. Proceedings of the 2020 IEEE International Conference on Multimedia and Expo ( ICME’20), 2020, Jul 6 - 10, London, UK. Piscataway, NJ, USA: IEEE, 2020: 1 -6.

[24] CAI J R, GU S H, ZHANG L. Learning a deep single image contrast enhancer from multi-exposure images. IEEE Transactions on Image Processing, 2018, 27(4): 2049 -2062.

[25] ZHU M F, PAN P B, CHEN W, et al. EEMEFN: low-light image enhancement via edge-enhanced multi-exposure fusion network. Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI’20), 2020, Feb 7 - 12, New York, NY, USA. Menlo Park, CA, USA: AAAI Press, 2020: 13106 -13113.

[26] WANG R X, ZHANG Q, FU C W, et al. Underexposed photo enhancement using deep illumination estimation. Proceedings of the 2019 IEEE/ CVF Conference on Computer Vision and Pattern Recognition (CVPR’19), 2019, Jun 15 - 20, Long Beach, CA, USA. Piscataway, NJ, USA: IEEE, 2019: 6849 -6857.

[27] ZHANG Y H, ZHANG J W, GUO X J. Kindling the darkness: a practical low-light image enhancer. Proceedings of the 27th ACM International Conference on Multimedia ( MM’19), 2019, Oct 21 - 25, Nice, France. New York, NY, USA: ACM, 2019: 1632 -1640.

[28] JIANG Y F, GONG X Y, LIU D, et al. EnlightenGAN: deep light enhancement without paired supervision. IEEE Transactions on Image Processing, 2021, 30: 2340 -2349.

[29] YU S, PARK B, JEONG J. Deep iterative down-up CNN for image denoising. Proceedings of the 2019 IEEE/ CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’19), 2019, Jun 16 - 17, Long Beach, CA, USA. Piscataway, NJ, USA: IEEE, 2019: 2095 -2103.

[30] ZAMIR S W, ARORA A, KHAN S, et al. Learning digital camera pipeline for extreme low-light imaging. Neurocomputing, 2021, 452: 37 -47.

[31] KARADENIZ A S, ERDEM E, ERDEM A. Burst photography for learning to enhance extremely dark images. arXiv Preprint, arXiv: 2006. 09845, 2020.

[32] LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition ( CVPR’17 ), 2017, Jul 21 - 26, Honolulu, HI, USA.

Piscataway, NJ, USA: IEEE, 2017: 2117 -2125.

[33] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16), 2016, Jun 27 - 30, Las Vegas, NV, USA. Piscataway, NJ, USA: IEEE, 2016: 770 -778.

[34] MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a completely blind image quality analyzer. IEEE Signal Processing Letters, 2012, 20(3): 209 -212.

Metrics

Comments

Copyright © 2020 The Journal of China Universities of Posts and Telecommunications
　 Adress: P.O. Box 231,Beijing University of Posts and Telecommunications,10 Xi Tucheng Road,Beijing 100876,P.R.China　Post Code: 100081
Tel：86-010-62282493　Fax： 86-010-62283461　E-mail: jchupt@bupt.edu.cn
Support by: Beijing Magtech Co.Ltd

Multi-scale fusion residual encoder-decoder approach for low illumination image enhancement

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 8

Recommended Articles

Metrics

Comments

[1]	. Back in time: digital restoration techniques for the millennium Dunhuang murals [J]. The Journal of China Universities of Posts and Telecommunications, 2022, 29(2): 13-23.
[2]	DUAN Lian. Low-light image enhancement algorithm using a residual network with semantic information [J]. The Journal of China Universities of Posts and Telecommunications, 2022, 29(2): 52-62.
[3]	. Infrared image enhancement algorithm based on adaptive weighted guided filter [J]. The Journal of China Universities of Posts and Telecommunications, 2022, 29(2): 73-84.
[4]	Li Hui, Li Shanshan, Zou Borong, Chen Yannan. Modulation classification based on the collaboration of dual-channel CNN-LSTM and residual network [J]. The Journal of China Universities of Posts and Telecommunications, 2022, 29(1): 113-124.
[5]	Ming Yue, Li Wenmin, Xu Siya, Gao Lifang, Zhang Hua, Shao Sujie, Yang Huifeng. Liveness detection of occluded face based on dual-modality convolutional neural network [J]. The Journal of China Universities of Posts and Telecommunications, 2021, 28(4): 1-12.
[6]	Wu Xiaochu, Tang Guijin, Liu Xiaohua, Cui Ziguan, Luo Suhuai. Low-light color image enhancement based on NSST [J]. The Journal of China Universities of Posts and Telecommunications, 2019, 26(5): 41-48.
[7]	. Cancelable palmprint template generating algorithm based on adaptive threshold [J]. The Journal of China Universities of Posts and Telecommunications, 2019, 26(1): 1-11.
[8]	ZHANG Jie , JING Xiao-jun, CHEN Na, WANG Jian-li. ncomplete fingerprint recognition based on feature fusion and pattern entropy [J]. Acta Metallurgica Sinica(English letters), 2013, 20(3): 121-128.