Illumination robust image transformations for feature-based SLAM using photometric and feature matches loss

doi:10.19682/j.cnki.1005-8885.2022.1012

The Journal of China Universities of Posts and Telecommunications ›› 2022, Vol. 29 ›› Issue (3): 92-104.doi: 10.19682/j.cnki.1005-8885.2022.1012

School of Computer Science, Beijing University of Posts and Telecommunications, Beijing 100876, China

Received:2021-10-27 Revised:2022-04-07 Online:2022-06-30 Published:2022-06-30
Contact: Yan Danfeng E-mail:yandf@bupt.edu.cn

Abstract

Abstract: Simultaneous localization and mapping (SLAM) technology becomes more and more important in robot localization. The purpose of this paper is to improve the robustness of visual features to lighting changes and increase the recall rate of map re-localization under different lighting environments by optimizing the image transformation model. An image transformation method based on matches and photometric error (name the method as MPT) is proposed in this paper, and it is seamlessly integrated into the pre-processing stage of the feature-based visual SLAM framework. The results of the experiment show that the MPT method has a better matching effect on different visual features. In addition, the image transformation module encapsulated by a robot operating system (ROS) can be used with multiple visual SLAM systems and improve its re-localization effect under different lighting environments.

Key words: simultaneous localization and mapping (SLAM), image transformation, long-term visual localization

Zhang Miao, Wang Zixian, Yan Danfeng. Illumination robust image transformations for feature-based SLAM using photometric and feature matches loss[J]. The Journal of China Universities of Posts and Telecommunications, 2022, 29(3): 92-104.

References

1. MUR-ARTAL R, TARDÓS J D. ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Transactions on Robotics, 2017, 33(5): 1255-1262.

2. CAMPOS C, ELVIRA R, RODRÍGUEZ J J G, et al. ORB-SLAM3: An accurate open-source library for visual, visual-inertial, and multimap SLAM. IEEE Transactions on Robotics, 2021, 37(6): 1874-1890.

3. QIN T, LI P L, SHEN S J. VINS-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics, 2018, 34(4): 1004-1020.

4. CHURCHILL W, NEWMAN P. Experience-based navigation for long-term localisation. The International Journal of Robotics Research, 2013, 32(14): 1645-1661.

5. LINEGAR C, CHURCHILL W, NEWMAN P. Work smart, not hard: Recalling relevant experiences for vast-scale but time-constrained localisation. Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA’15), 2015, May 26-30, Seattle, WA, USA. Piscataway, NJ, USA: IEEE, 2015: 90-97.

6. PATON M, MACTAVISH K, WARREN M, et al. Bridging the appearance gap: Multi-experience localization for long-term visual teach and repeat. Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’16), 2016, Oct 9-14, Daejeon, Republic of Korea. Piscataway, NJ, USA: IEEE, 2016: 1918-1925.

7. PATON M, MACTAVISH K, BERCZI L P, et al. I can see for miles and miles: An extended field test of visual teach and repeat 2.0. Proceedings of the 11th Conference on Field and Service Robotics (FSR’17), 2017, Sept 12-15, Zurich, Switzerland. Berlin, Germany: Springer, 2018: 415-431.

8. DETONE D, MALISIEWICZ T, RABINOVICH A. SuperPoint: Self-supervised interest point detection and description. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW'18), 2018, Jun 18-22, Salt Lake City, UT, USA. Piscataway, NJ, USA: IEEE, 2018: 224-236.

9. SARLIN P E, CADENA C, SIEGWART R, et al. From coarse to fine: Robust hierarchical localization at large scale. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'19), 2019, Jun 15-20, Long Beach, CA, USA. Piscataway, NJ, USA: IEEE, 2019: 12716-12725.

10. LI D J, SHI X S, LONG Q W, et al. DXSLAM: A robust and efficient visual slam system with deep features. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’20), 2020, Oct 25-29, Las Vegas, NV, USA. Piscataway, NJ, USA: IEEE, 2020: 4958-4965.

11. ARANDJELOVIĆ R, GRONAT P, TORII A, et al. NetVLAD: CNN architecture for weakly supervised place recognition. Proceedings of the2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16), 2016, Jun 27-30, Las Vegas, NV, USA. Piscataway, NJ, USA: IEEE, 2016: 5297-5307.

12. TORII A, ARANDJELOVIĆ R, SIVIC J, et al. 24/7 place recognition by view synthesis. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15), 2015, Jun 7-12, Boston, MA, USA. Piscataway, NJ, USA: IEEE, 2015: 1808-1817.

13. WEYAND T, KOSTRIKOV I, PHILBIN J. PlaNet-photo geolocation with convolutional neural networks. Computer Vision: Proceedings of the 14th European Conference on Computer Vision (ECCV’16): Part VIII, 2016, Oct 11-14, Amsterdam, Netherlands. LNIP 9912. Berlin, Germany: Springer, 2016: 37-55.

14. SARLIN P E, DEBRAINE F, DYMCZYK M, et al. Leveraging deep visual descriptors for hierarchical efficient localization. Proceedings of the 2nd Conference on Robot Learning (CoRL’18), 2018, Oct 29-31, Zurich, Switzerland. 2018: 456-465.

15. CLEMENT L, KELLY J. How to train a CAT: Learning canonical appearance transformations for direct visual localization under illumination change. IEEE Robotics and Automation Letters, 2018, 3(3): 2447-2454.

16. CORKE P, PAUL R, CHURCHILL W, et al. Dealing with shadows: Capturing intrinsic scene appearance for image-based outdoor localisation. Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013, Nov 3-8, Tokyo, Japan. Piscataway, NJ, USA: IEEE, 2013: 2085-2092.

17. MCMANUS C, CHURCHILL W, MADDERN W, et al. Shady dealings: Robust, long-term visual localisation using illumination invariance. Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA'14), 2014, May 31-Jun 7, Hong Kong, China. Piscataway, NJ, USA: IEEE, 2014: 901-906.

18. CLEMENT L, GRIDSETH M, TOMASI J, et al. Learning matchable image transformations for long-term metric visual localization. IEEE Robotics and Automation Letters, 2020, 5(2): 1492-1499.

19. RUBLEE E, RABAUD V, KONOLIGE K, et al. ORB: An efficient alternative to SIFT or SURF. Proceedings of the 2011 International Conference on Computer vision, 2011, Nov 6-13, Barcelona, Spain. Piscataway, NJ, USA: IEEE, 2011: 2564-2571.

20. SATTLER T, MADDERN W, TOFT C, et al. Benchmarking 6DOF outdoor visual localization in changing conditions. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'18), 2018, Jun 18-23, Salt Lake City, UT, USA. Piscataway, NJ, USA: IEEE, 2018: 8601-8610.

21. TAIRA H, OKUTOMI M, SATTLER T, et al. InLoc: Indoor visual localization with dense matching and view synthesis. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'18), 2018, Jun 18-23, Salt Lake City, UT, USA. Piscataway, NJ, USA: IEEE, 2018: 7199-7209.

22. RATNASINGAM S, COLLINS S. Study of the photodetector characteristics of a camera for color constancy in natural scenes. Journal of the Optical Society of America A, 2010, 27(2): 286-294.

23. HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'16), 2016, Jun 27-30, Las Vegas, NV, USA. Piscataway, NJ, USA: IEEE, 2016: 770-778.

24. IOFFE S, SZEGEDY C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning (ICML’15), 2015, Jul 6-11, Lille, France. 2015: 448-456.

25. HE K M, ZHANG X Y, REN S Q, et al. Delving deep into rectifiers: Surpassing human-level performance on imageNet classification. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV'15), 2015, Dec 7-13, Santiago, Chile. Piscataway, NJ, USA: IEEE, 2015: 1026-1034.

26. ROSTEN E, DRUMMOND T. Machine learning for high-speed corner detection. Proceedings of the 9th European Conference on Computer Vision (ECCV‘06): Part I, 2006, May 7-13, Graz, Austria. LNIP 3951. Berlin, Germany: Springer, 2006: 430-443.

27. LOWE D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91-110.

28. HARRIS C G, STEPHENS M. A combined corner and edge detector. Proceedings of the 1988 Alvey Vision Conference (AVC’88), 1988, Aug 31-Sept 2, Manchester, UK. 1988: 6p.

29. CLEMENT L, KELLY J, BARFOOT T D. Robust monocular visual teach and repeat aided by local ground planarity and color-constant imagery. Journal of Field Robotics, 2017, 34(1): 74-97.

30. PATON M, POMERLEAU F, MACTAVISH K, et al. Expanding the limits of vision-based localization for long-term route-following autonomy. Journal of Field Robotics, 2017, 34(1): 98-122.

31. GAO X, ZHANG T, LIU Y, et al. 14 lectures on visual SLAM: From theory to practice. Beijing, China: Publishing House of Electronics Industry, 2017 (in Chinese).

32. GAIDON A, WANG Q, CABON Y, et al. Virtual worlds as proxy for multi-object tracking analysis. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'16), 2016, Jun 27-30, Las Vegas, NV, USA. Piscataway, NJ, USA: IEEE, 2016: 4340-4349.

33. STURM J, ENGELHARD N, ENDRES F, et al. A benchmark for the evaluation of RGB-D SLAM systems. Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012, Oct 7-12, Vilamoura, Portugal. Piscataway, NJ, USA: IEEE, 2012: 573-580.

34. SHI X S, LI D J, ZHAO P P, et al. Are we ready for service robots? The OpenLORIS-scene datasets for lifelong SLAM. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA’20), 2020, May 31-Jun 15, Paris, France. Piscataway, NJ, USA: IEEE, 2020: 3139-3145.

35. GEIGER A, ZIEGLER J, STILLER C. Stereoscan: Dense 3D reconstruction in real-time. Proceedings of the 2011 IEEE Intelligent Vehicles Symposium (IV’11), 2011, Jun 5-9, Baden-Baden, Germany. Piscataway, NJ, USA: IEEE, 2011: 963-968.

36. YU C, LIU Z X, LIU X J, et al. DS-SLAM: A semantic visual SLAM towards dynamic environments. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’18), 2018, Oct 1-5, Madrid, Spain. Piscataway, NJ, USA: IEEE, 2018: 1168-1174.

Metrics

Comments

Copyright © 2020 The Journal of China Universities of Posts and Telecommunications
　 Adress: P.O. Box 231,Beijing University of Posts and Telecommunications,10 Xi Tucheng Road,Beijing 100876,P.R.China　Post Code: 100081
Tel：86-010-62282493　Fax： 86-010-62283461　E-mail: jchupt@bupt.edu.cn
Support by: Beijing Magtech Co.Ltd

[1]	Zhang Guangwei, Zhao Bing, Li Ruifan. Multi-level fusion with deep neural networks for multimodal sentiment classification [J]. The Journal of China Universities of Posts and Telecommunications, 2022, 29(3): 25-33.
[2]	Yang Chao, Li Yimin, Li Tong, Xu Siya, Qi Jun, Zhang Yu. Intelligent Service Function Chain Mapping Framework for Cloud-and-Edge-Collaborative IoT [J]. The Journal of China Universities of Posts and Telecommunications, 2022, 29(3): 54-68.
[3]	Yang Jingjing, Guo Yuchun, Feng Tingting, Chen Yishuai. User Friendly Preferential Private Recommendation [J]. The Journal of China Universities of Posts and Telecommunications, 2022, 29(3): 43-53.
[4]	Shao Zihao, Qu Tianguang, Wang Huiqiang, Zou Yifan, Lv Hongwu. User selection based on user-union and relative entropy in mobile crowdsensing [J]. The Journal of China Universities of Posts and Telecommunications, 2022, 29(3): 34-42.
[5]	Miao Jiansong, Li Hairui, Zheng Ziyuan, Wang Chu, Zhao Zhenmin. Secrecy Energy Efficiency Maximization for UAV-Enabled Multi-Hop Mobile Relay System [J]. The Journal of China Universities of Posts and Telecommunications, 2022, 29(3): 81-91.
[6]	Tang Fei, Dong Kun, Ye Zhangtao, Ling Guowei. Authentication scheme for industrial Internet of things based on DAG blockchain [J]. The Journal of China Universities of Posts and Telecommunications, 2021, 28(6): 1-12.