Progressive framework for deep neural networks:from linear to non-linear

doi:10.1016/S1005-8885(16)60063-8

JOURNAL OF CHINA UNIVERSITIES OF POSTS AND TELECOM ›› 2016, Vol. 23 ›› Issue (6): 1-7.doi: 10.1016/S1005-8885(16)60063-8

Received:2016-09-08 Revised:2016-12-21 Online:2016-12-31 Published:2016-12-30

Abstract

Abstract: We propose a novel progressive framework to optimize deep neural networks. The idea is to try to combine the stability of linear methods and the ability of learning complex and abstract internal representations of deep learning methods. We insert a linear loss layer between the input layer and the first hidden non-linear layer of a traditional deep model. The loss objective for optimization is a weighted sum of linear loss of the added new layer and non-linear loss of the last output layer. We modify the model structure of deep canonical correlation analysis (DCCA), i.e., adding a third semantic view to regularize text and image pairs and embedding the structure into our framework, for cross-modal retrieval tasks such as text-to-image search and image-to-text search. The experimental results show the performance of the modified model is better than similar state-of-art approaches on a dataset of National University of Singapore (NUS-WIDE). To validate the generalization ability of our framework, we apply our framework to RankNet, a ranking model optimized by stochastic gradient descent. Our method outperforms RankNet and converges more quickly, which indicates our progressive framework could provide a better and faster solution for deep neural networks.

Key words: framework, neural network, DCCA, semantic, RankNet

References

1. Chua T, Tang J, Hong R, et al. NUS-WIDE: a real-world web image database from National University of Singapore. Proceedings of the 8th ACM International Conference on Image and Video Retrieval, Jul 8-10, 2009, Santorini, Greece. New York, NY, USA: ACM, 2009: 48-56

2. Burges C J, Shaked T, Renshaw E, et al. Learning to rank using gradient descent. Proceedings of the 22nd International Conference on Machine Learning (ICML’05), Aug, 2005, Bonn, Germany. New York, NY, USA: ACM, 2005: 89-96

3. Rasiwasia N, Pereira J C, Coviello E, et al. A new approach to cross-modal multimedia retrieval. Proceedings of the 18th ACM International Conference on Multimedia (MM’10), Oct 25-29, 2010, Firenze, Italy. New York, NY, USA: ACM, 2010: 251-260

4. Akaho S. A kernel method for canonical correlation analysis. Proceedings of the International Meeting of the Psychometric Society (IMPS’01), Jul 15-19, 2001, Osaka, Japan. arXiv preprint cs/0609071, 2007, 40(2): 263-269

5. Andrew G, Arora R, Bilmes J, et al. Deep canonical correlation analysis. Proceedings of the 30th International Conference on Machine Learning (ICML’13), Jun 16-21, 2013, Atlanta, GA, USA. New York, NY, USA: ACM, 2013: 1247-1255

6. Srivastava N, Salakhutdinov R. Learning representations for multimodal data with deep belief nets. Proceedings of the 29th International Conference on Machine Learning Workshop (ICML’12), Jun 26-Jul 1, 2012, Scotland, UK. New York, NY, USA: ACM, 2012: 8p

7. Welling M, Rosen-Zvi M, Hinton G E. Exponential family harmoniums with an application to information retrieval. Proceedings of the Advances in Neural Information Processing Systems (NIPS’04), Dec 13-18, 2004, Vancouver, Canada. 2004: 1481-1488

8. Hinton G E, Salakhutdinov R R. Replicated softmax: an undirected topic model. Proceedings of the Advances in Neural Information Processing Systems (NIPS’09), Dec 7-10, 2009, Vancouver, Canada. 2009: 1607-1614

9. Feng F X, Wang X J, Li R F. Cross-modal retrieval with correspondence autoencoder. Proceedings of the 22nd ACM International Conference on Multimedia (MM’14), Nov 3-7, 2014, Orlando, FL, USA. New York, NY, USA: ACM, 2014: 7-16

10. Burges C J C. From ranknet to lambdarank to lambdamart: an overview. Learning, 2010, 11: 23-581

11. Cao Z, Qin T, Liu T Y, et al. Learning to rank: from pairwise approach to listwise approach. Proceedings of the 24th International Conference on Machine Learning (ICML’07), Jun 20-24, 2007, Corvallis, OR, USA. New York, NY, USA: ACM, 2007: 129-136

12. Taylor M, Guiver J, Robertson S, et al. SoftRank: optimizing non-smooth rank metrics. Proceedings of the International Conference on Web Search and Data Mining, Feb 11-12, 2008, Palo Alto, CA, USA. 2008: 77-86

13. Vincent P, Larochelle H, Bengio Y, et al. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine Learning (ICML’08), Jun 5-9, 2008, Helsinki, Finland. New York, NY, USA: ACM, 2008: 1096-1103

14. Nocedal J, Wright S J. Numerical optimization. 2nd ed. New York, NY, USA: Springer, 2006

15. Ngiam J, Khosla A, Kim M, et al. Multimodal deep learning. Proceedings of the 28th International Conference on Machine Learning (ICML’11), Jun 28-Jul 2, 2011, Bellevue, WA, USA. New York, NY, USA: ACM, 2011: 689-696

16. van der Maaten L, Hinton G. Visualizing data using t-SNE. Journal of Machine Learning Research, 2008, 9: 2579-2605

Metrics

Comments

Copyright © 2020 The Journal of China Universities of Posts and Telecommunications
　 Adress: P.O. Box 231,Beijing University of Posts and Telecommunications,10 Xi Tucheng Road,Beijing 100876,P.R.China　Post Code: 100081
Tel：86-010-62282493　Fax： 86-010-62283461　E-mail: jchupt@bupt.edu.cn
Support by: Beijing Magtech Co.Ltd