L2,1-norm robust regularized extreme learning machine for regression using CCCP method

doi:10.19682/j.cnki.1005-8885.2023.0004

中国邮电高校学报(英文版) ›› 2023, Vol. 30 ›› Issue (2): 61-72.doi: 10.19682/j.cnki.1005-8885.2023.0004

• Artificial Intelligence • 上一篇下一篇

L2,1-norm robust regularized extreme learning machine for regression using CCCP method

吴青¹,王凡¹,范九伦²,侯静¹

1. 西安邮电大学
2. 西安邮电学院信息与控制系手机 13002985988，029-85383403（办）

收稿日期:2022-03-31 修回日期:2022-11-07 出版日期:2023-04-30 发布日期:2023-04-27
通讯作者: 吴青 E-mail:xiyouwuq@126.com
基金资助:
中国国家自然科学基金;陕西省重点研究项目;陕西省自然科学基金;陕西省教育厅科研专项计划项目

L2,1-norm robust regularized extreme learning machine for regression using CCCP method

Wu Qing, Wang Fan, Fan Jiulun, Hou Jing

Received:2022-03-31 Revised:2022-11-07 Online:2023-04-30 Published:2023-04-27
Contact: Qing Wu E-mail:xiyouwuq@126.com
Supported by:
National Natural Science Foundation of China;Key Research Project of Shaanxi Province;Natural Science Foundation of Shaanxi Province of China;Special Scientific Research Plan Project of Shaanxi Province Education Department

摘要/Abstract

摘要：

As a way of training a single hidden layer feedforward network (SLFN),extreme learning machine (ELM) is rapidly becoming popular due to its efficiency. However, ELM tends to overfitting, which makes the model sensitive to noise and outliers. To solve this problem, L2,1-norm is introduced to ELM and an L2,1-norm robust regularized ELM (L2,1-RRELM) was proposed. L2,1-RRELM gives constant penalties to outliers to reduce their adverse effects by replacing least square loss function with a non-convex loss function. In light of the non-convex feature of L2,1-RRELM, the concave-convex procedure (CCCP) is applied to solve its model. The convergence of L2,1-RRELM is also given to show its robustness. In order to further verify the effectiveness of L2,1-RRELM, it is compared with the three popular extreme learning algorithms based on the artificial dataset and University of California Irvine (UCI) datasets. And each algorithm in different noise environments is tested with two evaluation criterions root mean square error (RMSE) and fitness. The results of the simulation indicate that L2,1-RRELM has smaller RMSE and greater fitness under different noise settings. Numerical analysis shows that L2,1-RRELM has better generalization performance, stronger robustness, and higher anti-noise ability and fitness.

关键词: extreme learning machine (ELM)| non-convex loss| L2,1 -norm| concave-convex procedure (CCCP)

Abstract:

Key words:

extreme learning machine (ELM)| non-convex loss| L2,1 -norm| concave-convex procedure (CCCP)

中图分类号:

TP181

Wu Qing, Wang Fan, Fan Jiulun, Hou Jing. L2,1-norm robust regularized extreme learning machine for regression using CCCP method [J]. The Journal of China Universities of Posts and Telecommunications, 2023, 30(2): 61-72.

参考文献

[1] ESHTAY M, FARI H, OBEID N. Improving extreme learning machine by competitive swarm optimization and its application for medical diagnosis problems. Expert Systems with Application, 2018, 104: 134 -152.

[2] LESHNO M, LIN V Y, PINKUS A, et al. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 1993, 6(6): 861 -867.

[3] HUANG G B, ZHU Q Y, SIEW C K. Extreme learning machine: theory and applications. Neurocomputing, 2005, 70 (1/2/3): 489 -501.

[4] HUANG G B, CHEN L, SIEW C K. Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Transactions on Neural Networks, 2006, 17(4): 879 -892.

[5] FENG G R, QIAN Z X, ZHANG X P. Evolutionary selection extreme learning machine optimization for regression. Soft Computing, 2012, 16 (9): 1485 -1491.

[6] XU J T, ZHOU H M, HUANG G B. Extreme learning machine based fast object recognition. Proceedings of the 15th International Conference on Information Fusion, 2012, Jul 9 - 12, Singapore. Piscataway, NJ, USA: IEEE, 2012: 1490 -1496.

[7] CAI W Q, NIAN R, HE B, et al. A fast sonar-based benthic object recognition model via extreme learning machine. Proceedings of the OCEANS 2015-MTS/ IEEE Washington, 2015, Oct 19 - 22, Washington, DC, USA. Piscataway, NJ, USA: IEEE, 2015: 1 -4.

[8] ERGUL U, BILHIN G. HCKBoost: hybridized composite kernel boosting with extreme learning machines for hyperspectral image classification. Neurocomputing, 2019, 334: 100 -113.

[9] ZHOU Y C, PENG J T, CHEN C L P. Extreme learning machine with composite kernels for hyperspectral image classification. IEEE Journal of Selected Topics in Applied Earth Observations & Remote Sensing, 2015, 8(6): 2351 -2360.

[10] YANG J, YU H L, YANG X B, et al. Imbalanced extreme learning machine based on probability density estimation. Proceedings of the 9th International Workshop on Multi- disciplinary Trends in Artificial Intelligence (MIWAI'15), 2015, Nov 13 - 15, Fuzhou, China. LNAI 9426. Berlin, Germany: Springer, 2015: 160 -167.

[11] GAO M, HONG X, CHEN S, et al. Probability density function estimation based over-sampling for imbalanced two-class problems. Proceedings of the 2012 International Joint Conference on Neural Networks (IJCNN'12), 2012, Jun 10 -15, Brisbane, Australia. Piscataway, NJ, USA: IEEE, 2012: 1 -8.

[12] LI X D, XIE H R, WANG R, et al. Empirical analysis: stock market prediction via extreme learning machine. Neural Computing and Applications, 2016, 27(1): 67 -78.

[13] XIONG Z B. Stock price prediction based on sparse Bayesian extreme learning machine. Journal of Jiaxing University, 2018, 30(5): 106 -113 (in Chinese).

[14] RUMELHART D E, HINTON G E, WILLIAMS R J. Learningrepresentations by back propagating errors. Nature, 1986, 323(6088): 533 -536.

[15] MICHE Y, SORJAMAA A, BAS P, et al. OP-ELM: optimally pruned extreme learning machine. IEEE Transactions on Neural Networks, 2010, 21(1): 158 -162.

[16] MICHE Y, VAN HEESWIJK M, BAS P, et al. TROP-ELM: a double-regularized ELM using LARS and Tikhonov regularization. Neurocomputing, 2011, 74(16): 2413 -2421.

[17] DENG W Y, ZHENG Q H, CHEN L. Regularized extreme learning machine. Proceedings of the 2009 IEEE Symposium on Computational Intelligence and Data Mining (CIDM'09), 2009, Mar 30 - Apr 2, Nashville, TN, USA. Piscataway, NJ, USA: IEEE, 2009: 389 -395.

[18] WANG Y B, LI D, DU Y, et al. Anomaly detection in traffic using L1-norm minimization extreme learning machine. Neurocomputing, 2015, 149(1): 415 -425.

[19] ZHOU S H, LIU X W, LIU Q, et al. Random Fourier extreme learning machine with L2,1 -norm regularization. Neurocomputing, 2016, 174(6): 143 -153.

[20] HASTIE T, TIBSHIRANI R, FRIEDMAN J. The elements of statistical learning: data mining, inference and prediction. Berlin, Germany: Springer, 2003.

[21] SHEN X, NIU L F, QI Z Q, et al. Support vector machine classifier with truncated pinball loss. Pattern Recognition, 2017, 68: 199 -210.

[22] WANG L, JIA H D, LI J. Training robust support vector machine with smooth Ramp loss in the primal space. Neurocomputing, 2008, 71(13/14/15): 3020 -3025.

[23] ZHAO Y P, SUN J G. Robust support vector regression in the primal. Neural Networks, 2008, 21(10): 1548 -1555.

[24] SINGH A, POKHAREL R, PRINCIPE J. The C-loss function for pattern classification, Pattern Recognition. 2014, 47(1): 441 - 453.

[25] ZHAO Y P, TAN J F, WANG J J, et al. C-loss based extreme learning machine for estimating power of small-scale turbojet engine. Aerospace Science and Technology, 2019, 89: 407 - 419.

[26] GUPTA D, HAZARIKA B B, BERLIN M. Robust regularized extreme learning machine with asymmetric Huber loss function. Neural Computing and Applications, 2020, 32(3/4): 12971 - 12998.

[27] BALASUNDARAM S, MEENA Y. Robust support vector regression in primal with asymmetric Huber loss. Neural Processing Letters, 2019, 49(3): 1399 -1431.

[28] WANG K N, ZHONG P. Robust non-convex least squares loss function for regression with outliers. Knowledge-Based Systems, 2014, 71: 290 -302.

[29] XU G B, HU B G, PRINCIPE J C. Robust C-Loss kernel classifiers. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29(3): 510 -522.

[30] YUILLE A L, RANGARAJAN A. The concave-convex procedure (CCCP). Neural Computation, 2003, 15(4): 915 -936.

[31] DEMSAR J. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 2006, 7(1): 1 - 30.

L2,1-norm robust regularized extreme learning machine for regression using CCCP method

L2,1-norm robust regularized extreme learning machine for regression using CCCP method

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 5

编辑推荐

Metrics

本文评价

[1]	Wu Qing, Li Feiyan, Zhang Hengchang, Fan Jiulun, Gao Xiaofeng. Least squares twin support vector machine with asymmetric squared loss[J]. 中国邮电高校学报(英文版), 2023, 30(1): 1-16.
[2]	Wu Qing, Fu Yanlin, Fan Jiulun, Ma Tianlu. Structural regularized twin support vector machine based on within-class scatter and between-class scatter[J]. 中国邮电高校学报(英文版), 2021, 28(4): 39-52.
[3]	何明枢, 金磊, 王小娟, 李源. Web log classification framework with data augmentation based on GANs[J]. 中国邮电高校学报(英文版), 2020, 27(5): 34-46.
[4]	李端张洪欣李强 Zhao Xinjie He Pengfei. Electromagnetic side_channel attack based on PSO Directed Acyclic Graph SVM[J]. Acta Metallurgica Sinica(English letters), 2015, 22(5): 10-15.
[5]	平源周亚建薛超杨义先. Efficient representation of text with multiple perspectives[J]. Acta Metallurgica Sinica(English letters), 2012, 19(1): 101-111.