An Improved Echo State Network via L1-Norm Regularization
-
摘要: 针对回声状态网络存在的病态解以及模型规模控制问题,本文提出一种基于L1范数正则化的改进回声状态网络.该方法通过在目标函数中添加L1范数惩罚项,提高模型求解的数值稳定性,同时借助于L1范数正则化的特征选择能力,控制网络的复杂程度,防止出现过拟合.对于L1范数正则化的求解,采用最小角回归算法计算正则化路径,通过贝叶斯信息准则进行模型选择,避免估计正则化参数.将模型应用于人造数据和实际数据的时间序列预测中,仿真结果证明了本文方法的有效性和实用性.Abstract: Considering the ill-posed problem and the model scale control of echo state network, an improved echo state network based on L1-norm regularization is proposed. In order to improve the numerical stability, the proposed method adds an L1-norm penalty term in the objective function. Meanwhile, the method can also control the complexity of the network and prevent overfitting by using feature selection capability of L1-norm regularization. To solve the L1-norm regularization model, we adopt the least angle regression algorithm to calculate regularization path and select suitable model through Bayesian information criterion, which can avoid the estimations of regularization parameter. The model is applied to the time series predictions of both synthetic dataset and practical dataset. The simulation results show the effectiveness and practicality of the proposed method.
-
[1] Jaeger H, Hass H. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science, 2004, 304(5667): 78-80 [2] Qiao Jun-Fei, Bo Ying-Chun, Han Guang. Application of ESN-based multi indices dual heuristic dynamic programming on wastewater treatment process. Acta Automatica Sinica, 2013, 39(7): 1146-1151 (乔俊飞, 薄迎春, 韩广. 基于ESN的多指标DHP控制策略在污水处理过程中的应用. 自动化学报, 2013, 39(7): 1146-1151) [3] Ongenae F, Van Looy S, Verstraeten D, Verplancke T, Benoit D, De Turck F, Dhaene T, Schrauwen B, Decruyenaere J. Time series classification for the prediction of dialysis in critically ill patients using echo state networks. Engineering Applications of Artificial Intelligence, 2013, 26(3): 984-996 [4] Li G Q, Niu P F, Zhang W P, Zhang Y. Control of discrete chaotic systems based on echo state network modeling with an adaptive noise canceler. Knowledge-Based Systems, 2012, 35: 35-40 [5] Peng Yu, Wang Jian-Min, Peng Xi-Yuan. Researches on time series prediction with echo state networks. Acta Electronica Sinica, 2010, 38(2A): 148-154 (彭宇, 王建民, 彭喜元. 基于回声状态网络的时间序列预测方法研究. 电子学报, 2010, 38(2A): 148-154) [6] Lukosevicius M, Jaeger H. Reservoir computing approaches to recurrent neural network training. Computer Science Review, 2009, 3(3): 127-149 [7] Rong H J, Ong Y S, Tan A H, Zhu Z. A fast pruned-extreme learning machine for classification problem. Neurocomputing, 2008, 72(1-3): 359-366 [8] Dutoit X, Schrauwen B, Van Campenhout J, Stroobandt D, Van Brussel H, Nuttin M. Pruning and regularization in reservoir computing. Neurocomputing, 2009, 72(7-9): 1534-1546 [9] Miche Y, Sorjamaa A, Bas P, Simula O, Jutten C, Lendasse A. OP-ELM: optimally pruned extreme learning machine. IEEE Transactions on Neural Networks, 2010, 21(1): 158-162 [10] Kump P, Bai E W, Chan K S, Eichinger B, Li K. Variable selection via RIVAL (removing irrelevant variables amidst LASSO iterations) and its application to nuclear material detection. Automatica, 2012, 48(9): 2107-2115 [11] Liu Qiao, Qin Zhi-Guang, Chen Wei, Zhang Feng-Li. Zero-norm penalized feature selection support vector machine. Acta Automatica Sinica, 2011, 37(2): 252-256 (刘峤, 秦志光, 陈伟, 张凤荔. 基于零范数特征选择的支持向量机模型. 自动化学报, 2011, 37(2): 252-256) [12] Han Min, Li De-Cai. An norm 1 regularization term ELM algorithm based on surrogate function and Bayesian framework. Acta Automatica Sinica, 2011, 37(11): 1344-1350 (韩敏, 李德才. 基于替代函数及贝叶斯框架的1范数ELM算法. 自动化学报, 2011, 37(11): 1344-1350) [13] Liu Jian-Wei, Li Shuang-Cheng, Luo Xiong-Lin. Classification algorithm of support vector machine via p-norm regularization. Acta Automatica Sinica, 2012, 38(1): 76-87 (刘建伟, 李双成, 罗雄麟. p范数正则化支持向量机分类算法. 自动化学报, 2012, 38(1): 76-87) [14] Miche Y, Van Heeswijk M, Bas P, Simula O, Lendasse A. TROP-ELM: a double-regularized ELM using LARS and Tikhonov regularization. Neurocomputing, 2011, 74(16): 2413-2421 [15] Friedman J H. Fast sparse regression and classification. International Journal of Forecasting, 2012, 28(3): 722-738 [16] Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 1996, 58(1): 267-288 [17] Peng Yi-Gang, Suo Jin-Li, Dai Qiong-Hai, Xu Wen-Li. From compressed sensing to low-rank matrix recovery: theory and applications. Acta Automatica Sinica, 2013, 39(7): 981-994(彭义刚, 索津莉, 戴琼海, 徐文立. 从压缩传感到低秩矩阵恢复: 理论与应用. 自动化学报, 2013, 39(7): 981-994) [18] Efron B, Hastie T, Johnstone I, Tibshirani R. Least angle regression. The Annals of Statistics, 2004, 32(2): 407-499 [19] Stoica P, Selen Y. Model-order selection: a review of information criterion rules. IEEE Signal Processing Magazine, 2004, 21(4): 36-47 [20] Watanabe S. A widely applicable Bayesian information criterion. Journal of Machine Learning Research, 2013, 14(1): 867-897 [21] Wu C L, Chau K W. Prediction of rainfall time series using modular soft computing methods. Engineering Applications of Artificial Intelligence, 2013, 26(3): 997-1007 [22] Box G E P, Jenkins G M, Reinsel G C. Time Series Analysis: Forecasting and Control. New Jersey, USA: John Wiley & Sons, 2008. 677-678
点击查看大图
计量
- 文章访问数: 1868
- HTML全文浏览量: 64
- PDF下载量: 1022
- 被引次数: 0