An Effective Online Sparse Learning Algorithm for Echo State Networks
-
摘要: 为克服传统储备池方法缺乏良好在线学习算法的问题, 同时考虑到储备池本身存在的不适定问题, 本文提出一种储备池在线稀疏学习算法, 对储备池目标函数施加L1正则化约束,并采用截断梯度算法在线近似求解.所提算法在对储备池输出权值进行在线调整的同时, 可对储备池输出权值的稀疏性进行有效控制, 有效保证了网络的泛化性能.理论分析和仿真实例证明所提算法的有效性.Abstract: In order to overcome the lack of effective online learning method for echo state networks and to solve the ill-posed problem of reservoir, an effective online sparse learning algorithm is proposed for echo state networks in this paper. An L1 regularization constraint is added to the objective function of reservoir, and a truncated gradient algorithm is used to approximately solve the problem online. The proposed method can adjust the output weights of reservoir online, control the sparsity of the output weights, and ensure the generalization performance.Theoretical analysis and simulation results demonstrate the effectiveness of the algorithm.
-
Key words:
- Recurrent neural networks /
- echo state networks (ESNs) /
- sparse /
- online /
- optimization
-
[1] Jaeger H, Haas H. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science, 2004, 304(5667): 78-80[2] Maass W, Natschlager T, Markram H. Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Computation, 2002, 14(11): 2531-2560[3] Han Min, Wang Ya-Nan. Multivariate time series online predictor with Kalman filter trained reservoir. Acta Automatica Sinica, 2010, 36(1): 169-173 (韩敏, 王亚楠. 基于Kalman滤波的储备池多元时间序列在线预报器. 自动化学报, 2010, 36(1): 169-173)[4] Jaeger H, Maass W, Principe J. Special issue on echo state networks and liquid state machines. Neural Networks, 2007, 20(3): 287-289[5] Liu Ying, Zhao Jun, Wang Wei, Wu Yi-Ping, Chen Wei-Chang. Improved echo state network based on data-driven and its application to prediction of blast furnace gas output. Acta Automatica Sinica, 2009, 35(6): 731-738 (刘颖, 赵珺, 王伟, 吴毅平,陈伟昌. 基于数据的改进回声状态网络在高炉煤气发生量预测中的应用. 自动化学报, 2009, 35(6): 731-738)[6] Roseschies B, Igel C. Structure optimization of reservoir networks. Logic Journal of the IGPL, 2010, 18(5): 635-669[7] Shi Z W, Han M. Support vector echo-state machine for chaotic time-series prediction. IEEE Transactions on Neural Networks, 2007, 18(2): 359-372[8] Steil J J. Online stability of backpropagation-decorrela-tion recurrent learning. Neurocomputing, 2006,69(7-9): 642-650[9] Jaeger H. Reservoir riddles: suggestions for echo state network research. In: Proceedings of the IEEE International Joint Conference on Neural Networks. Montreal, Canada: IEEE, 2005. 1460-1462[10] Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B (Methodological), 1996, 58(1): 267-288[11] Shevade S K, Keerthi S S. A simple and efficient algorithm for gene selection using sparse logistic regression. Bioinformatics, 2003, 19(17): 2246-2253[12] Langford J, Li L, Zhang T. Sparse online learning via truncated gradient. The Journal of Machine Learning Research, 2009,10: 777-801[13] Donoho D L. For most large underdetermined systems of linear equations the minimal L1-norm solution is also the sparsest solution. Communications on Pure and Applied Mathematics, 2006,59(6): 797-829
点击查看大图
计量
- 文章访问数: 2330
- HTML全文浏览量: 59
- PDF下载量: 953
- 被引次数: 0