Design and Application of Continuous Deep Belief Network
-
摘要: 针对深度信念网(Deep belief network, DBN)学习连续数据时预测精度较差问题, 提出一种双隐层连续型深度信念网. 该网络首先对输入数据进行无监督训练, 利用连续型传递函数实现数据特征提取, 设计基于对比分歧算法的权值训练方法, 并通过误差反传对隐层权值进行局部寻优, 给出稳定性分析, 保证训练输出结果稳定在规定区域. 利用 Lorenz 混沌序列、CATS 序列和大气 CO2 预测实验对该网络进行测试, 结果表明, 连续型深度信念网具有结构精简、 收敛速度快、 预测精度高等优点.Abstract: A continuous deep belief network (cDBN) with two hidden layers is proposed to solve the problem of low accuracy of traditional DBN in modeling continuous data. The whole process is to train the input data in an unsupervised way using continuous version of transfer function, to design the contrastive divergence in hidden-layer training process, and then to fine-tune the net by back propagation. Besides, hyper-parameters are analyzed according to stability analysis, as is given in the paper, to make sure the network finds the optimal. Experiments on Lorenz, CATS benchmark simulation and CO2 forecasting show a simplified structure, fast convergence speed and accuracy of this cDBN.
-
Key words:
- Deep learning /
- neural networks /
- structural design /
- stability /
- time series forecasting
-
[1] Hinton G E, Salakhutdinov R R. Reducing the dimensionality of data with neural networks. Science, 2006, 313(5786): 504-507 [2] Hinton G E, Osindero S, Teh Y W. A fast learning algorithm for deep belief nets. Neural Computation, 2006, 18(7): 1527-1554 [3] Deselaers T, Hasan S, Bender O, Ney H. A deep learning approach to machine transliteration. In: Proceedings of the 4th EACL Workshop on Statistical Machine Translation. Athens, Greece: Association for Computational Linguistics, 2009. 233-241 [4] Rumelhart D E, Hinton G E, Williams R J. Learning representations by back-propagating errors. Nature, 1986, 323(6088): 533-536 [5] Bengio Y, Lamblin P, Popovici D, Larochelle H. Greedy layer-wise training of deep networks. In: Proceedings of the 20th Advances in Neural Information Processing Systems. Vancouver, British Columbia, Canada: NIPS, 2007. 153-160 [6] Arel I, Rose D C, Karnowski T P. Deep machine learninga new frontier in Artificial intelligence research. IEEE Computational Intelligence Magazine, 2010, 5(4): 13-18 [7] Dahl G E, Dong Y, Deng L, Acero A. Large vocabulary continuous speech recognition with context-dependent DBN-HMMS. In: Proceedings of the 2011 IEEE International Conference on Acoustics, Speech and Signal Processing. Prague: IEEE, 2011. 4688-4691 [8] Fasel I, Berry J. Deep belief networks for real-time extraction of tongue contours from ultrasound during speech. In: Proceedings of the 20th International Conference on Pattern Recognition. Istanbul: IEEE, 2010. 1493-1496 [9] Zhang S L, Bao Y B, Zhou P, Jiang H, Dai L R. Improving deep neural networks for LVCSR using dropout and shrinking structure. In: Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Florence: IEEE, 2014. 6849-6853 [10] Bengio Y. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2009, 2(1): 1-127 [11] Wei Q L, Wang F Y, Liu D R, Yang X. Finite-approximation-error-based discrete-time iterative adaptive dynamic programming. IEEE Transactions on Cybernetics, 2014, 44(12): 2820-2833 [12] Liu D R, Wei Q L. Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems. IEEE Transactions on Neural Networks and Learning Systems, 2014, 25(3): 621-34 [13] Zhang H G, Qin C B, Jiang B, Luo Y H. Online adaptive policy learning algorithm for H∞ state feedback control of unknown affine nonlinear discrete-time systems. IEEE Transactions on Cybernetics, 2014, 44(12): 2706-2718 [14] Schmidhuber J. Deep learning in neural networks: an overview. Neural Nerwork, 2015, 61: 85-117 [15] Längkvist M, Karlsso L, Loutfi A. A review of unsupervised feature learning and deep learning for time-series modeling. Pattern Recognition Letters, 2014, 42: 11-24 [16] Taylor G M, Hinton G E, Roweis S. Modeling human motion using binary latent variables. In: Proceedings of the 20th Advances in Neural Information Processing Systems. Vancouver, British Columbia, Canada: NIPS, 2007. 1345-1352 [17] Chen H, Murray A F. A continuous restricted Boltzmann machine with an implementable training algorithm. IEEE Proceedings Vision, Image and Signal Processing, 2003, 150(3): 153-158 [18] Zhang R, Shen F R, Zhao J X. A model with fuzzy granulation and deep belief networks for exchange rate forecasting. In: Proceedings of the 2014 International Joint Conference on Neural Networks (IJCNN). Beijing, China: IEEE, 2014. 366-373 [19] Chen J F, Jin Q J, Chao J. Design of deep belief networks for short-term prediction of drought index using data in the Huaihe river basin. Mathematical Problems in Engineering, 2012, 2012: 235929 [20] Song R Z, Xiao W D, Zhang H G, Sun C Y. Adaptive dynamic programming for a class of complex-valued nonlinear systems. IEEE Transactions on Neural Networks and Learning Systems, 2014, 25(9): 1733-1739 [21] Song R Z, Lewis F, Wei Q L, Zhang H G, Jiang Z P, Levine D. Multiple actor-critic structures for continuous-time optimal control using input-output data. IEEE Transactions on Neural Networks and Learning Systems, 2015, 26(4): 851-865 [22] Hinton G E. Training products of experts by minimizing contrastive divergence. Neural Computation, 2002, 14(8): 1771-1800 [23] Bergstra J, Bengio Y. Random search for hyper-parameter optimization. The Journal of Machine Learning Research, 2012, 13(1): 281-305 [24] Wu Zhi-Wei, Chai Tian-You, Wu Yong-Jian. A hybrid prediction model of energy consumption per ton for fused magnesia. Acta Automatica Sinica, 2013, 39(12): 2002-2011(吴志伟, 柴天佑, 吴永建. 电熔镁砂产品单吨能耗混合预报模型. 自动化学报, 2013, 39(12): 2002-2011) [25] Patan K. Stability analysis and the stabilization of a class of discrete-time dynamic neural networks. IEEE Transactions on Neural Networks, 2007, 18(3): 660-673 [26] Crone S F, Nikolopoulos K. Results of the NN3 neural network forecasting competition. In: Proceedings of the 27th International Symposium on Forecasting Program. New York, USA: ISF, 2007. 1-129 [27] Chen Q L, Chai W, Qiao J F. A stable online self-constructing recurrent neural network. Advances in Neural Networks --ISNN 2011. Berlin Heidelberg: Springer, 2011, 6677: 122-131 [28] Chang L C, Chen P A, Chang F J. Reinforced two-step-ahead weight adjustment technique for online training of recurrent neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2012, 23(8): 1269-1278 [29] Zhang G P. Time series forecasting using a hybrid ARIMA and neural network model. Neurocomputing, 2003, 50: 159-175 [30] Lmaury A, Oja E, Simula O, Verleysen M. Time series prediction competition: the CATS benchmark. In: Proceedings of IJCNN'2004 --International Joint Conference on Neural Networks. Budapest: IJCNN, 2004. 1615-1620
点击查看大图
计量
- 文章访问数: 2075
- HTML全文浏览量: 129
- PDF下载量: 1762
- 被引次数: 0