2.765

2022影响因子

(CJCR)

  • 中文核心
  • EI
  • 中国科技核心
  • Scopus
  • CSCD
  • 英国科学文摘

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于RCNN-LSTM的脑电情感识别研究

柳长源 李文强 毕晓君

柳长源, 李文强, 毕晓君. 基于RCNN-LSTM的脑电情感识别研究. 自动化学报, 2022, 48(3): 917−925 doi: 10.16383/j.aas.c190357
引用本文: 柳长源, 李文强, 毕晓君. 基于RCNN-LSTM的脑电情感识别研究. 自动化学报, 2022, 48(3): 917−925 doi: 10.16383/j.aas.c190357
Liu Chang-Yuan, Li Wen-Qiang, Bi Xiao-Jun. Research on EEG emotion recognition based on RCNN-LSTM. Acta Automatica Sinica, 2022, 48(3): 917−925 doi: 10.16383/j.aas.c190357
Citation: Liu Chang-Yuan, Li Wen-Qiang, Bi Xiao-Jun. Research on EEG emotion recognition based on RCNN-LSTM. Acta Automatica Sinica, 2022, 48(3): 917−925 doi: 10.16383/j.aas.c190357

基于RCNN-LSTM的脑电情感识别研究

doi: 10.16383/j.aas.c190357
基金项目: 国家自然科学基金(5177090001)资助
详细信息
    作者简介:

    柳长源:哈尔滨理工大学测控技术与通信工程学院副教授. 主要研究方向为机器学习方法研究与改进, 智能优化算法及脑电信号智能诊断技术. 本文通信作者.E-mail: liuchangyuan@hrbust.edu.cn

    李文强:哈尔滨理工大学测控技术与通信工程学院硕士研究生. 主要研究方向为深度学习与情感识别. E-mail: lwqpost@163.com

    毕晓君:中央民族大学信息工程学院教授, 中国人工智能学会自然计算专委会成员, 黑龙江省生物医学工程学会常务副理事长. 主要研究方向为信息智能处理技术, 深度学习及智能优化算法.E-mail: bixiaojun@hrbeu.edu.cn

Research on EEG Emotion Recognition Based on RCNN-LSTM

Funds: Supported by National Natural Science Foundation of China (5177090001)
More Information
    Author Bio:

    LIU Chang-Yuan Associate professor at the College of Measurement and Control Technology and Communication Engineering, Harbin University of Science and Technology. His research interest covers research and improvement of machine learning methods, intelligent optimization algorithms, and EEG intelligent diagnosis technology. Corresponding author of this paper

    LI Wen-Qiang Master student at the College of Measurement and Control Technology and Communication Engineering, Harbin University of Science and Technology. His research interest covers deep learning and emotion recognition

    BI Xiao-Jun Professor at the School of Information Engineering, Minzu University of China, member of the Natural Computing Committee of Chinese Artificial Intelligence Society, and Executive Vice Chairman of Heilongjiang Biomedical Engineering Society. Her research interest covers information intelligent processing technology, deep learning, and intelligent optimization algorithm

  • 摘要: 情感作为人脑的高级功能, 对人们的个性特征和心理健康有很大的影响, 利用网上公开的脑电情感数据库(DEAP (Database for emotion analysis using physiological signals)数据库), 根据心理效价和激励唤醒度等级进行情感划分, 对压力和平静等5种情感进行研究分析. 针对脑电信号时空特征结合的特点, 把深度学习中的卷积神经网络(Convolutional neural network, CNN)和长短期记忆网络(Long short term memory, LSTM)两者作为基本前提, 并在此基础之上设计了一个RCNN-LSTM的脑电情感信号分类模型. 利用循环卷积神经网络(Recurrent convolutional neural network, RCNN)自动提取脑电信号中的抽象特征, 省去了人工选择与降维的过程, 然后结合LSTM网络对脑电情感信号进行分类识别. 实验结果表明, 利用该方法对5种情感类别的平均分类识别率达到了96.63%, 证明了该方法的有效性.
  • 图  1  一个含有内部环的RNN

    Fig.  1  An RNN with an inner ring

    图  2  RCNN结构分布图

    Fig.  2  RCNN structure distribution

    图  3  RCL展开图

    Fig.  3  RCL expansion diagram

    图  4  LSTM结构示意图

    Fig.  4  LSTM structure diagram

    图  5  RCNN-LSTM结构示意图

    Fig.  5  RNCN-LSTM structure diagram

    图  6  GA-SVM的适应度曲线

    Fig.  6  GA-SVM fitness curve

    表  1  RCNN网络模型结构

    Table  1  RCNN network model structure

    网络层 参数设置
    输入层 输入结构: (数据长度, 1)
    标准卷积层 核数量: 90; 大小: 3; 步长: 1
    最大池化层 核大小: 4 步长: 2
    RCL层 核数量: 90; 大小: 3; 步长: 1
    最大池化层 核大小: 4 步长: 2
    RCL层 核数量: 90; 大小: 3; 步长: 1
    最大池化层 核大小: 4 步长: 2
    RCL层 核数量: 90; 大小: 3; 步长: 1
    最大池化层 核大小: 4 步长: 2
    RCL层 核数量: 90; 大小: 3; 步长: 1
    最大池化层 核大小: 4 步长: 2
    下载: 导出CSV

    表  2  DEAP数据收集统计表

    Table  2  DEAP data collection statistics

    顺序 内容
    1 把电极放在适当的实验位置, 通过铃声来通知开始实验
    2 接着调整基线约 2 min, 直到符合要求 (主要目的就是为了使
    受试人员能够保持放松)
    3 显示 2 s 的实验编号来提示具体的进程
    4 收集 3 s 的基线数据信息
    5 播放 1 min 的音乐, 并且详细地记录这个时间段的数据信息
    6 对收集到的数据信息进行统计整理, 并且对各个相应的指标进
    行全面评估
    下载: 导出CSV

    表  3  不同算法下的召回率和分类识别率(%)

    Table  3  Classification recognition rate under differrnt algorithms (%)

    轻松 沮丧 快乐 压力 平静 总体识别率
    RCNN + LSTM 100 96.6 95.7 96.5 94.3 96.63
    CNN + LSTM 79.2 77.1 86.1 92.0 85.4 83.83
    CNN 72.8 76.3 82.6 84.9 81.0 79.46
    LSTM 73.6 73.7 77.4 81.4 86.2 78.45
    SVM 68.0 75.4 83.4 75.2 77.2 75.92
    BP 76.0 72.8 75.6 84.1 74.7 76.59
    下载: 导出CSV

    表  4  不同算法下的kappa值和方差

    Table  4  Kappa values and variances under different algorithms

    kappa 系数 方差 (×10−2)
    RCNN + LSTM 0.928 0.04
    CNN + LSTM 0.763 0.06
    CNN 0.70 0.12
    LSTM 0.718 0.16
    SVM 0.658 0.27
    BP 0.675 0.16
    下载: 导出CSV

    表  5  已有研究成果与本文的对比

    Table  5  The existing research results are compared with our method

    方法 分类类别 识别率 (%)
    SVM 2 70.1
    KNN 2 77.8
    CNN + RNN 2 73.09
    SAE + RNN 4 79.26
    DGCNN 3 90.4
    3D-CNN 2 87.965
    本文RCNN-LSTM 5 96.63
    下载: 导出CSV
  • [1] 耿雪青, 佘青山, 韩笑, 孟明. 基于人工蜂群优化高斯过程的运动想象脑电信号分类. 传感技术学报, 2017, 30(03): 378-384 doi: 10.3969/j.issn.1004-1699.2017.03.008

    Geng Xue-Qing, She Qing-Shan, Han Xiao, Meng Ming. Classification of motor imagery EEG based on gaussian process optimized with artificial bee colony. Chinese Journal of Sensors and Actuators, 2017, 30(03): 378-384 doi: 10.3969/j.issn.1004-1699.2017.03.008
    [2] 王薇蓉, 张雪英, 孙颖, 畅江. 关于脑电信号的情感识别优化仿真. 计算机仿真, 2018, 35(6): 426-431 doi: 10.3969/j.issn.1006-9348.2018.06.093

    Wang Wei-Rong, Zhang Xue-Ying, Sun Ying, Chang Jiang. Emotion optimization identification emulation for EEG signal. Computer Simulation, 2018, 35(6): 426-431 doi: 10.3969/j.issn.1006-9348.2018.06.093
    [3] Krisnandhika B, Faqih A, Pumamasari P D, Kusumoputro B. Emotion recognition system based on EEG signals using relative wavelet energy features and a modified radial basis function neural networks. In: Proceedings of the 2017 International Conference on Consumer Electronics and Devices (ICCED). London, UK: IEEE, 2017. 50−54
    [4] Duan R N, Zhu J Y, Lu B L. Differential entropy feature for EEG-based emotion classification. In: Proceedings of the 6th Internation IEEE EMBS Conference on Neural Engineering (NER). San Diego, California, USA: IEEE, 2013. 81−84
    [5] Murugappan M, Nagarajan R, Yaacob S. Combining spatial filtering and wavelet transform for classifying human emotions using EEG signals. Journal of Medical and Biological Engineering, 2011, 31(1): 45-51 doi: 10.5405/jmbe.710
    [6] 黄柠檬. 基于EEG的情绪识别 [硕士学位论文]. 华南理工大学, 中国, 2016.

    Huang Ning-Meng. Emotion Recognition Based on EEG [Master thesis]. South China University of Technology, China, 2016.
    [7] 杨默涵, 陈万忠, 李明阳. 基于总体经验模态分解的多类特征的运动想象脑电识别方法研究. 自动化学报, 2017, 43(05): 743-752

    Yang Mo-Han, Chen Wan-Zhong, Li Ming-Yang, Multiple feature extraction based on ensemble empirical mode decomposition for motor imagery EEG recognition tasks. Acta Automatica Sinica, 2017, 43(05): 743-752
    [8] Zhang T, Chen W Z, Li M Y. AR based quadratic feature extraction in the VMD domain for the automated seizure detection of EEG using random forest classifier. Biomedical Signal Processing and Control, 2017, 100(31): 550-559
    [9] 孙颖, 马江河, 张雪英. 结合非线性全局特征和谱特征的脑电情感识别. 计算机工程与应用, 2018, 54(17): 116-121 doi: 10.3778/j.issn.1002-8331.1803-0027

    Sun Ying, Ma Jiang-He, Zhang Xue-Ying. EEG emotion recognition based on nonlinear global features and spectral feature. Computer Engineering and Applications, 2018, 54(17): 116-121 doi: 10.3778/j.issn.1002-8331.1803-0027
    [10] 吴志勇, 丁香乾, 许晓伟, 鞠传香. 基于深度学习和模糊C均值的心电信号分类方法. 自动化学报, 2018, 44(10): 1913-1920

    Wu Zhi-Yong, Ding Xiang-Qian, Xu Xiao-Wei, Ju Chuan-Xiang. A method for ECG classification using deep learning and fuzzy C-means. Acta Automatic Sinica, 2018, 44(10): 1913-1920
    [11] Jirayucharoensak S, Setha P N, Pasin I. EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation [Online], available: https://doi.org/10.1155/2014/627892, March 25 2020
    [12] 杨豪, 张俊然, 蒋小梅, 刘飞. 基于深度信念网络脑电信号表征情绪状态的识别研究. 生物医学工程学杂志, 2018, 35(02): 182-190

    Yang Hao, Zhang Jun-Ran, Jiang Xiao-Mei, Liu Fei. Research of electroencephalography representational emotion recognition based on deep belief networks. Journal of Biomedical Engineering, 2018, 35(02): 182-190
    [13] Cheng C L, Wei X W, Zhou J. Emotion recognition algorithm based on convolution neural network. In: Proceedings of the 12th International Conference on Intelligent Systems and Knowledge Engineering (ISKE). Nanjing, China: IEEE, 2017. 1−5
    [14] Wang Y, Huang Z Y, Mccane B, Neo P. EmotioNet: A 3-D convolutional neural network for EEG-based emotion recognition. In: Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN). Rio de Janeiro, Brazil: IEEE, 2018. 1−7
    [15] 阚威, 李云. 基于LSTM的脑电情绪识别模型[J]. 南京大学学报(自然科学), 2019, 55(01): 110-116

    Kan Wei, Li Yun. Emotion recognition from EEG signals by using LSTM recurrent neural networks. Journal of Nanjing University(Natural Science), 2019, 55(01): 110-116
    [16] 李勇, 林小竹, 蒋梦莹. 基于跨连接LeNet-5网络的面部表情识别. 自动化学报, 2018, 44(1): 176-182

    Li Yong, Lin Xiao-Zhu, Jiang Meng-Ying. Facial expression recognition with cross-connect LeNet-5 network. Acta Automatica Sinica, 2018, 44(1): 176-182
    [17] Dutta K K. Multi-class time series classification of EEG signals with recurrent neural networks. In: Proceedings of the 9th International Conference on Cloud Computing, Data Science and Engineering (Confluence). Noida, India: IEEE, 2019. 337−341
    [18] Liang M, Hu X L. Recurrent convolutional neural network for object recognition. In: Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA: IEEE, 2015. 3367−3375
    [19] 汤鹏杰, 王瀚漓, 许恺晟. LSTM逐层多目标优化及多层概率融合的图像描述. 自动化学报, 2018, 44(7): 1237-1249

    Tang Peng-Jie, Wang Han-Li, Xu Kai-Sheng. Multi-objective layer-wise optimization and multi-level probability fusion for image description generation using LSTM. Acta Automatica Sinica, 2018, 44(7): 1237-1249.
    [20] Koelstra S, Muhl C, Soleymani M, Lee J S, Yazdani A, Ebrahimi T, et al. DEAP: a database for emotion analysis; using physiological signals. IEEE Transactions on Affective Computing, 2012, 3(1): 18-31. doi: 10.1109/T-AFFC.2011.15
    [21] 陈明. 基于脑电信号的情绪识别 [硕士学位论文], 杭州电子科技大学, 中国, 2017.

    Chen Ming. EEG-based Emotion Recognition [Master thesis], Hangzhou Dianzi University, China, 2017.
    [22] 谢佳利. 多尺度熵算法在情感脑电识别中的应用 [硕士学位论文], 燕山大学, 中国, 2016.

    Xie Jia-Li. Multi-Scale Entropy Algorithm and Its Application in Emotion Recognition [Master thesis], Yanshan University, China, 2016.
    [23] 李霞, 卢官明, 闫静杰, 张正言. 多模态维度情感预测综述. 自动化学报, 2018, 44(12): 2142-2159

    Li Xia, Lu Guan-Ming, Yan Jing-Jie, Zhang Zheng-Yan. A survey of dimensional emotion prediction by multimodal cues. Acta Automatica Sinica, 2018, 44(12): 2142-2159
    [24] Bastos-Filho T F, Ferreira A, Atencio A C, Argunan S, Kumar D. Evaluation of feature extraction techniques in emotional state recognition. In: Proceedings of the 4th International Conference on Intelligent Human Computer Interaction (IHCI). Kharagpur, India: IEEE, 2012. 1−6
    [25] 苏建新. 基于脑电信号的情绪识别研究 [硕士学位论文], 南京邮电大学, 中国, 2015.

    Su Jian-Xin. Emotion Recognition Research Based on EEG Signal [Master thesis]. Nanjing University of Posts and Telecommunications, China, 2015.
    [26] Li X, Song D W, Zhang P, Yu G L, Hou Y X, Hu B. Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. In: Proceedings of the 2016 IEEE International Conference on Bioinformatics and Bio-medicine (BIBM). Shenzhen, China: IEEE, 2016. 352−359
    [27] 李幼军, 黄佳进, 王海渊, 钟宁. 基于SAE 和LSTM RNN的多模态生理信号融合和情感识别研究. 通信学报, 2017, 38(12): 109-120 doi: 10.11959/j.issn.1000-436x.2017294

    Li You-Jun, Huang Jia-Jin, Wang Hai-Yuan, Zhong Ning. Study of emotion recognition based on fusion multi-modal bio-signal with SAE and LSTM recurrent neural network. Journal on Communications, 2017, 38(12): 109-120 doi: 10.11959/j.issn.1000-436x.2017294
    [28] Song T F, Zheng W M, Song P, Cui Z. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing, 2020, 11(3): 532−541
    [29] Shawky E, El-Khoribi R, Shoman M, Wahby Shalaby M. EEG-Based Emotion Recognition using 3D Convolutional Neural Networks. International Journal of Advanced Computer Science and Applications, 2018, 9(8): 329-337
  • 加载中
图(6) / 表(5)
计量
  • 文章访问数:  1734
  • HTML全文浏览量:  650
  • PDF下载量:  404
  • 被引次数: 0
出版历程
  • 收稿日期:  2019-05-12
  • 录用日期:  2020-03-25
  • 网络出版日期:  2022-02-16
  • 刊出日期:  2022-03-25

目录

    /

    返回文章
    返回