2.765

2022影响因子

(CJCR)

  • 中文核心
  • EI
  • 中国科技核心
  • Scopus
  • CSCD
  • 英国科学文摘

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于MHSA和句法关系增强的机器阅读理解方法研究

张虎 王宇杰 谭红叶 李茹

张虎, 王宇杰, 谭红叶, 李茹. 基于MHSA和句法关系增强的机器阅读理解方法研究. 自动化学报, 2022, 48(11): 2718−2728 doi: 10.16383/j.aas.c200951
引用本文: 张虎, 王宇杰, 谭红叶, 李茹. 基于MHSA和句法关系增强的机器阅读理解方法研究. 自动化学报, 2022, 48(11): 2718−2728 doi: 10.16383/j.aas.c200951
Zhang Hu, Wang Yu-Jie, Tan Hong-Ye, Li Ru. Research on machine reading comprehension method based on MHSA and syntactic relations enhancement. Acta Automatica Sinica, 2022, 48(11): 2718−2728 doi: 10.16383/j.aas.c200951
Citation: Zhang Hu, Wang Yu-Jie, Tan Hong-Ye, Li Ru. Research on machine reading comprehension method based on MHSA and syntactic relations enhancement. Acta Automatica Sinica, 2022, 48(11): 2718−2728 doi: 10.16383/j.aas.c200951

基于MHSA和句法关系增强的机器阅读理解方法研究

doi: 10.16383/j.aas.c200951
基金项目: 国家重点研发计划(2018YFB1005103), 国家自然科学基金(62176145), 山西省自然科学基金(201901D111028)资助
详细信息
    作者简介:

    张虎:山西大学计算机与信息技术学院副教授. 2014 年于山西大学计算机与信息技术学院获得工学博士学位. 主要研究方向为人工智能与自然语言处理. 本文通信作者. E-mail: zhanghu@sxu.edu.cn

    王宇杰:山西大学计算机与信息技术学院博士研究生. 主要研究方向为自然语言处理. E-mail: init_wang@foxmail.com

    谭红叶:山西大学计算机与信息技术学院教授. 2008年于哈尔滨工业大学计算机学院获得博士学位. 主要研究方向为人工智能, 自然语言处理. E-mail: tanhongye@sxu.edu.cn

    李茹:山西大学计算机与信息技术学院教授. 2011年于山西大学计算机与信息技术学院获得工学博士学位. 主要研究方向为人工智能与自然语言处理. E-mail: liru@sxu.edu.cn

Research on Machine Reading Comprehension Method Based on MHSA and Syntactic Relations Enhancement

Funds: Supported by National Key Research and Development Program of China (2018YFB1005103), National Natural Science Foundation of China (62176145), and Natural Science Foundation of Shanxi Province (201901D111028)
More Information
    Author Bio:

    ZHANG Hu Associate professor at the School of Computer and Information Technology, Shanxi University. He received his Ph.D. degree from the School of Computer and Information Technology, Shanxi University in 2014. His research interest covers artificial intelligence and natural language processing. Corresponding author of this paper

    WANG Yu-Jie Ph.D. candidate at the School of Computer and Information Technology, Shanxi University. His main research interest is natural language processing

    TAN Hong-Ye Professor at the School of Computer and Information Technology, Shanxi University. She received her Ph.D. degree from the School of Computer, Harbin Institute of Technology in 2008. Her research interest covers artificial intelligence and natural language processing

    LI Ru Professor at the School of Computer and Information Technology, Shanxi University. She received her Ph.D. degree from the School of Computer and Information Technology, Shanxi University in 2011. Her research interest covers artificial intelligence and natural language processing

  • 摘要: 机器阅读理解 (Machine reading comprehension, MRC)是自然语言处理领域中一项重要研究任务, 其目标是通过机器理解给定的阅读材料和问题, 最终实现自动答题. 目前联合观点类问题解答和答案依据挖掘的多任务联合学习研究在机器阅读理解应用中受到广泛关注, 它可以同时给出问题答案和支撑答案的相关证据, 然而现有观点类问题的答题方法在答案线索识别上表现还不是太好, 已有答案依据挖掘方法仍不能较好捕获段落中词语之间的依存关系. 基于此, 引入多头自注意力(Multi-head self-attention, MHSA)进一步挖掘阅读材料中观点类问题的文字线索, 改进了观点类问题的自动解答方法; 将句法关系融入到图构建过程中, 提出了基于关联要素关系图的多跳推理方法, 实现了答案支撑句挖掘; 通过联合优化两个子任务, 构建了基于多任务联合学习的阅读理解模型. 在2020中国“法研杯”司法人工智能挑战赛(China AI Law Challenge 2020, CAIL2020)和HotpotQA数据集上的实验结果表明, 本文提出的方法比已有基线模型的效果更好.
    1)  1 https://github.com/baidu/lac
    2)  1 https://github.com/baidu/lac2 https://github.com/baidu/DDParser3 https://github.com/explosion/spaCy
    3)  3 https://github.com/explosion/spaCy
    4)  4 https://github.com/china-ai-law-challenge/CAIL2020/tree/master/ydlj5 https://github.com/neng245547874/cail2020-mrc6 https://github.com/hotpotqa/hotpot
    5)  5 https://github.com/neng245547874/cail2020-mrc
    6)  6 https://github.com/hotpotqa/hotpot
  • 图  1  CAIL2020阅读理解数据集样例

    Fig.  1  Sample of CAIL2020 MRC dataset

    图  2  MJL-model模型结构

    Fig.  2  Model architecture of MJL-model

    图  3  多跳推理层结构图

    Fig.  3  Model architecture of multi-hop reasoning layer

    图  4  注意力可视化样例

    Fig.  4  Sample of attention visualization

    图  5  关联要素关系图样例

    Fig.  5  Sample of related element graph

    图  6  多跳推理注意力可视化样例图

    Fig.  6  Visible sample of multi-hop reasoning attention

    表  1  CAIL2020数据集实验结果(%)

    Table  1  Results on the CAIL2020 dataset (%)

    模型Ans_F1Sup_F1Joint_F1
    Baseline_BERT 70.40 65.74 49.25
    Baseline_RoBERTa 71.81 71.11 55.74
    Baseline_DPCNN 77.43 75.07 61.80
    Cola 74.63 73.68 59.62
    DFGN_CAIL 68.79 72.34 53.82
    MJL-model 78.83 75.51 62.72
    下载: 导出CSV

    表  2  HotpotQA实验结果(%)

    Table  2  Results on the HotpotQA dataset (%)

    模型Ans_F1Sup_F1Joint_F1
    Baseline 58.28 66.66 40.86
    QFE 68.70 84.70 60.60
    DFGN 69.34 82.24 59.86
    SAE 74.81 85.27 66.45
    MJL-Model 70.92 85.96 62.87
    下载: 导出CSV

    表  3  消融实验结果(%)

    Table  3  Results of ablation experiments (%)

    模型Ans_F1Sup_F1Joint_F1
    MJL-model78.8375.5162.72
    Question_answering76.36
    Answer_evidence73.42
    –MHSA 76.28 75.11 61.16
    –RCNN 75.96 75.05 60.96
    –Syntax & Similarity 77.61 74.39 60.80
    下载: 导出CSV
  • [1] 曾帅, 王帅, 袁勇, 倪晓春, 欧阳永基. 面向知识自动化的自动问答研究进展. 自动化学报, 2017, 43(9): 1491-150.

    Zeng Shuai, Wang Shuai, Yuan Yong, Ni Xiao-Chun, Ouyang Yong-Ji. Towards knowledge automation: a survey on question answering systems. ACTA AUTOMATICA SINICA, 2017, 43(9): 1491-1508(in Chinese).
    [2] 奚雪峰, 周国栋. 面向自然语言处理的深度学习研究. 自动化学报, 2016, 42(10): 1445-1465.

    Xi Xue-Feng, Zhou Guo-Dong. A survey on deep learning for natural language processing. ACTA AUTOMATICA SINICA, 2016, 42(10): 1445-1465(in Chinese).
    [3] Devlin J, Chang M W, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis, USA: ACL, 2019. 4171−4186
    [4] Rajpurkar P, Zhang J, Lopyrev K, Liang P. SQUAD: 100 000+ questions for machine comprehension of text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, USA: ACL, 2016. 2383−2392
    [5] Rajpurkar P, Jia R, Liang P. Know what you don't know: Unanswerable questions for squad. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne, Australia: ACL, 2018.784−789
    [6] Reddy S, Chen D Q, Manning C D, CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 2019, 7: 249−266
    [7] Duan X Y, Wang B X, Wang Z Y, Ma W T, Cui Y M, Wu D Y, et al. CJRC: A reliable human-annotated benchmark dataset for Chinese judicial reading comprehension. In: Proceedings of the 2019 China National Conference on Chinese Computational Linguistics. Kunming, China: Springer, 2019. 439−451
    [8] Yang Z L, Qi P, Zhang S Z, Bengio Y, Cohen W W, Salakhutdinov R, et al. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: ACL, 2018. 2369−2380
    [9] Lai S W, Xu L H, Liu K, Zhao J. Recurrent convolutional neural networks for text classification. In: Proceedings of the 2015 AAAI Conference on Artificial Intelligence. Austin, USA: AAAI, 2015. 2267−2273
    [10] Xiao Y X, Qu Y R, Qiu L, Zhou H, Li L, Zhang W N, Yu Y. Dynamically fused graph network for multi-hop reasoning. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: ACL, 2019. 6140−6150
    [11] Lai G K, Xie Q Z, Liu H X, Yang Y M, Hovy E. RACE: Large-scale reading comprehension dataset from examinations. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark: ACL, 2017. 785−794
    [12] He W, Liu K, Liu J, Lv Y J, Zhao S Q, Xiao X Y, et al. Dureader: A Chinese machine reading comprehension dataset from real-world applications. In: Proceedings of the 2018 Workshop on Machine Reading for Question Answering. Melbourne, Australia: ACL, 2018. 37−46
    [13] Chen D Q, Bolton J, Manning C D. A thorough examination of the CNN/daily mail reading comprehension task. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Berlin, Germany: ACL, 2016. 2358−2376
    [14] Seo M, Kembhavi A, Farhadi A, Hajishirzi H. Bidirectional attention flow for machine comprehension. In: Proceedings of the 5th International Conference on Learning Representations. Toulon, France: ICLR, 2017.
    [15] Wang W H, Yang N, Wei F R, Chang B B, Zhou M. Gated self-matching networks for reading comprehension and question answering. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouver, Canada: ACL, 2017. 189−198
    [16] Yu A W, Dohan D, Luong M T. QANet: Combining local convolution with global self-attention for reading comprehension. In: Proceedings of the 6th International Conference on Learning Representations. Vancouver, Canada: ICLR, 2018.
    [17] Liu Y H, Ott M, Goyal N, Du J F, Joshi M, Chen D Q, et al. RoBERTa: A robustly optimized BERT pretraining approach. arXiv: 1907.11692, 2019.
    [18] Lan Z Z, Chen M D, Goodman S, Gimpel K, Sharma P, Soricut R. ALBERT: A lite BERT for self-supervised learning of language representations. In: Proceedings of the 8th International Conference on Learning Representations. Addis Ababa, Ethiopia: ICLR, 2020.
    [19] Sun Y, Wang S H, Li Y K, Feng S K, Chen X Y, Zhang H, et al. ERNIE: Enhanced representation through knowledge integration. arXiv: 1904.09223, 2019.
    [20] Cui Y M, Che W X, Liu T, Qin B, Yang Z Q. Pre-training with whole word masking for Chinese BERT. IEEE Transactions on Audio, Speech, and Language Processing, 2021, 29: 3504−3514
    [21] Ding M, Zhou C, Chen Q B, Yang H X, Tang J. Cognitive graph for multi-hop reading comprehension at scale. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: ACL, 2019. 2694−2703
    [22] Tu M, Wang G T, Huang J, Tang Y, He X D, Zhou B W. Multi-hop reading comprehension across multiple documents by reasoning over heterogeneous graphs. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: ACL, 2019. 2704−2713
    [23] Nishida K, Nishida K, Nagata M, Otsuka A, Saito I, Asano H, et al. Answering while summarizing: multi-task learning for multi-hop QA with evidence extraction. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: ACL, 2019. 2335−2345
    [24] Tu M, Huang K, Wang G T, Huang J, He X D, Zhou B W. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. In: Proceedings of the 32nd Innovative Applications of Artificial Intelligence Conference. New York, USA: AAAI, 2020. 9073−9080
    [25] Johnson R, Zhang T. Deep pyramid convolutional neural networks for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouver, Canada: ACL, 2017. 562−570
    [26] Pennington J, Socher R, Manning C D. GloVe: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Doha, Qatar: ACL, 2014. 1532−1543
    [27] 刘康, 张元哲, 纪国良, 来斯惟, 赵军. 基于表示学习的知识库问答研究进展与展望. 自动化学报, 2016, 42(6): 807-818.

    Liu Kang, Zhang Yuan-Zhe, Ji Guo-Liang, Lai Si-Wei, Zhao Jun. Representation learning for question answering over knowledge base: An Overview. ACTA AUTOMATICA SINICA, 2016, 42(6): 807-818(in Chinese).
  • 加载中
图(6) / 表(3)
计量
  • 文章访问数:  1363
  • HTML全文浏览量:  330
  • PDF下载量:  158
  • 被引次数: 0
出版历程
  • 收稿日期:  2020-11-16
  • 网络出版日期:  2021-05-25
  • 刊出日期:  2022-11-22

目录

    /

    返回文章
    返回