2.765

2022影响因子

(CJCR)

  • 中文核心
  • EI
  • 中国科技核心
  • Scopus
  • CSCD
  • 英国科学文摘

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

面向非独立同分布数据的自适应联邦深度学习算法

张泽辉 李庆丹 富瑶 何宁昕 高铁杠

张泽辉, 李庆丹, 富瑶, 何宁昕, 高铁杠. 面向非独立同分布数据的自适应联邦深度学习算法. 自动化学报, 2023, 49(12): 2493−2506 doi: 10.16383/j.aas.c201018
引用本文: 张泽辉, 李庆丹, 富瑶, 何宁昕, 高铁杠. 面向非独立同分布数据的自适应联邦深度学习算法. 自动化学报, 2023, 49(12): 2493−2506 doi: 10.16383/j.aas.c201018
Zhang Ze-Hui, Li Qing-Dan, Fu Yao, He Ning-Xin, Gao Tie-Gang. Adaptive federated deep learning with non-IID data. Acta Automatica Sinica, 2023, 49(12): 2493−2506 doi: 10.16383/j.aas.c201018
Citation: Zhang Ze-Hui, Li Qing-Dan, Fu Yao, He Ning-Xin, Gao Tie-Gang. Adaptive federated deep learning with non-IID data. Acta Automatica Sinica, 2023, 49(12): 2493−2506 doi: 10.16383/j.aas.c201018

面向非独立同分布数据的自适应联邦深度学习算法

doi: 10.16383/j.aas.c201018
基金项目: 国家科技重大专项基金(2018YFB0204304), 天津市研究生科研创新基金(2019YJSB067)资助
详细信息
    作者简介:

    张泽辉:南开大学软件学院博士研究生. 2019年获得武汉理工大学硕士学位. 主要研究方向为联邦学习, 故障诊断和智能船舶控制. E-mail: zhangtianxia918@163.com

    李庆丹:南开大学软件学院硕士研究生. 主要研究方向为图像加密, 信息安全. E-mail: lqd18812745024@163.com

    富瑶:南开大学软件学院硕士研究生. 主要研究方向为云端数据完整性验证, 信息安全. E-mail: fuyao_tj@163.com

    何宁昕:南开大学软件学院硕士研究生. 2020年获得河北经贸大学学士学位. 主要研究方向为信息安全, 联邦学习. E-mail: ningxinhe1998@163.com

    高铁杠:南开大学软件学院教授. 1991年获华中理工大学应用数学专业硕士学位, 2005年获南开大学博士学位. 主要研究方向为联邦学习, 图像水印, 信息隐藏和云端数据安全. 本文通信作者. E-mail: gaotiegang@nankai.edu.cn

Adaptive Federated Deep Learning With Non-IID Data

Funds: Supported by National Science and Technology Major Project of China (2018YFB0204304) and Tianjin Research Innovation Project for Postgraduate Students (2019YJSB067)
More Information
    Author Bio:

    ZHANG Ze-Hui Ph.D. candidate at the College of Software, Nankai University. He received his master degree from Wuhan University of Technology in 2019. His research interest covers federated learning, fault diagnosis and intelligent ship control

    LI Qing-Dan Master student at the College of Software, Nankai University. Her research interest covers image encryption and information security

    FU Yao Master student at the College of Software, Nankai University. Her research interest covers cloud data integrity verification and information security

    HE Ning-Xin Master student at the College of Software, Nankai University. She received her bachelor degree from Hebei University of Economics and Business in 2020. Her research interest covers information security and federated learning

    GAO Tie-Gang Professor at the College of Software, Nankai University. He received his master degree in applied mathematics from Huazhong University of Science and Technology in 1991, and Ph.D. degree from Nankai University in 2005. His research interest covers federated learning, image watermarking, information hiding and cloud data security. Corresponding author of this paper

  • 摘要: 近些年, 联邦学习(Federated learning, FL)由于能够打破数据壁垒, 实现孤岛数据价值变现, 受到了工业界和学术界的广泛关注. 然而, 在实际工程应用中, 联邦学习存在着数据隐私泄露和模型性能损失的问题. 为此, 首先对这两个问题进行数学描述与分析. 然后, 提出一种自适应模型聚合方案, 该方案能够设定各参与者的Mini-batch值和自适应调整全局模型聚合间隔, 旨在保证模型精度的同时, 提高联邦学习训练效率. 并且, 混沌系统被首次引入联邦学习领域中, 用于构建一种基于混沌系统和同态加密的混合隐私保护方案, 从而进一步提升系统的隐私保护水平. 理论分析与实验结果表明, 提出的联邦学习算法能够保证参与者的数据隐私安全. 并且, 在非独立同分布数据的场景下, 该算法能够在保证模型精度的前提下提高训练效率, 降低系统通信成本, 具备实际工业场景应用的可行性.
  • 图  1  多层神经网络模型

    Fig.  1  Multi-layer neural network model

    图  2  不同比例数据泄露的图片

    Fig.  2  Images of different proportion data leakage

    图  3  本文所提出的联邦学习系统结构图

    Fig.  3  The structure diagram of the proposed federated learning system

    图  4  联邦学习训练过程交互图

    Fig.  4  Interaction diagram of the federated learning system

    图  5  加密参数的推理数据图

    Fig.  5  Inferring data of the encrypted parameters

    图  6  在CIFAR10上不同联邦学习模型的实验曲线

    Fig.  6  Experiment curves of the different federated learning models on CIFAR10

    图  7  在F-MNIST上不同联邦学习模型的实验曲线

    Fig.  7  Experiment curves of the different federated learning models on F-MNIST

    图  8  在CIFAR10和F-MNIST数据集的Mini-batch设定消融实验曲线

    Fig.  8  Experiment curves of the Mini-batch size setting on CIFAR10 and F-MNIST

    图  9  CIFAR10自适应模型更新间隔消融实验曲线

    Fig.  9  Experiment curves of the adaptive model aggregation interval on CIFAR10

    图  10  F-MNIST自适应模型更新间隔消融实验曲线

    Fig.  10  Experiment curves of the adaptive model aggregation interval on F-MNIST

    表  1  加密/解密算法的执行时间

    Table  1  Execution time of the encryption/ decryption operations

    操作类型500个参数2000个参数54000个参数
    随机数生成12.05 ms25.50 ms0.40 s
    CKKS 加密9.37 ms9.68 ms0.54 s
    CKKS 解密1.56 ms17.18 ms0.03 s
    CKKS 密文加法0.15 ms0.15 ms0.02 s
    Paillier 加密3.82 s14.61 s410.32 s
    Paillier 解密1.06 s4.22 s115.92 s
    Paillier 密文加法7.87 ms30.03 ms0.87 s
    下载: 导出CSV

    表  2  加密/解密算法的执行次数

    Table  2  Execution numbers of the encryption/ decryption operations

    模型更新间隔50次80次100次
    $\tau=1$[18]78001250015600
    $\tau=4$[16]155024803100
    $\tau=15$[33]5008001000
    下载: 导出CSV

    表  3  不同联邦学习方案的功能分析

    Table  3  The functionality analysis of the different FLs

    功能PFLAFLMFLAPFL
    隐私保护××
    自适应调整$\tau $××
    Mini-batch 设定×××
    动量项加速××
    下载: 导出CSV

    表  4  CIFAR10上不同联邦学习模型的分类结果(%)

    Table  4  Classification results of the different federated learning models on CIFAR10 (%)

    方法准确率精准率召回率Devavg聚合次数
    CL63.3663.9263.29
    $ {\rm{FL} }\; (\tau= 15) $[33]25.769.3425.8749.91250
    ${\rm{FL} }\; (\tau= 4)$[16]27.6450.1427.7645.041100
    $ {\rm{FL} }\; (\tau= 1) $[18]61.7862.7661.771.914400
    APFL63.6663.4963.642.022758
    下载: 导出CSV

    表  5  F-MNIST上不同联邦学习模型的分类结果(%)

    Table  5  Classification results of the different federated learning models on F-MNIST (%)

    方法准确率精准率召回率Devavg聚合次数
    CL90.1590.0790.15
    ${\rm{FL} }\; (\tau= 15)$[33]65.9962.1865.9931.43350
    $ {\rm{FL} }\; (\tau= 4) $[16]72.7765.2472.7723.161350
    ${\rm{FL} }\; (\tau= 1)$[18]89.1089.2589.100.885250
    APFL89.3689.3089.360.872339
    下载: 导出CSV

    表  6  CIFAR10下的Mini-batch设定消融实验结果(%)

    Table  6  Ablation experiment results of the Mini-batch size setting on CIFAR10 (%)

    方法AccuracyPrecisionRecallDevavg
    CL63.3663.9263.29
    ${\rm{FL} }\; (\tau= 15)$[33]25.769.3425.8749.91
    ${\rm{FL} }\; (\tau= 15)+{\rm{mbs} }$25.709.1425.7850.07
    ${\rm{FL} }\;(\tau= 4)$27.6450.1427.7645.04
    ${\rm{FL} }\;(\tau= 4)+{\rm{mbs} }$63.6660.9336.0632.90
    ${\rm{FL} }\;(\tau= 1)$[18]61.7862.7661.771.91
    ${\rm{FL} }\;(\tau= 1)+{\rm{mbs} }$63.0264.0862.271.53
    下载: 导出CSV

    表  7  F-MNIST下的Mini-batch设定消融实验结果(%)

    Table  7  Ablation experiment results of the Mini-batch size setting on F-MNIST (%)

    方法AccuracyPrecisionRecallDevavg
    CL90.1590.0790.15
    ${\rm{FL} }\; (\tau= 15)$[33]65.9962.1865.9931.43
    ${\rm{FL} }\; (\tau= 15)+{\rm{mbs} }$69.9964.2969.9926.05
    ${\rm{FL} }\;(\tau= 4)$27.7650.1427.7645.04
    ${\rm{FL} }\;(\tau= 4)+{\rm{mbs} }$76.2384.8476.2314.85
    ${\rm{FL} }\;(\tau= 1)$[18]89.1089.2589.100.88
    ${\rm{FL} }\;(\tau= 1)+{\rm{mbs} }$89.2789.2589.270.99
    下载: 导出CSV

    表  8  CIFAR10下的自适应更新间隔消融实验结果(%)

    Table  8  Ablation experiment results of the adaptive model aggregation interval on CIFAR10 (%)

    方法AccuracyPrecisionRecallDevavg聚合次数
    CL63.3663.9263.29
    ${\rm{FL} }\; (\tau= 15) $[33]25.769.3425.8749.91250
    ${\rm{FL} }\; (\tau= 4) $[16]27.6450.1427.7645.041100
    ${\rm{FL} }\; (\tau= 1)$[18]61.7862.7661.771.914000
    APFL (no mbs)61.1062.0061.363.271742
    下载: 导出CSV

    表  9  F-MNIST下的自适应更新间隔消融实验结果(%)

    Table  9  Ablation experiment results of the adaptive model aggregation interval on F-MNIST (%)

    方法AccuracyPrecisionRecallDevavg聚合次数
    CL90.1590.0790.15
    ${\rm{FL} }\; (\tau= 15) $[33]65.9962.1865.9931.43250
    ${\rm{FL} }\; (\tau= 4) $[16]72.7765.2472.7723.161100
    ${\rm{FL} }\; (\tau= 1) $[18]89.1089.2589.100.884400
    APFL (no mbs)89.4889.4289.480.841336
    下载: 导出CSV
  • [1] 孙长银, 穆朝絮. 多智能体深度强化学习的若干关键科学问题. 自动化学报, 2020, 46(7): 1301−1312

    Sun Chang-Yin, Mu Chao-Xu. Important scientific problems of multi-agent deep reinforcement learning. Acta Automatica Sinica, 2020, 46(7): 1301−1312
    [2] 金侠挺, 王耀南, 张辉, 刘理, 钟杭, 贺振东. 基于贝叶斯CNN和注意力网络的钢轨表面缺陷检测系统. 自动化学报, 2019, 45(12): 2312−2327

    Jin Xia-Ting, Wang Yao-Nan, Zhang Hui, Liu Li, Zhong Hang, He Zhen-Dong. DeepRail: Automatic visual detection system for railway surface defect using Bayesian CNN and attention Network. Acta Automatica Sinica, 2019, 45(12): 2312−2327
    [3] Zhang Z H, Guan C, Liu Z Y. Real-time optimization energy management strategy for fuel cell hybrid ships considering power sources degradation. IEEE Access, 2020, 8: 87046−87059 doi: 10.1109/ACCESS.2020.2991519
    [4] Chen H, Zhang Z H, Guan C, Gao H B. Optimization of sizing and frequency control in battery/supercapacitor hybrid energy storage system for fuel cell ship. Energy, 2020, 197: 117285 doi: 10.1016/j.energy.2020.117285
    [5] 鲜征征, 李启良, 黄晓宇, 吕威, 陆寄远. 基于差分隐私和SVD++的协同过滤算法. 控制与决策, 2019, 34(1): 43−54

    Xian Zheng-Zheng, Li Qi-Liang, Huang Xiao-Yu, Lv Wei, Lu Ji-Yuan. Collaborative filtering via SVD++ with differential privacy. Control and Decision, 2019, 34(1): 43−54
    [6] Li J, Kuang X H, Lin S J, Ma X, Tang Y. Privacy preservation for machine learning training and classification based on homomorphic encryption schemes. Information Sciences, 2020, 526: 166−179 doi: 10.1016/j.ins.2020.03.041
    [7] Gong M G, Pan K, Xie Y, Qin A K, Tang Z D. Preserving differential privacy in deep neural networks with relevance-based adaptive noise imposition. Neural Networks, 2020, 125: 131−141 doi: 10.1016/j.neunet.2020.02.001
    [8] 张超, 李强, 陈子豪, 黎祖睿, 张震. Medical Chain: 联盟式医疗区块链系统. 自动化学报, 2019, 45(8): 1495−1510

    Zhang Chao, Li Qiang, Chen Zi-Hao, Li Zu-Rui, Zhang Zhen. Medical Chain: Alliance medical blockchain system. Acta Automatica Sinica, 2019, 45(8): 1495−1510
    [9] Yang Q, Liu Y, Chen T J, Tong Y X. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology, 2019, 10(2): Article No. 12
    [10] Li T, Sahu A K, Talwalkar A, Smith V. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 2020, 37(3): 50−60 doi: 10.1109/MSP.2020.2975749
    [11] Zhang W, Li X, Ma H, Luo Z, Li X. Federated learning for machinery fault diagnosis with dynamic validation and self-supervision. Knowledge-Based Systems, 2021, 213: Article No. 106679 doi: 10.1016/j.knosys.2020.106679
    [12] Sheller M J, Edwards B, Reina G A, Martin J, Pati S, Kotrotsou A, et al. Federated learning in medicine: Facilitating multi-institutional collaborations without sharing patient data. Scientific Reports, 2020, 10(1): Article No. 12598 doi: 10.1038/s41598-020-69250-1
    [13] Kwon D, Jeon J, Park S, Kim J, Cho S. Multiagent DDPG-based deep learning for Smart Ocean federated learning IoT networks. IEEE Internet of Things Journal, 2020, 7(10): 9895−9903 doi: 10.1109/JIOT.2020.2988033
    [14] Rothchild D, Panda A, Ullah E, Ivkin N, Stoica I, Braverman V, et al. FetchSGD: Communication-efficient federated learning with sketching. In: Proceedings of the 37th International Conference on Machine Learning. Vienna, Austria: JMLR.org, 2020. Article No. 764
    [15] Duan M M, Liu D, Chen X Z, Liu R P, Tan Y J, Liang L. Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems, 2021, 32(1): 59−71 doi: 10.1109/TPDS.2020.3009406
    [16] Liu W, Chen L, Chen Y F, Zhang W Y. Accelerating federated learning via momentum gradient descent. IEEE Transactions on Parallel and Distributed Systems, 2020, 31(8): 1754−1766 doi: 10.1109/TPDS.2020.2975189
    [17] Wang S Q, Tuor T, Salonidis T, Leung K K, Makaya C, He T, et al. Adaptive federated learning in resource constrained edge computing systems. IEEE Journal on Selected Areas in Communications, 2019, 37(6): 1205−1221 doi: 10.1109/JSAC.2019.2904348
    [18] Li Q B, Wen Z Y, Wu Z M, Hu S X, Wang N B, Li Y, et al. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(4): 3347−3366
    [19] 张泽辉, 富瑶, 高铁杠. 支持数据隐私保护的联邦深度神经网络模型研究. 自动化学报, 2022, 48(5): 1273−1284

    Zhang Ze-Hui, Fu Yao, Gao Tie-Gang. Research on federated deep neural network model for data privacy preserving. Acta Automatica Sinica, 2022, 48(5): 1273−1284
    [20] Lyu L J, Li Y T, Nandakumar K, Yu J S, Ma X J. How to democratise and protect AI: Fair and differentially private decentralised deep learning. IEEE Transactions on Dependable and Secure Computing, 2022, 19(2): 1003−1017
    [21] Wang Y F, Gu M, Ma J H, Jin Q. DNN-DP: Differential privacy enabled deep neural network learning framework for sensitive crowdsourcing data. IEEE Transactions on Computational Social Systems, 2020, 7(1): 215−224 doi: 10.1109/TCSS.2019.2950017
    [22] Carpov S, Gama N, Georgieva M, Troncoso-Pastoriza J R. Privacy-preserving semi-parallel logistic regression training with fully homomorphic encryption. BMC Medical Genomics, 2019, 13(7): Article No. 88
    [23] 宋蕾, 马春光, 段广晗, 袁琪. 基于数据纵向分布的隐私保护逻辑回归. 计算机研究与发展, 2019, 56(10): 2243−2249 doi: 10.7544/issn1000-1239.2019.20190414

    Song Lei, Ma Chun-Guang, Duan Guang-Han, Yuan Qi. Privacy-preserving logistic regression on vertically partitioned data. Journal of Computer Research and Development, 2019, 56(10): 2243−2249 doi: 10.7544/issn1000-1239.2019.20190414
    [24] Phong L T, Aono Y, Hayashi T, Wang L H, Moriai S. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security, 2018, 13(5): 1333−1345 (本条文献与第18条文献重复, 请联系作者确认) doi: 10.1109/TIFS.2017.2787987
    [25] Ou W, Zeng J H, Guo Z J, Yan W Q, Liu D W, Fuentes S. A homomorphic-encryption-based vertical federated learning scheme for rick management. Computer Science and Information Systems, 2020, 17(3): 819−834 doi: 10.2298/CSIS190923022O
    [26] Chen H, Chillotti I, Song Y. Improved bootstrapping for approximate homomorphic encryption. In: Proceedings of the 38th Annual International Conference on the Theory and Applications of Cryptographic Techniques. Darmstadt, Germany: Springer, 2019. 34−54
    [27] Xiao X D, Wu T, Chen Y F, Fan X Y. Privacy-preserved approximate classification based on homomorphic encryption. Mathematical and Computational Applications, 2019, 24(4): Article No. 92 doi: 10.3390/mca24040092
    [28] Zhang Z H, Yao F, Gao T G. A hybrid image encryption algorithm based on chaos system and simplified advanced encryption system. International Journal of Multimedia Data Engineering and Management (IJMDEM), 2020, 11(4): Article No. 1 doi: 10.4018/IJMDEM.2020100101
    [29] Luo Y Q, Yu J, Lai W R, Liu L F. A novel chaotic image encryption algorithm based on improved baker map and logistic map. Multimedia Tools and Applications, 2019, 78(15): 22023−22043 doi: 10.1007/s11042-019-7453-3
    [30] Sathiyamurthi P, Ramakrishnan S. Speech encryption algorithm using FFT and 3D-Lorenz–logistic chaotic map. Multimedia Tools and Applications, 2020, 79(25): 17817−17835
    [31] Sattler F, Müller K, Samek W. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 2020, 32(8): 3710−3722
    [32] Al-Sharman M, Murdoch D, Cao D P, Lv C, Zweiri Y, Rayside D, et al. A sensorless state estimation for a safety-oriented cyber-physical system in urban driving: Deep learning approach. IEEE/CAA Journal of Automatica Sinica, 2021, 8(1): 169−178 doi: 10.1109/JAS.2020.1003474
    [33] Weng J S, Weng J, Zhang J L, Li M, Zhang Y, Luo W Q, et al. DeepChain: Auditable and privacy-preserving deep learning with blockchain-based incentive. IEEE Transactions on Dependable and Secure Computing, 2021, 18(5): 2438−2455
    [34] Sattler F, Wiedemann S, Müller K R, Samek W. Robust and communication-efficient federated learning from non-i.i.d. data. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(9): 3400−3413 doi: 10.1109/TNNLS.2019.2944481
    [35] Xu G W, Li H W, Zhang Y, Xu S M, Ning J T, Deng R H. Privacy-preserving federated deep learning with irregular users. IEEE Transactions on Dependable and Secure Computing, 2022, 19(2): 1364−1381
    [36] Teng S H, Wu N Q, Zhu H B, Zhang W. SVM-DT-based adaptive and collaborative intrusion detection. IEEE/CAA Journal of Automatica Sinica, 2018, 5(1): 108−118 doi: 10.1109/JAS.2017.7510730
  • 加载中
图(10) / 表(9)
计量
  • 文章访问数:  2749
  • HTML全文浏览量:  1812
  • PDF下载量:  412
  • 被引次数: 0
出版历程
  • 收稿日期:  2020-12-08
  • 录用日期:  2021-03-19
  • 网络出版日期:  2021-06-04
  • 刊出日期:  2023-12-27

目录

    /

    返回文章
    返回