-
摘要: 近些年, 联邦学习由于能够打破数据壁垒, 实现孤岛数据价值变现, 受到了工业界和学术界的广泛关注. 然而, 在实际工程应用中, 联邦学习存在着数据隐私泄露和模型性能损失的问题. 对此, 首先对这两个问题进行了数学描述与分析. 然后, 提出了一种自适应模型聚合方案, 该方案能够设定各参与者的mini-batch值和自适应调整全局模型聚合间隔, 旨在保证模型精度的同时, 提高联邦学习训练效率. 并且, 混沌系统被首次引入联邦学习领域中, 用于构建一种基于混沌系统和同态加密的混合隐私保护方案, 从而进一步提升系统的隐私保护水平. 理论分析与实验结果表明, 提出的联邦学习算法能够保证参与者的数据隐私安全. 并且, 在非独立同分布数据的场景下, 该算法够在保证模型精度的前提下提高训练效率, 降低系统通信成本, 具备实际工业场景应用的可行性.Abstract: In recent years, federated learning that can break data barriers and realize the value of isolated data, has been received wide-spread attention from industry and academia. However, in real industry applications, federated learning has problems with privacy leakage and model accuracy loss, which is analyzed through mathematical demonstration in this study. To solve the issues, this paper proposes an adaptive global model aggregation scheme that can adaptively set the mini-batch value of each participant and the global model aggregation interval for the parameter server, which aims to improve the training efficiency while ensuring the accuracy of the model. Moreover, this paper introduces the chaos system into the federated learning field, which is used to construct a hybrid privacy protection scheme based on chaos system and homomorphic encryption, thereby further improving the privacy protection level. Theoretical analysis and experimental results show that the proposed approach can guarantee the data privacy security of participants. Moreover, in the non-independent and identically distributed (Non-IID) data scenario, the proposed method can improve the training efficiency and reduce communication cost while ensuring the model accuracy, which is feasible for real industrial applications.
-
Key words:
- Federated learning /
- deep learning /
- privacy-preserving /
- homomorphic encryption /
- chaos system
-
表 1 加密/解密算法的执行时间
Table 1 Execution time of the encryption/decryption operations
参数个数 500 2000 54000 随机数生成 12.05 ms 25.50 ms 0.40 s CKKS加密 9.37 ms 9.68 ms 0.54 s CKKS解密 1.56 ms 17.18 ms 0.03 s CKKS密文加法 0.15 ms 0.15 ms 0.02 s Paillier加密 3.82 s 14.61 s 410.32 s Paillier解密 1.06 s 4.22 s 115.92 s Paillier密文加法 7.87 ms 30.03 ms 0.87 s 表 2 加密/解密算法的执行次数
Table 2 Execution numbers of the encryption/decryption operations
表 3 不同联邦学习方案的功能对比
Table 3 The functionality analysis of the different FLs
功能 PFL AFL MFL APFL 隐私保护 √ × × √ 自适应调整$\tau $ × √ × √ mini-batch设定 × × × √ 动量项加速 × × √ √ 表 4 CIFAR10上不同联邦学习模型的分类结果
Table 4 Classification results of the different federated learning models on CIFAR10
表 5 F-MNIST上不同联邦学习模型的分类结果
Table 5 Classification results of the different federated learning models on F-MNIST
表 6 CIFAR-10下的mini-batch设定消融实验结果
Table 6 Ablation experiment results of the mini-batch size setting on CIFAR-10
表 7 F-MNIST下的mini-batch设定消融实验结果
Table 7 Ablation experiment results of the mini-batch size setting on F-MNIST
表 8 CIFAR-10下的自适应更新间隔消融实验结果
Table 8 The ablation experiment results of the adaptive model aggregation interval on CIFAR-10
表 9 F-MNIST下的自适应更新间隔消融实验结果
Table 9 Ablation experiment results of the adaptive model aggregation interval on F-MNIST
-
[1] 孙长银, 穆朝絮. 多智能体深度强化学习的若干关键科学问题. 自动化学报, 2020, 46(07): 1301-1312.Sun Chang-Yin, Mu Chao-Xu. Important scientific problems of multi-agent Deep Reinforcement Learning. Acta Automatica Sinica, 2020, 46(07): 1301-1312. [2] 金侠挺, 王耀南, 张辉, 等. 基于贝叶斯CNN和注意力网络的钢轨表面缺陷检测系统. 自动化学报, 2019, 45(12): 2312-2327.Jin Xia-Ting, Wang Yao-Nan, Zhang Hui, et al. DeepRail: automatic visual detection system for railway surface defect using bayesian CNN and attention Network. Acta Automatica Sinica, 2019, 45(12): 2312-2327. [3] Zhang Z, Guan C, Liu Z. Real-time optimization energy management strategy for fuel cell hybrid ships considering power sources degradation[J]. IEEE Access, 2020, 8: 87046-87059. doi: 10.1109/ACCESS.2020.2991519 [4] Chen H, Zhang Z, Guan C, et al. Optimization of sizing and frequency control in battery/supercapacitor hybrid energy storage system for fuel cell ship. Energy, 2020, 197: 117285. doi: 10.1016/j.energy.2020.117285 [5] 鲜征征, 李启良, 黄晓宇, 等. 基于差分隐私和SVD++的协同过滤算法. 控制与决策, 2019, 34(01): 43-54.Xian Zheng-Zheng, Li Qi-Liang, Huang Xiao-Yu, et al. Collaborative filtering via SVD++ with differential privacy. Control and Decision, 2019, 34(01): 43-54. [6] Jing L, Xk C, Sl A, et al. Privacy preservation for machine learning training and classification based on homomorphic encryption schemes. Information Sciences, 2020, 526: 166-179. doi: 10.1016/j.ins.2020.03.041 [7] Gong M, Pan K, Xie Y, et al. Preserving differential privacy in deep neural networks with relevance-based adaptive noise imposition. Neural Networks, 2020, 125: 131-141. doi: 10.1016/j.neunet.2020.02.001 [8] 张超, 李强, 陈子豪, 等. Medical Chain: 联盟式医疗区块链系统. 自动化学报, 2019, 45(08): 1495-1510.Zhang Chao, Li Qiang, Chen Zi-Hao, et al. Medical Chain: alliance medical blockchain system. Acta Automatica Sinica, 2019, 45(08): 1495-1510. [9] Yang Q, Liu Y, Chen T, et al. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 2019, 10(2): 1-19. [10] Li T, Sahu A K, Talwalkar A, et al. Federated learn-ing: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 2020, 37(3): 50-60. doi: 10.1109/MSP.2020.2975749 [11] Zhang W, Li X, Ma H, et al. Federated learning for machinery fault diagnosis with dynamic validation and self-supervision. Knowledge-Based Systems, 2021, 213: 106679. doi: 10.1016/j.knosys.2020.106679 [12] Sheller M J, Edwards B, Reina G A, et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific reports, 2020, 10(1): 1-12. doi: 10.1038/s41598-019-56847-4 [13] Kwon D, Jeon J, Park S, et al. Multiagent DDPG-Based Deep Learning for Smart Ocean Federated Learning IoT Networks. IEEE Internet of Things Journal, 2020, 7(10): 9895-9903. doi: 10.1109/JIOT.2020.2988033 [14] Rothchild D, Panda A, Ullah E, et al. Fetchsgd: Communication-efficient federated learning with sketching[C]//International Conference on Machine Learning. PMLR, 2020: 8253−8265. [15] Duan M, Liu D, Chen X, et al. Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems, 2020, 32(1): 59-71. [16] Liu W, Chen L, Chen Y, et al. Accelerating Federated Learning via Momentum Gradient Descent. IEEE Transactions on Parallel and Distributed Systems, 2020, 31(8): 1754-1766. doi: 10.1109/TPDS.2020.2975189 [17] Wang S, Tuor T, Salonidis T, et al. Adaptive federated learning in resource constrained edge computing systems. IEEE Journal on Selected Areas in Communications, 2019, 37(6): 1205-1221. doi: 10.1109/JSAC.2019.2904348 [18] Aono Y, Hayashi T, Wang L, et al. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security, 2017, 13(5): 1333-1345. [19] 张泽辉, 富瑶, 高铁杠. 支持数据隐私保护的联邦深度神经网络模型研究. 自动化学报, 2020: 1-14.Zhang Ze-Hui, Fu Yao, Gao Tie-Gang. Research on federated deep neural network model for data privacy protection. Acta Automatica Sinica, 2020: 1-14. [20] Lyu L, Li Y, Nandakumar K, et al. How to democratise and protect AI: fair and differentially private decentralised deep learning. IEEE Transactions on Dependable and Secure Computing, 2020. [21] Wang Y, Gu M, Ma J, et al. DNN-DP: Differential Privacy Enabled Deep Neural Network Learning Framework for Sensitive Crowdsourcing Data. IEEE Transactions on Computational Social Systems, 2019, 7(1): 215-224. [22] Carpov S, Gama N, Georgieva M, et al. Privacy-preserving semi-parallel logistic regression training with Fully Homomorphic Encryption. 2019: 101. [23] 宋蕾, 马春光, 段广晗, 等. 基于数据纵向分布的隐私保护逻辑回归. 计算机研究与发展, 2019, 56(10): 2243-2249. doi: 10.7544/issn1000-1239.2019.20190414Song Lei, Ma Chun-Guang, Duan Guang-Han, et al. Privacy-preserving logistic regression on vertically partitioned data. Computer Research and Development, 2019, 56(10): 2243-2249. doi: 10.7544/issn1000-1239.2019.20190414 [24] Aono Y, Hayashi T, Wang L, et al. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security, 2017, 13(5): 1333-1345. [25] Ou W, Zeng J, Guo Z, et al. A homomorphic-encryption-based vertical federated learning scheme for rick management. Computer Science and Information Systems, 2020, 17(3): 819-834. doi: 10.2298/CSIS190923022O [26] Chen H, Chillotti I, Song Y. Improved Bootstrapping for Approximate Homomorphic Encryption. Springer, Cham, 2019. [27] Xiao X, Wu T, Chen Y, et al. Privacy-Preserved Ap-proximate Classification Based on Homomorphic En-cryption. Mathematical and Computational Applications, 2019, 24(4): 92. doi: 10.3390/mca24040092 [28] Zehui Z, Fu Y, Gao T. A Hybrid Image Encryption Algorithm Based on Chaos System and Simplified Ad-vanced Encryption System. International Journal of Multimedia Data Engineering and Management (IJMDEM), 2020, 11(4): 1-24. doi: 10.4018/IJMDEM.2020100101 [29] Luo Y, Yu J, Lai W, et al. A novel chaotic image encryption algorithm based on improved baker map and logistic map. Multimedia Tools and Applications, 2019, 78(15): 22023-22043. doi: 10.1007/s11042-019-7453-3 [30] Sathiyamurthi P, Ramakrishnan S. Speech encryption algorithm using FFT and 3D-Lorenz–logistic chaotic map. Multimedia Tools and Applications, 2020, 79(3). [31] Sattler F, Müller K, Samek W. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 2020. [32] Al-Sharman M, Murdoch D, Cao D, et al. A sensor-less state estimation for a safety-oriented cyber-physical system in urban driving: deep learning approach. IEEE/CAA Journal of Automatica Sinica, 2020. [33] Weng J, Weng J, Zhang J, et al. Deepchain: Auditable and privacy-preserving deep learning with blockchain-based incentive. IEEE Transactions on Dependable and Secure Computing, 2019. [34] Sattler F, Wiedemann S, Müller K, et al. Robust and communication-efficient federated learning from non-iid data. IEEE transactions on neural networks and learning systems, 2019. [35] Xu G, Li H, Zhang Y, et al. Privacy-preserving federated deep learning with irregular users. IEEE Transactions on Dependable and Secure Computing, 2020. [36] Teng S, Wu N, Zhu H, et al. SVM-DT-based adaptive and collaborative intrusion detection. IEEE/CAA Journal of Automatica Sinica, 2017, 5(1): 108-118. [37] Wang F Y, Fundamental Issues in Research of Computing with Words and Linguistic Dynamic Systems. Acta Automatica Sinica (Periodical style), 2005, 31(6): 844--852. [38] Roychoudhury R, Bandyopadhyay S, Paul K. Adistributed mechanism for topology discovery in ad hoc wireless networks using mobile agents. In: Proceeding of IEEE First Annual Workshop on Mobile and Ad hoc Networking and Computing (Conference Proceedings style), Piscataway, USA: IEEE Press, 2000.145—146. [39] Hryniewicz O. An evaluation of the reliability of complex systems using shadowed sets and fuzzy lifetime data. International Journal of Automation and Computing (Periodical style—Accepted for publication), to be published. [40] Zhang W. Reinforcement Learning for Job-Shop Scheduling. [Ph. D. Dissertation], Oregon State University, 1996. [41] IEEE Criteria for Class IE Electric Systems (Standards style), IEEE Standard 308, 1969. [42] Jones J. Networks, 2nd ed. (Online Sources style). [Online], available: http://www.atm.com, May 10, 1991. -

计量
- 文章访问数: 2057
- HTML全文浏览量: 1332
- 被引次数: 0