基于相对阶解耦的多智能体系统安全强化学习

黄頔; 张斯伦

doi:10.16383/j.aas.c260090

基于相对阶解耦的多智能体系统安全强化学习

doi: 10.16383/j.aas.c260090 cstr: 32138.14.j.aas.c260090

黄頔^1,,
张斯伦^2,

1.
湖北珞珈实验室武汉 430072 中国
2.
瑞典皇家理工学院数学系斯德哥尔摩 SE 114 28 瑞典

基金项目: 中国航空科学基金(202300010S5001), 国家自然科学基金(61903278)资助

详细信息

作者简介:
黄頔：湖北珞珈实验室副教授. 2017年获得北京大学力学系统与控制专业博士学位. 主要研究方向为航天器智能姿态与轨道控制, 航天器编队控制, 航天器故障诊断及容错控制.本文通信作者. E-mail: dhuang@whu.edu.cn

张斯伦：瑞典皇家理工学院数学系助理教授. 2019年获得瑞典皇家理工学院优化与系统理论专业博士学位. 主要研究方向为非线性控制, 大规模复杂系统, 激励驱动算法和风险规避/安全强化学习. E-mail: silunz@kth.se

计量
- 文章访问数: 261
- HTML全文浏览量: 175
- PDF下载量: 31
- 被引次数: 0
出版历程
- 收稿日期: 2026-02-03
- 录用日期: 2026-04-12
- 网络出版日期: 2026-09-08
- 刊出日期: 2026-10-25

Safe Reinforcement Learning for Multi-agent Systems Based on Relative-degree Decoupling

HUANG Di^1
,,
ZHANG Si-Lun^2
,

1.
Hubei Luojia Laboratory, Wuhan 430072, China
2.
Department of Mathematics, KTH Royal Institute of Technology, Stockholm SE 114 28, Sweden

Funds: Supported by the Aeronautical Science Foundation of China (202300010S5001) and National Natural Science Foundation of China (61903278)

More Information

Author Bio:
HUANG Di　Associate professor at the Hubei Luojia Laboratory. He received his Ph.D. degree in mechanical systems and control from Peking University in 2017. His research interests include spacecraft intelligent attitude and orbit control, spacecraft formation control, and fault diagnosis and fault-tolerant control of spacecraft. Corresponding author of this paper

ZHANG Si-Lun　Assistant professor in the Department of Mathematics, KTH Royal Institute of Technology. He received his Ph.D. degree in optimization and systems theory from KTH Royal Institute of Technology in 2019. His research interests include nonlinear control, large-scale complex systems, incentive-driven algorithms, and risk-averse/safe reinforcement learning

摘要

摘要: 针对存在模型不确定性与外部扰动的多智能体协同控制问题, 提出一种基于模型相对阶解耦的安全强化学习框架. 首先, 利用非线性系统中的相对阶理论, 将高阶状态安全约束映射为输入显式出现的仿射不等式; 在凸多面体参数不确定与有界扰动条件下, 进一步给出由有限个线性不等式交集表征的可认证凸安全动作集, 为在线投影与约束求解提供显式可行域. 其次, 为解决单步屏蔽易产生后继不可行的问题, 设计带终端集的滚动安全滤波器: 在每个时刻维护有限时域备份控制序列, 并通过末端约束与终端控制器保证递归可行性与无限时域安全性. 最后, 利用相对阶标准型构建策略分解架构, 以LQR标称控制器提供稳定热启动, 强化学习仅负责对未建模动态进行补偿, 并采用参数共享的独立近端策略优化算法学习补偿量. 卫星编队姿态控制的仿真结果表明: 在执行器效率不确定与扰动条件下, 所提方法能够将约束违规率维持在接近零的水平, 同时实现更快的训练收敛与更低的跟踪误差.
- 多智能体系统 /
- 安全强化学习 /
- 相对阶 /
- 控制屏障函数 /
- 安全滤波器
Abstract: To address issues in cooperative multi-agent control subject to model uncertainty and external disturbances, this paper proposes a safe reinforcement learning framework based on model relative-degree decoupling. First, utilizing relative-degree theory in nonlinear systems, high-order state safety constraints are mapped into affine inequalities where the control input appears explicitly. Under conditions of convex-polytope parameter uncertainty and bounded disturbances, a certifiable convex safe action set characterized by the intersection of finite linear inequalities is derived to provide an explicit feasible region for online projection and constraint solving. Second, to overcome the issues of subsequent infeasibility often associated with single-step shielding, a receding-horizon safety filter incorporating a terminal set is designed. By maintaining a finite-horizon backup control sequence at each time step, this filter guarantees recursive feasibility and infinite-horizon safety via terminal constraints and a terminal controller. Finally, a policy decomposition architecture is constructed based on the relative-degree normal form. A linear quadratic regulator nominal controller provides a stable warm start, while reinforcement learning is only responsible for compensating for unmodeled dynamics using a parameter-sharing independent proximal policy optimization (IPPO) algorithm to learn the compensation. Simulation results on satellite formation attitude control demonstrate that, under actuator efficiency uncertainty and external disturbances, the proposed method maintains a near-zero constraint violation rate while achieving faster training convergence and reduced tracking error.
- multi-agent systems /
- safe reinforcement learning /
- relative degree /
- control barrier functions /
- safety filter

HTML全文

图 1 卫星拓扑图

Fig. 1 Satellite topology diagram

下载: 全尺寸图片幻灯片

图 2 不同方法的训练奖励曲线

Fig. 2 Training reward curves of different methods

下载: 全尺寸图片幻灯片

图 3 不同方法的评估误差曲线

Fig. 3 Evaluation error curves of different methods

下载: 全尺寸图片幻灯片

图 4 不同方法的安全约束违规率对比

Fig. 4 Comparison of safety-constraint violation rate for different methods

下载: 全尺寸图片幻灯片

图 5 不同方法的最小安全距离对比

Fig. 5 Comparison of minimum safety distance for different methods

下载: 全尺寸图片幻灯片

图 6 不同方法的最大侵入时间对比

Fig. 6 Comparison of maximum intrusion time for different methods

下载: 全尺寸图片幻灯片

表 1 训练超参数设置

Table 1 Training hyperparameter settings

参数	符号	取值
智能体数量	$ N $	4
网络结构	—	2层隐藏层(每层256)
激活函数	—	$ \tanh $
姿态追踪权重	$ w_{{\rm{track}}} $	8.0
角速度惩罚	$ w_{{\rm{rate}}} $	0.05
优化器	—	Adam
Actor学习率	—	$ 3\times 10^{-5} $
Critic学习率	—	$ 5\times 10^{-4} $
相对熵早停阈值	—	0.015
预测步长	$ H $	6

下载: 导出CSV

参考文献(25)

[1]	江碧涛, 温广辉, 周佳玲, 郑德智. 智能无人集群系统跨域协同技术研究现状与展望. 中国工程科学, 2024, 26(1): 117−126 doi: 10.15302/J-SSCAE-2024.01.015 Jiang Bi-Tao, Wen Guang-Hui, Zhou Jia-Ling, Zheng De-Zhi. Cross-domain cooperative technology of intelligent unmanned swarm systems: Current status and prospects. Strategic Study of CAE, 2024, 26(1): 117−126 doi: 10.15302/J-SSCAE-2024.01.015
[2]	Zhang J, Yang D S, Li W H, Zhang H G, Li G D, Gu P. Resilient output control of multiagent systems with DoS attacks and actuator faults: Fully distributed event-triggered approach. IEEE Transactions on Cybernetics, 2024, 54(12): 7681−7690 doi: 10.1109/TCYB.2024.3404010
[3]	Zhou P P, Chen B M. Distributed optimal solutions for multiagent pursuit-evasion games for capture and formation control. IEEE Transactions on Industrial Electronics, 2024, 71(5): 5224−5234 doi: 10.1109/TIE.2023.3283684
[4]	Zhao R, Li Y, Gao F, Gao Z H, Zhang T Y. Multi-agent constrained policy optimization for conflict-free management of connected autonomous vehicles at unsignalized intersections. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(6): 5374−5388 doi: 10.1109/TITS.2023.3331723
[5]	Guo R, Feng J W, Zhao Y, Huang T W, Liu X Z, Wang J Y. Robust model predictive control for perturbed nonlinear multi-agent systems via dynamic event-triggered scheme. Science China Information Sciences, 2026, 69(4): Article No. 142205 doi: 10.1007/s11432-024-4807-5
[6]	Wen G H, Fu J J, Wang B W, Wang X D, Shen H. Robust formation tracking of ASVs with nonconservative collision avoidance: A control barrier function-based approach. IEEE Transactions on Industrial Electronics, 2026, 73(6): 9174−9186 doi: 10.1109/TIE.2025.3649849
[7]	Wen G H, Yu W W, Xia Y Q, Yu X H, Hu J Q. Distributed tracking of nonlinear multiagent systems under directed switching topology: An observer-based protocol. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2017, 47(5): 869−881 doi: 10.1109/TSMC.2016.2564929
[8]	Li Y H, Xie G M, Lu Z Q. Revisiting cooperative off-policy multi-agent reinforcement learning. In: Proceedings of the 42nd International Conference on Machine Learning. Vancouver, Canada: PMLR, 2025. Article No. 1439
[9]	Mai Y X, Yin Q Y, Ni W C, Xu P, Huang K Q. Constructive conflict-driven multi-agent reinforcement learning for strategic diversity. In: Proceedings of the 34th International Joint Conference on Artificial Intelligence. Montreal, Canada: ijcai.org, 2025. Article No. 24
[10]	Li D P, Lou N, Xu Z W, Zhang B, Fan G L. Efficient communication in multi-agent reinforcement learning with implicit consensus generation. In: Proceedings of the 39th AAAI Conference on Artificial Intelligence. Philadelphia, USA: AAAI Press, 2025. 23240−23248
[11]	Tessler C, Mankowitz D J, Mannor S. Reward constrained policy optimization. In: Proceedings of the 7th International Conference on Learning Representations. New Orleans, USA: OpenReview.net, 2019.
[12]	Fulton N, Platzer A. Safe reinforcement learning via formal methods: Toward safe control through proof and learning. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence. New Orleans, USA: AAAI Press, 2018. 4485−4491
[13]	Fisac J F, Lugovoy N F, Rubies-Royo V, Ghosh S, Tomlin C J. Bridging Hamilton-Jacobi safety analysis and reinforcement learning. In: Proceedings of the International Conference on Robotics and Automation (ICRA). Montreal, Canada: IEEE, 2019. 8550−8556
[14]	Polymenakos K, Abate A, Roberts S. Safe policy search using Gaussian process models. In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems. Montreal, Canada: International Foundation for Autonomous Agents and Multiagent Systems, 2019. 1565−1573
[15]	Cheng R, Khojasteh M J, Ames A D, Burdick J W. Safe multi-agent interaction through robust control barrier functions with learned uncertainties. In: Proceedings of the 59th IEEE Conference on Decision and Control (CDC). Jeju, Korea (South): IEEE, 2020. 777−783
[16]	Qin Z Y, Zhang K Q, Chen Y X, Chen J K, Fan C C. Learning safe multi-agent control with decentralized neural barrier certificates. In: Proceedings of the 9th International Conference on Learning Representations. Austria: OpenReview.net, 2021. Qin Z Y, Zhang K Q, Chen Y X, Chen J K, Fan C C. Learning safe multi-agent control with decentralized neural barrier certificates. In: Proceedings of the 9th International Conference on Learning Representations. Austria: OpenReview.net, 2021.
[17]	Ahmad H M S, Sabouni E, Wasilkoff A, Budhraja P, Guo Z J, Zhang S Y, et al. HMARL-CBF-hierarchical multi-agent reinforcement learning with control barrier functions for safety-critical autonomous systems. In: Proceedings of the 39th International Conference on Neural Information Processing Systems. San Diego, USA: 2025. Ahmad H M S, Sabouni E, Wasilkoff A, Budhraja P, Guo Z J, Zhang S Y, et al. HMARL-CBF-hierarchical multi-agent reinforcement learning with control barrier functions for safety-critical autonomous systems. In: Proceedings of the 39th International Conference on Neural Information Processing Systems. San Diego, USA: 2025.
[18]	Hsu K C, Nguyen D P, Fisac J F. ISAACS: Iterative soft adversarial actor-critic for safety. In: Proceedings of the 5th Learning for Dynamics and Control Conference. Philadelphia, USA: PMLR, 2023. 90−103
[19]	Chung S J, Bandyopadhyay S, Foust R, Subramanian G P, Hadaegh F Y. Review of formation flying and constellation missions using nanosatellites. Journal of Spacecraft and Rockets, 2016, 53(3): 567−578 doi: 10.2514/1.A33291
[20]	Boumeftah A, Ben Yahia O, Frigon J F, Falco G, Kurt G K. Adaptive detection of on-orbit jamming for securing GEO satellite links. In: Proceedings of the 12th Advanced Satellite Multimedia Systems Conference and the 18th Signal Processing for Space Communications Workshop (ASMS/SPSC). Sitges, Spain: IEEE, 2025. 1−8
[21]	Isidori A. Exact linearization methods. Nonlinear Control Systems: An Introduction. London: Springer, 1995. 178−253 Isidori A. Exact linearization methods. Nonlinear Control Systems: An Introduction. London: Springer, 1995. 178−253
[22]	Xiao W, Belta C. High-order control barrier functions. IEEE Transactions on Automatic Control, 2022, 67(7): 3655−3662 doi: 10.1109/TAC.2021.3105491
[23]	Ames A D, Xu X R, Grizzle J W, Tabuada P. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control, 2017, 62(8): 3861−3876 doi: 10.1109/TAC.2016.2638961
[24]	van den Berg J, Lin M, Manocha D. Reciprocal velocity obstacles for real-time multi-agent navigation. In: Proceedings of the IEEE International Conference on Robotics and Automation. Pasadena, USA: IEEE, 2008. 1928−1935
[25]	van Wijk D E J, Coogan S, Molnar T G, Majji M, Hobbs K L. Disturbance-robust backup control barrier functions: Safety under uncertain dynamics. IEEE Control Systems Letters, 2024, 8: 2817−2822 doi: 10.1109/LCSYS.2024.3514998