面向安全与性能动态平衡的自适应流形约束强化学习

刘正堂; 吴越; 胡春鹤; 卢仁智; 程骋; 郑一力

doi:10.16383/j.aas.c260122

面向安全与性能动态平衡的自适应流形约束强化学习

doi: 10.16383/j.aas.c260122 cstr: 32138.14.j.aas.c260122

刘正堂^{1, 2, 3,},
吴越^{1, 2, 3,},
胡春鹤^{1, 2, 3,},
卢仁智^4,,
程骋^4,,
郑一力^{1, 2, 3,}

1.
北京林业大学工学院北京 100083
2.
林木资源高效生产全国重点实验室北京 100083
3.
林业装备与自动化国家林业和草原局重点实验室北京 100083
4.
华中科技大学人工智能与自动化学院武汉 430074

基金项目: 国家自然科学基金(62273053)资助

详细信息

作者简介:
刘正堂：北京林业大学工学院硕士研究生. 主要研究方向为安全强化学习和多机器人协同控制. E-mail: liuzhengtang@bjfu.edu.cn

吴越：北京林业大学工学院副教授. 主要研究方向为多机器人协同控制与强化学习. 本文通信作者. E-mail: wuyue_a@bjfu.edu.cn

胡春鹤：北京林业大学工学院副教授. 主要研究方向为机器人安全强化学习, 机器人自主控制. E-mail: huchunhe@bjfu.edu.cn

卢仁智：华中科技大学人工智能与自动化学院副教授. 主要研究方向为强化学习与无人系统. E-mail: rzlu@hust.edu.cn

程骋：华中科技大学人工智能与自动化学院副教授. 主要研究方向为系统辨识与智能混线装配. E-mail: c_cheng@hust.edu.cn

郑一力：北京林业大学工学院教授.主要研究方向为智慧林业监测和机器人控制. E-mail: zhengyili@bjfu.edu.cn

中图分类号: Y
计量
- 文章访问数: 346
- HTML全文浏览量: 286
- PDF下载量: 10
- 被引次数: 0
出版历程
- 收稿日期: 2026-02-10
- 录用日期: 2026-05-07
- 网络出版日期: 2026-07-02

Adaptive Manifold Constrained Reinforcement Learning for Dynamic Balance Between Safety and Performance

LIU Zheng-Tang^{1, 2, 3
,},
WU Yue^{1, 2, 3
,},
HU Chun-He^{1, 2, 3
,},
LU Ren-Zhi^4
,,
CHENG Cheng^4
,,
ZHENG Yi-Li^{1, 2, 3
,}

1.
School of Technology, Beijing Forestry University, Beijing 100083
2.
State Key Laboratory of Efficient Production of Forest Resources, Beijing 100083
3.
Key Laboratory of National Forestry and Grassland Administration on Forestry Equipment and Automation, Beijing 100083
4.
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074

Funds: Supported by National Natural Science Foundation of China (62273053)

More Information

Author Bio:
LIU Zheng-Tang　Master student at the School of Technology, Beijing Forestry University. Her research interests include safety-reinforcement learning and multi-robot cooperative control

WU Yue　Associate professor at the School of Technology, Beijing Forestry University. His research interests include multi-robot cooperative control and reinforcement learning. Corresponding author of this paper

HU Chun-He　Associate professor at the School of Technology, Beijing Forestry University. His research interests include safe reinforcement learning for robots and autonomous robot control

LU Ren-Zhi　Associate professor at the School of Artificial Intelligence and Automation, Huazhong University of Science and Technology. His research interests include reinforcement learning and unmanned systems

CHENG Cheng　Associate professor at the School of Artificial Intelligence and Automation, Huazhong University of Science and Technology. Her research interests include system identification and intelligent mixed-line assembly

ZHENG Yi-Li　Professor at the School of Technology, Beijing Forestry University. His research interests include smart forestry monitoring and robot control

摘要

摘要: 机器人走向实际场景时, 如何严格满足安全约束并兼顾学习探索效率, 是机器人实用化的重要挑战. 现有安全学习方法依赖固定控制增益, 难以在训练中动态平衡探索效率与约束满足, 易导致过度保守或约束保持不足. 为此, 提出面向安全−性能动态平衡的状态相关增益调度自适应 ATACOM 方法, 简称 SGA-ATACOM. 在保持原有的约束流形切空间探索结构不变的基础上, 构建约束层与控制器层双层在线调度机制. 其中, 约束层依据约束状态、变化趋势及短时预测, 自适应调节可行性约束参数; 控制器层结合约束残差与速度风险指标, 更新误差修正增益和速度可行域增益, 从而实现在风险指标较高时增强约束回拉与速度保护、在风险指标较低时释放更有效的探索空间的自适应协调. 在 CircularMotion 和 Planar Air Hockey 机械臂任务上, 采用不同强化学习算法开展仿真实验. 结果表明, 该算法在不同任务和强化学习算法下具有通用性, 在维持较低约束违反的同时取得更优的安全−性能平衡. 消融实验验证了约束层与控制器层调度的互补作用. 该算法为受约束强化学习的安全−性能协同优化提供了具有几何可解释性和结构保持性的改进方案.
- 安全强化学习 /
- 流形约束 /
- ATACOM /
- 状态相关增益调度 /
- 安全−性能平衡
Abstract: In the deployment of robots to real-world scenarios, balancing strict safety constraint satisfaction and learning exploration efficiency has become a critical challenge restricting the practical application of robotic systems. Existing safe learning methods rely on fixed control-gain designs, making it difficult to dynamically balance exploration efficiency and constraint satisfaction during training, which may lead to overly conservative behavior or insufficient constraint maintenance. To address this issue, this paper proposes a state-dependent gain-scheduled adaptive ATACOM method for dynamic safety-performance balancing, termed SGA-ATACOM. Without altering ATACOM＇s original tangent-space exploration structure on the constraint manifold, a two-layer online scheduling mechanism is developed. Specifically, the constraint layer adaptively adjusts viability-constraint parameters according to the current constraint state, its variation trend, and short-term predictions, while the controller layer updates the error-correction gain and velocity-feasibility gain online based on the constraint residual and velocity-risk indicators. Thus, stronger constraint recovery and velocity protection are activated at high risk, whereas more effective exploration space is released at low risk. Simulations are conducted on the CircularMotion task and the Planar Air Hockey manipulator task using different reinforcement learning algorithms. The results show that the method exhibits good generality across different tasks and reinforcement learning algorithms, achieving a better safety-performance trade-off while maintaining low constraint violations. Ablation studies verify the complementary roles of constraint-layer and controller-layer scheduling mechanisms. Overall, the proposed method provides a geometrically interpretable and structure-preserving improvement for coordinated safety-performance optimization in constrained reinforcement learning.
- Safe reinforcement learning /
- constraint manifold /
- ATACOM /
- state-dependent gain scheduling /
- safety-performance balance

HTML全文

图 1 SGA-ATACOM的整体框架

Fig. 1 The overall framework of SGA-ATACOM

下载: 全尺寸图片幻灯片

图 2 仿真实验环境示意图

Fig. 2 Schematic diagram of the simulation environments

下载: 全尺寸图片幻灯片

图 3 CircularMotion环境下不同方法的性能与约束对比

Fig. 3 Comparison of performance and constraints of different methods in the CircularMotion environment

下载: 全尺寸图片幻灯片

图 4 CircularMotion环境下SGA-ATACOM在不同强化学习算法上的训练曲线对比

Fig. 4 Training curves of SGA-ATACOM under different reinforcement learning algorithms in the CircularMotion environment

下载: 全尺寸图片幻灯片

图 5 Planar Air Hockey环境下不同方法的性能与约束对比

Fig. 5 Performance and constraint comparison of different methods in the Planar Air Hockey environment

下载: 全尺寸图片幻灯片

图 6 Planar Air Hockey环境下SGA-ATACOM在不同强化学习算法上的训练曲线对比

Fig. 6 Comparison of training curves of SGA-ATACOM on different reinforcement learning algorithms in the Planar Air Hockey environment

下载: 全尺寸图片幻灯片

图 7 CircularMotion环境下SGA-ATACOM的消融实验结果

Fig. 7 Ablation results of SGA-ATACOM in the CircularMotion environment

下载: 全尺寸图片幻灯片

图 8 训练过程中的增益演化曲线

Fig. 8 Gain evolution curve during training

下载: 全尺寸图片幻灯片

图 9 约束层状态−参数对应曲线

Fig. 9 Constrained layer state-parameter correspondence curves

下载: 全尺寸图片幻灯片

图 10 控制器层状态−参数对应曲线

Fig. 10 Controller layer state-parameter correspondence curves

下载: 全尺寸图片幻灯片

表 1 SGA-ATACOM 的关键增益调度参数

Table 1 Key gain-scheduling parameters of SGA-ATACOM

实验环境	参数类别	约束层		控制器层
实验环境	参数类别	$K_f$	$K_g$	$K_c$	$K_q$
CircularMotion	基线增益	$0.1$	$2.0$	$100$	$20$
	相对增益范围	$[0.35,\;1.0]$	$[0.40,\;1.0]$	$[1.0,\;2.0]$	$[1.0,\;2.0]$
	更新速率	$0.25$	$0.30$	$0.15$	$0.20$
Planar Air Hockey	基线增益	$0.5$	$1.0$	$240$	$2a_{\max}/v_{\max}$
	相对增益范围	$[0.45,\;1.0]$	$[0.50,\;1.0]$	$[1.0,\;1.8]$	$[1.0,\;1.8]$
	更新速率	$0.18$	$0.16$	$0.12$	$0.16$

下载: 导出CSV

表 2 CircularMotion 环境下不同方法的定量结果

Table 2 Quantitative results of different methods in the CircularMotion environment

算法	方法	$ R $	$ c_{\mathrm{avg}} $	$ c_{\max} $	$ c_{\dot{q},\;\max} $
DDPG	ATACOM	$ 235.701\;5 $	$ 0.006\;402 $	$ 0.031\;223 $	$ 0 $
	SGA-ATACOM	$ 247.178\;7 $	$ 0.003\;576 $	$ 0.012\;514 $	$ 0 $
	ErrorCorrection	$ \;\;74.954\;0 $	$ 0.031\;649 $	$ 0.140\;175 $	$ 0 $
	Terminated	$ -95.739\;3 $	$ 0.118\;951 $	$ 0.428\;035 $	$ 0.498\;038 $
PPO	ATACOM	$ 278.928\;2 $	$ 0.006\;523 $	$ 0.031\;222 $	$ 0 $
	SGA-ATACOM	$ 300.638\;1 $	$ 0.002\;561 $	$ 0.012\;518 $	$ 0 $
	ErrorCorrection	$ 192.177\;3 $	$ 0.028\;069 $	$ 0.139\;112 $	$ 0 $
	Terminated	$ \;\;71.208\;9 $	$ 0.101\;304 $	$ 0.401\;823 $	$ 0.046\;248 $
TRPO	ATACOM	$ 264.615\;3 $	$ 0.006\;437 $	$ 0.031\;204 $	$ 0 $
	SGA-ATACOM	$ 293.580\;4 $	$ 0.003\;045 $	$ 0.012\;515 $	$ 0 $
	ErrorCorrection	$ 109.174\;2 $	$ 0.039\;597 $	$ 0.135\;171 $	$ 0 $
	Terminated	$ \;\;62.178\;4 $	$ 0.180\;728 $	$ 0.405\;737 $	$ 0.313\;938 $

下载: 导出CSV

表 3 Planar air hockey 环境下不同方法的定量结果

Table 3 Quantitative results of different methods in the Planar Air Hockey environment

算法	方法	$ R $	$ c_{\mathrm{avg}} $	$ c_{\max} $	$ c_{\dot{q},\;\max} $
PPO	ATACOM	$ 440.112\;4 $	$ 0 $	$ 0 $	$ 0 $
	SGA-ATACOM	$ 525.228\;6 $	$ 0 $	$ 0 $	$ 0 $
	Unconstrained	$ 280.524\;0 $	$ 0.000\;182 $	$ 0.033\;132 $	$ 1.178\;097 $
TRPO	ATACOM	$ 306.453\;7 $	$ 0 $	$ 0 $	$ 0 $
	SGA-ATACOM	$ 438.229\;5 $	$ 0 $	$ 0 $	$ 0.055\;845 $
	Unconstrained	$ 215.184\;6 $	$ 0.019\;395 $	$ 0.374\;530 $	$ 1.178\;097 $

下载: 导出CSV

表 4 CircularMotion 环境下消融实验的定量结果

Table 4 Quantitative results of ablation experiments in CircularMotion environment

方法	$ R $	$ c_{\mathrm{avg}} $	$ c_{\max} $	$ c_{\dot{q},\;\max} $
ATACOM	$ 264.615\,3 $	$ 0.006\,437 $	$ 0.031\,204 $	$ 0 $
仅控制器层调度	$ 254.084\,3 $	$ 0.004\,486 $	$ 0.024\,616 $	$ 0 $
仅约束层调度	$ 235.940\,3 $	$ 0.004\,151 $	$ 0.015\,066 $	$ 0 $
SGA-ATACOM	$ 293.580\,4 $	$ 0.003\,045 $	$ 0.012\,515 $	$ 0 $

下载: 导出CSV

参考文献(29)

[1]	Gu S D, Yang L, Du Y L, Chen G, Walter F, Wang J, et al. A review of safe reinforcement learning: Methods, theories, and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(12): 11216−11235 doi: 10.1109/TPAMI.2024.3457538
[2]	Lee J, Schroth L, Klemm V, Bjelonic M, Reske A, Hutter M. Exploring constrained reinforcement learning algorithms for quadrupedal locomotion. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Abu Dhabi, United Arab Emirates: IEEE, 2024. 11132−11138
[3]	Keyumarsi S, Atman M W S, Gusrialdi A. LiDAR-based online control barrier function synthesis for safe navigation in unknown environments. IEEE Robotics and Automation Letters, 2024, 9(2): 1043−1050 doi: 10.1109/LRA.2023.3339059
[4]	周毅, 张浩, 施孟佶, 林伯先, 石磊. 未知环境中基于控制障碍函数的机器人安全控制研究综述. 电子科技大学学报, 2025, 54(1): 29−38 doi: 10.12178/1001-0548.2023296 Zhou Yi, Zhang Hao, Shi Meng-Jie, Lin Bo-Xian, Shi Lei. A review of research on robot safety control based on control barrier functions in unknown environments. Journal of University of Electronic Science and Technology of China, 2025, 54(1): 29−38 doi: 10.12178/1001-0548.2023296
[5]	Könighofer B, Bloem R, Jansen N, Junges S, Pranger S. Shields for safe reinforcement learning. Communications of the ACM, 2025, 68(11): 80−90 doi: 10.1145/3715958
[6]	Dawood M, Shokry A, Bennewitz M. A dynamic safety shield for safe and efficient reinforcement learning of navigation tasks. In: Proceedings of the 7th Annual Learning for Dynamics & Control Conference. Ann Arbor, USA: PMLR, 2025. 686−697
[7]	董明泽, 温庄磊, 陈锡爱, 杨炅坤, 曾涛. 安全凸空间与深度强化学习结合的机器人导航方法. 兵工学报, 2024, 45(12): 4372−4382 doi: 10.12382/bgxb.2023.0982 Dong Ming-Ze, Wen Zhuang-Lei, Chen Xi-Ai, Yang Jiong-Kun, Zeng Tao. Research on robot navigation method integrating safe convex space and deep reinforcement learning. Acta Armamentarii, 2024, 45(12): 4372−4382 doi: 10.12382/bgxb.2023.0982
[8]	Liu P Z, Tateo D, Ammar H B, Peters J. Robot reinforcement learning on the constraint manifold. In: Proceedings of the 5th Conference on Robot Learning. London, UK: PMLR, 2022. 1357−1366
[9]	张昌昕, 张兴龙, 徐昕, 陆阳. 安全强化学习及其在机器人系统中的应用综述. 控制理论与应用, 2023, 40(12): 2090−2103 doi: 10.7641/CTA.2023.30247 Zhang Chang-Xin, Zhang Xing-Long, Xu Xin, Lu Yang. Safe reinforcement learning and its applications in robotics: A survey. Control Theory & Applications, 2023, 40(12): 2090−2103 doi: 10.7641/CTA.2023.30247
[10]	Altman E. Constrained Markov Decision Processes. New York: Routledge, 2021. 11−15
[11]	Achiam J, Held D, Tamar A, Abbeel P. Constrained policy optimization. In: Proceedings of the 34th International Conference on Machine Learning. Sydney, Australia: JMLR.org, 2017. 22−31
[12]	Liu Y S, Ding J X, Liu X. IPO: Interior-point policy optimization under constraints. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. New York, USA: AAAI Press, 2020. 4940−4947
[13]	Tessler C, Mankowitz D J, Mannor S. Reward constrained policy optimization. In: Proceedings of the 7th International Conference on Learning Representations. New Orleans, USA: OpenReview.net, 2019.
[14]	Stooke A, Achiam J, Abbeel P. Responsive safety in reinforcement learning by PID lagrangian methods. In: Proceedings of the 37th International Conference on Machine Learning. JMLR.org, 2020. Article No. 847
[15]	Ding D S, Wei X H, Yang Z R, Wang Z R, Jovanović M R. Provably efficient safe exploration via primal-dual policy optimization. In: Proceedings of the 24th International Conference on Artificial Intelligence and Statistics. San Diego, USA: PMLR, 2021. 3304−3312
[16]	Chow Y, Nachum O, Duenez-Guzman E, Ghavamzadeh M. A lyapunov-based approach to safe reinforcement learning. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montrél, Canada: Curran Associates Inc., 2018. 8103-8112
[17]	陈谋, 刘伟, 张鹏. 性能约束下的四旋翼无人机协同吊挂系统分布式避碰跟踪控制. 自动化学报, 2024, 50(12): 2392−2406 doi: 10.16383/j.aas.c240349 Chen Mou, Liu Wei, Zhang Peng. Distributed collision avoidance tracking control for quadrotor cooperative suspension system under performance constraints. Acta Automatica Sinica, 2024, 50(12): 2392−2406 doi: 10.16383/j.aas.c240349
[18]	Garcia J, Fernández F. Safe exploration of state and action spaces in reinforcement learning. Journal of Artificial Intelligence Research, 2012, 45(1): 515−564 doi: 10.1613/jair.3761
[19]	Alshiekh M, Bloem R, Ehlers R, Künighofer B, Niekum S, Topcu U. Safe reinforcement learning via shielding. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence. New Orleans, USA: AAAI Press, 2018.
[20]	Hans A, Schneega\beta D, Schäfer A M, Udluft S. Safe exploration for reinforcement learning. In: Proceedings of the 16th European Symposium on Artificial Neural Networks. Bruges, Belgium, 2008. 143−148
[21]	赵静, 裴子楠, 姜斌, 陆宁云, 赵斐, 陈树峰. 基于深度强化学习的无人机虚拟管道视觉避障. 自动化学报, 2024, 50(11): 2245−2258 doi: 10.16383/j.aas.c230728 Zhao Jing, Pei Zi-Nan, Jiang Bin, Lu Ning-Yun, Zhao Fei, Chen Shu-Feng. Virtual tube visual obstacle avoidance for UAV based on deep reinforcement learning. Acta Automatica Sinica, 2024, 50(11): 2245−2258 doi: 10.16383/j.aas.c230728
[22]	Dalal G, Dvijotham K, Vecerik M, Hester T, Paduraru C, Tassa Y. Safe exploration in continuous action spaces. arXiv preprint arXiv: 1801.08757, 2018
[23]	Cheng R, Orosz G, Murray R M, Burdick J W. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Honolulu, USA: AAAI Press, 2019. 3387−3395
[24]	Koller T, Berkenkamp F, Turchetta M, Krause A. Learning-based model predictive control for safe exploration. In: Proceedings of the IEEE Conference on Decision and Control (CDC). Miami, USA: IEEE, 2018. 6059−6066
[25]	Hewing L, Wabersich K P, Menner M, Zeilinger M N. Learning-based model predictive control: Toward safe learning in control. Annual Review of Control, Robotics, and Autonomous Systems, 2020, 3: 269−296 doi: 10.1146/annurev-control-090419-075625
[26]	Liu P Z, Bou-Ammar H, Peters J, Tateo D. Safe reinforcement learning on the constraint manifold: Theory and applications. IEEE Transactions on Robotics, 2025, 41: 3442−3461 doi: 10.1109/TRO.2025.3567477
[27]	Liu P Z, Zhang K, Tateo D, Jauhri S, Hu Z Y, Peters J, et al. Safe reinforcement learning of dynamic high-dimensional robotic tasks: Navigation, manipulation, interaction. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). London, United Kingdom: IEEE, 2023. 9449−9456
[28]	张楠杰, 陈玉全, 季茂沁, 孙运康, 王冰. 面向不同粗糙程度地面的四足机器人自适应控制方法. 自动化学报, 2025, 51(7): 1585−1598 doi: 10.16383/j.aas.c240738 Zhang Nan-Jie, Chen Yu-Quan, Ji Mao-Qin, Sun Yun-Kang, Wang Bing. Adaptive control method for quadruped robot facing floors of different roughness. Acta Automatica Sinica, 2025, 51(7): 1585−1598 doi: 10.16383/j.aas.c240738
[29]	Miki T, Lee J, Hwangbo J, Wellhausen L, Koltun V, Hutter M. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science robotics, 2022, 7(62): eabk2822 doi: 10.1126/scirobotics.abk2822