Time-varying Formation Tracking Control for Heterogeneous Air-sea Unmanned Swarms: A Single Network Adaptive Critic Reinforcement Learning Approach
-
摘要: 针对模型完全未知的异构空海无人集群时变编队跟踪控制问题, 提出基于单网络自适应评判强化学习的分层分布式控制方案. 首先, 为无人装备设计完全分布式自适应观测器, 仅依赖在线交互信息, 即可协同估计期望轨迹. 在此基础上, 引入复合学习技术, 以在有限时间内估计系统不确定性. 其次, 构建分层控制框架, 将异构集群编队控制问题解耦为位置与姿态的上下层控制子问题, 并利用单网络自适应评判强化学习技术为各层设计完全无需模型知识的最优控制策略. 理论分析表明, 提出的控制方案能保证跟踪误差和参数估计误差的一致最终有界. 最后, 仿真实验验证了所提方案在实现异构集群时变编队跟踪方面的有效性.Abstract: The time-varying formation tracking control problem for heterogeneous air-sea unmanned swarms with completely unknown dynamics is considered, a hierarchical distributed control scheme is proposed based on single network adaptive critic (SNAC) reinforcement learning. First, fully distributed adaptive observers are designed for vehicles, enabling them to cooperatively estimate the reference trajectories by utilizing only online exchanged information. Based on this, composite learning is introduced for estimation of system uncertainties within finite time. Then, a hierarchical control framework is established, which decouples the formation control problem for the heterogeneous swarms into upper-layer and lower-layer subproblems of position and attitude, and the SNAC reinforcement learning technique is employed to design model-free optimal control policies for each layer. Theoretical analysis demonstrates that the proposed control scheme guarantees the uniform ultimate boundedness of both the tracking errors and parameter estimation errors. Finally, simulation results verify the effectiveness of the proposed scheme in achieving time-varying formation tracking for the heterogeneous swarms.
-
图 2 参数估计误差范数的演化轨迹
Fig. 2 Evolution Trajectories of the parameter estimation error norms
图 4 跟踪误差和相应的参数估计演化曲线
Fig. 4 Tracking error and corresponding parameter estimation evolution curves
-
[1] 王耀南, 华和安, 张辉, 钟杭, 樊叶心, 梁鸿涛, 等. 性能函数引导的无人机集群深度强化学习控制方法. 自动化学报, 2025, 51(5): 905−916 doi: 10.16383/j.aas.c240519Wang Yao-Nan, Hua He-An, Zhang Hui, Zhong Hang, Fan Ye-Xin, Liang Hong-Tao, et al. Performance function-guided deep reinforcement learning control for UAV swarm. Acta Automatica Sinica, 2025, 51(5): 905−916 doi: 10.16383/j.aas.c240519 [2] 华长春, 穆殿瑞, 陈健楠, 魏饶. 无人机引导下多消防机器人协同作业. 自动化学报, 2026, 52(2): 267−283 doi: 10.16383/j.aas.c250420Hua Chang-Chun, Mu Dian-Rui, Chen Jian-Nan, Wei Rao. Collaborative operation of multiple firefighting robots guided by UAVs. Acta Automatica Sinica, 2026, 52(2): 267−283 doi: 10.16383/j.aas.c250420 [3] Wen G H. Model identification and control of autonomous surface vessels: An overview of recent advances and future perspectives. IEEE Systems, Man, and Cybernetics Magazine, 2026, 12(1): 65−79 doi: 10.1109/MSMC.2025.3542254 [4] Wen G H, Fu J J, Wang B W, Wang X D, Shen H. Robust formation tracking of ASVs with nonconservative collision avoidance: A control barrier function-based approach. IEEE Transactions on Industrial Electronics, 2026, 73(6): 9174−9186 doi: 10.1109/TIE.2025.3649849 [5] 温广辉, 余星火, 黄廷文, 周艳. 模型参数不确定下多无人艇系统固定时间二分编队跟踪控制. 自动化学报, 2025, 51(3): 669−677 doi: 10.16383/j.aas.c240473Wen Guang-Hui, Yu Xing-Huo, Huang Ting-Wen, Zhou Yan. Fixed-time bipartite formation tracking control for multi-USV systems with uncertain model parameters. Acta Automatica Sinica, 2025, 51(3): 669−677 doi: 10.16383/j.aas.c240473 [6] 江碧涛, 温广辉, 周佳玲, 郑德智. 智能无人集群系统跨域协同技术研究现状与展望. 中国工程科学, 2024, 26(1): 117−126 doi: 10.15302/J-SSCAE-2024.01.015Jiang Bi-Tao, Wen Guang-Hui, Zhou Jia-Ling, Zheng De-Zhi. Cross-domain cooperative technology of intelligent unmanned swarm systems: Current status and prospects. Strategic Study of CAE, 2024, 26(1): 117−126 doi: 10.15302/J-SSCAE-2024.01.015 [7] 闫敬, 关新平. 海上无人系统跨域集群发展现状及其关键技术. 自动化学报, 2025, 51(4): 744−761 doi: 10.16383/j.aas.c240334Yan Jing, Guan Xin-Ping. Development status and key techniques for cross-domain swarm of maritime unmanned systems. Acta Automatica Sinica, 2025, 51(4): 744−761 doi: 10.16383/j.aas.c240334 [8] Wang H T, Wang H Y, Guo X. Distributed adaptive fault-tolerant formation control for heterogeneous USV-AUV swarms based on dynamic event triggering. Journal of Marine Science and Engineering, 2025, 13(11): Article No. 2116 doi: 10.3390/jmse13112116 [9] Liu H T, Huang H T, Tian X H, Zhang J. Distributed fixed-time formation control for UAV-USV multiagent systems based on the FEWNN with prescribed performance. Ocean Engineering, 2025, 328: Article No. 120996 doi: 10.1016/j.oceaneng.2025.120996 [10] Huang D P, Li H Y, Li X. Formation of generic UAVs-USVs system under distributed model predictive control scheme. IEEE Transactions on Circuits and Systems II: Express Briefs, 2020, 67(12): 3123−3127 doi: 10.1109/tcsii.2020.2983096 [11] Zhu Y K, Li S L, Guo G, Yuan P F, Bai J G. Formation control of UAV–USV based on distributed event-triggered adaptive MPC with virtual trajectory restriction. Ocean Engineering, 2024, 294: Article No. 116850 doi: 10.1016/j.oceaneng.2024.116850 [12] 温广辉, 杨涛, 周佳玲, 付俊杰, 徐磊. 强化学习与自适应动态规划: 从基础理论到多智能体系统中的应用进展综述. 控制与决策, 2023, 38(5): 1200−1230 doi: 10.13195/j.kzyjc.2022.1933Wen Guang-Hui, Yang Tao, Zhou Jia-Ling, Fu Jun-Jie, Xu Lei. Reinforcement learning and adaptive/approximate dynamic programming: A survey from theory to applications in multi-agent systems. Control and Decision, 2023, 38(5): 1200−1230 doi: 10.13195/j.kzyjc.2022.1933 [13] Padhi R, Unnikrishnan N, Wang X H, Balakrishnan S N. A single network adaptive critic (SNAC) architecture for optimal control synthesis for a class of nonlinear systems. Neural Networks, 2006, 19(10): 1648−1660 doi: 10.1016/j.neunet.2006.08.010 [14] Zhou Y, Wen G H, Zhou J L, Liu H, Lv J H. Data-driven output consensus tracking control for heterogeneous multiagent systems with a dynamic leader. IEEE Transactions on Control of Network Systems, 2025, 12(3): 2029−2039 doi: 10.1109/TCNS.2025.3549066 [15] Zhao W B, Liu H, Valavanis K P, Lewis F L. Fault-tolerant formation control for heterogeneous vehicles via reinforcement learning. IEEE Transactions on Aerospace and Electronic Systems, 2022, 58(4): 2796−2806 doi: 10.1109/TAES.2021.3139260 [16] Zhao W B, Liu H, Wan Y, Lin Z L. Data-driven formation control for multiple heterogeneous vehicles in air-ground coordination. IEEE Transactions on Control of Network Systems, 2022, 9(4): 1851−1862 doi: 10.1109/TCNS.2022.3181254 [17] Liu D Y, Liu H, Lv J H, Lewis F L. Time-varying formation of heterogeneous multiagent systems via reinforcement learning subject to switching topologies. IEEE Transactions on Circuits and Systems I: Regular Papers, 2023, 70(6): 2550−2560 doi: 10.1109/TCSI.2023.3250516 [18] Liu H, Ma T, Lewis F L, Wan Y. Robust formation control for multiple quadrotors with nonlinearities and disturbances. IEEE Transactions on Cybernetics, 2020, 50(4): 1362−1371 doi: 10.1109/TCYB.2018.2875559 [19] Wen G H, Lam J, Fu J J, Wang S. Distributed MPC-based robust collision avoidance formation navigation of constrained multiple USVs. IEEE Transactions on Intelligent Vehicles, 2024, 9(1): 1804−1816 doi: 10.1109/TIV.2023.3315367 [20] 周艳, 温广辉, 郑德智. 基于命令滤波反推方法的水面无人艇容错编队跟踪控制. 指挥与控制学报, 2024, 10(4): 412−419 doi: 10.3969/j.issn.2096-0204.2024.04.0412Zhou Yan, Wen Guang-Hui, Zheng De-Zhi. Fault-tolerant formation tracking control of unmanned surface vehicles based on command filtered backstepping approach. Journal of Command and Control, 2024, 10(4): 412−419 doi: 10.3969/j.issn.2096-0204.2024.04.0412 [21] Zhou Y, Zhou J L, Wang S, Wen G H. Practical fixed-time attitude consensus tracking for multi-quadrotor systems: A composite learning backstepping approach. IEEE Transactions on Circuits and Systems II: Express Briefs, 2024, 71(6): 3066−3070 doi: 10.1109/tcsii.2024.3354839 [22] Liu Y, Li H Y, Lu R Q, Zuo Z Y, Li X D. An overview of finite/fixed-time control and its application in engineering systems. IEEE/CAA Journal of Automatica Sinica, 2022, 9(12): 2106−2120 doi: 10.1109/jas.2022.105413 [23] Slotine J J E, Li W P. Applied Nonlinear Control. Englewood Cliffs: Prentice Hall, 1991. [24] Wang D, He H B, Liu D R. Adaptive critic nonlinear robust control: A survey. IEEE Transactions on Cybernetics, 2017, 47(10): 3429−3451 doi: 10.1109/TCYB.2017.2712188 [25] Zhao W B, Liu H, Lewis F L. Robust formation control for cooperative underactuated quadrotors via reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(10): 4577−4587 doi: 10.1109/TNNLS.2020.3023711 [26] Liu H, Li D J, Zuo Z Y, Zhong Y S. Robust three-loop trajectory tracking control for quadrotors with multiple uncertainties. IEEE Transactions on Industrial Electronics, 2016, 63(4): 2263−2274 -
下载:
计量
- 文章访问数: 72
- HTML全文浏览量: 52
- 被引次数: 0
