-
摘要: 针对四轮独立转向与驱动(4WISD)移动机器人在复杂地形中依赖理想约束与参数整定、易受轮地非理想效应影响而泛化受限的问题, 构建一种无需显式模型计算的端到端深度强化学习底层控制策略. 该方法基于近端策略优化算法, 结合课程学习与域随机化构建多地形、多扰动训练过程, 并通过结构化状态–动作定义与物理启发奖励塑形, 使策略由本体观测直接生成四轮转向角与轮速命令, 实现多轮协同与侧滑抑制. 基于Isaac Sim高保真仿真平台, 在起伏粗糙地形和低摩擦平面两类工况下与PID-LS、MPC-LS级联基线进行对比. 结果表明, 该策略在工况变化下性能波动较小, 在粗糙地形中线速度跟踪更接近任务目标, 且在线计算开销较低, 单步耗时0.304 ms. 室外平整水泥地面实机验证表明, 该策略可基于ROS-PLC控制链路稳定在线运行, 三轴速度RMSE分别为0.169 m/s、0.213 m/s和0.116 rad/s; 单步策略推理耗时0.62 ms, 传感器同步与状态构建平均耗时56.15 ms. 结果验证了端到端数据驱动底层控制在4WISD平台无精确建模条件下的可行性, 为高自由度轮式移动平台复杂环境运动控制提供了创新路径.Abstract: To address the limited generalization of traditional control methods for four-wheel independent steering and driving (4WISD) mobile robots in complex terrains, caused by their reliance on idealized constraints and parameter tuning as well as their susceptibility to non-ideal wheel-ground interactions, an end-to-end deep reinforcement learning low-level control policy without explicit model-based computation is developed. Based on proximal policy optimization, the method integrates curriculum learning and physics-based domain randomization to construct a multi-terrain and multi-disturbance training process. With a structured state-action formulation and physically motivated reward shaping, the policy directly maps proprioceptive observations to four-wheel steering-angle and wheel-speed commands, enabling coordinated multi-wheel control and slip suppression. Experiments in the high-fidelity Isaac Sim simulator compare the proposed policy with PID-LS and MPC-LS cascaded baselines under rough undulating terrains and low-friction flat terrains. Results show that the proposed policy exhibits smaller performance fluctuations across conditions, achieves more accurate linear-velocity tracking on rough terrains, and maintains low online computational cost, with a single-step latency of 0.304 ms. Real-world validation on an outdoor flat concrete surface further demonstrates stable online execution through a ROS-PLC control pipeline, with velocity root mean square error (RMSE) values of 0.169 m/s, 0.213 m/s, and 0.116 rad/s for the three velocity components. The policy inference latency is 0.62 ms, while sensor synchronization and state construction take 56.15 ms on average. These results verify the feasibility of end-to-end data-driven low-level control for 4WISD platforms without precise modeling, offering an innovative approach to motion control of high-DOF wheeled mobile robots in complex environments.
-
图 1 端到端底层控制框架
注: 策略在复杂地形中由本体观测直接生成轮组命令, 无需 解析模型在线求解.
Fig. 1 End-to-end low-level control framework
图 2 四轮独立转向与驱动移动机器人系统结构
Fig. 2 Schematic of the four-wheel independent steering and driving mobile robot system architecture
图 3 基于Isaac Lab的端到端训练流程
注: 训练环境通过域随机化和外部扰动构造多地形交互场景, 并结合观测、动作、指令、奖励、终止与课程管理完成策略训练.
Fig. 3 End-to-end training pipeline based on Isaac Lab
图 4 4WISD移动机器人实物图(上)以及基于URDF (左下)与USD (右下)格式的仿真模型
Fig. 4 Physical photograph of the 4WISD mobile robot (top), and its simulation models in URDF (bottom left) and USD (bottom right) formats
图 5 训练过程中课程等级与平均回报的变化趋势
Fig. 5 The changing trends of curriculum level and mean reward during training
图 6 训练过程中六项关键奖励与惩罚指标的变化趋势
Fig. 6 The changing trends of six key reward and penalty components during training
图 7 不同地形下的端到端控制执行过程
注: 绿色箭头表示期望机体速度方向, 蓝色箭头表示实际执 行速度方向.
Fig. 7 End-to-end control execution process under different terrains
图 8 在室外平整水泥地面上的实机速度指令跟踪实验
Fig. 8 Real-world velocity-command tracking experiment on an outdoor flat concrete surface
图 9 实机端到端控制链路与ROS节点通信关系
注: 遥控手柄输入、轮式里程计、惯性传感器、关节状态和里 程计估计构成策略输入, 端到端控制节点输出轮速命令, 并由位置控制器和速度控制器执行.
Fig. 9 Real-world end-to-end control pipeline and ROS node communication relationship
表 1 奖励函数与训练相关关键超参数汇总
Table 1 Summary of key hyperparameters related to the reward function and training
符号 含义 取值 $ \sigma^2 $ 速度误差标准化系数 0.25 $ w_1 $ 线速度跟踪权重 2.0 $ w_2 $ 角速度跟踪权重 1.0 $ w_3 $ 垂直振动惩罚权重 1.0 $ w_4 $ 横滚/俯仰角速度惩罚权重 1.0 $ w_5 $ 动作变化率惩罚权重 0.2 $ w_6 $ 侧滑惩罚权重 0.001 
下载: 导出CSV
表 2 两类地形及其子配置下端到端控制策略与基线方法的控制性能对比(均值$ \pm $标准差)
Table 2 Control performance comparison between the end-to-end control policy and baseline methods under two terrain types and their sub-configurations (mean $ \pm $ standard deviation)
地形 方法 $ E_{\mathrm{lin}} $ (m/s) $ E_{\mathrm{ang}} $ (rad/s) $ Z_{\mathrm{rms}} $ (m/s) $ RP_{\mathrm{rms}} $ (rad/s) $ T_{\mathrm{ctrl}} $ (ms) $ {\cal{T}}_{1a} $ PID-LS 0.246 ± 0.102 0.120 ± 0.062 0.086 ± 0.008 0.130 ± 0.026 0.850 ± 0.021 MPC-LS 0.145 ± 0.125 0.070 ± 0.064 0.101 ± 0.012 0.160 ± 0.044 2.913 ± 0.088 end-to-end (PPO) 0.124 ± 0.057 0.116 ± 0.066 0.106 ± 0.009 0.173 ± 0.055 0.308 ± 0.018 $ {\cal{T}}_{1b} $ PID-LS 0.291 ± 0.137 0.132 ± 0.067 0.106 ± 0.012 0.159 ± 0.036 0.851 ± 0.015 MPC-LS 0.217 ± 0.151 0.098 ± 0.062 0.121 ± 0.015 0.209 ± 0.045 2.880 ± 0.089 end-to-end (PPO) 0.149 ± 0.070 0.119 ± 0.069 0.125 ± 0.018 0.205 ± 0.071 0.303 ± 0.020 $ {\cal{T}}_{2a} $ PID-LS 0.222 ± 0.088 0.101 ± 0.053 0.075 ± 0.007 0.061 ± 0.010 0.852 ± 0.019 MPC-LS 0.096 ± 0.094 0.050 ± 0.049 0.083 ± 0.008 0.064 ± 0.017 2.866 ± 0.094 end-to-end (PPO) 0.105 ± 0.070 0.105 ± 0.064 0.085 ± 0.008 0.081 ± 0.029 0.305 ± 0.019 $ {\cal{T}}_{2b} $ PID-LS 0.210 ± 0.090 0.136 ± 0.062 0.078 ± 0.009 0.060 ± 0.013 0.848 ± 0.021 MPC-LS 0.098 ± 0.079 0.063 ± 0.040 0.085 ± 0.009 0.062 ± 0.019 2.870 ± 0.100 end-to-end (PPO) 0.098 ± 0.057 0.102 ± 0.067 0.084 ± 0.006 0.085 ± 0.029 0.301 ± 0.030
下载: 导出CSV
-
[1] Li D Y, Song Y D, Huang D, Chen H N. Model-independent adaptive fault-tolerant output tracking control of 4WS4WD road vehicles. IEEE Transactions on Intelligent Transportation Systems, 2013, 14(1): 169−179 doi: 10.1109/TITS.2012.2211016 [2] Guo J H, Luo Y G, Li K Q. An adaptive hierarchical trajectory following control approach of autonomous four-wheel independent drive electric vehicles. IEEE Transactions on Intelligent Transportation Systems, 2018, 19(8): 2482−2492 doi: 10.1109/TITS.2017.2749416 [3] Liang Y, Li Y, Zheng L, Yu Z, Ren Y. Yaw rate tracking-based path-following control for four-wheel independent driving and four-wheel independent steering autonomous vehicles considering the coordination with dynamics stability. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 2021, 235(1): 260−272 doi: 10.1177/0954407020938490 [4] Liu X J, Wang G L, Chen K. Nonlinear model predictive tracking control with C/GMRES method for heavy-duty AGVs. IEEE Transactions on Vehicular Technology, 2021, 70(12): 12567−12580 doi: 10.1109/TVT.2021.3123176 [5] Ding T, Zhang Y H, Ma G C, Cao Z H, Zhao X W, Tao B. Trajectory tracking of redundantly actuated mobile robot by MPC velocity control under steering strategy constraint. Mechatronics, 2022, 84: Article No. 102779 doi: 10.1016/j.mechatronics.2022.102779 [6] Liu X X, Wang W, Li X L, Liu F S, He Z H, Yao Y Z, et al. MPC-based high-speed trajectory tracking for 4WIS robot. ISA Transactions, 2022, 123: 413−424 doi: 10.1016/j.isatra.2021.05.018 [7] Jeong Y, Kim D H, Youn M H, Park S, Li X J, Lee T. Model predictive control-based path tracking with four-wheel independent steering, driving, and braking autonomous vehicles on low friction road. In: Proceedings of the 23rd International Conference on Control, Automation and Systems (ICCAS). Busan, Korea: IEEE, 2023. 1427−1432 [8] Nguyen N T, Gangavarapu P T, Mandel N, Bruder R, Ernst F. Motion planning for 4WS vehicle with autonomous selection of steering modes via an MIQP-MPC controller. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Yokohama, Japan: IEEE, 2024. 9765−9771 [9] Mnih V, Badia A P, Mirza M, Graves A, Lillicrap T, Harley T, et al. Asynchronous methods for deep reinforcement learning. In: Proceedings of the International Conference on Machine Learning (ICML). New York, USA: PMLR, 2016. 1928−1937 [10] Lee J, Hwangbo J, Wellhausen L, Koltun V, Hutter M. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 2020, 5(47): Article No. eabc5986 doi: 10.1126/scirobotics.abc5986 [11] Miki T, Lee J, Hwangbo J, Wellhausen L, Koltun V, Hutter M. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 2022, 7(62): Article No. eabk2822 doi: 10.1126/scirobotics.abk2822 [12] Belmonte-Baeza A, Lee J, Valsecchi G, Hutter M. Meta reinforcement learning for optimal design of legged robots. IEEE Robotics and Automation Letters, 2022, 7(4): 12134−12141 doi: 10.1109/LRA.2022.3211785 [13] Margolis G B, Agrawal P. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In: Proceedings of the 6th Conference on Robot Learning (CoRL). Atlanta, USA: PMLR, 2023. 22−31 [14] Margolis G B, Yang G, Paigwar K, Chen B, Agrawal P. Rapid locomotion via reinforcement learning. The International Journal of Robotics Research, 2024, 43(4): 572−587 doi: 10.1177/02783649231224053 [15] 陈词, 余纪宇, 李超, 陆豪健, 高洪波, 熊蓉, 等. 基于预训练−微调框架的四足机器人结构−控制协同设计. 机器人, 2025, 47(5): 625−635 doi: 10.13973/j.cnki.robot.240239Chen Ci, Yu Ji-Yu, Li Chao, Lu Hao-Jian, Gao Hong-Bo, Xiong Rong, et al. Structure-control co-design of quadruped robots based on pre-training-fine-tuning framework. Robot, 2025, 47(5): 625−635 doi: 10.13973/j.cnki.robot.240239 [16] 张楠杰, 陈玉全, 季茂沁, 孙运康, 王冰. 面向不同粗糙程度地面的四足机器人自适应控制方法. 自动化学报, 2025, 51(7): 1585−1598 doi: 10.16383/j.aas.c240738Zhang Nan-Jie, Chen Yu-Quan, Ji Mao-Qin, Sun Yun-Kang, Wang Bing. Adaptive control method for quadruped robot facing floors of different roughness. Acta Automatica Sinica, 2025, 51(7): 1585−1598 doi: 10.16383/j.aas.c240738 [17] Li Z Y, Cheng X X, Peng X B, Abbeel P, Levine S, Berseth G, et al. Reinforcement learning for robust parameterized locomotion control of bipedal robots. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Xi'an, China: IEEE, 2021. 2811−2817 [18] Siekmann J, Godse Y, Fern A, Hurst J. Sim-to-real learning of all common bipedal gaits via periodic reward composition. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Xi'an, China: IEEE, 2021. 7309−7315 [19] Jeon S H, Heim S, Khazoom C, Kim S. Benchmarking potential based rewards for learning humanoid locomotion. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). London, UK: IEEE, 2023. 9204−9210 [20] Zhang Q, Cui P, Yan D, Sun J K, Duan Y Q, Han G, et al. Whole-body humanoid robot locomotion with human reference. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Abu Dhabi, UAE: IEEE, 2024. 11225−11231 [21] 武黎明, 王明明, 罗建军, 张大羽, 梁澄汐. 基于强化学习的空间机器人在轨装配柔顺控制方法. 机器人, 2025, 47(2): 239−248 doi: 10.13973/j.cnki.robot.240135Wu Li-Ming, Wang Ming-Ming, Luo Jian-Jun, Zhang Da-Yu, Liang Cheng-Xi. Compliant control method for on-orbit assembly by space robots based on reinforcement learning. Robot, 2025, 47(2): 239−248 doi: 10.13973/j.cnki.robot.240135 [22] Xu D G, Chen P, Zhou X H, Wang Y Z, Tan G Z. Deep reinforcement learning based mapless navigation for industrial AMRs: Advancements in generalization via potential risk state augmentation. Applied Intelligence, 2024, 54(19): 9295−9312 doi: 10.1007/s10489-024-05679-5 [23] Wang Y Z, Xie Y F, Xu D G, Shi J H, Fang S Y, Gui W H. Heuristic dense reward shaping for learning-based map-free navigation of industrial automatic mobile robots. ISA Transactions, 2025, 156: 579−596 doi: 10.1016/j.isatra.2024.10.026 [24] Wang Y Z, Xu D G, Xie Y F. Universal multi-mode kinematic controller for four-wheel independent steering and driving mobile robots. In: Proceedings of the China Automation Congress (CAC). Harbin, China: IEEE, 2025. [25] Mittal M, Yu C, Yu Q, Liu J, Rudin N, Hoeller D, et al. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 2023, 8(6): 3740−3747 doi: 10.1109/LRA.2023.3270034 -
下载:
计量
- 文章访问数: 253
- HTML全文浏览量: 129
- PDF下载量: 2
- 被引次数: 0
