-
摘要: 针对四轮独立转向与驱动(four-wheel independent steering and driving, 4WISD)移动机器人在复杂地形中依赖理想约束与参数整定、易受轮地非理想效应影响而泛化受限的问题, 构建一种无需显式模型计算的端到端深度强化学习(deep reinforcement learning, DRL)底层控制策略. 该方法基于近端策略优化(proximal policy optimization, PPO)算法, 结合课程学习与物理域随机化构建多地形、多扰动训练过程, 并通过结构化状态–动作定义与物理启发奖励塑形, 使策略由本体观测直接生成四轮转向角与轮速命令, 实现多轮协调与侧滑抑制. 基于Isaac Sim高保真仿真平台, 在起伏粗糙地形和低摩擦平面两类工况下与PID-LS、MPC-LS级联基线进行对比. 结果表明, 该策略在工况变化下性能波动较小, 在粗糙地形中线速度跟踪更接近任务目标, 且在线计算开销较低, 单步耗时为0.30 ms. 室外平整水泥地面实机验证表明, 该策略可基于ROS-PLC控制链路稳定在线运行, 三轴速度RMSE分别为0.169 m/s、0.213 m/s和0.116 rad/s; 单步策略推理耗时为0.62 ms, 传感器同步与状态构建平均耗时为56.15 ms. 结果验证了端到端数据驱动底层控制在4WISD平台无精确建模条件下的可行性, 为高自由度轮式移动平台复杂环境运动控制提供了创新路径.Abstract: To address the limited generalization of traditional control methods for four-wheel independent steering and driving (4WISD) mobile robots in complex terrains, caused by their reliance on idealized constraints and parameter tuning as well as their susceptibility to non-ideal wheel-ground interactions, an end-to-end deep reinforcement learning (DRL) low-level control policy without explicit model-based computation is developed. Based on Proximal Policy Optimization (PPO), the method integrates curriculum learning and physics-based domain randomization to construct a multi-terrain and multi-disturbance training process. With a structured state-action formulation and physically motivated reward shaping, the policy directly maps proprioceptive observations to four-wheel steering-angle and wheel-speed commands, enabling coordinated multi-wheel control and slip suppression. Experiments in Isaac Sim compare the proposed policy with PID-LS and MPC-LS cascaded baselines under rough undulating terrains and low-friction flat terrains. Results show that the proposed policy exhibits smaller performance fluctuations across conditions, achieves more accurate linear-velocity tracking on rough terrains, and maintains low online computational cost, with a single-step latency of 0.30 ms. Real-world validation on an outdoor flat concrete surface further demonstrates stable online execution through a ROS-PLC control pipeline, with velocity RMSE values of 0.169 m/s, 0.213 m/s, and 0.116 rad/s for the three velocity components. The measured policy inference latency is 0.62 ms, while sensor synchronization and state construction take 56.15 ms on average. These results verify the feasibility of end-to-end data-driven low-level control for 4WISD platforms without precise modeling, offering an innovative approach to motion control of high-DOF wheeled mobile robots in complex environments.
-
图 1 端到端底层控制框架, 策略在复杂地形中由本体观测直接生成轮组控制命令, 无需解析模型在线求解
Fig. 1 End-to-end low-level control framework, where the policy directly maps proprioceptive observations to wheel control commands on complex terrains without online analytical-model solving
图 2 四轮独立转向与驱动(4WISD)机器人系统结构示意
Fig. 2 Schematic of the four-wheel independent steering and driving (4WISD) robot system architecture
图 3 基于Isaac Lab的端到端训练流程, 训练环境通过域随机化和外部扰动构造多地形交互场景, 并结合观测、动作、指令、奖励、终止与课程管理完成策略训练
Fig. 3 End-to-end training pipeline in Isaac Lab, where domain randomization and external disturbances are used to construct multi-terrain interaction scenes, and policy training is performed through observation, action, command, reward, termination, and curriculum management
图 4 四轮独立驱动、转向(4WISD)机器人实物图(上)以及基于URDF(左下)与USD(右下)格式的仿真模型
Fig. 4 Photograph of the four-wheel independent steering and driving (4WISD) robot platform (top), and its simulation models in URDF (bottom left) and USD (bottom right) formats
图 5 训练过程中课程等级与平均奖励的变化趋势
Fig. 5 Evolution of curriculum level and mean reward during training
图 6 训练过程中六项关键奖励与惩罚指标的变化趋势
Fig. 6 Evolution of six key reward and penalty components during training
图 7 不同地形下的端到端控制执行过程, 绿色箭头表示期望机体速度方向, 蓝色箭头表示实际执行速度方向
Fig. 7 End-to-end control results under different terrains, where green arrows indicate desired body-velocity directions and blue arrows indicate actual execution-velocity directions
图 8 在室外平整水泥地面上的实机速度指令跟踪实验
Fig. 8 Real-world velocity-command tracking experiment on an outdoor flat concrete surface
图 9 实机端到端控制链路与ROS节点通信关系, 遥控输入、轮式里程计、惯性传感器、关节状态和里程计估计构成策略输入, 端到端控制节点输出转向与驱动命令, 并由位置控制器和速度控制器执行
Fig. 9 Real-world end-to-end control pipeline and ROS node communication, where remote-control input, wheel odometry, inertial sensing, joint states, and odometry estimation form the policy input, and the end-to-end control node outputs steering and driving commands executed by the position and velocity controllers
图 10 实机速度跟踪结果, 分别给出机体系速度$ v_x $、$ v_y $和$ \omega_z $的指令值(cmd, 蓝线)、实测值(meas, 红线)及误差(err, 绿虚线)随时间的变化
Fig. 10 Real-world velocity tracking results of the body-frame velocities $ v_x $, $ v_y $, $ \omega_z $, showing the commanded values (cmd, blue), measured values (meas, red), and errors (err, green dashed) over time
表 1 奖励函数与训练相关关键超参数汇总
Table 1 Summary of key hyperparameters related to the reward function and training
符号 含义 取值 $ \sigma^2 $ 速度误差标准化系数 0.25 $ w_1 $ 线速度跟踪权重 2.0 $ w_2 $ 角速度跟踪权重 1.0 $ w_3 $ 垂直振动惩罚权重 1.0 $ w_4 $ 横滚/俯仰角速度惩罚权重 1.0 $ w_5 $ 动作变化率惩罚权重 0.2 $ w_6 $ 侧滑惩罚权重 0.001 
下载: 导出CSV
表 2 两类地形及其子配置下端到端策略与基线方法的控制性能对比(均值$ \pm $标准差)
Table 2 Control performance comparison between the end-to-end policy and baseline methods under two terrain types and their sub-configurations (mean $ \pm $ standard deviation)
地形 方法 $ E_{\mathrm{lin}} $ (m/s) $ E_{\mathrm{ang}} $ (rad/s) $ Z_{\mathrm{rms}} $ (m/s) $ RP_{\mathrm{rms}} $ (rad/s) $ T_{\mathrm{ctrl}} $ (ms) $ {\cal{T}}_{1a} $ PID-LS 0.246 ± 0.102 0.120 ± 0.062 0.086 ± 0.008 0.130 ± 0.026 0.850 ± 0.021 MPC-LS 0.145 ± 0.125 0.070 ± 0.064 0.101 ± 0.012 0.160 ± 0.044 2.913 ± 0.088 End-to-End (PPO) 0.124±0.057 0.116±0.066 0.106±0.009 0.173±0.055 0.308±0.018 $ {\cal{T}}_{1b} $ PID-LS 0.291±0.137 0.132±0.067 0.106±0.012 0.159±0.036 0.851±0.015 MPC-LS 0.217±0.151 0.098±0.062 0.121±0.015 0.209±0.045 2.880±0.089 End-to-End (PPO) 0.149±0.070 0.119±0.069 0.125±0.018 0.205±0.071 0.303±0.020 $ {\cal{T}}_{2a} $ PID-LS 0.222±0.088 0.101±0.053 0.075±0.007 0.061±0.010 0.852±0.019 MPC-LS 0.096±0.094 0.050±0.049 0.083±0.008 0.064±0.017 2.866±0.094 End-to-End (PPO) 0.105±0.070 0.105±0.064 0.085±0.008 0.081±0.029 0.305±0.019 $ {\cal{T}}_{2b} $ PID-LS 0.210±0.090 0.136±0.062 0.078±0.009 0.060±0.013 0.848±0.021 MPC-LS 0.098±0.079 0.063±0.040 0.085±0.009 0.062±0.019 2.870±0.100 End-to-End (PPO) 0.098±0.057 0.102±0.067 0.084±0.006 0.085±0.029 0.301±0.030
下载: 导出CSV
-
[1] Li D Y, Song Y D, Huang D, Chen H N. Model-independent adaptive fault-tolerant output tracking control of 4WS4WD road vehicles. IEEE Transactions on Intelligent Transportation Systems, 2013, 14(1): 169−179 doi: 10.1109/TITS.2012.2211016 [2] Guo J H, Luo Y G, Li K Q. An adaptive hierarchical trajectory following control approach of autonomous four-wheel independent drive electric vehicles. IEEE Transactions on Intelligent Transportation Systems, 2018, 19(8): 2482−2492 doi: 10.1109/TITS.2017.2749416 [3] Liang Y, Li Y, Zheng L, Yu Z, Ren Y. Yaw rate tracking-based path-following control for four-wheel independent driving and four-wheel independent steering autonomous vehicles considering the coordination with dynamics stability. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 2021, 235(1): 260−272 doi: 10.1177/0954407020938490 [4] Liu X J, Wang G L, Chen K. Nonlinear model predictive tracking control with C/GMRES method for heavy-duty AGVs. IEEE Transactions on Vehicular Technology, 2021, 70(12): 12567−12580 doi: 10.1109/TVT.2021.3123176 [5] Ding T, Zhang Y H, Ma G C, Cao Z H, Zhao X W, Tao B. Trajectory tracking of redundantly actuated mobile robot by MPC velocity control under steering strategy constraint. Mechatronics, 2022, 84: 102779 doi: 10.1016/j.mechatronics.2022.102779 [6] Liu X X, Wang W, Li X L, Liu F S, He Z H, Yao Y Z, Ruan H P, Zhang T. MPC-based high-speed trajectory tracking for 4WIS robot. ISA Transactions, 2022, 123: 413−424 doi: 10.1016/j.isatra.2021.05.018 [7] Jeong Y, Kim D H, Youn M H, Park S, Li X J, Lee T. Model predictive control-based path tracking with four-wheel independent steering, driving, and braking autonomous vehicles on low friction road. In: Proceedings of the 23rd International Conference on Control, Automation and Systems (ICCAS). Busan, Korea: IEEE, 2023. 1427-1432 [8] Nguyen N T, Tej Gangavarapu P, Mandel N, Bruder R, Ernst F. Motion planning for 4WS vehicle with autonomous selection of steering modes via an MIQP-MPC controller. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Yokohama, Japan: IEEE, 2024. 9765-9771 [9] Mnih V, Badia A P, Mirza M, Graves A, Lillicrap T, Harley T, Silver D, Kavukcuoglu K. Asynchronous methods for deep reinforcement learning. In: Proceedings of the International Conference on Machine Learning (ICML). New York, USA: PMLR, 2016. 1928-1937 [10] Lee J, Hwangbo J, Wellhausen L, Koltun V, Hutter M. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 2020, 5(47): Article No. eabc5986 doi: 10.1126/scirobotics.abc5986 [11] Miki T, Lee J, Hwangbo J, Wellhausen L, Koltun V, Hutter M. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 2022, 7(62): Article No. eabk2822 doi: 10.1126/scirobotics.abk2822 [12] Belmonte-Baeza A, Lee J, Valsecchi G, Hutter M. Meta reinforcement learning for optimal design of legged robots. IEEE Robotics and Automation Letters, 2022, 7(4): 12134−12141 doi: 10.1109/LRA.2022.3211785 [13] Margolis G B, Agrawal P. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In: Proceedings of the 6th Conference on Robot Learning (CoRL). Atlanta, USA: PMLR, 2023. 22-31 [14] Margolis G B, Yang G, Paigwar K, Chen B, Agrawal P. Rapid locomotion via reinforcement learning. The International Journal of Robotics Research, 2024, 43(4): 572−587 doi: 10.1177/02783649231224053 [15] 陈词, 余纪宇, 李超, 陆豪健, 高洪波, 熊蓉, 王越. 基于预训练—微调框架的四足机器人结构—控制协同设计. 机器人, 2025, 47(5): 625−635 doi: 10.13973/j.cnki.robot.240239Chen Ci, Yu Ji-Yu, Li Cao, Lu Hao-Jian, Gao Hong-Bo, Xiong Rong, Wang Yue. Structure-control co-design of quadruped robots based on pre-training-fine-tuning framework. Robot, 2025, 47(5): 625−635 doi: 10.13973/j.cnki.robot.240239 [16] 张楠杰, 陈玉全, 季茂沁, 孙运康, 王冰. 面向不同粗糙程度地面的四足机器人自适应控制方法. 自动化学报, 2025, 51(7): 1585−1598 doi: 10.16383/j.aas.c240738Zhang Nan-Jie, Chen Yu-Quan, Ji Mao-Qin, Sun Yun-Kang, Wang Bing. Adaptive control method for quadruped robot facing floors of different roughness. Acta Automatica Sinica, 2025, 51(7): 1585−1598 doi: 10.16383/j.aas.c240738 [17] Li Z Y, Cheng X X, Peng X B, Abbeel P, Levine S, Berseth G, Sreenath K. Reinforcement learning for robust parameterized locomotion control of bipedal robots. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Xi'an, China: IEEE, 2021. 2811-2817 [18] Siekmann J, Godse Y, Fern A, Hurst J. Sim-to-real learning of all common bipedal gaits via periodic reward composition. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Xi'an, China: IEEE, 2021. 7309-7315 [19] Jeon S H, Heim S, Khazoom C, Kim S. Benchmarking potential based rewards for learning humanoid locomotion. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). London, UK: IEEE, 2023. 9204-9210 [20] Zhang Q, Cui P, Yan D, Sun J K, Duan Y Q, Han G, Zhao W, Zhang W N, Guo Y J, Zhang A. Whole-body humanoid robot locomotion with human reference. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Abu Dhabi, UAE: IEEE, 2024. 11225-11231 [21] 武黎明, 王明明, 罗建军, 张大羽, 梁澄汐. 基于强化学习的空间机器人在轨装配柔顺控制方法. 机器人, 2025, 47(2): 239−248 doi: 10.13973/j.cnki.robot.240135Wu Li-Ming, Wang Ming-Ming, Luo Jian-Jun, Zhang Da-Yu, Liang Cheng-Xi. Compliant control method for on-orbit assembly by space robots based on reinforcement learning. Robot, 2025, 47(2): 239−248 doi: 10.13973/j.cnki.robot.240135 [22] Xu D G, Chen P, Zhou X H, Wang Y Z, Tan G Z. Deep reinforcement learning based mapless navigation for industrial AMRs: Advancements in generalization via potential risk state augmentation. Applied Intelligence, 2024, 54(19): 9295−9312 doi: 10.1007/s10489-024-05679-5 [23] Wang Y Z, Xie Y F, Xu D G, Shi J H, Fang S Y, Gui W H. Heuristic dense reward shaping for learning-based map-free navigation of industrial automatic mobile robots. ISA Transactions, 2025, 156: 579−596 doi: 10.1016/j.isatra.2024.10.026 [24] Wang Y Z, Xu D G, Xie Y F. Universal multi-mode kinematic controller for four-wheel independent steering and driving mobile robots. In: Proceedings of the China Automation Congress (CAC). Harbin, China: IEEE, 2025. [25] Mittal M, Yu C, Yu Q, Liu J, Rudin N, Hoeller D, et al. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 2023, 8(6): 3740−3747 doi: 10.1109/LRA.2023.3270034 -
下载:
计量
- 文章访问数: 5
- HTML全文浏览量: 3
- 被引次数: 0
