Reinforcement Learning Based Multi-Terrain Adaptive Whole-Body Control for a Quadruped Manipulator
A small Unitree Go1 robot dog fitted with a 1.5 kg custom arm walked over sandbags and down stairs sideways and then picked up objects named in plain language in 32 of 40 real trials (80%), using one learned controller for legs and arm, according to a University of Tokyo preprint. In simulation the controller failed 2 times in 4,096 arm-motion runs, while a conventional split controller failed in up to 13.77% of runs.
Can a single reinforcement-learned policy control the legs and arm of a small, payload-limited quadruped so that the gripper keeps a steady, task-consistent pose while the body pitches and rolls on slopes, stairs and soft ground?
On small quadrupeds an arm eats much of the payload budget and acts as a moving load. Terrain jolts travel through the body to the gripper, and posture changes that help the arm reach can undermine the balance needed to walk. Controllers that treat walking and reaching separately suffer most when the ground gets rough.
Earlier systems used whole-body optimisation, a locomotion controller with an inverse-kinematics arm tracker on top, or learned whole-body policies. Most were shown on flat ground, ramps or mild outdoor scenes; the paper's comparison table lists platforms from about 14.8 kg to about 90 kg, and only one, MLM at 19.9 kg, reports manipulation during stair or obstacle traversal.
The authors built a 13.5 kg system from a Unitree Go1 and a 1.5 kg Lite Arm with more than 60 cm reach, and trained one policy in Isaac Lab with task priorities: body regulation and contact first, locomotion second, manipulation third. The gripper's orientation is held relative to gravity while its position target is expressed in the body frame, a hybrid formulation meant to keep targets reachable while the body moves. Random payloads up to 0.5 kg and pushes were added in training. On the robot the policy runs on an NVIDIA Jetson Xavier NX at 49.9 Hz with 1.53 ms inference and 2.76 ms sensor-to-command latency; Grounding DINO and depth provide the target from a spoken-style text query.
In simulated repeated sideways arm motions at 0.4 and 0.8 m/s with 0.2 and 0.5 kg payloads, the split controller (locomotion plus IK arm) failed in 8.50% of runs in the slow, light case and 13.77% in the fast, heavy case, with gripper RMSE rising from 0.070 to 0.108 m; the whole-body policy failed twice in 4,096 runs (0.05%, MEKAWIRE) with RMSE of 0.057 to 0.069 m. Across terrain difficulty it also had lower body-motion error, gripper error and actuator overload exposure than the split controller. On the real robot it crossed a 30 degree slope and 10 cm stairs with the gripper held nearly level, and in language-guided grasping it succeeded 17 of 20 times on sandbag-and-sponge ground and 15 of 20 on a sideways stair descent, 32 of 40 overall. Of the 8 failures, 2 were falls, 4 were mislocated transparent objects, 1 a table collision and 1 a slipped grasp.
This is a preprint. Real-world comparisons with the robot's built-in MPC controller are qualitative, and the 80% grasp figure has no baseline run on the same course. Half the failures came from perceiving transparent objects, which the controller cannot fix. The platform is an older Go1 with a custom arm, payloads above 0.5 kg were not trained, and a heavier 3 kg arm raised actuator overload in simulation, so results may not transfer to commercial arms.
Inspection and site-patrol robots are moving from walking cameras to legged robots that also open, carry and pick. The result supports cheap, light arms on small quadrupeds instead of heavy arms on large ones, relevant to Unitree, to makers of compact arms and to Japanese plant and infrastructure operators facing staff shortages. Commercial use is likely two to three years out, after perception of difficult objects improves.
The full text is not republished here because the paper's license does not allow it. Read the original on arXiv.