PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviors

Can relative preference feedback steer a pretrained generative robot policy toward desirable behaviors that are extremely unlikely or absent under the initial policy distribution?

Users often want a robot to perform a task differently—faster, along a higher detour path, or with a different final state—but the initial policy rarely or never produces such behaviors. Existing preference-based adaptation methods mainly reinforce behaviors already represented in the initial policy, so they fail when desired rollouts never appear in the preference data.

Prior methods include reward-model-plus-RL pipelines such as FDPP, direct preference optimization (DPO/FlowPRO), imitation on filtered preferred rollouts (IFBC), DSRL, which reweights diffusion noise sampling, and CFGRL, which uses classifier-free guidance as a policy-improvement operator. These approaches primarily sharpen modes the policy already exhibits and do not reliably shift behavior into regions where the preferred trajectory has near-zero initial density.

PrefPI repeatedly alternates between generating rollouts, selecting the top m by user preference, and training two generative branches: one conditioned on those preferred rollouts and one on all accumulated successful rollouts. At inference, classifier-free guidance blends the two branches, amplifying the density ratio between the preferred and broad behavior distributions. The paper derives that top-m selection preserves latent utility order and that CFG monotonically improves expected utility under idealized assumptions. Repeating the loop lets previously inaccessible behaviors emerge as stepping stones.

On a real WidowX cube pick-and-place task, after fine-tuning PI0.5 with 100 demonstrations, PrefPI increased maximum object height from 10.7 cm to 19.8 cm with 150 preference-labeled trajectories over 3 iterations, and to 20.4 cm after 6 iterations. It reduced average task-completion time from 19.8 s to 10.8 s and increased throwing distance by 4.5x while success remained above 90%. In LIBERO simulations with PI0.5, speed improved 25–38% within 10 iterations (400 evaluated trajectories); detour height rose 65.1% within 320 trajectories and plate displacement rose 106% within 200 trajectories, though aggressive steering caused 20–30% success drops. On RoboMimic under matched feedback budgets, PrefPI achieved the strongest steering on CanDropPos and CanSpeed, while FDPP won on LiftDetour but was less consistent; IFBC and DSRL showed little shift and DPO shifted less. In a controlled diversity experiment, a high-diversity prior reached about 24 cm with >85% success versus about 20 cm and 50% success for a low-diversity prior.

The authors state that steering quality depends on the behavioral diversity of the initial policy; low-diversity priors lead to smaller shifts and lower success. Aggressive out-of-distribution steering trades off reliability, with simulation success drops of 20–30% and one RoboMimic task falling to roughly 30% success. Real-robot validation is limited to a single WidowX arm and a cube pick-and-place task; the paper does not report multi-task or multi-embodiment deployment, and no code release is mentioned.