Fiber: 휴머노이드 하나의 플래너로 상자, 문, 물체 잡기 학습, 성공률 42% 대 3.5% (아마존 FAR)

Abstract

Learning a single perceptive policy for diverse humanoid loco-manipulation requires unifying high-level task decisions without losing the state-dependent physical responses that make each interaction succeed. This makes the representation between high-level planning and whole-body control critical: kinematic or contact targets provide compact abstractions but do not directly encode interaction-specific control responses, while directly predicting low-level actions requires the planner to model high-frequency closed-loop feedback. We propose Fiber, a framework that uses the executable latent space of a frozen bfm (bfm) as a shared planning representation for heterogeneous interaction skills. Our lpp (lpp) maps independently acquired task-specific policies into this representation by identifying latent commands whose induced actions match their local control responses, enabling lower-rate planning while retaining the BFM’s high-frequency state feedback. After projection, we aggregate closed-loop latent rollouts from all skills and train a single perceptive flow planner entirely offline. We further introduce AH-BFM, an adaptive-horizon bfm trained with multi-\gamma fb (fb) learning to support both precise control and temporally extended whole-body behavior. We evaluate Fiber on box pickup, door opening, and object grasping in simulation and on a real Unitree G1 humanoid. Across 128 diverse variations per task in simulation, Fiber achieves 42.0\% mean success, more than tripling projected latent replay (11.6\%) and substantially outperforming kinematic planning (3.5\%). Real-world experiments further demonstrate sim-to-real transfer, skill chaining, and conditional steering through the same unified planner. Project website: https://fiberaa.github.io/.

I Introduction

Humanoid loco-manipulation requires both deciding what interaction to perform from visual and semantic context and determining how to realize it through balance, contact, and coordinated whole-body control. Recent work has achieved strong task-specific interaction skills, including object manipulation, articulated-object interaction, and object transport [1, 2], often using task-specific objectives, observations, and training procedures. Scaling these successes to a versatile policy requires unifying diverse task-level decisions without losing the physical responses that make each skill succeed.

A common strategy separates task-level decision making from whole-body control. A high-level policy predicts structured targets such as body motion, end-effector poses, or contacts, while a low-level controller realizes them on the robot. However, the same target may require different actions under different contacts, object states, or disturbances, so such representations do not directly capture the state-dependent control responses required for interaction. Predicting low-level actions avoids this abstraction, but offline learning then requires broad coverage of closed-loop corrective responses, which is difficult for contact-rich interaction. This exposes a central challenge for humanoid planning: how can we abstract low-level control complexity without abstracting away the physical responses that make interaction succeed?

A learned behavioral representation offers a way to resolve this tension. We build on a bfm that exposes an executable behavioral latent space. Given the current robot state and a latent command, bfm produces low-level actions at high frequency. A latent therefore indexes a state-dependent control policy rather than merely a desired motion, allowing high-level planning to operate over compact behavioral commands while the bfm retains high-frequency closed-loop physical feedback.

Using such an interface for versatile loco-manipulation still requires acquiring the interaction capabilities that the planner should invoke. One approach is to collect demonstrations for the desired interactions through a common data-acquisition pipeline [3, 4], but obtaining broad coverage of contact-rich and corrective behavior is expensive. Another approach is to first train task-specific interaction policies and distill them into a generalist. On-policy multi-expert distillation can improve closed-loop coverage by querying these policies at states visited by the generalist, but requires all policies and their environments to remain available during integration. We instead consider a decoupled setting: each interaction policy can be acquired independently, and only afterward incorporated into the shared behavioral interface.

Under this decoupled setting, different interaction policies need not share their observations, objectives, or task representations. They nevertheless share one quantity that directly describes their physical realization: the actions applied to the robot. A latent command from the frozen bfm is grounded in the same quantity, because it also produces robot actions conditioned on the current robot state. This common action space provides a direct way to relate an independently acquired interaction policy to the bfm latent representation.

Based on this insight, we propose Fiber, which represents independently acquired interaction policies, or experts, in the executable latent space of a frozen bfm. At the core of Fiber is lpp, which identifies latent commands by matching the actions induced by the bfm to an expert’s local state-dependent control responses. The resulting projected policy executes through the frozen bfm, preserving high-frequency state feedback while expressing heterogeneous interaction skills through a common lower-rate representation.

Because each expert is projected independently, the experts need not share an acquisition procedure or remain jointly available during subsequent training. After projection, we roll out the projected policies and collect their latent commands together with the perceptive and proprioceptive observations available to the final policy. These closed-loop latent rollouts are aggregated across skills to train a single perceptive planner entirely offline. At deployment, the planner predicts future latent commands from onboard observations and a high-level steering condition, while the frozen bfm provides the state-dependent whole-body responses required to execute them.

Finally, projection and planning impose different temporal requirements on the latent representation. Matching an expert’s control response favors precise short-horizon behavior, whereas lower-rate planning benefits from stable commands that preserve behavior over longer horizons. We therefore introduce AH-BFM, an adaptive-horizon bfm trained with multi-\gamma fb learning to support both control fidelity and temporally extended behavioral structure.

We evaluate Fiber on three heterogeneous interaction tasks—box pickup, door opening, and object grasping-in simulation and on a real Unitree G1. In simulation, lpp largely preserves expert performance on box pickup and door opening near the interaction regime. The unified perceptive planner achieves 42.0\% mean success, compared with 11.6\% for fixed projected latent replay and 3.5\% for kinematic planning. Real-world experiments further demonstrate direct sim-to-real transfer, interaction chaining, and waypoint-conditioned steering through the same planner.

Our contributions are threefold:

• A shared executable representation for humanoid interaction planning. We use the latent space of a frozen bfm as a behavioral interface that supports lower-rate high-level planning while retaining state-dependent interaction realization responses.
• Action-aligned latent policy projection. We introduce lpp, which maps independently acquired interaction experts into the shared bfm representation by aligning their local control responses in robot action space, decoupling skill acquisition from subsequent integration.
• Adaptive-horizon BFM and SF-guided behavior reuse. We introduce multi-\gamma fb learning and AH-BFM that balance precise control with temporally extended behavioral structure, together with sf (sf)-guided interaction-preserving behaviors reuse.

II Related Work

II-A Humanoid Controller and Behavioral Foundation Models

Humanoid control has increasingly moved from skill-specific policies toward shared controllers that support broad motor repertoires through a common conditioning interface. Motion-conditioned approaches unify locomotion and whole-body behaviors through dense or sparse motion specifications [5, 6], while recent systems extend unified control toward language-conditioned behaviors and humanoid loco-manipulation [7, 8]. These works demonstrate that diverse behaviors can be consolidated within a single controller, but the supported behaviors are typically acquired through the controller’s predefined training and conditioning interface. Our work instead considers a complementary setting: integrating interaction behaviors acquired independently of this interface into an already-trained controller.

bfms expose a learned behavioral latent space as such a shared control interface. BFM-Zero [9] uses fb to learn a promptable humanoid behavioral space, while subsequent behavioral inference methods show that expert behaviors can be recovered by optimizing within a frozen fb latent space [10]. These results establish latent inference as an effective mechanism for behavior reproduction. We build on this capability but study its use as an interface for interaction representation, where the latent sequence must capture temporally varying local control responses rather than only recover a stationary behavior or desired state trajectory. This setting further exposes a trade-off in conventional fb learning: the successor horizon favorable for precise local tracking differs from that required to preserve long-term behavior. Our adaptive-horizon bfm is designed to retain both properties within the same behavioral space.

II-B Humanoid Loco-Manipulation and Representation

Humanoid loco-manipulation requires specifying not only the desired motion of the robot, but also how it should respond to physical interaction with objects and the environment. A broad family of methods makes interaction explicit through structured task representations, including contact sequences, end-effector constraints, task-motion objectives, distance fields, and interaction-preserving references [11, 12, 13, 14, 15, 16, 17]. Such representations provide effective and often interpretable interfaces for learning diverse skills. However, they specify interaction through variables chosen in advance; the resulting task description does not directly encode the state-dependent control response induced by contact and environment dynamics.

Alternatively, interaction responses can be learned directly through task-specific rl (rl) or demonstrations [18, 19, 20, 21, 22, 23, 24, 25]. Teleoperation and human-video-based pipelines further broaden the sources from which such behaviors can be acquired [26, 27, 28, 29, 30, 31, 32]. These approaches can capture interaction-specific control responses through rl, but the acquired behavior generally remains tied to the specialist policy or downstream training process. In contrast, we decouple interaction acquisition from execution: demonstrated state-action responses are projected into the native latent interface of a frozen behavioral foundation controller, allowing independently acquired interactions to share a common executable substrate.

III Methodology

III-A Overview

Our goal is to express independently acquired interaction skills in a shared behavioral interface that preserves their state-dependent physical realization and supports unified perceptive planning. Fiber uses the latent space of a frozen bfm as this interface and maps each interaction expert into it by aligning the expert’s local control responses with the actions induced by bfm latent commands.

As illustrated in Fig. 1, Fiber proceeds in three stages. First, interaction skills are acquired independently with task-appropriate supervision. Second, lpp maps each expert into the frozen bfm latent space by matching the actions induced by latent commands to the expert’s local control responses. The resulting projected policies are rolled out directly through the shared behavioral interface, after which the original experts are no longer required. Finally, their closed-loop latent rollouts are aggregated to train a single perceptive flow planner entirely offline, conditioned on onboard observations and high-level steering commands.

This latent space must support both precise local interaction and temporally extended whole-body behavior. We therefore introduce AH-BFM, an adaptive-horizon bfm trained with multi-\gamma fb learning. We consider

\Gamma=\{\gamma_{s},\gamma_{l}\},\qquad\gamma_{s}<\gamma_{l},

where \gamma_{s} emphasizes local transition fidelity and \gamma_{l} preserves longer-horizon behavioral structure. We condition the forward representation F on \gamma while sharing the backward representation B, and normalize the corresponding values as

\widetilde{Q}_{\gamma}^{\mathrm{FB}}=(1-\gamma)Q_{\gamma}^{\mathrm{FB}}.

The shared actor adaptively aggregates the horizon-specific objectives,

J_{\pi}=\mathbb{E}\left[\tau\log\sum_{\gamma\in\Gamma}p(\gamma)\exp\left(\frac{\widetilde{Q}_{\gamma}^{\mathrm{FB}}(s,\pi_{\mathrm{BFM}}(s,z),z)}{\tau}\right)\right],

with gradient weights

w_{\gamma}\propto p(\gamma)\exp\left(\frac{\widetilde{Q}_{\gamma}^{\mathrm{FB}}(s,\pi_{\mathrm{BFM}}(s,z),z)}{\tau}\right).

Here, p(\gamma) defines a prior over horizons and \tau controls the aggregation sharpness. Because the weights depend on the current state and latent command, the shared actor can combine short- and long-horizon supervision within the same behavioral space.

III-B Heterogeneous Skill Acquisition

Fiber allows interaction skills to be acquired with task-appropriate supervision, observations, and training objectives. We consider three complementary acquisition routes.

Motion-based acquisition

For interactions with suitable humanoid–object motion references, such as grasping [33] and object pickup [29], we train task-specific experts by combining motion imitation with interaction objectives.

Reward-shaping-based acquisition

Some interactions admit a natural decomposition into successive stages and can therefore be acquired directly through staged reward shaping. For example, opening an articulated door [2] can be trained by progressively rewarding approach, contact establishment, handle manipulation, and door opening. This provides a direct route to task-specific experts without requiring motion references, while allowing each stage to use task-appropriate observations and rewards.

sf-guided behavior reuse

The pretrained bfm provides an additional route for extending acquired interactions with whole-body capabilities already represented in its latent space. For an interaction I, such as carrying box, we learn a latent adapter A_{\psi_{I}} while keeping the bfm frozen:

\hat{z}_{t}=A_{\psi_{I}}(s_{t},z^{N}),\qquad a_{t}=\pi_{\mathrm{BFM}}(s_{t},\hat{z}_{t}),

where z^{N} specifies the native behavior to be incorporated.

We optimize the adapter using the interaction reward together with a successor-feature-guided preservation term,

r_{t}=r_{t}^{I}+\lambda_{\mathrm{SF}}B(s_{t})^{\top}z^{N}.

For a fixed z^{N},

\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}B(s_{t})^{\top}z^{N}\right]=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}B(s_{t})\right]^{\top}z^{N},

so the second term encourages the adapted rollout to preserve the behavior specified by z^{N} while r_{t}^{I} maintains the interaction. This enables acquired interactions to be extended with native skills, such as running or kneeling while carrying.

Motion imitation, task-specific reward learning, and successor-feature-guided behavior reuse each produce task-specific experts, all of which are subsequently projected into the shared bfm latent space through lpp.

III-C Action-Aligned Latent Policy Projection

Externally acquired experts must be converted into a representation suitable for offline planner training. Kinematic trajectories provide compact behavioral targets, but do not uniquely determine the control responses required to realize an interaction. Expert actions provide this missing physical information, yet directly modeling high-frequency actions requires covering a broad state-dependent control distribution. We instead use the frozen bfm to map expert behavior into a lower-rate latent representation that preserves both its executable response and native behavioral semantics.

For expert \pi_{i}^{E}, we learn a local projector P_{\psi_{i}} that maps the expert observation at update time t_{k} to a latent,

z_{k}=P_{\psi_{i}}(o_{t_{k}}).

Between consecutive updates, we interpolate from the previous latent z_{k-1} to z_{k} using SLERP to obtain a latent command at each control step,

z_{t_{k}+h}=\operatorname{SLERP}\!\left(z_{k-1},z_{k},\alpha_{h}\right),\quad h=0,\ldots,H_{P}-1.

We identify the latent command primarily through its induced actions. Given expert actions

a_{t}^{E_{i}}=\pi_{i}^{E}(o_{i,t}^{E}),

we define the short-horizon action-alignment objective

\mathcal{L}_{\mathrm{act}}=\mathbb{E}_{P_{\psi_{i}}}\left[\frac{1}{H_{P}}\sum_{h=0}^{H_{P}-1}\left\|\pi_{\mathrm{BFM}}(s_{t_{k}+h},z_{t_{k}+h})-a_{t_{k}+h}^{E_{i}}\right\|_{2}^{2}\right].

Here, P_{\psi_{i}} denotes the on-policy state distribution induced by the current projector executed through the frozen BFM. This directly constrains the projected latent by matching the expert’s responses, rather than by a predefined kinematic or contact representation target.

Action alignment alone, however, can leave the latent underdetermined: different commands may induce similar local actions while corresponding to different behavioral directions in the bfm space. Recall that the bfm latent z acts as a policy index through the forward–backward objective F(s,a)^{\top}z, while B(s) provides the corresponding state-side behavioral representation. We therefore anchor each projected command to the native semantic representation of its local state,

\mathcal{L}_{\mathrm{sem}}=\mathbb{E}\left[\left\|z_{k}-B(s_{t_{k}})\right\|_{2}^{2}\right].

The complete projection objective is

\mathcal{L}_{\mathrm{proj}}=\mathcal{L}_{\mathrm{act}}+\lambda_{\mathrm{sem}}\mathcal{L}_{\mathrm{sem}}.

The action term determines the latent according to the expert’s required control response, while the semantic term selects solutions consistent with the native behavioral geometry of the bfm. This reduces action-equivalent latent ambiguity and yields more coherent latent trajectories for downstream planning. To obtain a robust projection beyond nominal expert rollouts, we train each P_{\psi_{i}} independently using single-skill on-policy data aggregation [35].

After projection, the original expert is no longer required. We roll out P_{\psi_{i}} under randomized initial states and environment configurations and collect

\mathcal{D}_{i}^{Z}=\left\{(d_{t},p_{t},u_{t},z_{t})\right\},

where d_{t} and p_{t} denote the perceptive and proprioceptive observations available to the downstream planner, and u_{t} denotes the corresponding high-level task condition. These latent rollouts provide executable and semantically regularized targets that can be combined across independently acquired skills for offline latent planning.

III-D Unified Latent Flow Planning

Given the collected latent rollouts, we train a single perceptive planner to generate future behavioral commands entirely offline. At each planning step, the policy is conditioned on onboard depth d_{t}, proprioception p_{t}, and a compact steering token c_{t}:

c_{t}=E_{\mathrm{steer}}(u_{t}).

The steering input u_{t} specifies the desired high-level behavior and can be instantiated by different conditioning modalities, such as a language command or a navigation waypoint. This separates the representation of task intent from the perceptive latent planner itself.

The planner predicts a horizon of future bfm commands

x_{1}=\operatorname{vec}\left(z_{t},z_{t+1},\ldots,z_{t+H-1}\right)

using conditional flow matching. Given Gaussian noise x_{0}\sim\mathcal{N}(0,I) and flow time \tau\sim\mathcal{U}(0,1), we construct the linear interpolation x_{\tau}=(1-\tau)x_{0}+\tau x_{1}. Then a conditional network predicts the clean latent, and is trained with

\mathcal{L}_{\mathrm{flow}}=\mathbb{E}\left[\left\|\hat{x}_{1}-x_{1}\right\|_{2}^{2}\right].

At deployment, the planner repeatedly observes the current depth and proprioceptive state, constructs the steering token from the desired high-level condition, and generates a future latent trajectory. The leading latent commands are executed through the frozen bfm, after which the planner replans from the newly observed state. This receding-horizon execution allows the same policy to continuously adapt its whole-body behavior to the perceived scene while following different steering conditions.

Through this pipeline, skills originating from heterogeneous acquisition procedures are converted into a common executable latent representation and subsequently learned by a single offline perceptive planner.

IV Experiments

We evaluate Fiber through simulation and real-world experiments to answer four questions:

Q1: How faithfully can heterogeneous interaction experts be projected into the shared latent space of a frozen BFM?

Q2: Can these projected skills be jointly learned by a single planner for versatile humanoid loco-manipulation?

Q3: Can the unified planner chain projected skills and flexibly steer their execution through various conditions?

Q4: Which components of Fiber are critical for reliable latent projection and downstream planning?

For each projected skill, we collect 20K episodes, and merge them across tasks for Fiber training. Language conditions are assigned according to the underlying task, while waypoint conditions are labeled from the robot trajectories in the collected rollouts.

IV-A Simulation Evaluation

We evaluate in IsaacSim, with 128 variations per task: box size and initial placement for pickup, door geometry for opening, and object identity for grasping. We do not report on-policy multi-expert distillation because the experts use different environments, observation spaces, and codebases, and therefore do not share the online rollout interface required for joint distillation.

Action-aligned latent projection

To answer Q1, we compare each task-specific expert with its lpp projector in Tab. I. lpp largely preserves expert performance on box pickup and door opening, while fine-grained grasping is more challenging, dropping from 97.0\% to 52.9\%.

We further compare representations using targets derived from the same successful expert trajectories. Reference-tracking baselines, including the bfm’s own backward representation B(s), perform poorly, whereas action-aligned projected replay achieves substantially higher success on box pickup and door opening. Notably, FALCON [21], despite learning force-adaptive end-effector tracking under randomized external forces, achieves only 0.4\%/2.7\% on door opening. These results show that kinematic, behavioral, or generic force-adaptive tracking alone does not recover the task-specific control responses required for interaction, motivating action-aligned latent projection.

Unified perceptive planning

To answer Q2, we compare replaying lpp-produced latent trajectories with the unified perceptive planner. Fiber improves mean success from 11.6\% to 42.0\%, showing that high-frequency bfm feedback alone is insufficient and that task-level latent replanning remains important. Under the same perceptive planning setup, the Action Planner fails on all tasks, while the Kinematic Planner reaches only 3.5\% mean success. Directly planning low-level actions removes high-frequency state feedback between planner updates, whereas predicting kinematic targets does not specify the interaction-specific control response, which leads to failure on contact-rich tasks; Fiber retains both through latent planning over bfm.

Skill chaining and conditional steering

To answer Q3, we further test whether the unified planner can compose projected skills under changing high-level commands. As shown in Fig. 2, the same policy sequentially opens the door, picks up the box, and carries it back through the doorway within a single rollout, demonstrating closed-loop chaining of heterogeneous interaction skills.

IV-B Real-World Evaluation

We further deploy Fiber on a Unitree G1 equipped with an onboard RealSense depth camera to evaluate real-world transfer of the unified planner (Q2) and interaction chaining and steering (Q3). Both the planner and AH-BFM run on a single NVIDIA RTX PRO 5000 Blackwell laptop. With TensorRT, the planner runs at 5 Hz using 10 Heun steps and rtc (rtc) (\sim20 ms per inference), while AH-BFM runs at 50 Hz (\sim1 ms per forward pass).

Tab. II reports success rates for box pickup, door opening, and object grasping. All experts, latent projectors, and the unified planner are trained entirely in simulation. Fiber succeeds in 6/10 box-pickup trials and 4/10 door-opening trials, demonstrating direct sim-to-real transfer of the latent planning interface. Object grasping is more challenging (2/10), where inaccurate depth estimates for the small object compound the fine-grained control difficulty. Beyond task success, Fig. 3 highlights the key advantage of our representation: planned latents remain grounded in closed-loop physical responses. When the box becomes unstable, the robot repositions its hands underneath for support in (a); when opening a heavy door, it leans its whole body into the interaction to generate sufficient force in (c).

To answer Q3, we further evaluate post-pickup behavior chaining and waypoint steering. After picking up a box, the same planner follows root-waypoint commands while maintaining the grasp; when disturbed during carrying, it can stabilize on its knees, recover, and continue execution, as shown in Fig. 3 (b). These results show that the unified planner can chain interaction and locomotion behaviors while remaining steerable through high-level waypoint commands.

IV-C Ablation and Analysis

To answer Q4, we analyze the two components underlying our latent planning interface: the adaptive-horizon BFM and the action-aligned latent representation.

Effect of the adaptive-horizon BFM

We first evaluate how the BFM temporal horizon affects both behavior representation and downstream projection.

For behavior representation, we evaluate AH-BFM using two complementary metrics. Joint-MAE measures instantaneous motion-tracking accuracy, while BE-EMD measures how well a fixed latent command preserves a reference behavior over increasing embedding temporal horizons: for each reference clip, we infer a single latent through lpp, roll out the BFM under this fixed command, and measure its discrepancy from the reference.

As shown in Tab. III, the short-horizon model \gamma=0.8 achieves the lowest BE-EMD at 0.2 s, whereas \gamma=0.98 better preserves behavior over longer horizons. AH-BFM achieves the lowest Joint-MAE and the lowest BE-EMD at 0.5, 1, and 2 s. Adaptive multi-\gamma learning therefore provides a better trade-off between precise local control and temporally extended behavioral structure than either fixed horizon alone.

This advantage also transfers to interaction projection: in Tab. I, replacing the fixed-\gamma BFM (\gamma=0.98) with AH-BFM increases mean lpp success from 68.8\% to 72.8\%, with substantial gains on box pickup and door opening.

Effect of action-aligned latent representation

The reference-prompted results in Tab. I isolate the importance of action alignment. B(s) tracking and projected replay are derived from the same successful expert trajectories, but the former identifies behavior from the expert state trajectory whereas lpp identifies latent commands from the expert’s action responses. Their large performance difference on pickup box and door opening shows that behavioral-state matching alone does not retain interaction-critical feedback. At the other extreme, the Action Planner fails in all tasks, showing that low-rate action planning cannot replace the high-frequency state feedback provided by the frozen bfm.

The gap between the feedback-conditioned lpp projector and fixed projected replay further shows that projected behaviors cannot generally be reduced to open-loop latent trajectories. Conversely, the improvement from projected replay to Fiber shows that a unified perceptive planner can recover task-level latent feedback from onboard observations. Together, these comparisons separate the roles of action-aligned representation, latent feedback, and perceptive replanning in Fiber.

V Conclusions and Limitations

We presented Fiber, an action-aligned latent planning framework for versatile humanoid loco-manipulation. lpp projects heterogeneous interaction experts into a frozen BFM by matching their short-horizon control responses, producing executable latent trajectories for unified offline perceptive planning. Adaptive multi-\gamma fb learning supports both precise control and temporally extended behavior, while successor-feature-guided learning enables interaction-preserving behavior reuse. Our simulation and real-world experiments demonstrate effective expert projection and perception-conditioned execution across diverse interaction tasks.

While our approach enables versatile humanoid loco-manipulation through action-aligned latent planning, extending the underlying BFM toward more dynamic whole-body behaviors, such as stair traversal, remains worth exploring. Moreover, incorporating richer visual and tactile feedback together with online latent adaptation could support interaction under more diverse contact conditions and facilitate the continual integration of new skills. We hope our work provides insights into scalable humanoid skill acquisition and opens new directions for unified whole-body planning.

References

1. [1] T. He, Z. Wang, H. Xue, Q. Ben, Z. Luo, W. Xiao, Y. Yuan, X. Da, F. Castañeda, S. Sastry, C. Liu, G. Shi, L. Fan, and Y. Zhu (2026) VIRAL: visual sim-to-real at scale for humanoid loco-manipulation. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13430–13441.
2. [2] H. Xue, T. He, Z. Wang, Q. Ben, W. Xiao, Z. Luo, X. Da, F. Castañeda, G. Shi, S. Sastry, L. J. Fan, and Y. Zhu (2026) Opening the sim-to-real door for humanoid pixel-to-action policy transfer. In Conference on Computer Vision and Pattern Recognition (CVPR),
3. [3] Z. Li, Z. Zhang, Y. Wei, W. Zhang, X. Yuan, P. Zhi, G. Li, X. Guo, F. Gao, J. Yang, and S. Zhang (2026) \omega-0: a latent predictive world action model for concurrent humanoid loco-manipulation. External Links: 2608.06375
4. [4] J. Zheng, T. Ma, Y. Fan, Z. Wang, S. Yang, and J. Liang (2026) MotionWAM: towards foundation world action models for real-time humanoid loco-manipulation. External Links: 2606.09215
5. [5] C. Tessler, Y. Guo, O. Nabati, G. Chechik, and X. B. Peng (2024) Maskedmimic: unified physics-based character control through masked motion inpainting. ACM Transactions on Graphics (TOG) 43 (6), pp. 1–21.
6. [6] Q. Liao, T. E. Truong, X. Huang, Y. Gao, G. Tevet, K. Sreenath, and C. K. Liu (2026) BeyondMimic: from motion tracking to versatile humanoid control via guided diffusion. Science Robotics 11 (117), pp. eadx8924. External Links: Document
7. [7] H. Jiang, J. Chen, Q. Bu, L. Chen, M. Shi, Y. Zhang, D. Li, C. Suo, C. Wang, Z. Peng, and H. Li (2026) WholeBodyVLA: towards unified latent vla for whole-body loco-manipulation control. In International Conference on Learning Representations (ICLR),
8. [8] S. Wei, H. Jing, B. Li, Z. Zhao, J. Mao, Z. Ni, S. He, S. Zang, X. Liu, K. Kang, J. Liu, W. Yuan, M. Pavone, D. Huang, and Y. Wang (2026) \Psi_{0}: An open foundation model towards universal humanoid loco-manipulation. In Robotics: Science and Systems (RSS), External Links: Document
9. [9] Y. Li, Z. Luo, T. Zhang, C. Dai, A. Kanervisto, A. Tirinzoni, H. Weng, K. Kitani, M. Guzek, A. Touati, A. Lazaric, M. Pirotta, and G. Shi (2026) BFM-Zero: a promptable behavioral foundation model for humanoid control using unsupervised reinforcement learning. In International Conference on Learning Representations (ICLR),
10. [10] M. Pirotta, A. Tirinzoni, A. Touati, A. Lazaric, and Y. Ollivier (2024) Fast imitation via behavior foundation models. In International Conference on Learning Representations (ICLR),
11. [11] C. Zhang, W. Xiao, T. He, and G. Shi (2024) WoCoCo: learning whole-body humanoid control with sequential contacts. In Conference on Robot Learning (CoRL), pp. 455–472.
12. [12] Y. Li, Y. Zhang, W. Xiao, C. Pan, H. Weng, G. He, T. He, and G. Shi (2025) Hold my beer: learning gentle humanoid locomotion and end-effector stabilization control. In Conference on Robot Learning (CoRL), Proceedings of Machine Learning Research, Vol. 305, pp. 4506–4523.
13. [13] J. Ren, J. Long, T. Huang, H. Wang, Z. Wang, F. Jia, W. Zhang, J. Wang, P. Luo, and J. Pang (2025) Humanoid goalkeeper: learning from position conditioned task-motion constraints. External Links: 2510.18002
14. [14] Y. Lin, J. Cui, Y. Li, B. Jia, Y. Zhu, and S. Huang (2026) LessMimic: long-horizon humanoid interaction with unified distance field representations. External Links: 2602.21723
15. [15] L. Yang, X. Huang, Z. Wu, A. Kanazawa, P. Abbeel, C. Sferrazza, C. K. Liu, R. Duan, and G. Shi (2026) Omniretarget: interaction-preserving data generation for humanoid whole-body loco-manipulation and scene interaction. In International Conference on Robotics and Automation (ICRA),
16. [16] S. Chen, S. Zhao, Z. Wu, J. Li, G. Shi, and C. K. Liu (2026) SceneBot: contact-prompted general humanoid whole body tracking with scene-interaction. External Links: 2606.27581
17. [17] R. Yu, X. Lin, J. Ma, Y. Wang, K. Luo, J. Ji, H. Wang, W. Wang, R. Zhang, P. Tan, T. Wu, R. Dai, Q. Chen, and L. Han (2026) OmniContact: chaining meta-skills via contact flow for generalizable humanoid loco-manipulation. External Links: 2606.26201
18. [18] Y. Wang, J. Lin, A. Zeng, Z. Luo, J. Zhang, and L. Zhang (2023) PhySHOI: physics-based imitation of dynamic human-object interaction. External Links: 2312.04393
19. [19] Y. Wang, Q. Zhao, R. Yu, H. W. Tsui, A. Zeng, J. Lin, Z. Luo, J. Yu, X. Li, Q. Chen, J. Zhang, L. Zhang, and P. Tan (2025) Skillmimic: learning basketball interaction skills from demonstrations. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17540–17549.
20. [20] R. Yu, Y. Wang, Q. Zhao, H. W. Tsui, J. Wang, P. Tan, and Q. Chen (2025) Skillmimic-v2: learning robust and generalizable interaction skills from sparse and noisy demonstrations. In ACM SIGGRAPH Conference Papers, pp. 1–11.
21. [21] Y. Zhang, Y. Yuan, P. Gurunath, I. Gupta, S. Omidshafiei, A. Agha-mohammadi, M. Vazquez-Chanlatte, L. Pedersen, T. He, and G. Shi (2026) Falcon: learning force-adaptive humanoid loco-manipulation. In Learning for Dynamics and Control (L4DC), pp. 265–281.
22. [22] Y. Du, Y. Li, B. Jia, Y. Lin, P. Zhou, W. Liang, Y. Yang, and S. Huang (2026) Learning human-humanoid coordination for collaborative object carrying. In International Conference on Robotics and Automation (ICRA),
23. [23] Y. Wang, Q. Zhao, Y. F. Lau, R. Yu, H. W. Tsui, Q. Chen, J. Wang, J. Pang, and P. Tan (2026) HumanX: toward agile and generalizable humanoid interaction skills from human videos. External Links: 2602.02473
24. [24] D. Li, X. Chen, Q. Wu, B. Chen, S. Wu, H. Wu, G. Zhang, L. Li, M. Zhou, D. Xiang, J. Ma, Q. Zhang, and R. Xu (2026) HAIC: humanoid agile object interaction control via dynamics-aware world model. In Robotics: Science and Systems (RSS), External Links: Document
25. [25] Z. Wang, Z. Wu, P. Abbeel, R. Duan, J. Malik, C. Sferrazza, C. K. Liu, G. Shi, and A. Kanazawa (2026) Counterfactual video generation enables scalable humanoid loco-manipulation. External Links: 2609.38172
26. [26] T. He, Z. Luo, X. He, W. Xiao, C. Zhang, W. Zhang, K. Kitani, C. Liu, and G. Shi (2024) OmniH2O: universal and dexterous human-to-humanoid whole-body teleoperation and learning. In Conference on Robot Learning (CoRL), pp. 1516–1540.
27. [27] Y. Ze, Z. Chen, J. P. Araújo, Z. Cao, X. B. Peng, J. Wu, and C. K. Liu (2025) TWIST: teleoperated whole-body imitation system. In Conference on Robot Learning (CoRL), pp. 2143–2154.
28. [28] Y. Li, Y. Lin, J. Cui, T. Liu, W. Liang, Y. Zhu, and S. Huang (2025) CLONE: closed-loop whole-body humanoid teleoperation for long-horizon tasks. In Conference on Robot Learning (CoRL), Proceedings of Machine Learning Research, Vol. 305, pp. 4493–4505.
29. [29] H. Weng, Y. Li, N. Sobanbabu, Z. Wang, Z. Luo, T. He, D. Ramanan, and G. Shi (2025) Hdmi: learning interactive humanoid whole-body control from human videos. External Links: 2509.16757
30. [30] R. Nai, B. Zheng, J. Zhao, H. Zhu, S. Dai, Z. Chen, Y. Hu, Y. Hu, T. Zhang, C. Wen, and Y. Gao (2026) Humanoid manipulation interface: humanoid whole-body manipulation from robot-free demonstrations. External Links: 2602.06643
31. [31] M. Shi, S. Peng, J. Chen, H. Jiang, T. Li, P. Luo, D. Huang, H. Li, and L. Chen (2026) Unlocking in-the-wild loco-manipulation with robot-free egocentric demonstration. In Robotics: Science and Systems (RSS), External Links: Document
32. [32] T. Wu, X. Kong, Y. Chen, Q. Yu, H. Ye, J. Li, Y. Wang, and H. Dong (2026) SUGAR: a scalable human-video-driven generalizable humanoid loco-manipulation learning framework. External Links: 2605.20373
33. [33] T. Xie, H. Zhang, J. Park, Z. Wang, B. Wen, J. Li, X. Li, Q. Ben, H. Weng, Y. Ye, D. Minor, T. Wang, C. Jiang, S. Fidler, J. Kautz, L. Fan, Y. Zhu, Z. Luo, U. Iqbal, and Y. Yuan (2026) GRAIL: generating humanoid loco-manipulation from 3D assets and video priors. External Links: 2606.05160
34. [34] Z. Luo, Y. Yuan, T. Wang, C. Li, F. Castañeda, S. Chen, Z. Cao, J. Li, D. Minor, Q. Ben, J. Park, D. Sami, Z. Wang, X. Da, R. Ding, C. Hogg, L. Song, E. Lim, E. Jeong, T. He, H. Xue, W. Xiao, S. Yuen, J. Kautz, Y. Chang, U. Iqbal, L. J. Fan, and Y. Zhu (2026) SONIC: supersizing motion tracking for natural humanoid whole-body control. Science Robotics 11 (117), pp. eaed4592. External Links: Document
35. [35] S. Ross, G. Gordon, and D. Bagnell (2011) A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), Proceedings of Machine Learning Research, Vol. 15, pp. 627–635.
36. [36] I. Mason, S. Starke, and T. Komura (2022) Real-time style modelling of human locomotion via feature-wise transformations and local motion phases. Proceedings of the ACM on Computer Graphics and Interactive Techniques 5 (1). External Links: Document

Appendix A Adaptive-Horizon Behavioral Foundation Model

This appendix follows the order of the methodology in the main paper. We first detail the adaptive-horizon BFM that defines the shared executable space, then the three routes for skill acquisition, action-aligned latent policy projection, and finally unified offline latent planning.

The same BFM latent space serves two roles in Fiber: it is the executable interface used to project interaction experts and the prediction space of the unified planner. Projection favors locally precise responses, while latent planning benefits from commands with longer temporal coherence. The adaptive-horizon BFM is designed to support both requirements within one shared behavioral space.

A-A Executable Behavioral Interface and BFM Observations

The body controller commands the 29 actuated joints of the Unitree G1 at 50 Hz, with control interval \Delta t=0.02 s. Tasks using Dex3 hands additionally predict 14 hand-joint targets, seven per hand. These hand targets are produced by a separate branch and are not part of the BFM body action.

Let d_{z} denote the BFM latent dimension. Executable body commands lie on

\mathcal{Z}=\{z\in\mathbb{R}^{d_{z}}:\|z\|_{2}=\sqrt{d_{z}}\},\quad\Pi_{\mathcal{Z}}(v)=\sqrt{d_{z}}\frac{v}{\|v\|_{2}},

with a fixed nonzero reference vector used only for the degenerate zero-norm case. We use d_{z}=256. The fixed-radius parameterization keeps the average squared coordinate magnitude independent of the latent dimension.

The BFM actor consumes the current robot state together with causal history. Its current proprioceptive state is

s_{t}^{\mathrm{prop}}=\left[q_{t}-q^{\mathrm{default}};\dot{q}_{t};g_{t}^{\mathrm{body}};0.25\,\omega_{t}^{\mathrm{body}}\right]\in\mathbb{R}^{64},

where g_{t}^{\mathrm{body}} is the gravity direction and \omega_{t}^{\mathrm{body}} is the body angular velocity. The actor also receives the previous scaled body action and nine causal proprioceptive/action history frames. Each history frame contains 93 entries: 29 action entries, three angular velocities, 29 joint positions, 29 joint velocities, and three gravity entries. The actor therefore receives 64+29+9\times 93=930 proprioceptive/action coordinates before the latent command is appended.

The backward representation additionally receives 463 dimensions of heading-frame whole-body kinematics during BFM training and simulation-based projection. These privileged quantities are not consumed by the deployed BFM actor. Domain-randomization variables and external forces may likewise enter training-time representations and critics without entering the deployed policy.

A-B Multi-Horizon Forward–Backward Representation

For a latent-conditioned policy, the forward representation admits the successor-feature interpretation

F_{\gamma}(s_{t},a_{t},z)\approx\mathbb{E}\left[\sum_{h=0}^{\infty}\gamma^{h}B(s_{t+h+1})\;\middle|\;s_{t},a_{t},\pi_{\mathrm{BFM}}(\cdot,z)\right].

The corresponding FB value is Q_{\gamma}^{\mathrm{FB}}(s,a,z)=F_{\gamma}(s,a,z)^{\top}z. We condition the forward representation on \gamma while sharing both the backward map B and the executable actor across horizons. For every replay transition, a horizon is sampled from the configured set \Gamma, and the same \gamma is used in the online prediction and bootstrapped target. The output of B is projected to \mathcal{Z}.

For a minibatch of n transitions (s_{i},a_{i},s_{i}^{+},d_{i}), with termination indicator d_{i}, define

\displaystyle M_{ij} \\ \displaystyle=F_{\gamma_{i}}(s_{i},a_{i},z_{i})^{\top}B(s_{j}^{+}), \\ \displaystyle R_{ij} \\ \displaystyle=M_{ij}-\gamma_{i}(1-d_{i})\bar{M}_{ij},

where \bar{M}_{ij} is a stopped-gradient target constructed from target representations and a next action sampled from the latent-conditioned actor. Suppressing ensemble notation, the FB minibatch objective is

\mathcal{L}_{\mathrm{FB}}=\frac{\sum_{i\neq j}R_{ij}^{2}}{2n(n-1)}-\frac{1}{n}\sum_{i}R_{ii}.

The shared backward map is regularized through

\mathcal{L}_{\mathrm{orth}}=\frac{\sum_{i\neq j}(b_{i}^{\top}b_{j})^{2}}{2n(n-1)}-\frac{1}{n}\sum_{i}\|b_{i}\|_{2}^{2},\qquad b_{i}=B(s_{i}^{+}),

and the representation update minimizes \mathcal{L}_{\mathrm{FB}}+\lambda_{\mathrm{orth}}\mathcal{L}_{\mathrm{orth}}. Motion-discriminator and auxiliary-critic objectives follow the underlying BFM training procedure; the multi-horizon modification applies to the FB representation and the shared actor objective below.

A-C Adaptive Actor Objective

Values at different discounts have different natural scales. We therefore use the normalized horizon-specific objective

V_{\gamma}(s,z)=(1-\gamma)Q_{\gamma}^{\mathrm{FB}}\bigl(s,\pi_{\mathrm{BFM}}(s,z),z\bigr).

The factor (1-\gamma) removes the leading discounted-sum scale, making the short- and long-horizon terms directly comparable. The shared actor maximizes

J_{\mathrm{FB}}(s,z)=\tau\log\sum_{\gamma\in\Gamma}p(\gamma)\exp\!\left(\frac{V_{\gamma}(s,z)}{\tau}\right),

where p(\gamma)\geq 0,\sum_{\gamma}p(\gamma)=1, and \tau>0. Actor gradient is

\displaystyle\nabla_{\theta}J_{\mathrm{FB}} \\ \displaystyle=\sum_{\gamma\in\Gamma}w_{\gamma}(s,z)\nabla_{\theta}V_{\gamma}(s,z), \\ \displaystyle w_{\gamma}(s,z) \\ \displaystyle=\frac{p(\gamma)\exp(V_{\gamma}(s,z)/\tau)}{\sum_{\gamma^{\prime}\in\Gamma}p(\gamma^{\prime})\exp(V_{\gamma^{\prime}}(s,z)/\tau)}.

Thus the update remains a convex combination of the horizon-specific policy gradients, but the combination is selected from the current state and latent command rather than fixed globally.

Stability of the adaptive FB-to-actor objective

The relevant stability property is not only that the horizon aggregator is non-expansive, but that it does not amplify perturbations introduced through the FB backup. Let

\widetilde{Q}_{\gamma}=(1-\gamma)Q_{\gamma}^{\mathrm{FB}},

and collect the normalized action values as \widetilde{\mathbf{Q}}=[\widetilde{Q}_{\gamma}]_{\gamma\in\Gamma}. Holding the current backward map B fixed, let r_{z}^{B} denote the corresponding latent-conditioned reward. For a fixed latent policy \pi, the normalized FB Bellman operator is

\begin{split}&(\mathcal{B}^{\pi}\widetilde{\mathbf{Q}})_{\gamma}(s,a,z)\\
&\quad=\mathbb{E}\!\left[(1-\gamma)r_{z}^{B}(s^{\prime})+\gamma(1-d)\widetilde{Q}_{\gamma}\bigl(s^{\prime},\pi(s^{\prime},z),z\bigr)\mid s,a\right].\end{split}

Equip the product space with the norm

\|\widetilde{\mathbf{Q}}\|_{\Gamma,\infty}=\max_{\gamma\in\Gamma}\|\widetilde{Q}_{\gamma}\|_{\infty}.

For any two value collections \widetilde{\mathbf{Q}} and \widetilde{\mathbf{Q}}^{\prime}, the reward terms cancel, yielding

\begin{split}\|\mathcal{B}^{\pi}\widetilde{\mathbf{Q}}-\mathcal{B}^{\pi}\widetilde{\mathbf{Q}}^{\prime}\|_{\Gamma,\infty}\leq\gamma_{\max}\|\widetilde{\mathbf{Q}}-\widetilde{\mathbf{Q}}^{\prime}\|_{\Gamma,\infty},\end{split}

where \gamma_{\max}=\max\Gamma<1. Thus, for fixed B and \pi, the multi-horizon FB Bellman backup is a \gamma_{\max}-contraction.

We next define the adaptive actor-value functional

(\mathcal{A}_{\tau}^{\pi}\widetilde{\mathbf{Q}})(s,z)=\tau\log\sum_{\gamma\in\Gamma}p(\gamma)\exp\!\left(\frac{\widetilde{Q}_{\gamma}(s,\pi(s,z),z)}{\tau}\right).

The log-sum-exp map is monotone and translation equivariant. Let

\delta=\|\widetilde{\mathbf{Q}}-\widetilde{\mathbf{Q}}^{\prime}\|_{\Gamma,\infty}.

Then, for every (s,a,z) and \gamma\in\Gamma,

\widetilde{Q}^{\prime}_{\gamma}(s,a,z)-\delta\leq\widetilde{Q}_{\gamma}(s,a,z)\leq\widetilde{Q}^{\prime}_{\gamma}(s,a,z)+\delta.

Monotonicity and translation equivariance therefore imply

\|\mathcal{A}_{\tau}^{\pi}\widetilde{\mathbf{Q}}-\mathcal{A}_{\tau}^{\pi}\widetilde{\mathbf{Q}}^{\prime}\|_{\infty}\leq\|\widetilde{\mathbf{Q}}-\widetilde{\mathbf{Q}}^{\prime}\|_{\Gamma,\infty}.

Hence the adaptive actor-value functional is non-expansive with respect to perturbations of the horizon-specific FB values.

The same bound holds for the exact improved-value functional

(\mathcal{I}_{\tau}\widetilde{\mathbf{Q}})(s,z)=\max_{a}\tau\log\sum_{\gamma\in\Gamma}p(\gamma)\exp\!\left(\frac{\widetilde{Q}_{\gamma}(s,a,z)}{\tau}\right),

since

\left|\max_{a}f(a)-\max_{a}g(a)\right|\leq\max_{a}|f(a)-g(a)|.

This statement concerns the improved value functional rather than the \arg\max policy map itself.

Combining (32) and (36) gives

\begin{split}\|\mathcal{A}_{\tau}^{\pi}\mathcal{B}^{\pi}\widetilde{\mathbf{Q}}-\mathcal{A}_{\tau}^{\pi}\mathcal{B}^{\pi}\widetilde{\mathbf{Q}}^{\prime}\|_{\infty}\leq\gamma_{\max}\|\widetilde{\mathbf{Q}}-\widetilde{\mathbf{Q}}^{\prime}\|_{\Gamma,\infty}.\end{split}

Therefore, adaptive horizon aggregation preserves the contractive sensitivity of the fixed-policy FB backup: perturbations in the horizon-specific FB values are not amplified by the joint actor objective. Consequently, relative to a single-horizon objective with \gamma=\gamma_{\max}, adaptive aggregation introduces no worse worst-case sensitivity to critic perturbations while allowing the shared policy to exploit value estimates at multiple temporal horizons.

Adaptive aggregation versus a fixed weighted sum

A natural multi-\gamma alternative is the fixed convex average

J_{\mathrm{WS}}(s,z)=\sum_{\gamma\in\Gamma}p(\gamma)V_{\gamma}(s,z).

Both J_{\mathrm{WS}} and (26) are scale controlled after the (1-\gamma) normalization; non-expansiveness alone therefore does not distinguish the two. The key difference is that the weighted-sum baseline uses the same horizon weights for every state and behavior, whereas (26) adapts the effective weights to the current horizon-specific values. As \tau\rightarrow 0, the update concentrates on the largest normalized value; for large \tau, its gradient approaches the prior-weighted average. To isolate this effect from simply exposing the actor to multiple discounts, we additionally compare against (40) using the same multi-horizon representation. The adaptive objective empirically outperforms this fixed weighted-sum baseline, supporting state- and behavior-dependent horizon weighting rather than a fixed global mixture.

For intuition, a discount \gamma at control interval \Delta t has effective duration approximately \Delta t/(1-\gamma). At 50 Hz, \gamma=0.8 corresponds to approximately 0.1 s and \gamma=0.98 to approximately 1 s. This is an interpretation of the temporal weighting, not a hard execution horizon for the latent command.

A-D Architecture and Optimization

The actor, forward representations, and critics use residual MLP backbones. The forward representations and critics use two ensemble members; the actor is a single shared network conditioned on the current latent and outputs 29 body actions. The backward encoder maps proprioceptive and privileged kinematic features to the normalized behavioral space. The scalar discount is embedded by a small MLP before entering the forward representation and is not an input to the actor. Details are shown in Tab. IV.

The BFM is frozen during expert projection, projected-data collection, and offline planner training. These stages therefore use the same latent geometry, state convention, and body-action decoder.

Appendix B Heterogeneous Skill Acquisition

As in the main paper, skill acquisition is deliberately decoupled from the shared representation. Experts may use different supervision, observations, and training procedures; projection only requires access to the resulting robot actions. We use motion-based and reward-shaped experts for the base interactions, and SF-guided behavior reuse when we specifically want to create diverse compositions between an interaction and native BFM behaviors.

B-A Task-Specific Interaction Experts

Motion-based experts combine humanoid motion tracking with object and contact objectives. For box pickup and object grasping, the tracked quantities include both body motion and manipulated-object pose, while contact rewards encourage the demonstrated interaction to be realized physically. The door expert is trained with stage-specific objectives [2] for approach, contact establishment, handle manipulation, and opening. These experts may use different privileged observations and rewards; none of those task-specific inputs are required by the unified planner after projection.

For reproducibility, the HDMI-style motion expert uses a privileged observation encoder with a 256-dimensional embedding and a reference-action residual actor. Its actor hidden widths are [512,256,256], critic hidden widths are [512,256,128], and the activation is Mish. The PPO settings for this route are summarized in Tab. VI.

B-B SF-Guided Behavior Reuse for Skill Composition

The third acquisition route targets interaction–behavior composition. Given an acquired interaction, we use behaviors already represented by the BFM to create additional physically valid variants: the interaction reward preserves the task-specific physical requirements, while a native BFM latent specifies the whole-body behavior to be expressed simultaneously. This expands one interaction into a family of compositions without requiring a separate motion reference or task-specific expert for every combination.

In our experiments, we instantiate this idea with box carrying. By varying the requested native latent, the same carrying interaction can be composed with crouching, kneeling, walking, or running. For example, carrying while crouching requires the controller to maintain support of the box while realizing a substantially different whole-body configuration. The resulting rollouts add diverse compositional behavior to the downstream planner data.

For an interaction I, let z^{N}\in\mathcal{Z} denote the native BFM behavior to be composed with it. We freeze the BFM and learn an adapter A_{\psi_{I}} that outputs an executable latent,

\hat{z}_{t}=\Pi_{\mathcal{Z}}\left(z^{N}+A_{\psi_{I}}^{\mathrm{res}}(o_{t},z^{N})\right),\quad a_{t}=\pi_{\mathrm{BFM}}(s_{t},\hat{z}_{t}).

The residual parameterization lets the adapter modify the native command only as needed to satisfy the interaction. The deployable adapter uses causal robot inputs and the requested native latent; privileged object state and interaction errors may be used by the critic and reward computation during simulation training.

We optimize the adapter with

r_{t}=r_{t}^{I}+\lambda_{\mathrm{SF}}B(s_{t})^{\top}z^{N},

where r_{t}^{I} preserves the interaction and the SF term encourages the rollout to remain aligned with the requested native behavior. For fixed z^{N},

\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^{t}B(s_{t})^{\top}z^{N}\right]=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^{t}B(s_{t})\right]^{\top}z^{N},

so the preservation term evaluates the discounted state occupancy along the behavioral direction selected by z^{N}. The two terms therefore play distinct roles: r_{t}^{I} keeps the interaction valid, while the SF term determines which native behavior is composed with that interaction.

For the carrying compositions used in our experiments, r_{t}^{I} contains upper-body pose, object-position, object-orientation, and contact terms. The three tracking terms are

\displaystyle r_{\mathrm{pose}} \\ \displaystyle=\exp(-e_{q}/0.25^{2}), \\ \displaystyle r_{\mathrm{pos}} \\ \displaystyle=\exp(-\|e_{p}\|_{2}^{2}/0.10^{2}), \\ \displaystyle r_{\mathrm{ori}} \\ \displaystyle=\exp(-\theta_{\mathrm{obj}}^{2}/0.35^{2}),

with weights 1, 2, and 1, respectively; the contact term has weight 1. Here e_{q} is the mean squared upper-body joint-position error, e_{p} is the object-position error in the torso frame, and \theta_{\mathrm{obj}} is the object-orientation discrepancy in that frame. The SF term has weight 0.1, the latent-smoothness penalty has weight -0.5, and object-drop termination contributes -100. These interaction terms are shared across the composed variants; behavioral diversity is introduced by changing z^{N}.

Because both B(s_{t}) and z^{N} lie in the radius \sqrt{d_{z}} sphere,

\frac{B(s_{t})^{\top}z^{N}}{d_{z}}=\cos\angle\!\left(B(s_{t}),z^{N}\right).

The normalized and unnormalized forms therefore differ only by a constant scale absorbed into \lambda_{\mathrm{SF}}. Importantly, the preservation target is the requested native behavior z^{N}, not the adapted latent \hat{z}_{t}.

The adapter actor and critic use two residual feed-forward blocks of width 512 with feed-forward width 2048, GELU activations, and observation normalization. We additionally regularize rapid latent changes with the normalized cost

c_{\mathrm{smooth},t}=\frac{\|\hat{z}_{t}-\hat{z}_{t-1}\|_{2}^{2}}{2d_{z}},

which equals one minus cosine similarity on \mathcal{Z}.

Appendix C Action-Aligned Latent Policy Projection

After each expert is acquired, lpp maps its local closed-loop responses into the executable BFM space. This section follows the same order as the main method: first establish the shared action convention, then define the projector and short-horizon latent execution, the action/semantic objectives, and finally the on-policy aggregation used to obtain robust projected rollouts.

C-A Common Action Space

Experts may use different normalized action conventions, so action alignment is performed only after both expert and BFM outputs are converted to the same physical body joint-target space. The frozen BFM actor produces a_{t}=\pi_{\mathrm{BFM}}(s_{t},z_{t})\in\mathbb{R}^{29}. Under the BFM control profile,

q_{t}^{\mathrm{des}}=q^{\mathrm{default}}+0.25\,r_{q}\odot\operatorname{clip}(6a_{t},-6,6),

where r_{q} contains the per-joint action scales. Joint order, default pose, and scale vectors are kept consistent across expert conversion, projection, projected-rollout collection, and execution. The resulting targets are realized by the robot PD controller with the gains and actuator limits of the selected control profile.

C-B Projector and Latent Execution

Each expert is projected independently. At update time t_{k}, a projector P_{\psi_{i}} maps the expert observation to BFM latent,

z_{k}=P_{\psi_{i}}(o_{t_{k}}^{E_{i}}),\quad z_{k}\in\mathcal{Z}.

The projector is a pre-normalized residual MLP consisting of an input LayerNorm and linear projection, residual feed-forward blocks, an output LayerNorm, and a latent head followed by \Pi_{\mathcal{Z}}. A residual block is

\displaystyle\tilde{h}^{\ell} \\ \displaystyle=\operatorname{GELU}\left(W_{1}^{\ell}\operatorname{LN}(h^{\ell})+b_{1}^{\ell}\right), \\ \displaystyle h^{\ell+1} \\ \displaystyle=h^{\ell}+W_{2}^{\ell}\tilde{h}^{\ell}+b_{2}^{\ell}.

The hidden width is 1024 with feed-forward expansion ratio four. The motion-based projector uses four residual blocks and the door projector uses two.

Projection updates occur every five body-control steps, corresponding to a 0.1 s interval and a 10 Hz projector. Between two projector updates, the latent command is interpolated on the sphere,

z_{t_{k}+h}=\operatorname{SLERP}\left(z_{k-1},z_{k},\alpha_{h}\right),\quad h=0,\ldots,H_{P}-1,

where H_{P}=5. For endpoints a,b\in\mathcal{Z} with angular separation \vartheta,

\operatorname{SLERP}(a,b,\alpha)=\frac{\sin((1-\alpha)\vartheta)}{\sin\vartheta}a+\frac{\sin(\alpha\vartheta)}{\sin\vartheta}b.

The cosine used to compute \vartheta is clamped for numerical stability.

C-C Action Alignment and Semantic Regularization

Let T_{B} and T_{i} convert the BFM and expert outputs to the same physical body joint-target space. Over the short execution interval beginning at t_{k}, we use

\displaystyle e_{h}^{a} \\ \displaystyle=T_{B}\!\left(\pi_{\mathrm{BFM}}(s_{t_{k}+h},z_{t_{k}+h})\right)-T_{i}\!\left(\pi_{i}^{E}(o_{t_{k}+h}^{E_{i}})\right), \\ \displaystyle\mathcal{L}_{\mathrm{act}} \\ \displaystyle=\mathbb{E}_{P_{\psi_{i}}}\left[\frac{\sum_{h=0}^{H_{P}-1}m_{h}\|e_{h}^{a}\|_{2}^{2}}{29\sum_{h=0}^{H_{P}-1}m_{h}}\right],

where m_{h} masks samples beyond an episode boundary. Comparing physical joint targets prevents normalized-action conventions of different experts from changing the meaning of the alignment loss.

Action matching alone can leave the latent underdetermined. We therefore anchor the predicted latent to the native BFM representation of the same local state,

\mathcal{L}_{\mathrm{sem}}=\mathbb{E}\left[\|z_{k}-B(s_{t_{k}})\|_{2}^{2}\right].

For two vectors on \mathcal{Z},

\|z-B(s)\|_{2}^{2}=2d_{z}\left(1-\frac{z^{\top}B(s)}{d_{z}}\right),

so squared distance and cosine gap are equivalent up to a constant factor. The full objective is

\mathcal{L}_{\mathrm{proj}}=\mathcal{L}_{\mathrm{act}}+\lambda_{\mathrm{sem}}\mathcal{L}_{\mathrm{sem}}.

The action term identifies commands by the expert’s required local control response, whereas the semantic term resolves action-equivalent solutions in a way that remains coherent with the native BFM geometry. Training details in Tab. VII.

For skills with dexterous hands, a separate projector head regresses the expert’s physical hand-joint targets. Its loss is evaluated only when valid hand labels are available. The sphere constraint applies only to the body latent; hand outputs retain their physical joint-target parameterization.

C-D On-Policy Projection and Data Aggregation

Projection is trained with single-skill on-policy data aggregation. The current projector is executed through the frozen BFM under randomized task initializations. At the states it visits, we query the corresponding expert for body actions and record the BFM inputs, expert observations, and reset masks. Short execution windows are then replayed through the frozen BFM actor to evaluate (56) and (57). The projector is updated and new rollouts are collected with the updated policy. Gradients pass through the frozen BFM actor to the predicted latent, but no BFM parameter is changed and no simulator gradient is required. This DAgger-style procedure [35] exposes the projector to corrective states that are absent from nominal expert trajectories.

Exploration noise is applied to the executed latent before projection back to \mathcal{Z} and is never added to the expert action label. Once projection is complete, the original expert is no longer required for projected execution or planner-data collection.

The projected latent should be interpreted locally: it is identified by the control response induced by the frozen BFM on states visited during projection, rather than by assuming that the expert and BFM policies are globally identical. This is why projected closed-loop rollouts, rather than nominal expert trajectories alone, are used as the supervision for the unified planner.

Appendix D Unified Latent Flow Planning

Once each expert has been projected, its closed-loop latent rollouts can be combined because they now share the same executable representation. We train the unified planner entirely offline on these projected trajectories, matching the data construction, conditioning variables, and flow objective. Details of the planner network are shown in Tab. VIII.

D-A Projected Rollout Dataset and Prediction Target

For each projected skill, we collect 20K closed-loop episodes and aggregate them for offline planner training. Each record contains onboard depth, proprioception, the high-level condition, and the action-aligned latent actually executed by the projected policy. Replacing these targets with B(s) would change the supervision from projected control responses to behavioral-state encoding and is therefore not equivalent to our data construction.

The planner predicts a one-second sequence of future BFM commands. We use one latent target every 0.1 s, corresponding to five 50 Hz BFM control steps, and predict H=10 latent commands per sequence. Matching the trajectory notation used in the main paper,

Z_{t}=[z_{t},z_{t+1},\ldots,z_{t+H-1}]\in\mathbb{R}^{H\times d_{z}},\quad H=10,

Each future latent is represented by one temporal token in the transformer. For hand-bearing data, a separate target Y_{t}\in\mathbb{R}^{H\times 14} stores physical hand-joint targets in a fixed hand-major ordering. Hand targets are normalized with their own statistics and are not projected onto \mathcal{Z}.

D-B Observation and Steering Conditioning

Unlike the task-specific experts, the unified planner uses only onboard observations. Its proprioceptive input is p_{t}=[q_{t};\dot{q}_{t}]\in\mathbb{R}^{58}; for hand-bearing samples, the measured 14 hand-joint positions are included through a separate projection. Continuous features are standardized coordinate-wise using fixed training-set statistics, and the normalization statistics and joint ordering are stored with the model.

Depth

The planner processes a metric egocentric depth image of size 144\times 256, clipped to the valid range 0.12–3.0 m. A strided convolution with 12\times 16 kernel and stride produces a 12\times 16 grid of 192 spatial tokens. Learned spatial embeddings are added before two self-attention blocks. Eight learned register tokens participate in the depth encoder and are discarded afterward, leaving the 192 spatial tokens as conditioning memory.

Proprioception

The 58-dimensional joint state is standardized coordinate-wise and linearly embedded to the model width. For coordinate j, \bar{p}_{j}=(p_{j}-\mu_{j})/\max(\sigma_{j},\epsilon). When hands are enabled, their measured joint positions are standardized separately and added through an additional projection.

Language

Task instructions are encoded by a frozen T5-base text encoder. Masked pooling produces a 768-dimensional sentence embedding, followed by LayerNorm and a two-layer SiLU MLP that maps it to the planner width. The text encoder is not updated by the planner objective.

Waypoint

Waypoint supervision is derived from future root poses. At horizons \delta\in\{0.5,1.0\} s, we use

u_{t}^{\delta}=[\Delta x_{t}^{\delta},\Delta y_{t}^{\delta},\sin\Delta\psi_{t}^{\delta},\cos\Delta\psi_{t}^{\delta},h_{t+\delta}],

where displacement is expressed in the current heading frame, \Delta\psi is relative yaw, and h is future pelvis height. Concatenating the two horizons gives a ten-dimensional waypoint condition. Missing waypoint conditions are masked rather than represented by a zero-valued physical command.

Steering representation

Language and waypoint embeddings are converted to a small fixed set of steering tokens by a self-attention encoder. The latent transformer cross-attends to these steering tokens, the 192 spatial depth tokens, one proprioceptive state token, and eight learned conditioning-register tokens. Unavailable language or waypoint modalities are masked explicitly.

D-C Conditional Latent Transformer

Each noisy future latent is linearly embedded and combined with a learned temporal-position embedding. If hand targets are present, a separate projection of the noisy hand target is added to the corresponding temporal token. Flow time is sinusoidally embedded and passed through an MLP. Each transformer block contains temporal self-attention, cross-attention to the observation/steering memory, and a feed-forward MLP. Adaptive LayerNorm modulation supplies shift, scale, and residual-gate vectors from the flow-time condition. For example,

h^{\prime}=h+g_{\tau}\odot\operatorname{Attn}\left((1+s_{\tau})\odot\operatorname{LN}(h)+b_{\tau}\right).

Separate linear heads decode the clean body latents and optional hand targets.

D-D Offline Flow-Matching Objective

Let \xi_{t} denote the encoded onboard observations and available steering conditions. We sample x_{0}\sim\mathcal{N}(0,I) and \tau\sim\mathcal{U}(0,1), and construct

x_{\tau}=(1-\tau)x_{0}+\tau x_{1}.

The network predicts the clean endpoint f_{\eta}(x_{\tau},\tau,\xi_{t}) and is trained with

\mathcal{L}_{\mathrm{body}}=\mathbb{E}\left[\frac{1}{Hd_{z}}\|f_{\eta}(x_{\tau},\tau,\xi_{t})-x_{1}\|_{2}^{2}\right].

The clean latent targets lie on \mathcal{Z}, but the Gaussian interpolation and raw endpoint prediction are carried out in Euclidean coordinates. Sphere projection is applied only when sampled outputs are converted to executable BFM commands.

An analogous Gaussian interpolation is applied to the hand targets when available. The complete loss is

\mathcal{L}_{\mathrm{planner}}=\mathcal{L}_{\mathrm{body}}+\lambda_{\mathrm{hand}}\mathcal{L}_{\mathrm{hand}},

where \mathcal{L}_{\mathrm{hand}} is evaluated only on valid hand samples. No decoded body-action auxiliary loss is used in the reported planner training; action alignment is already incorporated when constructing the projected latent supervision.

An endpoint predictor induces the velocity field

v_{\eta}(x_{\tau},\tau,\xi_{t})=\frac{f_{\eta}(x_{\tau},\tau,\xi_{t})-x_{\tau}}{1-\tau},\quad\tau<1.

For a numerical grid 0=\tau_{0}<\cdots<\tau_{N}=1, Euler integration uses

x_{j+1}=x_{j}+(\tau_{j+1}-\tau_{j})v_{\eta}(x_{j},\tau_{j},\xi_{t}).

Heun integration uses the corresponding predictor-corrector update. After sampling, every body latent is independently projected by \Pi_{\mathcal{Z}}, while hand outputs are denormalized using their own statistics.

D-E Planner Optimization

Shown in Tab. IX, training windows are sampled within episode boundaries, and train/validation partitions are defined at the episode level. The model checkpoint stores the architecture, optimizer state, proprioceptive normalization, and hand-target normalization. In the exported configuration, state-condition dropout is 0.2 and waypoint-condition dropout is 0.1; waypoint tokens are omitted for episodes without waypoint supervision.

D-F Receding-Horizon Execution

At deployment, the planner executes the leading segment of the sampled latent sequence and then replans from the new onboard observation. Consecutive latent updates are connected by SLERP to supply the BFM with a command at each 50 Hz body-control step. The BFM actor therefore retains high-frequency state-dependent feedback while the planner operates at a lower rate. Hand joint targets follow the same receding-horizon schedule as the corresponding body-latent sequence.

Appendix E Evaluation and Real-World Deployment

E-A Simulation Evaluation Protocol

We evaluate each task over diverse object or environment variations in IsaacSim. For pickup and door opening, the w/ appr. setting initializes the humanoid approximately 0.5–1.5 m from the interaction target and requires it to approach before execution, whereas w/o appr. initializes the humanoid directly in front of the target. Pickup succeeds if the box is lifted by at least 0.3 m and maintained for at least 1 s; grasping succeeds if the object is lifted by at least 0.1 m and maintained for at least 1 s. Door opening succeeds when the door is opened and the humanoid passes through the doorway. Reported \pm values are computed from repeated evaluations with different random seeds to account for simulation stochasticity. Detailed object variation configurations used in evaluation are shown in Tab. X.

E-B Real-World Execution

The real-world system uses onboard RealSense depth and proprioception. Both the unified planner and the frozen BFM run on a single NVIDIA RTX PRO 5000 Blackwell laptop. The TensorRT planner replans at 5 Hz using 10 Heun integration steps, with approximately 20 ms inference time per planner call, while the BFM runs at 50 Hz with approximately 1 ms per forward pass.

We use rtc during deployment to avoid discontinuities between consecutively sampled latent trajectories. At each replanning step, the flow planner predicts a chunk of future latent commands while the robot continues executing the current chunk. The newly generated chunk overlaps temporally with the remaining portion of the previous prediction, and rtc conditions the overlapping prefix of the new sample toward the previously committed trajectory during flow integration. This preserves continuity across replanning boundaries while still allowing later latent commands in the chunk to adapt to the latest onboard observation. In this way, rtc decouples the planner inference latency from continuous robot execution and avoids abrupt latent changes caused by independently sampling each plan.