VICAR: Video Imitation with Contact-Aware Retargeting for Humanoid Interactive Skills
Third-person human videos offer a scalable alternative to task-specific reward engineering, teleoperation, and manually scripted demonstrations for humanoid loco-manipulation. Recent video-to-humanoid systems have shown that noisy human motion, object trajectories, and contacts can be extracted from unstructured videos and refined into deployable skills.
From the abstract. Our summary is in progress.
Full paper
VICAR: Video Imitation with Contact-Aware Retargeting for Humanoid Interactive Skills
Published under CC BY 4.0. Reproduced with attribution; original at arXiv:2610.13079 (PDF).
Abstract
Third-person human videos offer a scalable alternative to task-specific reward engineering, teleoperation, and manually scripted demonstrations for humanoid loco-manipulation. Recent video-to-humanoid systems have shown that noisy human motion, object trajectories, and contacts can be extracted from unstructured videos and refined into deployable skills. However, a key gap remains for contact-rich imitation: the task is often defined by world-frame contacts whose poses, timing, and speed must survive embodiment mismatch and runtime scene changes. Towards this, we present VICAR (Video Imitation with Contact-Aware Retargeting), a pipeline for learning humanoid interactive skills from third-person human videos. The system first recovers human motion, reconstructs task-relevant obstacles from monocular video, and then retargets the motion to a humanoid while preserving global contact poses, motion speed, and collision clearance. Unlike retargeting methods that only remove penetration or preserve local interaction geometry, our formulation represents user-specified contact poses and collision clearance as high-priority Lagrangian penalty terms and optimizes the full motion to remain smooth through contact-rich transitions. The resulting motions are used to train a contact-conditioned motion generator whose inputs include movable contact points enabling runtime adaptation to new contact positions. We demonstrate the framework on ten diverse third-person video skills, including six table-tennis serves, three object-pickup tasks, and ladder climbing, and deploy the learned skills on hardware. These results suggest that contact-preserving retargeting is a practical step toward humanoids that learn useful interactive behavior by watching humans.
Project website: ember-center-berkeley.github.io/VICAR
I Introduction
Humans routinely learn interaction skills by watching other humans. A child does not need an adult to physically guide their body through every action, in the way teleoperation is often used to train robots; they can learn by observing that a cup should be reached on the table, that the table edge can support the body while crouching, or that a tossed ball must be met at a precise point in space. For humanoid robots, however, this mode of learning remains difficult: third-person videos provide abundant motion cues, but the robot must execute them with a different morphology, different joint limits, and different contact geometry. These tasks are organized around sparse contact events: a hand must arrive at an object, a foot must land on a rung, or a racket must meet a ball at the right pose and time. A trajectory can therefore resemble the source motion yet remain unusable if those events drift in the world frame.
The embodiment gap becomes especially severe for contact-rich tasks. Consider a standing adult picking up an object from a table. For the adult, the table may be near waist height; for a shorter humanoid such as Unitree G1, the same table may be near the chest, as shown in Figure 2. A retargeted motion that only matches human joint or keypoint motion will miss the object. The same issue appears when a robot crouches to pick up an object under a table while using the other hand for support: success depends on preserving the support contact, avoiding the table, and keeping the motion dynamically trackable. Collision avoidance alone does not resolve this ambiguity: several collision-free poses may exist, but only a small subset preserves the demonstrated contact and approach speed.
Many video-imitation pipelines rely on egocentric or robot-view demonstrations, but the web and everyday recordings contain a plethora of third-person human videos that can provide strong priors for humanoid whole-body motion learning. Recent systems show that monocular videos can provide scalable skill data for humanoids. VideoMimic [1] reconstructs human-scene geometry from casual videos and uses the resulting meshes and retargeted motions for contextual control. HDMI [2] learns humanoid-object interaction policies from monocular RGB videos by co-tracking robot and object states. OmniRetarget [3] improves interaction-preserving retargeting by preserving human-object-terrain relations through an interaction mesh and hard kinematic constraints. These works make major progress, but our target setting places a different emphasis: we want the robot to preserve specific global contact poses that may move at runtime, such as the ball-hit point in a table-tennis serve or the object position in a pickup task. The challenge is thus not only to reconstruct what the human did, but also to identify which interaction quantities must remain invariant after embodiment transfer.
We introduce VICAR, a video-to-humanoid learning pipeline centered on contact-aware retargeting. Given a third-person video, VICAR extracts human motion using GENMO/GEM [4], reconstructs task-relevant obstacle geometry using monocular depth and semantic segmentation. The selected regions are unprojected into a point cloud, voxelized into collision geometry, and scaled following the human-scene alignment ideas of VideoMimic [1]. We then retarget the motion in two stages: an initial OmniRetarget-style [3] pass gives a feasible robot reference, and a second optimization uses high-priority collision and global contact-pose penalties while preserving motion speed.
Our retargeted motions are not merely static references. They seed a motion generator conditioned on movable contact points that can be adjusted online during inference: for example, the ball-hit point in a table-tennis serve can move with the perceived ball position, and object-pick motions can be adapted to different object placements. A downstream policy then tracks the generated reference using proprioception and gravity observations, allowing deployment on hardware.
This paper makes the following contributions:
- We formulate contact- and speed-preserving retargeting for third-person video imitation, using high-priority global contact-pose and collision penalties derived from monocularly reconstructed obstacles.
- We connect this retargeting stage to a contact-conditioned motion generator that supports dynamic contact locations and contact-point augmentation.
- We evaluate the pipeline on ten diverse skills, including six table-tennis serves, object pickups above and below a table, bimanual box pickup and placement, and ladder climbing. All skills are deployed on hardware except ladder climbing.
II Related Work
Humanoid learning from human videos
Human video is an attractive supervision source because it is easier to collect than teleoperation, motion capture, or robot demonstrations. Recent systems learn pose or locomotion from large video collections [5, 6, 7, 8, 1, 9], adapt robot-free videos to manipulation [10, 11, 12, 13], or target humanoid-object interaction [2, 14, 15, 16]. HDMI [2] extracts human and object trajectories for co-tracking policies, while VideoMimic [1] reconstructs human-scene geometry for contextual skills. HumanX and SUGAR scale video-derived interaction learning [15, 16], but do not explicitly preserve task-specified global contact poses that can change at runtime. These systems primarily expand reference acquisition and policy learning; our focus is the intermediate geometric adaptation needed to keep sparse, task-defining contacts valid after embodiment transfer.
Human-to-humanoid retargeting
Kinematic imitation uses retargeted human motion as reference data for characters and robots [17, 18, 19, 20]. PHC [5] fits SMPL-style pose to a robot embodiment, and GMR [21] matches human pose through relative keypoint geometry. Video-to-humanoid systems such as HumanPlus and ResMimic use retargeted references for tracking [6, 14], but do not directly preserve global contacts or avoid task obstacles. OmniRetarget [3] preserves local human–object–terrain relations with an interaction mesh and hard joint, velocity, collision, and stance constraints. VICAR instead represents selected contacts as world-frame pose objectives and optimizes the whole clip to preserve source speed through contact. The distinction matters when collision avoidance admits several geometrically feasible solutions: in tabletop pickup, for example, entering and exiting the platform through different faces produces a discontinuous reference despite satisfying local clearance.
Scene reconstruction for robot learning
Monocular depth and segmentation provide useful but noisy scene geometry [22, 23, 24]. Video-to-robot pipelines also use body recovery, object reconstruction, 6D pose tracking, and registration to extract interaction priors [25, 26, 27, 28, 29, 16]. Similar to VideoMimic [1], we unproject depth and resolve scale from the human-scene geometry, but reconstruct only semantic regions likely to constrain the robot. This task-focused reconstruction avoids requiring a complete scene model while retaining the surfaces needed for collision and contact reasoning.
Contact-conditioned motion generation
Interactive humanoid policies often track reference robot and object states [2, 14] or reconstructed scene-conditioned references [1]. SUGAR [16] improves generalization with a hierarchical command generator after physics refinement. SceneBot [30] and ContactMimic [31] show that per-link or binary contact commands can condition the tracking policy directly when scene interaction is represented implicitly. This policy-level conditioning is sufficient when the controller can resolve the desired interaction from a motion and contact mode. In contrast, retargeting-layer anchors are needed when the world exogenously determines a specific \mathrm{SE}(3) pose—such as a measured ball location or fixed ladder rung—that must first be embedded in the reference motion and interpolated across contact variants. VICAR places a contact-conditioned generator between retargeting and policy learning, separating geometric adaptation from low-level control.
- PHC [5]
- –
- \times
- \times
- \times
- \times
- \times
- GMR [21]
- –
- \times
- \times
- \times
- \times
- \times
- VideoMimic [1]
- Penalty
- \times
- \checkmark
- \times
- \times
- \times
- OmniRetarget [3]
- Hard
- \checkmark
- \checkmark
- \checkmark
- Relative
- \times
- VICAR (Ours)
- Penalty
- \checkmark
- \checkmark
- \checkmark
- \checkmark
- \checkmark
III Method
The input to VICAR is a monocular third-person RGB video of a human performing an interaction skill. The output is a policy that tracks a generated humanoid reference while allowing selected contact poses to vary at inference time. The pipeline has five stages.
III-A Human Motion and Contact Annotation
We extract global human motion from the input video using GENMO/GEM [4]. The resulting sequence provides source body poses, global root motion, and task-relevant body keypoints. For the current prototype, contact frames, contacted body parts, and contact-frame orientations are labeled manually. These annotations define tuples
\mathcal{A}=\{(\tau_{k},b_{k},\hat{\mathbf{p}}_{k},\hat{\mathbf{R}}_{k})\}_{k=1}^{K},where \tau_{k} is a frame, b_{k} is a robot link or end-effector, and (\hat{\mathbf{p}}_{k},\hat{\mathbf{R}}_{k})\in\mathbb{R}^{3}\times\mathrm{SO}(3) is the desired global contact pose. Thus, a contact means that a specified robot body reaches a world-frame \mathrm{SE}(3) pose at a specified timestep; it is a pose objective only and does not model force, contact mode, or sliding. In table tennis, for example, we annotate one pose for the ball toss and one for the racket–ball hit, and the hit pose can move online with the estimated ball state.
III-B Obstacle Reconstruction from Video
The obstacle reconstruction stage identifies the portions of the scene that constrain the robot motion. We first estimate per-frame monocular depth using an off-the-shelf metric depth model: UniDepth [22], and segment the image into semantic regions using Semantic-SAM [23, 24]. We then query GPT-4o mini [32] with the segmented frame and task description to select regions likely to be in contact with the human, such as a tabletop, table edge, floor patch, or support surface.
For each selected region, we unproject depth pixels into camera-frame points and transform them into the task world frame. Following VideoMimic’s use of human-scene alignment and metric scale [1], we estimate a scale factor from the recovered human height from GENMO/GEM [4] and estimated height from human semantic region depths. The selected points are filtered and converted into voxelized mesh obstacles:
\mathbf{x}_{u,v}=\alpha\,\mathbf{T}_{wc}\,\mathbf{K}^{-1}\begin{bmatrix}u&v&1\end{bmatrix}^{\top}D(u,v),where \mathbf{x}_{u,v}\in\mathbb{R}^{3}, D(u,v)\in\mathbb{R}_{+} is depth, \mathbf{K}\in\mathrm{GL}(3) is the camera intrinsic matrix, \mathbf{T}_{wc}\in\mathrm{SE}(3) is the camera-to-world transform, and \alpha\in\mathbb{R}_{+} is the scale factor. In this stage, a voxel grid is regarded as a set of cuboid obstacles. We use voxel grid size of 0.05m side-length cuboids with a threshold of 5 points to consider as occupied with obstacle.
III-C Contact- and Speed-Preserving Retargeting
We retarget in two stages. First, we obtain an initial robot trajectory \bar{q}_{1:T} using an OmniRetarget-style keypoint formulation [3] with obstacle penalties disabled. This gives a reasonable motion prior under robot joint limits. Second, we optimize the trajectory with high-priority contact and collision penalties while preserving the source motion’s speed profile at the end-effectors.
III-C1 Stage 1: Interaction-Mesh Initialization
The first pass follows OmniRetarget’s interaction-mesh formulation but omits its obstacle-avoidance and penetration penalties. For keypoint i at time t, the interaction mesh encodes local geometry using the Laplacian coordinate
L(\mathbf{p}_{t,i})=\mathbf{p}_{t,i}-\sum_{j\in\mathcal{N}(i)}w_{ij}\mathbf{p}_{t,j},\qquad w_{ij}=\frac{1}{|\mathcal{N}(i)|},where \mathcal{N}(i) is the neighborhood of keypoint i. The deformation loss matches the source and robot Laplacian coordinates:
E_{L}(q_{t})=\sum_{i}\left\|L(\mathbf{p}^{\mathrm{src}}_{t,i})-L(\mathbf{p}^{\mathrm{rob}}_{t,i}(q_{t}))\right\|_{2}^{2}.We solve this motion-fitting problem over the active joints with temporal regularization and joint feasibility:
\displaystyle\bar{q}_{t}=\arg\min_{q_{t}}\quad \\ \displaystyle E_{L}(q_{t})+\|q_{t}-\bar{q}_{t-1}\|_{Q}^{2} \\ \displaystyle q_{\min}\leq q_{t}\leq q_{\max}, \\ \displaystyle v_{\min}\Delta t\leq q_{t}-q_{t-1}\leq v_{\max}\Delta t.Unlike the full OmniRetarget program, Eq. (5) does not include a signed-distance penalty. In implementation, this is equivalent to running the batched joint-angle solver with an empty obstacle list. The resulting \bar{q}_{1:T} supplies a motion prior without prematurely forcing the robot around potentially mismatched scene geometry.
III-C2 Stage 2: Contact- and Speed-Preserving Refinement
Let q_{t}\in\mathcal{Q} denote the robot floating-base and joint configuration at time t. For a set of key links \mathcal{B}, forward kinematics gives world positions \mathbf{p}^{b,R}_{t}(q_{t})\in\mathbb{R}^{3} and orientations \mathbf{R}^{b,R}_{t}(q_{t})\in\mathrm{SO}(3). The source human positions and orientations are denoted \mathbf{p}^{b,H}_{t}\in\mathbb{R}^{3} and \mathbf{R}^{b,H}_{t}\in\mathrm{SO}(3) after semantic correspondence between human and robot keypoints. We solve
\displaystyle q^{\star}_{1:T}=\arg\min_{q_{1:T}}\quad \\ \displaystyle\mathcal{L}_{p}(q)+\lambda_{R}\mathcal{L}_{R}(q)+\lambda_{s}\mathcal{L}_{s}(q) \\ \displaystyle{}+\lambda_{\mathrm{col}}\mathcal{L}_{\mathrm{col}}(q)+\lambda_{a}\mathcal{L}_{\mathrm{anchor}}(q)\displaystyle\mathcal{L}_{p}(q)={} \\ \displaystyle\sum_{b\in\mathcal{B}}\sum_{t=2}^{T}\Big(\big\|\Delta\mathbf{p}^{b,H}_{t}\big\|_{2} \\ \displaystyle\hskip 45.00006pt{}-\big\|\Delta\mathbf{p}^{b,R}_{t}(q)\big\|_{2}\Big)^{2}, \\ \displaystyle\mathcal{L}_{R}(q)={} \\ \displaystyle\sum_{b\in\mathcal{B}}\sum_{t=2}^{T}\Big(d_{R}\!\left(\Delta\mathbf{R}^{b,H}_{t},\mathbf{I}_{3}\right) \\ \displaystyle\hskip 32.00002pt{}-d_{R}\!\left(\Delta\mathbf{R}^{b,R}_{t}(q),\mathbf{I}_{3}\right)\Big)^{2}, \\ \displaystyle\mathcal{L}_{s}(q)={} \\ \displaystyle\sum_{t=2}^{T}\|q_{t}-q_{t-1}\|_{2}^{2}, \\ \displaystyle\mathcal{L}_{\mathrm{col}}(q)={} \\ \displaystyle\sum_{t,j}\left[\max\!\left(0,-\phi_{j}(q_{t})\right)\right]^{2}, \\ \displaystyle\mathcal{L}_{\mathrm{anchor}}(q)={} \\ \displaystyle\sum_{k=1}^{K}\left(\|\mathbf{e}^{p}_{k}\|_{2}^{2}+\beta_{R}(e^{R}_{k})^{2}\right), \\ \displaystyle q_{\min}\leq q_{t}\leq q_{\max},\quad\forall t, \\ \displaystyle v_{\min}\Delta t\leq q_{t}-q_{t-1}\leq v_{\max}\Delta t,\quad\forall t.Here \Delta\mathbf{p}^{b,\cdot}_{t}=\mathbf{p}^{b,\cdot}_{t}-\mathbf{p}^{b,\cdot}_{t-1}, \Delta\mathbf{R}^{b,\cdot}_{t}=\mathbf{R}^{b,\cdot}_{t}(\mathbf{R}^{b,\cdot}_{t-1})^{\top}, and d_{R} is the geodesic distance on \mathrm{SO}(3). The anchor errors are \mathbf{e}^{p}_{k}=\mathbf{p}^{b_{k},R}_{\tau_{k}}(q_{\tau_{k}})-\hat{\mathbf{p}}_{k} and e^{R}_{k}=d_{R}(\mathbf{R}^{b_{k},R}_{\tau_{k}}(q_{\tau_{k}}),\hat{\mathbf{R}}_{k}). The first two terms preserve translational and angular speed, while \mathcal{L}_{s} smooths the trajectory. The collision and anchor terms are high-priority Lagrangian (KKT-style) penalties: their large weights prioritize clearance and the desired global contact pose, but gradient descent does not guarantee exact equality. If an annotation conflicts with feasible geometry, the optimizer returns the nearest feasible pose and we report its residual explicitly.
In practice, we optimize the full trajectory over each task’s active joints. Collision penalties are computed against cuboid or mesh obstacles, anchor costs act over a small temporal window, and joint clamps enforce limits.
III-D Contact-Conditioned Motion Generator
The optimized trajectory dataset is used to train a motion generator G_{\theta}. At each step, the generator receives a motion style vector \mathbf{s}, movable contact conditions \mathbf{c}, and a temporal context, and predicts reference robot and object motion:
(\hat{\mathbf{m}}_{t:t+\Delta},\hat{\mathbf{o}}_{t:t+\Delta})=G_{\theta}(\mathbf{s},\mathbf{c},\mathbf{m}_{\leq t},\mathbf{o}_{\leq t}).The style vector is a one-hot identifier for the source video or motion family. For the six table-tennis serves, this vector selects among simple forehand, simple backhand, forehand chop, backhand chop, forehand side spin, and backhand side spin. The contact vector stores task-dependent contact positions: toss and hit points for serving, object pickup contact pose for manipulation, and hand/foot support contacts for ladder climbing.
Contact-point augmentation.
For serving, we augment the racket–ball hit point on a 3D grid around the demonstrated location while preserving the toss and serve style; one-hot style vectors and per-style contact ranges represent the six serves. For object pickup, we vary the object pose within task-specific ranges. Bimanual variants keep both hands synchronized with the box, while ladder variants preserve the sequential hand and foot support anchors. The resulting optimized variants train the generator and policy over movable contact poses.
III-E RL Environment and Training
We train the tracking policy in Isaac Lab with a Unitree G1 model and joint-position actions. The simulator uses a 0.005 s physics step and action decimation of 4, yielding a nominal 50 Hz policy rate; the low-frequency variant runs at 25 Hz. Each episode follows the motion length. We use 4096 parallel environments with 2.5 m spacing.
Policy observations and actions.
The observation concatenates the reference command; motion-anchor position and orientation in the robot anchor frame; body positions and orientations relative to the anchor; base linear and angular velocity; relative joint position and velocity; and the previous action. Observation corruption uses uniform noise with ranges \pm 0.25 m for anchor position, \pm 0.05 for anchor orientation, \pm 0.5 m/s for base linear velocity, \pm 0.2 rad/s for base angular velocity, \pm 0.01 rad for joint position, and \pm 0.5 rad/s for joint velocity. Actions are joint-position targets about the default pose, scaled per actuator using effort limits and stiffness.
PPO and actor–critic architecture.
We use PPO [33] with empirical observation normalization. The actor and critic are MLPs with hidden dimensions [512,256,128], ELU activations, and initial action-noise standard deviation 1.0. PPO uses 24 steps per environment, 5 learning epochs, 4 minibatches, an adaptively scheduled learning rate of 10^{-3}, clipping parameter 0.2, entropy coefficient 0.005, value-loss coefficient 1.0, discount \gamma=0.99, GAE \lambda=0.95, desired KL 0.01, and maximum gradient norm 1.0. Training runs for up to 30{,}000 iterations, with checkpoints every 500 iterations.
Domain randomization.
Following the minimal randomization strategy of OmniRetarget [3], we randomize robot and contact parameters without task-specific curricula. We sample static friction in [0.3,1.6], dynamic friction in [0.3,1.2], restitution in [0,0.5], joint default offsets in \pm 0.01 rad, and torso center-of-mass offsets of \pm 0.025 m in x and \pm 0.05 m in y,z. Random pushes occur every 1–3 s, with linear velocity ranges x,y\in[-0.5,0.5] m/s and z\in[-0.2,0.2] m/s, and angular velocity ranges of [-0.52,0.52] rad/s in roll/pitch and [-0.78,0.78] rad/s in yaw.
III-F Rewards
Following OmniRetarget [3] and BeyondMimic [34], the reward combines anchor, body-pose, body-velocity, and optional object tracking with action-rate, soft joint-limit, and undesired-contact penalties:
r_{t}=\sum_{i}w_{i}r_{i}-w_{a}\|\mathbf{a}_{t}-\mathbf{a}_{t-1}\|_{2}^{2}-w_{j}r_{\mathrm{joint}}-w_{c}r_{\mathrm{contact}}.Tracking terms use
r_{\mathrm{exp}}(\mathbf{e};\sigma)=\exp\!\left(-\frac{\|\mathbf{e}\|_{2}^{2}}{\sigma^{2}}\right),where orientation errors use the geodesic quaternion distance. Table II gives the weights used for the G1 policy.
- Global anchor position
- 0.5
- 0.3
- Tracks torso/anchor world position.
- Global anchor orientation
- 0.5
- 0.4
- Tracks anchor orientation.
- Relative body position
- 1.0
- 0.3
- Tracks selected body-link positions relative to the anchor.
- Relative body orientation
- 1.0
- 0.4
- Tracks selected body-link orientations.
- Body linear velocity
- 1.0
- 1.0
- Tracks selected body-link linear velocities.
- Body angular velocity
- 1.0
- \pi
- Tracks selected body-link angular velocities.
- Action rate
- -0.1
- –
- Penalizes \|\mathbf{a}_{t}-\mathbf{a}_{t-1}\|_{2}^{2}.
- Soft joint limit
- -10.0
- –
- Penalizes excursions beyond soft joint-position limits.
- Undesired contact
- -0.1
- –
- Penalizes non-allowed links with contact force above 1 N; feet and wrists are excluded.
Episodes terminate on timeout or when anchor-height error exceeds 0.25 m, projected-gravity error exceeds 0.8, or ankle/wrist height error exceeds 0.25 m.
IV Experiments
Our evaluation is designed to answer four questions: (i) whether contact-aware retargeting reduces contact error and collision while preserving motion smoothness; (ii) whether it outperforms OmniRetarget-style retargeting on tasks where global contact pose matters; (iii) whether contact-conditioned generation can adapt the same motion to novel contact points; and (iv) whether the generated references successfully transfer to the real Unitree G1 robot.
IV-A Tasks
We evaluate our approach on ten third-person video skills, summarized in Table III. A skill is one source video and its contact-conditioned motion family; the six serves are distinct skills with different styles. The first six skills have movable toss and ball-hit points, the next three vary object-pick poses, and ladder climbing has sequential hand and foot supports.
- 1
- Simple forehand serve
- Toss
- Racket hit pose
- 2
- Simple backhand serve
- Toss
- Racket hit pose
- 3
- Forehand chop serve
- Toss
- Racket hit pose
- 4
- Backhand chop serve
- Toss
- Racket hit pose
- 5
- Forehand side-spin serve
- Toss
- Racket hit pose
- 6
- Backhand side-spin serve
- Toss
- Racket hit pose
- 7
- Pick object from tabletop
- -
- Object grasp pose
- 8
- Pick object under table
- Right-hand table-edge support
- Left-hand object grasp pose
- 9
- Bimanual box pickup from ground and placement on table
- Synchronized two-hand box grasp
- -
- 10
- Ladder climbing
- Hand and foot rung supports
- -
IV-B Baselines and Metrics
We compare VICAR against OmniRetarget [3], GMR [21], PHC [5], VideoMimic [1], and an ablation without speed preservation. Identical contact-pose penalty terms are added post-hoc to every baseline, and every method uses the same anchor-pose augmentations. Downstream tracking policies use identical PPO hyperparameters, training steps, domain randomization, and rewards. The comparison therefore isolates retargeting quality rather than contact formulation, augmentation, or policy compute. We report the following metrics:
- Penetration and residuals: mean signed penetration depth; mean position residual at annotated anchors; and infeasibility rate, defined as the fraction of frames with penetration greater than
1cm. - Smoothness: mean joint jerk and maximum joint velocity in motion (to measure discontinuity in motions). For optimized joint trajectories
q^{(n)}_{1:T}overNsamples andJjoints, mean joint jerk is computed as the average absolute third finite difference,\displaystyle J_{\mathrm{jerk}}\displaystyle=\frac{1}{N(T-3)J}\sum_{n=1}^{N}\sum_{t=1}^{T-3}\sum_{j=1}^{J}\left|q^{(n)}_{t+3,j}-3q^{(n)}_{t+2,j}\right.(13)\displaystyle\left.{}+3q^{(n)}_{t+1,j}-q^{(n)}_{t,j}\right|,reported in rad/step3. - Speed preservation: error between source and retargeted keypoint displacement magnitudes of end-effectors (wrist and feet). We compute
\displaystyle E_{\mathrm{speed}}\displaystyle=\frac{1}{|\mathcal{B}_{\mathrm{ee}}|(T-1)}\sum_{b\in\mathcal{B}_{\mathrm{ee}}}\sum_{t=2}^{T}(14)\displaystyle\left(\left\|\Delta\mathbf{p}^{b,H}_{t}\right\|_{2}-\left\|\Delta\mathbf{p}^{b,R}_{t}\right\|_{2}\right)^{2},where\mathcal{B}_{\mathrm{ee}}is the set of evaluated end-effectors and\Delta\mathbf{p}^{b,\cdot}_{t}=\mathbf{p}^{b,\cdot}_{t}-\mathbf{p}^{b,\cdot}_{t-1}. - Downstream RL Policy success rate: success rate of trained policy in simulation over 1000 trials.
IV-C Retargeting Results
Table IV reports the quantitative comparison. Table-tennis success depends strongly on augmentation and runtime contact estimation, but a smooth reference remains a prerequisite: GMR attains 97.2\% versus VICAR’s 99.8\%. The clearest gains occur on obstacle-critical tasks, where downstream success rises from 76.2\% to 99.8\% for tabletop pickup and from 56.0\% to 95.5\% for ladder climbing relative to OmniRetarget. In tabletop pickup, OmniRetarget’s initial reference enters and exits the platform through different faces, so a later non-penetration correction becomes discontinuous; VICAR instead preserves contact and speed while finding a smooth path (Figure 3).
- Method
- Pen. (cm) \downarrow
- Anchor res. (cm) \downarrow
- Infeas. (%) \downarrow
- Avg jerk (rad/step3) \downarrow
- Max vel. (rad/step) \downarrow
- Speed err. (m/step) \downarrow
- RL success (%) \uparrow
- PHC [5]
- 0.00/3.54/0.75
- –
- –
- 0.059/.0020/.0016
- 1.496/.205/.096
- .014/.073/.013
- 41.0/0.0/0.0
- GMR [21]
- 0.00/4.21/7.43
- –
- –
- .033/.0049/.0045
- .831/1.159/.523
- .0018/.397/.023
- 97.2/52.4/46.8
- VideoMimic [1]
- 0.00/.53/.37
- –
- –
- .033/.0051/.0054
- .894/1.204/.614
- .0018/.464/.120
- 96.3/99.7/94.6
- OmniRetarget [3]
- 0.00/.03/.06
- –
- –
- .062/.0046/.0020
- 1.597/.833/.176
- .014/.946/.013
- 99.8/100/98.1
- VICAR
- 0.00/.04/.01
- .12/.42/.52
- 0.0/0.0/0.0
- .032/.0026/.0024
- .787/.249/.098
- .000/.0014/.00019
- 99.8/100/99.2
- Pick from table / Ladder climbing
- No data
- No data
- No data
- No data
- No data
- No data
- No data
- Method
- Pen. (cm) \downarrow
- Anchor res. (cm) \downarrow
- Infeas. (%) \downarrow
- Avg jerk (rad/step3) \downarrow
- Max vel. (rad/step) \downarrow
- Speed err. (m/step) \downarrow
- RL success (%) \uparrow
- PHC [5]
- 8.52/.54
- –
- –
- .0012/.0043
- .086/.195
- .0015/.278
- 0.0/0.0
- GMR [21]
- 8.15/1.32
- –
- –
- .0019/.0091
- .40/.748
- .013/.297
- 11.2/0.0
- VideoMimic [1]
- .21/.85
- –
- –
- .0037/.0096
- .42/.728
- .058/.310
- 62.2/0.0
- OmniRetarget [3]
- .02/.08
- –
- –
- .0032/.0078
- .37/.530
- .35/.419
- 76.2/56.0
- VICAR
- .00/.05
- .19/1.5
- 0.0/2.6
- .0024/.0087
- .10/.303
- .027/.012
- 99.8/95.5
IV-D Contact-Point Generalization
Figure 6 reports held-out contact-point performance as the number of augmented trajectories increases; the corresponding augmentation procedure is described in Section III-D.
IV-E Hardware Deployment
We deploy the policies on hardware. No real-robot training data are used, so deployment is zero-shot sim-to-real in that specific sense, not visual sim-to-real. The policy tracks generated references using proprioception and projected gravity, while an external motion-capture system supplies the global ball, object positions used to update contact conditions. The Dex3-1 hand is used for object pickup. Hardware success is 21/30 (70.0\%) for table-tennis serves, 10/10 (100\%) for tabletop pickup, 7/10 (70.0\%) for under-table pickup, and 6/10 (60.0\%) for bimanual box pickup.
V Limitations
This framework has several limitations. Contact frames, body parts, and \mathrm{SE}(3) poses are manually annotated, requiring approximately 20–30 minutes per skill; new task families still require per-task annotation and scene reconstruction. Motions with unreachable contacts, unsupported balance states, or dexterous manipulation beyond the hand model can fail. Ladder hardware deployment failed because the same blind tracking policy stumbled, even though the retargeted trajectory was smooth. Deployment also requires substantial motion-capture engineering to recover global pelvis, object, and ladder poses. Finally, robustness to monocular depth errors and Semantic-SAM or GPT-4o mini hallucinations is outside our present contribution. Future work will automate contact labeling, integrate onboard perception, and train vision-conditioned policies in reconstructed simulation twins.
VI Conclusion
We introduced VICAR, a pipeline for learning humanoid interactive skills from third-person videos through contact-aware retargeting. The core idea is to make the useful part of a human demonstration explicit: not only the pose sequence, but also the global contacts that make the task work. By reconstructing task-relevant obstacles, preserving contact poses and speed during retargeting, and training a generator conditioned on movable contact points, VICAR can adapt video-derived skills to new object or ball positions before a tracking policy executes them on hardware. The current results span ten contact-rich motions, from table-tennis serves to pickup tasks and ladder climbing.