EgoExplorer: A Vision-Motion-Touch Observatory for Human Loco-Manipulation In Situ

Abstract

Learning humanoid loco-manipulation from human demonstrations rests on observing how locomotion, posture, and object interaction evolve together as tasks unfold. Scaling collection across diverse environments, in turn, favors lightweight, portable capture systems. Yet, existing datasets provide only partial views of this coordination. Thus, we introduce EgoExplorer, a 100-hour vision-motion-touch observatory of human loco-manipulation built with a wearable system. With an emphasis on professional activities, its continuous recordings align egocentric video with reconstructed whole-body motion, fine-grained finger articulation, and tactile observations, accompanied by step-level descriptions. To achieve this coverage with sparse body sensing, we develop the Calibrate-Complete-Constrain pipeline. Concretely, reusable calibration fixes participant geometry and sensor attachments before motion inference to disentangle mounting offsets from articulation. A temporal prior then completes unobserved joint motion, while layered constraints promote measurement agreement, temporal continuity, and geometric consistency, properties that motion plausibility alone does not guarantee. Against tracker-fitted references, the method reduces body-point error by 71.9% relative to the strongest baseline; matched evaluations also show gains in hand stability and contact estimation. Preliminary studies explore multimodal generation and conditioning alongside humanoid retargeting with whole-body control in simulation.

Homepage | Toolkit | Data

1 Introduction

What a humanoid needs to reach, grasp, or carry shapes where it moves and how it positions its body (Shi et al., 2026). Learning such loco-manipulation skills calls for demonstrations spanning diverse environments, objects, and task sequences. Robot teleoperation provides such data, but dedicated hardware, on-site deployment, and sustained operator effort constrain its scalability (Zhu et al., 2026; Punamiya et al., 2026). Human activity offers a rich alternative across everyday and professional settings, and recent work shows that egocentric demonstrations can improve robot manipulation and generalization to novel scenarios (Kareer et al., 2025; Qiu et al., 2025; Shi et al., 2026). Realizing this potential for loco-manipulation hinges on datasets that jointly record visual context, whole-body and hand motion, and measured contact throughout continuous tasks.

Building such datasets at scale entails balancing environmental coverage with the richness and fidelity of the observations. Systems built around fixed camera arrays for optical motion capture (Cao et al., 2026) or multi-view reconstruction (Bhatnagar et al., 2022; Zhang et al., 2022) can provide detailed observations, but require each recording space to be instrumented and calibrated (Taheri et al., 2020; Kim et al., 2025). Portable capture eases deployment, yet existing datasets often offer only partial views of this coordination. Particularly, egocentric video preserves diverse visual context but leaves much of the body unobserved (Grauman et al., 2022). Wearable body tracking adds body motion, often without fine finger articulation or physical contact measurements (Ma et al., 2024; DeTone et al., 2026). Conversely, hand-focused datasets capture detailed hand motion and, in some cases touch, but typically omit the accompanying whole-body movement (Hoque et al., 2026; Zhou et al., 2026; Song et al., 2025). These partial records obscure the coupling central to loco-manipulation and therefore fall short of capturing its execution as a coherent whole.

To this end, we introduce EgoExplorer, an egocentric dataset that captures the whole-body coordination underlying human loco-manipulation through portable vision-motion-touch sensing. As depicted in Fig. 1, it aligns first-person video with reconstructed body motion, glove-measured finger articulation, and tactile observations. Collected across five environments with an emphasis on professional activities, the recordings preserve the transitions between approaching targets, adjusting posture, manipulating objects, and relocating. Step-level descriptions link these synchronized observations to the actions and progression of each task.

To support this multimodal capture, our lightweight wearable rig combines five trackers, instrumented gloves, and a head-mounted camera. Recovering whole-body trajectories requires inferring unobserved joint motion from these sparse device poses. Yet direct fitting is ambiguous: each pose depends jointly on its attachment transform and the body’s kinematic chain, allowing different combinations of mounting offsets and joint configurations to explain the same sparse measurements (Jiang et al., 2022; Ponton et al., 2023). Differently, our \mathrm{C}^{3} pipeline (Calibrate-Complete-Constrain) disentangles fixed body-sensor geometry from motion through a calibration protocol: prescribed participant movements are matched to a reference motion library. Using these correspondences, we refine initial estimates of body shape and scale and fit body-sensor transforms, forming a profile reusable across tasks with unchanged sensor placement. A temporal prior model then completes unobserved articulation from the calibrated anchors, yielding an initial whole-body trajectory, i.e., the motion proposal. We refine this proposal under layered constraints to promote measurement agreement and temporal continuity while penalizing self-interpenetration and ground penetration. Finally, we incorporate hand motion, touch, and expert-guided step annotations and check multimodal consistency. The resulting dataset comprises 2,715 egocentric recordings totaling 100 hours (Tab. 1).

We evaluate annotation accuracy on a purpose-built Evaluation Subset and reconstruction consistency on the Natural Recordings. Compared with the strongest evaluated body-reconstruction baseline, our system reduces body-point error by 71.9%. Hand and contact tests also show better holding stability, kinematic target recovery, and regional contact estimation than the evaluated alternatives, supporting the benefits of calibrated reconstruction and direct hand sensing. Beyond annotation quality, we conduct a preliminary exploration of training utility through Cosmos-based (NVIDIA, 2026) multimodal prediction and human-to-robot motion transfer using ZeroWBC-style (Yang et al., 2026a) whole-body control in simulation. These initial studies suggest benefits from combining synchronized modalities and illustrate the potential of the recordings for policy learning.

We summarize our contributions as follows: (i) EgoExplorer, a vision-motion-touch dataset that achieves rich multimodal coverage and high-fidelity motion annotations with lightweight instrumentation, capturing continuous human loco-manipulation; (ii) the \mathrm{C}^{3} pipeline, which decouples body-sensor calibration from motion inference and combines temporal completion with layered constraints to recover metric whole-body motion from sparse trackers. (iii) experiments demonstrating improved annotation quality and the effectiveness of our design choices, complemented by an initial assessment of the recordings’ utility for embodied learning.

2 Related Work

Multimodal human datasets. Calibrated capture systems provide detailed body, hand, and interaction geometry, but require instrumented spaces (Taheri et al., 2020; Bhatnagar et al., 2022; Zhang et al., 2022; Kim et al., 2025; Cao et al., 2026). Egocentric video datasets broaden activity coverage (Damen et al., 2022; Grauman et al., 2022; Grauman et al., 2024), with manipulation-oriented collections increasingly adding hand-motion annotations (Hoque et al., 2026; Punamiya et al., 2026; Li et al., 2026b; Li et al., 2026a). Wearable body tracking adds full-body trajectories but generally lacks direct finger and tactile measurements (Ma et al., 2024; Fan et al., 2025; DeTone et al., 2026). Hand-focused datasets capture articulation, object geometry, or contact, typically without the accompanying whole-body motion (Banerjee et al., 2025; Zhao et al., 2025; Zhou et al., 2026; Song et al., 2025). EgoExplorer combines these signals in portable recordings of continuous loco-manipulation.

Motion reconstruction from partial observations. Video-based reconstruction remains sensitive to occlusion, depth ambiguity, and camera motion (Shen et al., 2024; Zhang et al., 2025; Xu et al., 2026). Sparse-tracking methods complete motion from observations using learned priors (Jiang et al., 2022; Du et al., 2023; Ponton et al., 2023; Jiang et al., 2024b; Dai et al., 2024). Kinematic fitting and physical regularization improve plausibility (Jiang et al., 2022; Zheng et al., 2023; Jiang et al., 2024a). Yet matching sparse anchors leaves intermediate joints ambiguous, and mount errors can be absorbed into articulation. Our \mathrm{C}^{3} pipeline combines participant and mount calibration, motion completion, and constrained refinement to address sensor mismatch, self-intersection, and floor penetration.

Human data for embodied learning. Human demonstrations support cross-embodiment manipulation learning (Kareer et al., 2025; Qiu et al., 2025; Punamiya et al., 2026), mobile and humanoid policy generalization (Zhu et al., 2026; Zhong et al., 2025; Shi et al., 2026), and whole-body control through motion retargeting and contact-aware references (Yang et al., 2026b; Yang et al., 2026a; Yu et al., 2026). EgoExplorer preserves visual context, body coordination, and motion-contact relationships throughout continuous tasks, providing a resource for these learning settings.

3 The EgoExplorer Dataset

Egocentric video captures task semantics at scale, yet leaves key signals for loco-manipulation under-constrained: hand articulation is obscured; whole-body motion, especially lower-body support, extends beyond the camera’s field of view; and appearance alone cannot reveal tactile contact. EgoExplorer addresses this asymmetry by pairing a head-mounted camera with sparse wearable sensing, grounding body and hand motion in sensor measurements and corresponding touch records. The dataset contains 2,715 recordings (100 h) across ten task templates and five environments, spanning laboratories, factories, and homes. Following the workflow in Fig. 2, we first introduce the capture hardware (Sec. 3.1), then describe sensor calibration, body completion, and constrained refinement of fine-grained body reconstruction (Sec. 3.2), and enrich the episodes with expert-guided task boundaries and annotations (Sec. 3.3). Sec. 4 offers a glimpse of the dataset.

3.1 Capture Hardware

Requirements. Fine-grained loco-manipulation capture requires accurate measurements of body motion, hand articulation, and contact in a lightweight, portable system. Studio motion capture (Taheri et al., 2020; Kim et al., 2025; Cao et al., 2026) provides dense measurements but confines collection to a calibrated multi-camera volume, whereas readily deployable video-only capture (Shen et al., 2024; Zhang et al., 2025) must infer occluded motion and visually ambiguous contact. We therefore complement sparse tracking at key body joints with direct finger sensing and optional touch for whole-body reconstruction and fine-grained interaction annotation.

Hardware. Five VIVE Ultimate Trackers locate the head, hands, and ankles (App. A), using onboard cameras for 6-DoF inside-out tracking without external base stations. Head and hand trackers establish metric camera and end-effector trajectories, while ankle trackers capture foot placement and constrain stance and support beyond the head camera’s view. Custom 3D-printed mounts rigidly couple the head tracker to a GoPro HERO13 with Media Mod and attach the others to the glove dorsum and lateral ankles, with straps for added fixation. Two interchangeable glove configurations share this layout: MANUS Metagloves measure finger articulation, while Wuji gloves additionally record tactile signals.

Deployment. The compact, reusable rig simplifies sensor setup and donning, facilitating collection across environments. Subject and mount calibration is reused across task recordings as long as the rig remains mounted. Sec. 3.2 details calibration and whole-body reconstruction from sparse measurements; App. G covers cost and deployment trade-offs.

3.2 Whole-Body Reconstruction from Sparse Measurement

\mathbf{C}^{3}: Calibrate-Complete-Constrain. Whole-body reconstruction calls for inferring the body joint motion from the rig’s sparse tracker poses. Yet, coupled ambiguities arise: tracker poses depend jointly on body articulation and rigid attachment to anatomical anchors (i.e., head, wrists, and ankle joint frames), allowing mounting errors to be absorbed into estimated motion. Even calibrated anchors leave intermediate joints undetermined. We address these with \mathrm{C}^{3} (Calibrate-Complete-Constrain; Fig. 2), separating fixed body-sensor geometry from time-varying motion. Protocol-guided calibration aligns a prescribed movement sequence (i.e., the calibration protocol) with a reference motion library by phase. Robust fitting estimates body shape, scale, and sensor-to-anchor transforms from these correspondences once per mounting configuration. Prior-guided motion completion uses a temporal motion prior, a model for completing unobserved articulation, to generate a motion proposal: an initial whole-body trajectory from anchors. Layered constraints then jointly refine skeletal measurement agreement, surface clearance, and ground non-penetration. We fuse glove-measured finger articulation and touch as direct interaction evidence.

3.2.1 Protocol-guided calibration

Calibration problem definition. Calibration estimates a participant-and-mount profile \Phi=(\beta,s,\{A_{i},a_{i}\}_{i=1}^{5}). Here \beta\in\mathbb{R}^{10} specifies SMPL-X body shape (Pavlakos et al., 2019) and s>0 converts model dimensions to the participant’s metric dimensions. For tracker i, a_{i}\in\mathbb{R}^{3} is the displacement from the anchor to the device origin. A_{i}\in\mathrm{SO}(3) expresses the device’s axes in that same frame, where \mathrm{SO}(3) denotes proper 3D rotation matrices. The five attachments correspond to the head, wrists, and ankles. These parameters remain shared across calibration actions and subsequent tasks with unchanged mounts; joint articulation varies. We estimate \Phi by anchoring body dimensions, matching action phases, and fitting one attachment per device across the matched poses.

Body shape and scale initialization. Visual evidence provides an initial estimate of body proportions; direct height measurement anchors its metric size. We obtain \beta^{\mathrm{vis}}\in\mathbb{R}^{10} from a short participant video using GVHMR (Shen et al., 2024) and measure the participant’s height h^{\star}. Let H(\beta) be the unscaled model height in neutral pose, then \beta_{0}=\beta^{\mathrm{vis}} initializes the shape fit. Taking the visual shape as our starting point, we rescale it to the measured height s_{0}=\frac{h^{\star}}{H(\beta^{\mathrm{vis}})}. The resulting (\beta_{0},s_{0}) initializes the geometry fit below. The measurements are collected for the reusable profile.

Protocols. To disentangle body proportions, sensor offsets, and world-frame alignment, we use a calibration protocol comprising standing, A- and T-poses, forward arm extension, head and wrist rotations, elbow flexion, shallow squats, leg raises, and ankle motion. A reference motion library provides consistent anatomical targets without requiring an exact match to the participant’s motion. To accommodate differences in execution speed, we automatically match ordered holds and motion peaks in recorded tracker signals to reference actions, pairing observed device poses with corresponding reference joint poses. The library is cached and reused, limiting participant-specific work to protocol recording and profile fitting. The full protocol and sample-selection rules are detailed in Sec. C.1.

Forward model for calibration. A tracker’s pose is jointly determined by the body shape, the pose locating the anchor, and the attachment placing the device relative to it. Let n=1,\ldots,N_{\mathrm{cal}} index matched calibration samples, with r(n) identifying the recording containing sample n. Each sample corresponds to reference action k_{n} at normalized phase \alpha_{n}\in[0,1] (0: start; 1: end). Phase-based correspondence accommodates differences in execution speed, and each action may contribute multiple samples. For body shape \beta, the matched reference pose specifies anchor i’s pelvis-relative position J_{n,i}(\beta)=J_{i}^{k_{n}}(\alpha_{n};\beta) and orientation G_{n,i}=G_{i}^{k_{n}}(\alpha_{n}) in the reference body frame. Let s denote body scale, and let a_{i} and A_{i} denote the fixed attachment translation and rotation expressed in anchor i’s frame. Recording-specific yaw rotation Q_{r} and pelvis translation u_{r} then give the predicted tracker position and orientation in the tracking world frame:

\widehat{p}_{n,i}=u_{r(n)}+Q_{r(n)}\!\left[sJ_{n,i}(\beta)+G_{n,i}a_{i}\right],\ \widehat{R}_{n,i}=Q_{r(n)}G_{n,i}A_{i}.

Here \widehat{p}_{n,i} and \widehat{R}_{n,i} are the predicted position and orientation of tracker i at matched sample n. The term sJ_{n,i}(\beta) locates the joint at metric scale and G_{n,i}a_{i} adds the mounting offset, \widehat{p}_{n,i} enters the position loss to fit profile term \Theta; the relation for \widehat{R}_{n,i} is then used separately to recover A_{i}.

Joint calibration objective. We define a calibration loss to jointly estimate body geometry, attachment displacements, and recording alignments, collected in \Theta=(\beta,s,\{a_{i}\}_{i=1}^{5},\{Q_{r},u_{r}\}_{r}). Anthropometric measurements and geometric priors constrain this fit to discourage compensation between limb dimensions and mounting offsets. Initializing (\beta,s) at (\beta_{0},s_{0}), we solve:

\Theta^{\mathrm{fit}}=\arg\min_{\Theta:\,s>0}\mathcal{L}_{\mathrm{cal}}(\Theta),\ \mathcal{L}_{\mathrm{cal}}(\Theta)=\lambda_{\mathrm{obs}}\mathcal{L}_{\mathrm{robust\mbox{-}pos}}+\lambda_{\mathrm{anth}}\mathcal{L}_{\mathrm{anth}}+\mathcal{R}_{\mathrm{geom}}.

Here \Theta^{\mathrm{fit}} denotes the fitted parameters and the non-negative \lambda coefficients weight the loss terms. \mathcal{L}_{\mathrm{robust\mbox{-}pos}} compares predicted tracker positions \widehat{p}_{n,i} from Eq. 1 with tracker-measured positions p_{n,i}; \mathcal{L}_{\mathrm{anth}} constrains height and arm span to their measurements; and \mathcal{R}_{\mathrm{geom}} anchors shape to the visual estimate and regularizes attachments, scale, and yaw. Reference rotations remain fixed during optimization. Loss definitions, squat-translation handling, and solver settings are in Sec. C.1.

Attachment rotation recovery. We next estimate the remaining rotation A_{i} by matching predicted tracker orientations \widehat{R}_{n,i} to measured ones R_{n,i}. With Q_{r(n)} and G_{n,i} fixed, the rotation-calibration routine f_{\mathrm{rot}} fits one shared A_{i} through the orientation model in Eq. 1 across matched samples. We summarize this fitting step as A_{i}=f_{\mathrm{rot}}(\{\widehat{R}_{n,i},R_{n,i}\}_{n}), completing \Phi (Sec. C.1).

3.2.2 Prior-guided motion completion

Reconstruction requirements. With the calibrated profile \Phi fixed, we reconstruct \mathcal{X}=\{x_{t}\}_{t=1}^{T}, where x_{t}=(\omega_{t},\theta_{t},u_{t}) comprises the global pelvis orientation, the local rotations of the 21 SMPL-X joints relative to their respective parents, and the pelvis position in the world frame. Joint positions follow from forward kinematics with the calibrated body shape \beta and scale s. A temporal motion prior completes unobserved articulation from the five anchors, yielding an initial motion proposal. A proposal that is plausible under the prior may still deviate from calibrated anchor poses or exhibit body self-interpenetration and ground penetration. We therefore refine it under three constraint layers: skeletal motion, participant-specific surface clearance, and body-ground consistency.

Motion completion. The calibrated attachments map measured tracker poses (p_{t,i},R_{t,i}) to anatomical anchor frames: \widetilde{R}_{t,i}=R_{t,i}A_{i}^{\top} and \widetilde{p}_{t,i}=p_{t,i}-\widetilde{R}_{t,i}a_{i}. These frames locate the head, wrists, and ankles but leave intermediate articulation underdetermined. To complete the unobserved motion, we extend AvatarPoser (Jiang et al., 2022) from head-and-hand inputs to all five anchors and train the transformer on AMASS (Mahmood et al., 2019). The resulting motion prior f_{\mathrm{prior}} maps an L-frame observation history to initial root orientation and local body-joint rotations: (\omega_{t}^{0},\theta_{t}^{0})=f_{\mathrm{prior}}(\mathbf{z}_{t-L+1:t}), where \mathbf{z}_{t} encodes calibrated positions, rotations, and their temporal increments for all five anchors. We recover the pelvis position by aligning the predicted head position with its calibrated anchor: u_{t}^{0}=\widetilde{p}_{t,\mathrm{head}}-s\bar{J}_{\mathrm{head}}(\omega_{t}^{0},\theta_{t}^{0};\beta), where \bar{J}_{\mathrm{head}} is the unscaled pelvis-to-head displacement under the predicted root orientation and joint rotations. Together, these estimates form the initial trajectory \mathcal{X}^{0}=\{(\omega_{t}^{0},\theta_{t}^{0},u_{t}^{0})\}_{t=1}^{T}. Model training is detailed in Sec. C.2.

3.2.3 Layered constraints

Constraint layer 1: skeletal motion. The learned completion provides an initial articulation; refinement fits it to the current recording’s measurements. With shape, scale, and attachments fixed, forward kinematics maps each candidate state x_{t} to predicted tracker poses (\widehat{p}_{t,i},\widehat{R}_{t,i}) through Eq. 1. Position and orientation losses \mathcal{L}_{p} and \mathcal{L}_{R} compare these predictions with valid raw tracker observations. To regularize the fit, \mathcal{L}_{\mathrm{prior}} discourages unsupported deviations from \mathcal{X}^{0}, \mathcal{L}_{\mathrm{time}} regularizes temporal changes in articulation, and \mathcal{L}_{\mathrm{joint}} penalizes extreme poses through soft joint limits. Together, these terms define \mathcal{L}_{\mathrm{skel}}=\sum_{k}\lambda_{k}\mathcal{L}_{k}, with details in Sec. C.3.

Constraint layer 2: surface clearance. Skeletal agreement does not account for body thickness: an arm may intersect the torso even when its wrist matches the target. We therefore constrain the posed SMPL-X surface using the participant’s calibrated shape and scale. For nearby points sampled from distinct body parts, \mathcal{L}_{\mathrm{surf}} penalizes insufficient oriented clearance along the target surface normal, distinguishing points outside the surface from those that have crossed it. Aggregating the largest violations per part pair concentrates corrections in problematic regions, discouraging self-intersection while respecting participant-specific geometry (Sec. C.4).

Constraint layer 3: body-ground consistency. An ankle anchor above the floor does not ensure that the sole, heel, or knee also clears it. We apply \mathcal{L}_{\mathrm{floor}} to all posed body vertices, penalizing their depth below the estimated ground plane. This constrains the body’s placement relative to the environment, including regions that approach the floor during crouching or kneeling. The term penalizes penetration without imposing foot-contact or foot-locking constraints.

Joint refinement. The three layers act on the same trajectory through a shared objective:

\mathcal{X}^{\star}=\arg\min_{\mathcal{X}}\mathcal{L}_{\mathrm{body}}(\mathcal{X};\Phi),\ \mathcal{L}_{\mathrm{body}}(\mathcal{X};\Phi)=\mathcal{L}_{\mathrm{skel}}+\lambda_{\mathrm{surf}}\mathcal{L}_{\mathrm{surf}}+\lambda_{\mathrm{floor}}\mathcal{L}_{\mathrm{floor}}.

The non-negative \lambda coefficients weight the surface and ground terms; \mathcal{L}_{\mathrm{skel}} includes its component weights. Starting from \mathcal{X}^{0}, the solver evaluates all three layers at every iteration and differentiates through SMPL-X forward kinematics and skinning to update (\omega_{t},\theta_{t},u_{t}), keeping \Phi fixed. Optimization runs in temporal chunks, whose fitted states form \mathcal{X}^{\star} (Sec. C.6).

3.3 Multimodal integration and annotation

Hand and touch fusion. We directly fuse glove-measured finger geometry to the SMPL-X hands in wrist-local coordinates. Tactile observations are aligned in time and associated with the corresponding hand regions. Hand fitting and tactile processing are detailed in Sec. D.1.

Expert-guided procedural context. We condition off-the-shelf VLM tools (Peng et al., 2026) on egocentric video, task instructions, and expert-provided skill documents outlining workflow stages, typical progression, and relevant equipment and operations (Sec. D.2). The tools produce subtask segments with start/end times, skill labels, object/tool roles, observed outcomes, and concise descriptions; uncertain outcomes remain unspecified. Video timestamps align the annotations with motion and tactile streams. Finally, operators manually review video, reconstructed motion, available touch, and text for temporal alignment and semantic consistency.

4 Data Statistics

EgoExplorer comprises 2,715 recordings (100 hours) spanning ten task templates across five environments, with an emphasis on professional activities. Fig. 3 summarizes session duration and the balance between regular and recovery episodes, including responses to interruptions or failed attempts. Figs. 4 and 5 illustrate synchronized body–touch observations and procedural text. Further statistics on sensing coverage, motion, contact, text, and reconstructed environments are provided in App. E.

5 Experiments

We organize evaluation around annotation quality, preliminary training utility, and key design choices.

Evaluation setup. Our purpose-built Evaluation Subset covers whole-body motion, hand articulation, and regional contact. Evaluation addresses three aspects: kinematic accuracy, through end-point and full-body point errors; temporal quality, through tracking continuity and trajectory smoothness; and contact accuracy, through regional IoU and F1. References are modality-specific, with full-body errors measured against tracker-fitted labels. The remaining Natural Recordings lack matched accuracy references and are assessed for sensor agreement, temporal quality, and geometric consistency. Subsets, metrics, and full-body references are detailed in Secs. F.1, F.4 and F.2.

5.1 Annotation Reliability

Whole-body reconstruction. Tab. 2 compares three baseline settings: public pretrained checkpoints with calibrated sensor-to-anchor conversion, direct use of raw tracker poses, and fine-tuning with five tracker anchors. EgoExplorer outperforms all three settings across every reported metric, achieving 18.0 mm body-point error and 20.3 mm at non-tracked points. Its lowest errors extend beyond the observed anchors to other body points, including elbows and knees, indicating better recovery of full-body articulation. Visual comparisons are provided in Fig. 20.

Hand motion and touch capture. Tabs. 4 and 4 compare our glove settings with visual and kinematic inference. In matched-configuration hand tests, our glove setting yields 0.4 mm holding RMS, versus 4.5 mm for HaWoR (Zhang et al., 2025) and 11.3 mm for HandFlow (Xu et al., 2026), and achieves 100% kinematic target recovery. For regional contact, our glove setting achieves 0.915 IoU, compared with 0.621 for HACO (Jung and Lee, 2025) and 0.324 for hand-pose-only inference. Together, these tests support direct sensing of finger articulation and contact.

Step text quality. Expert review of model-generated annotations (Tab. 5) finds factual errors in 11.09% of reviewed segments and unsupported claims in 2.46%. The audit also reports missed- and extra-step rates of 8.16% and 2.05%, respectively, with boundary corrections in 3.08% of segments. These residual errors motivate manual verification and correction in our workflow before release.

5.2 Preliminary Training Study

We adapt Cosmos3-Edge (NVIDIA, 2026) to generate video, SMPL-X-derived action, and touch (Tab. 6). With common video/action/touch history, we compare separate and joint generation using sampled predictions at 5K training steps. Joint generation improves video PSNR from 15.94 to 16.62 dB and contact F1 from 0.610 to 0.694 over the separate models, with a lower action MSE. Conditioning action generation on recorded video over the prediction interval reduces action MSE from 0.0478 to 0.0277; adding touch reduces it to 0.0264. These conditions reveal complementary information across modalities. Retargeting two recordings to a Unitree G1 and tracking them with a ZeroWBC-style (Yang et al., 2026a) whole-body controller provides a preliminary demonstration of human-to-robot motion transfer in simulation (Fig. 6). These initial studies illustrate multimodal learning and simulated motion tracking; protocols are detailed in Secs. F.6 and F.9.

5.3 Ablation Study

Calibration and layered reconstruction. Tab. 7 shows complementary gains from successively adding Layers 1–3: body-point error drops from 79.0 to 16.1 mm, crossing pairs from 72.8 to 38.7 per frame, and floor violations from 61.4% to 6.0%, respectively. Full has the lowest Avg. (28.4; mean of nine reported values), despite higher body-point error and jerk than Layer 1 alone. Replacing calibrated shape and scale with the mean body raises body-point error from 18.0 to 59.5 mm and floor violations from 6.0% to 50.9%. Omitting mount calibration during training raises body-point error to 95.0 mm, while omitting it only at test time yields 98.1 mm. These results support fixing body-sensor geometry before motion recovery. Fig. 7 shows reduced self-intersection and floor penetration.

6 Conclusion

This paper introduces EgoExplorer, an egocentric dataset aligning whole-body motion, hand articulation, contact, and descriptions for human loco-manipulation. The proposed \mathrm{C}^{3} pipeline recovers metric body motion from sparse wearable sensing through calibration, temporal completion, and layered constraints. Evaluations show the benefits of direct sensing, and support this portable approach to capturing and studying coordinated movement and manipulation.

Limitations. Reconstruction cannot guarantee exact recovery of all joints, while tactile sensing has variable reliability. Preliminary studies leave benefits for real-world humanoid learning unverified.

Ackonwledgement

This project is supported by Shanghai Artificial Intelligence Laboratory. We are grateful to all data collection contributors for their dedicated efforts in large-scale data acquisition. We also acknowledge the support and guidance from Prof. Minyi Guo and Prof. Hongzi Zhu.

References

1. Araujo et al. (2025) J. P. Araujo, Y. Ze, P. Xu, J. Wu, and C. K. Liu Retargeting matters: general motion retargeting for humanoid motion tracking. arXiv preprint arXiv:2510.02252.
2. Banerjee et al. (2025) P. Banerjee, S. Shkodrani, P. Moulon, S. Hampali, S. Han, F. Zhang, L. Zhang, J. Fountain, E. Miller, S. Basol, R. Newcombe, R. Wang, J. J. Engel, and T. Hodan HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7061–7071. External Links: Link
3. Bhatnagar et al. (2022) B. L. Bhatnagar, X. Xie, I. A. Petrov, C. Sminchisescu, C. Theobalt, and G. Pons-Moll BEHAVE: Dataset and Method for Tracking Human Object Interactions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15914–15925. External Links: Document
4. Build AI (2025) Build AI Egocentric-10K. Note: Hugging Face dataset External Links: Link
5. Cadene et al. (2026) R. Cadene, S. Alibert, F. Capuano, M. Aractingi, A. Zouitine, P. Kooijmans, J. Choghari, M. Russi, C. Pascal, S. Palma, M. Shukor, J. Moss, A. Soare, D. Aubakirova, Q. Lhoest, Q. Gallouédec, and T. Wolf LeRobot: an open-source library for end-to-end robot learning. In The Fourteenth International Conference on Learning Representations, External Links: Link
6. Cao et al. (2026) Y. Cao, H. Xie, B. Wen, R. Yao, Y. Liu, Y. Huang, Z. Liao, Y. Wang, H. Liu, X. Tian, D. Su, L. Zhuo, D. Tao, X. Wang, L. Pan, and Z. Liu ACE-Data-0: human-centric ambient capture as embodied data engine. arXiv preprint arXiv:2607.28625.
7. Dai et al. (2024) P. Dai, Y. Zhang, T. Liu, Z. Fan, T. Du, Z. Su, X. Zheng, and Z. Li HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 874–884. External Links: Document
8. Damen et al. (2022) D. Damen, H. Doughty, G. M. Farinella, A. Furnari, E. Kazakos, J. Ma, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100. International Journal of Computer Vision 130 (1), pp. 33–55. External Links: Document
9. DeTone et al. (2026) D. DeTone, F. Bogo, E. Le, D. Frost, J. Straub, Y. Siddiqui, Y. Ye, J. Engel, R. Newcombe, and L. Ma NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Data. arXiv preprint arXiv:2603.18496. External Links: Link
10. Du et al. (2023) Y. Du, R. Kips, A. Pumarola, S. Starke, A. Thabet, and A. Sanakoyeu Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 481–490. External Links: Document
11. Fan et al. (2025) Z. Fan, P. Dai, Z. Su, X. Gao, Z. Lv, J. Zhang, T. Du, G. Wang, and Y. Zhang EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs. Proceedings of the AAAI Conference on Artificial Intelligence 39 (3), pp. 2879–2887. External Links: Document
12. Feng et al. (2024) H. Feng, W. Ma, Q. Gao, X. Zheng, N. Xue, and H. Xu Stratified avatar generation from sparse observations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
13. Grauman et al. (2022) K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, T. Do, M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, A. Gebreselasie, C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar, S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Southerland, Y. Sugano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Z. Zhao, Y. Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M. Farinella, C. Fuegen, B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo, K. Kitani, H. Li, R. Newcombe, A. Oliva, H. S. Park, J. M. Rehg, Y. Sato, J. Shi, M. Z. Shou, A. Torralba, L. Torresani, M. Yan, and J. Malik Ego4D: Around the World in 3,000 Hours of Egocentric Video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18973–18990. External Links: Document
14. Grauman et al. (2024) K. Grauman, A. Westbury, L. Torresani, K. Kitani, J. Malik, T. Afouras, K. Ashutosh, V. Baiyya, S. Bansal, B. Boote, E. Byrne, Z. Chavis, J. Chen, F. Cheng, F. Chu, S. Crane, A. Dasgupta, J. Dong, M. Escobar, C. Forigua, A. Gebreselasie, S. Haresh, J. Huang, M. M. Islam, S. Jain, R. Khirodkar, D. Kukreja, K. J. Liang, J. Liu, S. Majumder, Y. Mao, M. Martin, E. Mavroudi, T. Nagarajan, F. Ragusa, S. K. Ramakrishnan, L. Seminara, A. Somayazulu, Y. Song, S. Su, Z. Xue, E. Zhang, J. Zhang, A. Castillo, C. Chen, X. Fu, R. Furuta, C. González, P. Gupta, J. Hu, Y. Huang, Y. Huang, W. Khoo, A. Kumar, R. Kuo, S. Lakhavani, M. Liu, M. Luo, Z. Luo, B. Meredith, A. Miller, O. Oguntola, X. Pan, P. Peng, S. Pramanick, M. Ramazanova, F. Ryan, W. Shan, K. Somasundaram, C. Song, A. Southerland, M. Tateno, H. Wang, Y. Wang, T. Yagi, M. Yan, X. Yang, Z. Yu, S. C. Zha, C. Zhao, Z. Zhao, Z. Zhu, J. Zhuo, P. Arbeláez, G. Bertasius, D. Damen, J. Engel, G. Maria Farinella, A. Furnari, B. Ghanem, J. Hoffman, C. V. Jawahar, R. Newcombe, H. S. Park, J. M. Rehg, Y. Sato, M. Savva, J. Shi, M. Z. Shout, and M. Wray Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19383–19400. External Links: Document
15. Hartley et al. (2013) R. Hartley, J. Trumpf, Y. Dai, and H. Li Rotation averaging. International Journal of Computer Vision 103 (3), pp. 267–305. External Links: Document
16. Hoque et al. (2026) R. Hoque, P. Huang, D. J. Yoon, M. Sivapurapu, and J. Zhang EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video. In International Conference on Learning Representations (ICLR), External Links: Link
17. Huber (1964) P. J. Huber Robust estimation of a location parameter. The Annals of Mathematical Statistics 35 (1), pp. 73–101. External Links: Document
18. Jiang et al. (2024a) J. Jiang, P. Streli, X. Luo, C. Gebhardt, and C. Holz MANIKIN: Biomechanically Accurate Neural Inverse Kinematics for Human Motion Estimation. In European Conference on Computer Vision (ECCV), pp. 128–146. External Links: Document
19. Jiang et al. (2024b) J. Jiang, P. Streli, M. Meier, and C. Holz EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywhere. In European Conference on Computer Vision (ECCV), pp. 277–294. External Links: Document
20. Jiang et al. (2022) J. Jiang, P. Streli, H. Qiu, A. Fender, L. Laich, P. Snape, and C. Holz AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing. In European Conference on Computer Vision (ECCV), pp. 443–460. External Links: Document
21. Jung and Lee (2025) D. S. Jung and K. M. Lee Learning Dense Hand Contact Estimation from Imbalanced Data. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link
22. Kareer et al. (2025) S. Kareer, D. Patel, R. Punamiya, P. Mathur, S. Cheng, C. Wang, J. Hoffman, and D. Xu EgoMimic: Scaling Imitation Learning via Egocentric Video. In IEEE International Conference on Robotics and Automation (ICRA), pp. 13226–13233. External Links: Document
23. Kim et al. (2025) J. Kim, J. Kim, J. Na, and H. Joo ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1816–1828. External Links: Link
24. Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. In International Conference on Learning Representations,
25. Li et al. (2026a) Y. Li, X. Wei, J. Luo, Y. Xiao, Y. Bai, G. Zhou, T. Zou, C. Gui, J. Wen, H. Zhang, K. Chen, X. Pan, S. Liu, D. Wang, T. An, J. Li, S. Jin, W. Zhang, T. Wang, B. Wei, Z. Huang, F. Liu, R. Li, H. Zhang, A. Li, Y. Gong, P. Cao, J. Liang, and L. Lin EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks. arXiv preprint arXiv:2604.23570. External Links: Link
26. Li et al. (2026b) Z. Li, B. Yang, C. Miao, K. Zhu, H. Chen, Q. Guan, Z. Wu, W. Zhan, Y. Sun, Z. Huang, Z. Shan, Z. Jin, J. Hong, T. Wang, Y. Feng, Y. Liu, Y. Wang, Y. Yang, Z. Zhou, M. Luo, H. Cheng, B. Zhang, J. Li, J. Cai, G. Yao, J. Zhang, C. Lin, R. Xu, L. Yu, C. Shen, C. Shen, and Z. Li Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning. arXiv preprint arXiv:2607.14183. External Links: Link
27. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations,
28. Ma et al. (2024) L. Ma, Y. Ye, F. Hong, V. Guzov, Y. Jiang, R. Postyeni, L. Pesqueira, A. Gamino, V. Baiyya, H. J. Kim, K. Bailey, D. S. Fosas, C. K. Liu, Z. Liu, J. Engel, R. D. Nardi, and R. Newcombe Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild. In European Conference on Computer Vision (ECCV), pp. 445–465. External Links: Document
29. Mahmood et al. (2019) N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black AMASS: Archive of Motion Capture As Surface Shapes. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5441–5450. External Links: Document
30. NVIDIA (2026) NVIDIA Cosmos3 Generator Action. Note: Official model and training documentation External Links: Link
31. Pavlakos et al. (2019) G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black Expressive Body Capture: 3D Hands, Face, and Body From a Single Image. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10967–10977. External Links: Document
32. Peng et al. (2026) J. Peng, X. Yu, D. Feng, Y. Yang, W. Cai, J. Xiong, G. Yang, J. Zheng, J. Cao, X. Wei, J. Pang, Y. Shen, and T. Wang Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation. arXiv preprint arXiv:2607.05377. External Links: Link
33. Ponton et al. (2023) J. L. Ponton, H. Yun, A. Aristidou, C. Andujar, and N. Pelechano SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data. ACM Transactions on Graphics 43 (1), pp. 1–14. External Links: Document
34. Punamiya et al. (2026) R. Punamiya, S. Kareer, Z. Liu, J. Citron, R. Qiu, X. Cai, A. Gavryushin, J. Chen, D. Liconti, L. Y. Zhu, P. Aphiwetsa, B. Li, A. Cheluva, P. Kuppili, Y. Liu, D. Patel, A. Gao, H. Chung, R. Co, R. Zbizika, J. Liu, X. Xu, H. Xiong, G. Chen, S. Oliani, W. Xuan, C. Yang, X. Wang, J. Fort, R. Newcombe, J. Gao, J. Chong, G. Matsuda, A. Doriwala, M. Pollefeys, R. Katzschmann, X. Wang, S. Song, J. Hoffman, and D. Xu EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World. arXiv preprint arXiv:2604.07607. External Links: Link
35. Qi et al. (2017) C. R. Qi, L. Yi, H. Su, and L. J. Guibas PointNet++: deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 30.
36. Qiu et al. (2025) R. Qiu, S. Yang, X. Cheng, C. Chawla, J. Li, T. He, G. Yan, D. J. Yoon, R. Hoque, L. Paulsen, G. Yang, J. Zhang, S. Yi, G. Shi, and X. Wang Humanoid Policy \sim Human Policy. In Proceedings of the 9th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 305, pp. 2888–2906. External Links: Link
37. Romero et al. (2017) J. Romero, D. Tzionas, and M. J. Black Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics 36 (6), pp. 1–17. External Links: Document
38. Ropedia (2026) Ropedia Xperience-10M: A Large-Scale Egocentric Multimodal Dataset with Structured 3D/4D Annotations. Note: Hugging Face dataset External Links: Link
39. Sakoe and Chiba (1978) H. Sakoe and S. Chiba Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing 26 (1), pp. 43–49. External Links: Document
40. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
41. Shen et al. (2024) Z. Shen, H. Pi, Y. Xia, Z. Cen, S. Peng, Z. Hu, H. Bao, R. Hu, and X. Zhou World-Grounded Human Motion Recovery via Gravity-View Coordinates. In ACM SIGGRAPH Asia Conference Papers, pp. 1–11. External Links: Document
42. Shi et al. (2026) M. Shi, S. Peng, J. Chen, H. Jiang, T. Li, D. Huang, P. Luo, H. Li, and L. Chen EgoHumanoid: Unlocking In-the-Wild Loco-Manipulation with Robot-Free Egocentric Demonstration. arXiv preprint arXiv:2602.10106. External Links: Link
43. Shoemake (1985) K. Shoemake Animating rotation with quaternion curves. In Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 245–254. External Links: Document
44. Song et al. (2025) Y. R. Song, J. Li, R. Fu, D. Murphy, K. Zhou, R. Shiv, Y. Li, H. Xiong, C. E. Owens, Y. Du, Y. Luo, X. Cheng, A. Torralba, W. Matusik, and P. P. Liang OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction. arXiv preprint arXiv:2512.16842. External Links: Link
45. Taheri et al. (2020) O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas GRAB: A Dataset of Whole-Body Human Grasping of Objects. In European Conference on Computer Vision (ECCV), pp. 581–600. External Links: Document
46. Todorov et al. (2012) E. Todorov, T. Erez, and Y. Tassa MuJoCo: a physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5026–5033.
47. Umeyama (1991) S. Umeyama Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 13 (4), pp. 376–380. External Links: Document
48. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 30.
49. Xu et al. (2026) M. Xu, B. Duan, Y. Gu, Z. Shen, R. Xu, and Y. Yue HandFlow: Fully Generative 4D Hand Recovery with Flow Matching. arXiv preprint arXiv:2607.11221. External Links: Link
50. Yang et al. (2026a) H. Yang, J. Bao, Y. Xin, H. Song, Y. Tian, B. Zhao, D. Wang, and X. Li ZeroWBC: Learning Natural Whole-Body Humanoid Interaction from Human Egocentric Data. arXiv preprint arXiv:2603.09170. External Links: Link
51. Yang et al. (2026b) L. Yang, X. Huang, Z. Wu, A. Kanazawa, P. Abbeel, C. Sferrazza, C. K. Liu, R. Duan, and G. Shi OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction. In IEEE International Conference on Robotics and Automation (ICRA), External Links: Link
52. Yu et al. (2026) R. Yu, X. Lin, J. Ma, Y. Wang, K. Luo, J. Ji, H. Wang, W. Wang, R. Zhang, P. Tan, T. Wu, R. Dai, Q. Chen, and L. Han OmniContact: Chaining Meta-Skills via Contact Flow for Generalizable Humanoid Loco-Manipulation. arXiv preprint arXiv:2606.26201. External Links: Link
53. Zhang et al. (2025) J. Zhang, J. Deng, C. Ma, and R. A. Potamias HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1805–1815. External Links: Document
54. Zhang et al. (2022) S. Zhang, Q. Ma, Y. Zhang, Z. Qian, T. Kwon, M. Pollefeys, F. Bogo, and S. Tang EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices. In European Conference on Computer Vision (ECCV), pp. 180–200. External Links: Document
55. Zhang et al. (2021) Y. Zhang, Z. Li, L. An, M. Li, T. Yu, and Y. Liu Lightweight multi-person total motion capture using sparse multi-view cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5540–5549. External Links: Document, Link
56. Zhao et al. (2025) Y. Zhao, T. Kwon, P. Streli, M. Pollefeys, and C. Holz EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 27727–27738. External Links: Link
57. Zheng et al. (2023) X. Zheng, Z. Su, C. Wen, Z. Xue, and X. Jin Realistic full-body tracking from sparse observations via joint-level modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),
58. Zhong et al. (2025) R. Zhong, Y. Sun, J. Wen, J. Li, C. Cheng, W. Dai, Z. Zeng, H. Lu, Y. Zhu, and Y. Xu HumanoidExo: Scalable Whole-Body Humanoid Manipulation via Wearable Exoskeleton. arXiv preprint arXiv:2510.03022. External Links: Link
59. Zhou et al. (2026) J. Zhou, Z. Gao, F. Hong, Z. Liu, G. Zhang, W. Dai, R. Zhen, C. Lyu, H. Wu, Y. Mao, X. Wang, Y. Jiang, W. Ding, and S. Yang TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video. arXiv preprint arXiv:2605.13083. External Links: Link
60. Zhou et al. (2019) Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5738–5746.
61. Zhu et al. (2026) L. Y. Zhu, P. Kuppili, R. Punamiya, P. Aphiwetsa, D. Patel, S. Kareer, S. Ha, and D. Xu EMMA: Scaling Mobile Manipulation via Egocentric Human Data. IEEE Robotics and Automation Letters 11 (3), pp. 3087–3094. External Links: Document

Appendix Contents

AAppendix A Capture Hardware . A

BAppendix B Data Format . B

CAppendix C Whole-Body Reconstruction from Sparse Measurement . C

C.1C.1 Protocol-guided calibration . C.1

C.2C.2 Prior-guided motion completion . C.2

C.3C.3 Constraint layer 1: skeletal motion . C.3

C.4C.4 Constraint layer 2: surface clearance . C.4

C.5C.5 Constraint layer 3: body-ground consistency . C.5

C.6C.6 Joint refinement . C.6

DAppendix D Multimodal Integration and Annotation . D

D.1D.1 Hand and touch fusion . D.1

D.2D.2 Expert-provided procedural context . D.2

D.3D.3 Example text prompt . D.3

D.4D.4 Human verification and experimental interpretation . D.4

EAppendix E Data Statistics . E

FAppendix F Experiments . F

F.1F.1 Evaluation setup . F.1

F.2F.2 Whole-body reconstruction: inputs and references . F.2

F.3F.3 Hand motion and contact capture . F.3

F.4F.4 Evaluation metrics . F.4

F.5F.5 Expert review of text annotations . F.5

F.6F.6 Preliminary training study . F.6

F.7F.7 Calibration and layered reconstruction ablations . F.7

F.8F.8 Professional-skill ablation . F.8

F.9F.9 Humanoid retargeting . F.9

F.10F.10 Qualitative examples . F.10

GAppendix G Discussion on Capture Trade-offs . G

Appendix A Capture Hardware

Extending Sec. 3.1, the capture system comprises a head-mounted camera, five wearable body trackers, and articulation gloves; the Wuji configuration additionally provides touch sensing. As in Sec. 3.1, custom 3D-printed mounts rigidly couple the head tracker to the camera and attach the remaining trackers to the glove dorsum and the lateral ankles, with straps for added fixation (). The trackers directly measure device poses, while anatomical anchors are obtained as joint frames through calibrated device-to-joint transformations. Tab. 8 summarizes the acquisition settings and manufacturer-specified characteristics of the articulation gloves. Each sensing stream preserves its native timestamps for temporal alignment, and subsequent resampling only interpolates existing measurements without introducing additional observations. The calibration parameters are reused as long as the corresponding mounts remain fixed and are recomputed after remounting.

The camera operates at a resolution of 1920\times 970 pixels, with intrinsic matrix:

K=\begin{bmatrix}1012.0&0&960.0\\
0&1016.0&430.0\\
0&0&1\end{bmatrix},

where the focal lengths and principal-point coordinates are expressed in pixels. For body–hand fusion, glove joint positions are transformed into the wrist-local coordinate frame. The tactile sensing matrix follows row-major indexing and is mapped to 17 anatomical regions: two regions on the thumb, three on each remaining finger, and three on the palm. The contact extraction procedure used for quantitative evaluation is specified in Sec. F.3, while the corresponding export interface is described in Sec. D.1. illustrates the five body trackers, the rigid head mount coupling the camera to the head tracker, and the two interchangeable glove configurations.

Appendix B Data Format

The release is organized according to the LeRobotDataset v3.0 specification (Cadene et al., 2026), with all streams aligned to a common 30 Hz temporal grid. Each row corresponds to one episode timestamp, and associates the corresponding visual observation, motion, tactile, and available annotation streams. The canonical fields episode_index, frame_index, index, timestamp, and task_index denote the episode, local frame, global row, episode-relative time, and task, respectively. Timestamps are episode-relative and reset to zero at the beginning of each episode. Frame-level numerical data and textual annotations are stored in Parquet, whereas egocentric frames are encoded as MP4 and retrieved via episode metadata and temporal offsets. A shard may contain multiple episodes. The release layout is as follows:

meta/info.json; meta/stats.json; meta/tasks.parquet meta/episodes/chunk-000/file-000.parquet data/chunk-000/file-000.parquet videos/observation.images.ego/chunk-000/file-000.mp4

meta/info.json specifies fps=30, feature shapes and dtypes, and shard paths; meta/stats.json stores feature statistics; and meta/tasks.parquet provides a mapping from task indices to task descriptions. Episode metadata specify row ranges and video offsets, together with anonymized session, split, and calibration-profile identifiers. Tab. 9 lists the per-row fields.

Step annotations will accompany every row in their annotated interval through additional annotation.* fields; their original boundaries will be retained in seconds. Episode-level task descriptions and step-level text remain distinct. Fixed shape, scale, and mount parameters \Phi will be referenced by calibration-profile ID instead of repeated in every row. Measurements, reconstructions, and reference labels will retain separate provenance; unavailable streams will be marked by validity masks rather than encoded as measured zeros.

Body and tracker positions use a metric, right-handed, +Y-up world frame. Tracker quaternions use (x,y,z,w) order and active device-to-world rotations on column vectors. SMPL-X rotations are local axis-angle vectors in radians; the root orientation is world-relative. The release translation field stores the world pelvis position u_{t} defined in Sec. 3.2.2, rather than the SMPL-X library’s additive translation parameter. The conversion is specified in Sec. C.2.

Temporal alignment and validity. Native host timestamps are retained and converted to episode-relative seconds using their recorded time origins. The 30 Hz grid uses linear interpolation for positions, spherical linear interpolation (SLERP) (Shoemake, 1985) for rotations, and nearest recorded timestamps for RGB. Interpolated or held values retain source-time and validity information; they are not new sensor measurements. For contact, each output interval retains any valid native-rate detection within that interval; a bin without valid samples remains unavailable (Sec. D.1). Original step boundaries are stored in seconds, with per-row annotations assigned by interval membership.

Appendix C Whole-Body Reconstruction from Sparse Measurement

This section supplements \mathrm{C}^{3} in Sec. 3.2, following the same calibration, completion, and constraint stages. We retain its profile \Phi, calibration variables \Theta, motion state x_{t}, and losses in Eqs. 1, 2 and 3. All distances inside the optimization are in meters. Below, \operatorname{MS} denotes the mean of squared scalar entries, [\,\cdot\,]_{+}=\max(\cdot,0), and \|\cdot\|_{2} and \|\cdot\|_{F} are Euclidean and Frobenius norms.

C.1 Protocol-guided calibration

Reference protocol. Fig. 8 shows the 17 ordered stages used by Sec. 3.2.1, named as in the figure: (A) body poses and articulation, (01) neutral stance, (02) head rotations, (03) A-pose, (04) T-pose, (05) forward reach, (06) elbow flexion, (07) wrist rotation, (08) shallow squat, (09) left leg raise, (10) right leg raise, and (11) ankle motion; (B) hand articulation, (12) spread/close, (13) fist/release, (14) pinch/oppose, and (15) finger flexion; and (C) root motion, (16) left/right 180^{\circ} turns and (17) two steps forward, two backward. Stages (01)–(11) cover the body actions listed in the main text, where neutral stance corresponds to standing and forward reach to forward arm extension. A VPoser decoder (Pavlakos et al., 2019) fits body landmark targets, followed by local elbow, wrist, head, and ankle corrections. Finger targets use the MANO-compatible hand model (Romero et al., 2017). Turning and stepping add prescribed root motion and alternating support. The cached library provides the reference joint positions J_{n,i}(\beta) and rotations G_{n,i} already defined in Eq. 1; joint positions are reevaluated at the participant’s shape during fitting.

Trajectory-based alignment and sample selection. Following Sec. 3.2.1, each sample n comes from recording r(n) and is paired with reference action k_{n} at phase \alpha_{n}. The ordered protocol identifies the action, while motion patterns and reviewed boundaries locate its interval in the recording. Within that action, we match scale-normalized pairwise tracker distances and relative orientation changes using monotonic dynamic time warping (DTW) (Sakoe and Chiba, 1978). Reviewed events in a reference recording link characteristic poses to library phases; the matching path transfers these correspondences to recording r(n) while accommodating different speeds and pauses. Thus \alpha_{n} is the matched reference phase, rather than a fixed fraction of elapsed recording time. The resulting observed poses p_{n,i},R_{n,i} are paired with the reference joint poses for (k_{n},\alpha_{n}) used in Eq. 1.

For static holds, we aggregate stable observations after removing transitions; for dynamic actions, we aggregate valid observations within \pm 0.11 s of each matched instant. Positions use coordinatewise medians and orientations use rotation averaging (Hartley et al., 2013). Invalid or geometrically implausible tracker streams are masked by m_{n,i}; holds and dynamic samples receive weights w_{n}=1 and 0.1, respectively. Finger, turning, and stepping stages are retained for motion checks but excluded from the body-profile fit. Matched phases, timestamps, and validity are recorded with each sample.

Step 1: jointly fit body geometry and attachment positions. With the action correspondences fixed, we solve Eq. 2, whose three terms are expanded in Eqs. 4, 5 and 6, for the main-text variables \Theta: body shape and scale, the five attachment translations a_{i}, and each recording’s alignment (Q_{r},u_{r}). Each a_{i} is one anchor-local vector shared across all matched poses; reference articulation remains fixed. At each iteration, we reevaluate the SMPL-X joints at the current shape, predict tracker positions using Eq. 1, and backpropagate the calibration loss through these operations. Adam then updates all components of \Theta jointly. The attachment rotations A_{i} are computed separately in Step 2.

We initialize shape and scale as in Sec. 3.2.1, set a_{i}=0, and initialize each Q_{r} from the horizontal left–right wrist direction in the T-pose. The initial u_{r} is the coordinatewise median of observed-minus-predicted tracker positions for that recording. We optimize \log s to keep scale positive. Only the first five shape coefficients are updated; the remaining five retain the visual estimate.

Robust tracker-position loss (\mathcal{L}_{\mathrm{robust\mbox{-}pos}}). A calibrated body and its fixed attachments should explain the measured tracker locations across the matched actions. Tracking spikes and imperfect execution of the reference poses can nevertheless produce large residuals. We use a Huber penalty (Huber, 1964) with threshold \tau=0.025 m: small errors receive a quadratic penalty for precise fitting, while large errors receive a linear penalty so that a few poor observations do not dominate the calibration. For squats, the pelvis also moves while the feet remain planted. A shared translation \delta_{n}, computed as the mean observed-minus-predicted position of valid ankle trackers, accounts for this displacement without changing the mounts. It is recomputed each iteration and is zero for other actions or when no ankle is valid. With e_{n,i}=\widehat{p}_{n,i}+\delta_{n}-p_{n,i} and predictions from Eq. 1, the weighted position loss is:

\mathcal{L}_{\mathrm{robust\mbox{-}pos}}=\frac{\sum_{n,i,c}w_{n}m_{n,i}\rho_{0.025}(e_{n,i,c})}{3\max(1,\sum_{n,i}w_{n}m_{n,i})}.

Here c indexes Cartesian coordinates, and the denominator averages over valid weighted observations and their three coordinates. Distances are in meters; \rho_{\tau}(e)=e^{2}/2 for |e|\leq\tau and \tau(|e|-\tau/2) otherwise. Invalid entries are removed before arithmetic.

Measured body-size loss (\mathcal{L}_{\mathrm{anth}}). Tracker agreement alone cannot reliably distinguish a longer limb from a larger mounting offset. We therefore require the fitted body to agree with the participant’s measured height and arm span. Using the main-text model height H(\beta), the unscaled T-pose arm span W(\beta), and measured values h^{\star},w^{\star}, this loss is:

\mathcal{L}_{\mathrm{anth}}=(sH(\beta)-h^{\star})^{2}+(sW(\beta)-w^{\star})^{2}.

Both measurements are in meters. The weights in Eq. 2 are \lambda_{\mathrm{obs}}=420 and \lambda_{\mathrm{anth}}=1800.

Geometry regularization (\mathcal{R}_{\mathrm{geom}}). Even with measured body size, several body–mount configurations may explain the sparse observations. We favor the visually estimated proportions, plausible attachment locations, and small scale and yaw corrections. Using the mean-square operator \operatorname{MS} defined above, the implemented regularizer is:

\displaystyle\mathcal{R}_{\mathrm{geom}}={} \\ \displaystyle 0.012\operatorname{MS}(\beta)+0.18\operatorname{MS}(\beta_{1:5}-\beta^{\mathrm{vis}}_{1:5}) \\ \displaystyle+\frac{280}{5}\sum_{i=1}^{5}[\|a_{i}\|_{2}-\ell_{i}]_{+}^{2} \\ \displaystyle+3\!\left[(\|a_{2}\|_{2}-\|a_{3}\|_{2})^{2}+(\|a_{4}\|_{2}-\|a_{5}\|_{2})^{2}\right] \\ \displaystyle+12\operatorname{MS}\!\left(\{G_{\mathrm{neutral},i}a_{i}-d_{i}\}_{i\in\{4,5\}}\right) \\ \displaystyle+0.18(\log s-\log s_{0})^{2}+0.015\operatorname{MS}(\{\psi_{r}\}_{r}).

Here \ell=(0.18,0.15,0.15,0.18,0.18) m gives soft mount-length limits in head, left/right wrist, left/right ankle order. The bilateral terms compare mount lengths, allowing mirrored local axes. G_{\mathrm{neutral},i} is anchor i’s reference orientation in the neutral-stance stage (01); it expresses the ankle offset in neutral body axes before comparison with d_{4}=(0.075,0.070,0) and d_{5}=(-0.075,0.070,0) m. The corresponding mean square averages all six ankle-offset coordinates. Finally, \psi_{r} is the yaw angle of Q_{r}. These soft penalties limit body–mount compensation while allowing the tracker observations and measured body dimensions to determine the fit.

Adam (Kingma and Ba, 2015) runs for 1,100 iterations, with learning rates 0.008 for log-scale/yaw, 0.018 for shape/recording translation, and 0.012 for attachment translation. After each update, we clamp scale to [0.72,1.15], optimized shape coefficients to [-3,3], and yaw to [-\pi,\pi]. This returns \Theta^{\mathrm{fit}}, including the shared a_{i}; no per-frame attachment is fitted.

Step 2: recover attachment rotations by robust averaging. With the fitted Q_{r}^{\mathrm{fit}} fixed, the orientation relation in Eq. 1 gives one candidate device-to-anchor rotation \widetilde{A}_{n,i} per valid observation. Let \mathcal{I}_{i} contain the retained samples for tracker i: static holds for wrists, and both static and selected dynamic samples for the head and ankles. Compute the mean candidate matrix M_{i}, take its singular value decomposition (SVD), and project it to a proper rotation:

\displaystyle\widetilde{A}_{n,i} \\ \displaystyle=(Q_{r(n)}^{\mathrm{fit}}G_{n,i})^{\top}R_{n,i}, \\ \displaystyle M_{i} \\ \displaystyle=\frac{1}{|\mathcal{I}_{i}|}\sum_{n\in\mathcal{I}_{i}}\widetilde{A}_{n,i}=U_{i}\Sigma_{i}V_{i}^{\top}, \\ \displaystyle A_{i} \\ \displaystyle=U_{i}\operatorname{diag}\!\left(1,1,\det(U_{i}V_{i}^{\top})\right)V_{i}^{\top}.

The determinant correction excludes reflections. For a fixed retained set, this gives the rotation minimizing the summed squared Frobenius distances to its candidates; since A_{i} is the right factor of \widehat{R}_{n,i} in Eq. 1 and orthogonal factors preserve the Frobenius norm, it equivalently minimizes \sum_{n\in\mathcal{I}_{i}}\|\widehat{R}_{n,i}-R_{n,i}\|_{F}^{2}, i.e., it matches the predicted orientations \widehat{R}_{n,i} to the measured R_{n,i} as in Sec. 3.2.1. It requires no gradient optimization.

We make the estimate robust with up to three rejection rounds. After each average, compute each candidate’s angular residual as the rotation angle of A_{i}^{\top}\widetilde{A}_{n,i}. Reject residuals above the larger of 12^{\circ} and the median plus 2.5 times the median absolute deviation (floored at 1^{\circ}), then recompute the average. Stop if the set is unchanged or a rejection would leave fewer than four samples. An empty initial set uses an identity fallback marked as low confidence. The resulting five A_{i}, together with the fitted shape, scale, and a_{i}, complete the reusable profile \Phi. Eq. 7 with this rejection rule is the explicit implementation of f_{\mathrm{rot}} in Sec. 3.2.1.

C.2 Prior-guided motion completion

Shared state and coordinate conventions. We use x_{t}=(\omega_{t},\theta_{t},u_{t}) and the calibrated anchor frames \widetilde{p}_{t,i},\widetilde{R}_{t,i} from Sec. 3.2.2. Here u_{t} is the world pelvis position. The SMPL-X library’s additive translation subtracts the scaled zero-translation pelvis location from u_{t}; conversion back to u_{t} is applied on export. This is an implementation coordinate conversion, not an additional motion variable. The five anchors are SMPL-X joints (15,20,21,7,8) in head, left/right wrist, left/right ankle order. For task-time fitting, Eq. 1 uses the rotation represented by \omega_{t} in place of Q_{r}, joint geometry evaluated from \theta_{t} in place of the reference pose, and u_{t} in place of u_{r}. No second yaw is applied, and metric attachment offsets a_{i} are never scaled again.

Input representation and model. The feature vector \mathbf{z}_{t} from the main text has 90 entries: for each anchor, six rotation coordinates, six relative-rotation coordinates, three positions, and three position increments. The six-dimensional rotation representation uses the first two matrix columns (Zhou et al., 2019); relative rotations are previous-rotation transpose times current rotation. Channels are packed by feature type for head/wrists (54), then ankles (36). Initial rotation increments are identity and position increments zero; increments are not divided by time.

The AvatarPoser-based (Jiang et al., 2022) transformer (Vaswani et al., 2017) prior f_{\mathrm{prior}} has width 256, three encoder layers, and eight heads. The last token of an L=40 frame trailing window predicts 22 rotations (root and 21 body joints) plus two auxiliary foot-contact logits. Initial windows are shorter. Rotation outputs are orthonormalized, interpolated back to tracker timestamps, and converted to capture coordinates. The head-alignment expression for u_{t}^{0} in Sec. 3.2.2 then gives \mathcal{X}^{0}; it is not a second fitting stage.

Training data and optimization. Compatible public AvatarPoser weights initialize the network; the added 36 ankle input channels are zero-initialized. The pool combines 2,700 AMASS-derived sequences (Mahmood et al., 2019) from HDM05, CMU, and BMLrub, VPoser synthetic motion, 20 sequences from the ten B01–B10 guides at two speeds/durations, and teacher-fitted real recordings, including repetitions 1–2 of the full-body evaluation actions (Sec. F.1). Synthetic tracker inputs use participant shape, scale, and mounts. VPoser latent random walks last 240 frames at 60 Hz, with 2 mm Gaussian position noise and 1.5% simulated dropout. Wrist-twist augmentation duplicates 30% of synthetic sequences with a \pm 25^{\circ} twist in one wrist observation and its target.

Teacher fits use 300 Adam iterations per chunk, learning rate 0.01, and (\lambda_{p},\lambda_{R},\lambda_{\mathrm{prior}})=(10000,60,0.5), retaining the other active constraints. They provide task-domain supervision rather than independent ground truth. Each real recording uses its own calibration and up to 2,000 uniformly spaced 40-frame windows. Training minimizes mean SmoothL1 loss (unit threshold) on 132 rotation coordinates plus 0.25 times mean binary cross-entropy on the two foot logits. Natural-sequence foot targets are zero; this auxiliary head is not used as support evidence in refinement. AdamW (Loshchilov and Hutter, 2019) runs for three epochs, with learning rate and weight decay 10^{-4}, batch size 32, and gradient-norm clipping at 1. The 60 Hz network timeline is internal; the release remains 30 Hz. Evaluation repetition 3 is held out as specified in Sec. F.1. Video and glove signals are not network inputs.

C.3 Constraint layer 1: skeletal motion

The weighted sum \mathcal{L}_{\mathrm{skel}} is defined in Secs. 3.2.3 and 3; we expand its five component losses here. Within one of the temporal chunks of Sec. 3.2.3, with B frames, t=1,\ldots,B is a local frame index and i=1,\ldots,5. Let m_{t,i}\in\{0,1\} indicate a valid tracker observation, and N_{\mathrm{obs}}=\max(1,\sum_{t=1}^{B}\sum_{i=1}^{5}m_{t,i}). The implemented measurement losses are coordinate-normalized squared errors:

\mathcal{L}_{p}=\frac{1}{3N_{\mathrm{obs}}}\sum_{t,i}m_{t,i}\|\widehat{p}_{t,i}-p_{t,i}\|_{2}^{2},\qquad\mathcal{L}_{R}=\frac{1}{9N_{\mathrm{obs}}}\sum_{t,i}m_{t,i}\|\widehat{R}_{t,i}-R_{t,i}\|_{F}^{2}.

Predictions come from Eq. 1 and are compared with raw device observations p_{t,i},R_{t,i}, as specified in the main text. The inferred anchor frames in Sec. 3.2.2 are used for completion. Dividing by 3 and 9 averages scalar components; it does not remove the difference in physical units. The rotational objective is chordal matrix distance, not the geodesic angle used by diagnostic reporting. For the chunk of \mathcal{X}^{0}, the regularizers are:

\displaystyle\mathcal{L}_{\mathrm{prior}} \\ \displaystyle=\operatorname{MS}(\theta-\theta^{0})+0.2\operatorname{MS}(\omega-\omega^{0})+0.2\operatorname{MS}(u-u^{0}), \\ \displaystyle\mathcal{L}_{\mathrm{time}} \\ \displaystyle=\operatorname{MS}(\{\theta_{t}-\theta_{t-1}\}_{t=2}^{B}).

Here \theta,\omega,u stack the chunk’s main-text states and superscript 0 denotes the proposal. With fixed shape, the library’s additive-translation offset cancels in u-u^{0}. The temporal term is zero for B=1. Rotation regularizers use axis-angle differences; none of these differences is normalized by elapsed time. The active configuration neither enforces temporal coupling across chunk boundaries nor uses overlapping chunks.

Let \kappa_{t,l} and \epsilon_{t,l} be geometric knee and elbow flexion angles, with zero at straight extension, and \eta_{t} the pelvis-to-neck angle from the upward axis. The active soft-angle loss is:

\begin{split}\mathcal{L}_{\mathrm{joint}}={}&\frac{1}{2B}\sum_{t,l}[\kappa_{t,l}-125\pi/180]_{+}^{2}+\frac{1}{2B}\sum_{t,l}[\epsilon_{t,l}-140\pi/180]_{+}^{2}\\
&+\frac{1}{B}\sum_{t}[\eta_{t}-15\pi/180]_{+}^{2}.\end{split}

Here l indexes the two sides. The torso threshold is an engineering bias, not an anatomical limit; genuine deep bends may be penalized. This term belongs to \mathcal{L}_{\mathrm{skel}} in Eq. 3, even though its angles are evaluated from posed joint geometry.

C.4 Constraint layer 2: surface clearance

We evaluate the posed SMPL-X surface already used in the main text’s surface and ground layers. Let v_{t,j}\in\mathbb{R}^{3} denote surface vertex j in the tracking world frame, obtained from x_{t} with fixed (\beta,s). Vertices are model outputs, not additional measurements or optimization variables. The six sampled parts are the torso, head, left arm, right arm, left leg, and right leg. Their 30 ordered distinct-part pairs form \mathcal{P}. Skinning weights assign a vertex to its strongest part when that part’s weight exceeds 0.8; two boundary-erosion steps exclude transition regions, and farthest-point sampling (Qi et al., 2017) selects 256 interior vertices per part on the fixed neutral mesh. Let \mathcal{S}_{b} contain the sampled vertex indices of part b; these are the “points sampled from distinct body parts” in Sec. 3.2.3. For j\in\mathcal{S}_{b}, let j^{\prime} be its nearest sampled vertex on target part b^{\prime}, and let \mathbf{N}_{t,j^{\prime}} be that vertex’s area-weighted unit surface normal. The signed clearance d_{t,j}^{b,b^{\prime}} is positive outside the target surface and negative once it is crossed; the 0.10 m gate restricts the penalty to nearby points, and TopK aggregates the largest violations per part pair. The adopted oriented penalty is:

\begin{split}d_{t,j}^{b,b^{\prime}}&=\langle v_{t,j}-v_{t,j^{\prime}},\mathbf{N}_{t,j^{\prime}}\rangle,\\
q_{t,j}^{b,b^{\prime}}&=\mathbf{1}\{\|v_{t,j}-v_{t,j^{\prime}}\|_{2}<0.10\}[0.018-d_{t,j}^{b,b^{\prime}}]_{+}^{2},\\
\mathcal{L}_{\mathrm{surf}}&=\frac{1}{B|\mathcal{P}|K}\sum_{t=1}^{B}\sum_{(b,b^{\prime})\in\mathcal{P}}\sum_{j\in\operatorname{TopK}(q_{t}^{b,b^{\prime}})}q_{t,j}^{b,b^{\prime}},\qquad K=8.\end{split}

Distances are in meters; TopK selects the eight largest penalties, including zeros when fewer than eight are active. Nearest indices, normals, and the 0.10 m support gate are recomputed each iteration without gradients; gradients flow through both selected vertex positions. The 0.018 m clearance discourages local penetration, but the gate can omit deep intersections and this local proxy is not an exact signed distance field or a collision-free guarantee.

C.5 Constraint layer 3: body-ground consistency

The estimated ground plane of Sec. 3.2.3 is the horizontal plane at height g in the +Y-up world frame. Summing over all posed body vertices v_{t,j} (Sec. C.4), the floor term is:

\mathcal{L}_{\mathrm{floor}}=\frac{1}{B}\sum_{t=1}^{B}\sum_{j}[g-v_{t,j}^{(y)}]_{+}^{2}.

The sum over vertices is intentionally not averaged, and the superscript (y) selects the vertical coordinate. Unless overridden, the launcher sets g to the fifth percentile of valid ankle-tracker heights minus 0.12 m; this heuristic height estimate is used instead of fitting a plane to scene geometry.

C.6 Joint refinement

The solver minimizes Eq. 3 using the component losses in Eqs. 8, 9, 10, 11, 12 and 13, with the calibrated profile fixed. In \mathcal{L}_{\mathrm{skel}}=\sum_{k}\lambda_{k}\mathcal{L}_{k}, the weights are (\lambda_{p},\lambda_{R},\lambda_{\mathrm{prior}},\lambda_{\mathrm{time}},\lambda_{\mathrm{joint}})=(3000,20,2,5,100); in Eq. 3, \lambda_{\mathrm{surf}}=3000 and \lambda_{\mathrm{floor}}=50. Contact, capsule, torso-hull, sole-only floor, and sole-flatness weights are zero. The batch wrapper uses Adam (Kingma and Ba, 2015) for 60 iterations per chunk with gradient-norm clipping at 2. Its initial learning rate/chunk size is (0.004,256); bounded fallbacks are (0.002,256), (0.001,256), (0.0005,256), (0.001,128), and (0.0005,128). Each chunk starts from its states in \mathcal{X}^{0}. Every Adam iteration evaluates all three layers and updates the state x_{t} through the library translation conversion in Sec. C.2; there are no separate skeletal, surface, or ground optimization passes. Concatenating the non-overlapping fitted chunks gives \mathcal{X}^{\star}. The argmin in Eq. 3 specifies the objective; the implementation returns its finite-iteration numerical estimate, without a global-optimum guarantee. Failed chunks use the bounded retry settings above.

Appendix D Multimodal Integration and Annotation

D.1 Hand and touch fusion

Hand alignment and fitting. As in Sec. 3.3, finger articulation is fitted in wrist-local coordinates after body reconstruction. Glove positions are translated to the measured wrist origin and rotated into its frame; wrist-local Wuji inputs enter directly. Both layouts retain four landmarks per finger, including the tip (20 non-wrist landmarks); the 25-node layout discards four extra palm/metacarpal nodes. A sequence-fixed similarity aligns these points to the calibrated model’s flat hand, using the median of up to 12 frames with the straightest fingers. Scale is the ratio of model/glove RMS radii about the wrist; rotation uses SVD with reflection correction (Umeyama, 1991). This hand scale is separate from body scale s.

The MANO-compatible SMPL-X hand (Romero et al., 2017; Pavlakos et al., 2019) optimizes its 45 local axis-angle coordinates per side. The fitting objective is:

\mathcal{L}_{\mathrm{hand}}=1000\mathcal{L}_{\mathrm{points}}+0.015\mathcal{L}_{\mathrm{directions}}+0.003\mathcal{L}_{\mathrm{flat}}+0.004\mathcal{L}_{\mathrm{continuity}}+0.02\mathcal{L}_{\mathrm{angle}}.

The point term averages squared coordinate errors over the 20 wrist-local landmarks in meters. The direction term averages one minus cosine similarity along the 20 finger bones, with vector norms floored at 10^{-8}. The flat-hand term is mean squared pose deviation from the model’s flat-hand pose. Continuity averages squared adjacent-frame pose differences and adds twice the mean squared difference from the preceding window’s final pose; the latter is omitted for the first window. The angle term averages squared excess of each of the 15 joint-rotation magnitudes above 2.7 radians.

Left/right objectives Eq. 14 are summed. At most 480 frames are sampled over the hands’ overlapping time range and fitted in 90-frame windows with 60 Adam iterations and learning rate 0.04. After accounting for host-time origins, hand rotations are SLERPed onto the body timeline; endpoint holds outside the fitted span retain coverage flags. Fusion replaces hand-pose arrays while preserving the fitted body, root, shape, scale, and attachments.

Tactile processing. Wearable tactile response weakens with use and depends on wearing tension and channel health. Raw amplitude therefore cannot by itself distinguish light contact from reduced sensitivity or missing data. The three interfaces in Tab. 10 retain native values (L0), baseline-relative changes (L1), and 17-region contact with separate health/validity (L2). Unresponsive taxels are excluded from regional support; insufficient support makes a region unavailable, not negative. Baselines, response normalization, and region mappings accompany each recording. These are contact indicators, not calibrated force or pressure measurements.

Contact is temporally associated with motion, not used to constrain body IK. To preserve brief touches on the 30,Hz grid, each region’s output is the logical OR of valid native-rate detections within the 1/30,s bin centered on the output timestamp. Bins without valid samples are masked. This release rule differs from the controlled press study’s 30,Hz signal averaging, baselines, thresholds, and scoring masks (Sec. F.3). The tactile L0–L2 interfaces are unrelated to the three body-constraint layers.

D.2 Expert-provided procedural context

The off-the-shelf annotation tools (Peng et al., 2026) in Sec. 3.3 receive video, task instructions, and expert-written procedural context. Tab. 11 summarizes preparation, reaction, monitoring, workup, extraction, concentration, and purification. These descriptions supply terminology and likely procedural transitions; observed activity determines which steps are annotated and in what order.

D.3 Example text prompt

The example below includes the procedural background itself, alongside the annotation request. It is adapted from the laboratory document to illustrate the input supplied to the model; it is not a verbatim archived run prompt.

Task. You are annotating a first-person recording of chemical laboratory work. Divide the observed activity into meaningful procedural steps and write a concise description of each step.

Episode instruction. <task description for this recording> Observations. <video or sampled frames with source timestamps>

Professional Skills: laboratory workflow context. The experiment may include the stages below. This information explains the procedure and the operations you may see; the recording may cover only a subset of them.

1. Preparation. Reagents, catalysts, vessels, and consumables are retrieved from storage, brought to the bench, and arranged by use. A balance may be tared before solids are weighed and transferred; pipettes may be rinsed before liquid handling.

2. Reaction setup. Materials are added to a reaction flask. The example procedure describes adding solvent, solid substrate, liquid reagent, and catalyst in sequence, followed by adjustment of heating and stirring and observation of the apparatus.

3. Monitoring. TLC involves sampling, spotting a plate, and UV inspection. HPLC sample preparation can involve liquid transfer, dilution, mixing, filtration into a sample vial, and labeling.

4. Workup. The reaction vessel may be moved from the stirring station to the fume hood. Subsequent operations include handling the receiving vessel, pouring, adjusting controls, and inspecting relevant readouts.

5. Extraction. A mixture is transferred into a separatory funnel and allowed to separate into layers. The stopcock is opened to drain a layer into a receiver and closed to stop drainage; the other layer is collected separately. Addition of extraction solvent, mixing, settling, and separation may be repeated.

6. Rotary evaporation. The flask is connected to the evaporator, controls are adjusted, and concentration is monitored. Later operations include stopping the apparatus, releasing vacuum, removing the flask, and weighing it.

7. Column chromatography. Sample and silica are mixed, material is loaded into a column, the column and tubing are installed, and the apparatus is operated to collect fractions.

Annotation instructions. Use this background to identify equipment and interpret the stage of the experiment. Follow the video when deciding which operations occurred and where their boundaries lie. Preserve visible relocation, repetition, and correction. Describe the action and its target concretely, such as “attaches the flask to the rotary evaporator.” Do not insert steps solely because they appear in the procedure. Leave unreadable quantities, uncertain identities, and unobserved outcomes unspecified.

Output. Return a time-ordered list of segments with start_time, end_time (source-video seconds), skill, description, object_roles, and observed_outcome. Use a short operation label for skill and null for an outcome that cannot be established from the video.

D.4 Human verification and experimental interpretation

As stated in Sec. 3.3, operators review the egocentric video, reconstructed motion, available touch, and draft text together for temporal alignment and semantic consistency. They check action and object identification, omitted or extra steps, unsupported statements, and temporal boundaries (Sec. F.5). The ablation in Tab. 15 adds or removes the procedural document while retaining the same annotation model and other inputs. It tests the benefit of professional background knowledge, with no model retraining. Its automated draft assessment remains distinct from the expert review in Tab. 5.

Appendix E Data Statistics

Collection scale and sensing coverage. EgoExplorer contains 2,715 recordings totaling 100 hours across ten task templates and five environments. Laboratory activities account for 41% of recordings and 46% of duration; the remaining recordings span the living room (8%), study (23%), food-preparation area (11%), and industrial workspace (17%). Figs. 3 and 9 distinguish regular and recovery episodes and summarize their durations and sensing coverage. Among regular recordings, head video and tracker streams are available throughout; MANUS and Wuji glove streams cover 58% and 41%, respectively. Measured touch accompanies the Wuji configuration and is available in 41% of regular recordings. These rounded percentages describe stream availability, not reconstruction accuracy. The analyses below use the signals and health masks required by each diagnostic; their cohorts should not be equated with the full collection.

Movement and task progression. Fig. 10 summarizes 3D head-path length, locomotion and manipulation bout duration, and the dynamics of all five trackers. Short locomotion bouts alternate with sustained manipulation, particularly in laboratory and industrial work. The study and living-room recordings include longer head paths, while ankle dynamics distinguish locomotion from stationary manipulation. Tracker speed, acceleration, and jerk describe recorded behavior; reconstruction quality is evaluated separately in Tab. 7. Fig. 11 illustrates the temporal structure in a reaction-setup episode, aligning ego frames with motion and detected posture across two consecutive intervals. Displayed speed is capped at 1,500 mm/s; the detected states are descriptive diagnostics.

Hand activity and contact. Fig. 12 relates contact events and hand activity to locomotion and stationary manipulation. Panel (a) reports time-weighted unilateral and bimanual occupancy over 545 regular Wuji recordings with healthy tactile streams on both hands. Panel (b) uses 565 recordings, comprising 1,110 healthy hand streams; the median event frequency across hands is 722 events per hour. Events last at least 0.1 s, with gaps shorter than 0.15 s merged. Panel (c) distinguishes locomotion from stationary periods with active or idle hands, marking hands active above 150 mm/s mean wrist speed. These summaries capture differences in task structure: bimanual contact occupies 62% of industrial recording time, whereas laboratory recordings show more left-only contact (52%).

Regional contact coverage. Fig. 13 resolves contact over the 17 anatomical regions in Fig. 18. Each value is the median active fraction among contact frames. Multiple regions can be active together; rows therefore need not sum to one. Parentheses give the number of analyzed hands by environment and side. A region is n/a if fewer than three recordings have at least 30% live taxels there, including unsupported right-hand index and central-palm regions.

Environmental context. Fig. 14 shows registered reconstructions of the five capture environments. Together with the synchronized body–touch examples in Fig. 4, these views place the recorded motions and interactions in their working spaces. Released geometry is limited by capture quality and privacy requirements.

Text coverage and vocabulary. Tab. 12 describes the caption corpus. Temporal coverage is the union of annotated intervals divided by total video duration, pooled over recordings; it reaches 99.9998%. Segment duration has median 7.486 s and 90th percentile 24.000 s. Captions average 98.87 characters, with an exact duplicate rate of 0.0402%, computed as (N-N_{\mathrm{unique}})/N over caption strings. These measures describe coverage and granularity; semantic quality is assessed in Secs. F.5 and F.8.

Fig. 15 complements these statistics with the frequency of terms in the step descriptions. Prominent words name laboratory containers and equipment, including beakers, flasks, funnels, and balances, together with actions such as weighing, holding, tilting, and adjusting. Spatial terms and references to instrument readings add context to these descriptions. This vocabulary reflects the strong laboratory presence in the analyzed corpus; Fig. 5 shows how the descriptions are attached to individual task steps.

Appendix F Experiments

F.1 Evaluation setup

The Evaluation Subset in Sec. 5 contains separate controlled body, hand, and contact tests (Tab. 13). Whole-body capture covers six static and four dynamic actions (Fig. 16), each repeated three times. Repetitions 1–2 support network adaptation; repetition 3 supplies ten held-out test recordings. Hand tests cover ten right-hand configurations, with three glove repetitions per configuration. The video comparison uses 30 recordings per method: 20 with a static camera and ten with camera motion. These are separately recorded matched configurations, not synchronized hand-pose references. Contact tests use button-marked press intervals and task-specific anatomical regions. The detailed settings below apply to the reported comparisons, rather than to the entire collection.

The remaining Natural Recordings have no matched accuracy references. Body ablations use a 200-recording development pool, of which eight support surface-term selection and the other 192 form the reported natural-data cohort. This separation concerns constraint selection; it is not an unseen-task or participant-generalization split. The protocol guides illustrate intended actions; fitted body references and measured hand/contact references remain separate from those illustrations.

F.2 Whole-body reconstruction: inputs and references

Inputs and baseline adaptations. Tab. 14 summarizes the inputs and controls of all main-text comparisons. Public three-anchor baselines observe the head and hands after calibrated device-to-joint conversion; rows marked as raw tracker poses omit that conversion. Our system and the five-anchor AvatarPoser∗ and HMD-Poser∗ variants also observe both ankles. AvatarPoser∗ uses a held-out checkpoint fine-tuned on repetitions 1–2 and calibration recordings, and is evaluated before geometric refinement. HMD-Poser (Dai et al., 2024) uses a local five-tracker adaptation trained on synthetic data without official pretrained weights. SparsePoser (Ponton et al., 2023) additionally receives a pelvis pose borrowed from our reconstruction. Comparisons use common body points and timestamps; different sensing inputs are retained as explicit system differences. The construction ablations instead fix the five-anchor inputs.

Full-body reference labels for the Evaluation Subset. The controlled full-body recordings of the evaluation participant (ten actions \times three repetitions, Tab. 13) do not have an independent optical reference. The accuracy numbers in Tabs. 2 and 7 are therefore measured against reference labels: the same five-tracker refinement run for 300 iterations (learning rate 0.01, position weight 10^{4}) with the participant’s calibration, plus a per-frame pose prior from the protocol library (weight 30 on hip, spine, knee, ankle and foot rotations) that selects the intended articulation where the tracker constraints alone admit several branches (forward bend, lunge, sit/stand, leg raise). These labels agree with the trackers to within 5 mm and were reviewed frame-by-frame; the scoring uses a common body model and coordinate frame. The labels also share our fitting model, calibration, and constraints, so agreement with them is not independent ground-truth accuracy and can favor related methods. Baselines that do not observe the feet are still scored on the same 22 body joints. Repetitions 1–2 train the participant-specific proposal network; repetition 3 forms the full-body portion of the Evaluation Subset (Fig. 20). For this participant the joint-limit block additionally contains a signed knee-hyperextension term, a pelvis-roll term (both weight 100) and a 90∘ torso-lean limit so that deep bends are admissible; evaluation on the Natural Recordings keeps the production configuration.

For visualization, one instant per recording is selected from the fitted reference, using the last of the candidate instants for peak deviation from the initial stance, deepest knee flexion, strongest trunk lean, and lowest pelvis position. All methods are shown at this same instant in Sec. F.10.

F.3 Hand motion and contact capture

Matched hand comparison. The configurations in Fig. 18 are captured with the articulation glove and a RealSense D435 RGB stream (1280\times 720, 30 fps), reconstructed by HaWoR and HandFlow (Zhang et al., 2025; Xu et al., 2026). All outputs use the same retargeter, filter, and 20-DoF Wuji hand model. Each recording’s held configuration is the coordinatewise median of finite wrist-local landmarks in its central hold interval. Ten glove-derived median robot configurations define the task prototypes. Glove trials are scored with the tested repetition excluded from prototype construction. Thus the comparison tests pose retention and kinematic target recovery, without a robot dynamics rollout.

Contact references and exclusions. Button-held intervals mark perceived contact, and the T01–T05 instructions specify the expected regions (Fig. 18). Scoring includes between-press negatives, excludes frames within \pm 100 ms of button transitions, and excludes initial presses without a preceding rest window. The same task-specific region and time masks apply to all methods; regions outside the task mask are not counted as negative examples.

Glove detector and comparison methods. For the controlled study, 120 Hz readings are averaged into 30 Hz video bins. A taxel’s baseline is its median during the 0.1–0.6 s preceding a press; its activation threshold is the maximum of 0.02, three robust rest standard deviations, and the 99.5th percentile of rest fluctuations. The health mask, estimated from earlier recordings, retains 526 of 744 taxels. A region is active when at least two and at least 5% of its valid taxels exceed their thresholds. HACO (Jung and Lee, 2025) vertex predictions use the same count/fraction rule; a dark-glove image mask supplies the hand box when its bare-hand detector fails. The pose-only rule uses the glove skeleton: fingers must flex at least 40^{\circ} relative to the most open pose and remain below 15 mm/s over a 0.25 s window for at least 0.2 s. Three or more flexed fingers indicate a grasp; otherwise only flexed fingertips are marked. The glove detector uses annotated rest windows, which should be considered when interpreting the system comparison.

F.4 Evaluation metrics

Body accuracy (Tabs. 2 and 7). Body-point error averages Euclidean distances over 22 SMPL-X joints (pelvis through wrists), in mm. Non-tracked error uses the 17 joints outside the head, wrists, and ankles; the pelvis is included, with its error also reported separately as Root. Root error is pelvis displacement in the fixed calibrated frame. PA-MPJPE centers each pose, matches its Frobenius norm to the reference, and applies the SVD rotation with reflection correction (Umeyama, 1991); the norm-matching scale is not separately optimized by least squares. Velocity error averages Euclidean differences of joint velocities, computed from adjacent positions divided by their timestamp intervals, in mm/s. Scores use matched timelines and finite observations, averaged within each recording and then equally across recordings. No per-frame alignment is applied except for PA-MPJPE.

Hand accuracy, stability, and continuity (Tab. 4). Tip averages the Euclidean discrepancy of the five held fingertips in wrist-local coordinates after matching hand length to the reference. Hold RMS (root mean square) measures within-hold scatter: subtract each fingertip’s median held position from its emitted positions, square the Euclidean distances, average across fingertips and valid hold frames, and take the square root (mm). It is a stability measure, not independent pose accuracy. Missing is the fraction of frames with no finite pose; Gap p95 is the 95th-percentile duration of contiguous missing runs pooled across recordings. Success assigns the retargeted median pose to the closest of ten glove-derived prototypes, using RMS landmark distance with weights 3 for tips, 0 for wrist, and 1 elsewhere. A trial succeeds when the nearest prototype matches its instructed configuration; chance is 10%.

Sensor agreement and geometry (Tab. 7). Tracker RMS squares Euclidean distances between predicted and observed device positions, averages over valid device–frame pairs, and takes the square root (mm); recording-level RMS values are then averaged equally. Predictions use each variant’s mount model, so this measures sensor agreement. Crossing counts intersecting triangle pairs between distinct, non-adjacent body parts using the same independent checker and 5 Hz sample times for all variants; frame counts are averaged within recordings, then across recordings. Floor is the fraction of frames with any posed vertex more than 5 mm below the shared ground plane.

Temporal quality (Tab. 7). Slide is mean horizontal foot speed during common support intervals, reported in mm/s. Support is inferred from raw ankles with horizontal speed below 100 mm/s, height within 30 mm of that ankle’s fifth-percentile height, and duration at least 0.3 s; pairs separated by more than 0.1 s are excluded. Jerk uses pelvis-relative joint positions in meters while preserving world orientation. After resampling to 30 Hz without bridging invalid samples or gaps above 0.1 s, take the third forward difference, multiply by 30^{3}, and average its vector norm over joint–frame pairs, then equally over recordings (m/s3). No smoothing is applied. Avg. is the unweighted arithmetic mean of the nine displayed values in Tab. 7, including jerk and both data cohorts. This summary depends on displayed units and has no single physical unit.

Regional contact (Tab. 4). After the shared masks in Sec. F.3, true positives (TP), false positives (FP), and false negatives (FN) are pooled over retained frame–region decisions, including non-contact time. IoU is TP/(TP+FP+FN) and F1 is 2TP/(2TP+FP+FN); task-wise scores are not averaged. Missing sensing support is excluded, not relabeled as no contact.

F.5 Expert review of text annotations

Language review. Tab. 5 reports expert review of model-generated text annotations prior to manual correction. The audit covers 487 segments. Factual errors concern incorrect actions, objects, tools, or outcomes; unsupported claims lack evidence in the recording; missed and extra steps denote omissions and spurious additions, respectively; unjudgeable cases are reported separately. Boundary error is the fraction of reviewed segments requiring temporal correction: 15 of 487 segments (3.08%), reported at the segment level rather than as absolute timing deviation, with no frame-accurate boundary ground truth available. Unjudgeable segments are those for which the available recording does not support a judgment. Factual-error, unsupported-claim, extra-step, boundary, and unjudgeable rates use reviewed segments; missed-step rate uses the reference steps that should be described. These denominators need not coincide. All rates reflect pre-correction audit findings; human verification and correction follow model annotation. The automated professional-skill ablation is reported separately in Tab. 15.

F.6 Preliminary training study

Pilot representation and metrics. Tab. 6 summarizes two preliminary Cosmos3-Edge (NVIDIA, 2026) studies: separate versus joint output generation and conditional action generation. For the output comparison, video-only, action-only, touch-only, and joint models all receive video/action/touch history. We report their 5,000-step checkpoints using the same sampled-evaluation implementation. The separate-model row combines the video, action, and touch scores of three independently trained models, one per output; it is not an ensemble prediction or an average across models. Action uses the model’s 57-dimensional hand-pose representation, derived from SMPL-X forward kinematics: head and wrist poses plus wrist-local fingertip positions. Each action coordinate value \upsilon is normalized as \operatorname{clip}(2(\upsilon-\upsilon_{1\%})/\max(\upsilon_{99\%}-\upsilon_{1\%},10^{-6})-1,-1,1), where \upsilon_{1\%} and \upsilon_{99\%} are its 1st and 99th percentiles computed from training sessions only. Action MSE is therefore an error in normalized feature space, distinct from the body-point and fingertip errors in millimeters used for annotation evaluation. Touch uses a separate token stream with regional contact, intensity, and validity information; missing regions are masked.

Training settings. The 5K configuration uses batch size 64 and AdamW with momentum coefficients (0.9,0.95), learning rates 5\times 10^{-5} for the unfrozen backbone and 5\times 10^{-4} for new modality heads. Backbone weight decay is 0.01; new heads use no decay. A 200-step warmup precedes cosine decay, with gradient-norm clipping at 1 and seed 0. Training predicts clean targets from noisy inputs: video-latent and action MSE have weight 1, regional contact MSE weight 3, and tactile-intensity MSE weight 1.5. Invalid contact regions are masked. Separate models activate only their own output losses, while the joint model activates all three streams.

Temporal split and scoring. The loader splits at session level within each task: sorted session indices congruent to five modulo ten form validation; all windows from a session remain in the same partition. The pilot uses 16 observed and 16 generated samples at 15 Hz, distinct from the release grid. Pixel-space PSNR evaluation uses 256 held-out windows, two samples per window, and 20 denoising steps. Prediction and target latents are decoded with the same clean historical context; PSNR scores the generated interval against the VAE-decoded target, not directly against uncompressed camera frames. PSNR is 10\log_{10}(1/\mathrm{MSE}_{\mathrm{pixel}}) in dB for pixels in [0,1], averaged over windows and samples. Action MSE averages the 57 normalized features, excluding padding channels. Contact predictions threshold the mean of sampled binary states above 0.5; F1 pools TP, FP, and FN over valid region–time entries as in Sec. F.4. The current output comparison in Tab. 6 reports higher video PSNR and contact F1 and lower action MSE for joint generation.

Conditional action generation. In the second study of Tab. 6, recorded video, or video and touch together, from the same interval as the generated action are supplied as conditions. The unconditioned row receives no such additional observations; the video/action/touch history remains available to all rows. Because these conditions are observed over the action interval, this test measures cross-modal conditioning rather than prediction from history alone. Action MSE is computed in the same normalized feature space as above.

F.7 Calibration and layered reconstruction ablations

Within each cohort of Tab. 7, layer variants share the same five-anchor inputs, proposal checkpoint, calibrated profile, and 60-iteration budget. Proposal only skips refinement; Layer 1, Layer 1+2, Layer 1+3, and Full activate the corresponding terms of Eq. 3 without retraining. Mean body replaces participant-specific shape and scale while retaining mounts and the reconstruction procedure; proposals are recomputed under that geometry. No mount calibration uses identity rotations and zero attachment translations in training, inference, and refinement, with a correspondingly trained network. Uncalibrated at test retains the calibrated network but omits attachment correction during inference and refinement.

The natural-data cohort retains its production proposals. The independently fine-tuned five-anchor AvatarPoser row in Tab. 2 has its own training pool and unrefined checkpoint; comparing its 64.1 mm with the 79.0 mm proposal-only row does not isolate refinement. The within-cohort layer rows provide that comparison. Geometric scores must be read alongside pose and tracker errors: moving the body toward sensor locations can reduce crossing or floor violations without improving articulation.

F.8 Professional-skill ablation

Tab. 15 compares unrevised annotation drafts generated with and without professional Skills documents (Sec. D.2). The annotation model, input videos, task instructions, and decoding settings are held fixed; no model retraining is involved. An automated judge flags factual errors and unsupported claims at the window level, while event omission is measured against model-provided reference events. Adding Skills documents lowers these three rates by 6.52, 5.52, and 7.05 percentage points, respectively. These automated assessments are distinct from the expert audit in Tab. 5; generated annotations still undergo human verification and correction.

F.9 Humanoid retargeting

Downstream humanoid control motivates preserving metric root translation, per-joint rotations, and hand pose at a common sampling rate rather than normalizing skeletons. Fig. 19 illustrates the transfer of a 135 s reaction-setup recording to a Unitree G1, including reconstructed SMPL-X motion, kinematic retargeting, and physics-based execution. The kinematic stage uses GMR (Araujo et al., 2025) whole-body inverse kinematics (IK) with position and orientation targets, followed by root-height correction to prevent sole penetration. We compare the corrected trajectories with GMR-scaled and offset human targets at nine corresponding body origins: the pelvis, feet, knees, elbows, and wrists. At temporally matched samples, the mean position errors for the reaction-setup and reagent-weighing clips are 6.8 and 7.0 cm, respectively; the corresponding wrist errors are 7.5 and 7.8 cm. These residuals include discrepancies between human joint centers and robot link origins. Position errors average Euclidean distances over the stated origins and matched frames. The 14-body robot-reference score uses pelvis, torso, and bilateral hip, knee, ankle, shoulder, elbow, and wrist link origins. Root orientation is the angle of the relative quaternion, averaged over frames; it is not a yaw-only error.

The physics-based stage follows the ZeroWBC (Yang et al., 2026a) training recipe and uses a reinforcement-learning tracking policy in MuJoCo (Todorov et al., 2012). Unlike the original architecture, our implementation feeds short- and long-horizon targets to an MLP and is trained only on the two clips. After 15,000 PPO (Schulman et al., 2017) iterations, full-clip evaluation at 50 Hz yields mean 14-body tracking errors of 4.6 and 3.6 cm after aligning torso horizontal position and yaw with the robot reference, compared with 6.6 and 4.8 cm without alignment. No falls occur. Global wrist errors are 5.8 and 4.0 cm. Root-position errors are 3.8 and 2.5 cm, and root-orientation errors are 1.8 and 1.3∘, respectively. Relative to the same human targets and nine body origins used above, the simulated trajectories have mean errors of 8.5 and 6.8 cm and wrist errors of 9.0 and 8.2 cm, without evaluation-time alignment. Thus, human-to-robot mapping error and robot-reference tracking error are measured against different references and should not be added.

Error sources. The post-IK floor correction raises the root by 2.7–6.7 cm across the two clips. Mean wrist error increases from approximately 5.6 cm before correction to 7.5 and 7.8 cm afterward. Removing pelvis translation alone still leaves wrist errors of 7.0 and 6.9 cm and elbow errors of 9.8 and 9.5 cm in the kinematic retargeting. Thus, global displacement does not explain the full residual, and the current body mapping and IK trade-offs leave substantial articulation error. For the control policy, wrist and foot position rewards use \exp(-\overline{e^{2}}/0.3^{2}), with distances measured in meters. A 5 cm deviation therefore yields a reward of approximately 0.97. The joint-position reward has weight 0.1, and no dedicated elbow- or knee-position rewards are used. These choices provide limited incentives for centimeter-level precision, and their individual effects have not been isolated by ablation. The results therefore demonstrate preliminary simulated motion transfer rather than precise object interaction. Joint optimization of ground contact and body targets, together with tighter tracking rewards, remains an important direction for future work.

Robustness, hands, and scope. A second rollout also remains fall-free under random base-velocity perturbations applied every 2–4 s, with horizontal components in [-500,500] mm/s and yaw velocity in [-0.5,0.5] rad/s. Fingers are retargeted independently to the three-finger hand by matching fingertip vectors in the palm frame, yielding fingertip errors of 1.1–2.2 cm. The thumb–index aperture follows the human aperture with correlations of 0.991–0.993 (Pearson correlation of thumb–index fingertip distances over matched frames). Hand poses are decoded using the same SMPL-X hand mean as in glove fitting. Fingers are replayed kinematically, while the tracking policy controls the 29 body joints. Evaluation replays the training clips and therefore does not establish generalization to unseen tasks. This example demonstrates transfer of recorded motion, not hardware deployment; deployment would additionally require state estimation and safety constraints, which are not evaluated here.

F.10 Qualitative examples

Fig. 20 compares EgoExplorer with six baselines at matched instants from the Evaluation Subset. Our results more closely recover the displayed forward bends, lunges, and leg raises, including articulation between tracked endpoints. Rows follow B01–B10 in Tab. 13; the lateral raises, square walk, and leg raises correspond to the cross-body reaches, rectangular step sequence, and straight-leg forward kicks. This visual comparison complements Tab. 2; the sensing inputs and fitted references are specified in Sec. F.2.

Temporal body and touch examples. Fig. 21 expands the ten sessions in Fig. 4, with six synchronized instants per row, evenly spaced from 10% to 90% of each recording. These sequences complement the text studies with continuous body motion and contact observations.

Appendix G Discussion on Capture Trade-offs

EgoExplorer combines portable deployment with accurate motion reconstruction and complementary visual, body, hand, and tactile observations. Its wearable rig supports collection in existing environments, while participant and mount calibration grounds the recovered motion in measured geometry. The body, hand, and contact comparisons in Tabs. 2, 4 and 4 support the quality of these annotations relative to the evaluated baselines. Together, portability, annotation quality, and sensing coverage motivate this design for capturing human loco-manipulation across tasks and workspaces. Tab. 16 summarizes the deployment trade-offs.

Deployment and sensing. Multi-view RGB reconstruction provides an affordable route to whole-body capture within a calibrated volume. FreeMoCap supports ordinary USB cameras and recommends three to four views for general motion capture. Such arrays require overlapping views, synchronization, and intrinsic/extrinsic calibration; moving the cameras requires renewed calibration. Multi-view methods can also recover hands, although occlusion and the small image scale of fingers make fine articulation challenging (Zhang et al., 2021). Direct glove sensing complements body capture and can additionally provide measured contact. Hybrid systems such as ParaHome combine cameras, an inertial suit, and gloves (Kim et al., 2025); HOT3D uses optical markers for hands and objects (Banerjee et al., 2025); Nymeria combines an inertial body suit with wearable Aria devices (Ma et al., 2024). Our rig avoids external camera-array and base-station setup and reuses its body–sensor calibration while mounts remain fixed. Its inside-out trackers still depend on suitable visual conditions.

Hardware costs. Tab. 17 reports costs in USD, excluding computers, software licenses, mounting/cabling accessories, taxes, shipping, and setup labor; prices are public listings recorded in September 2026.

At USD 69.99 per C920s, four to six RGB cameras cost less than the USD 1,184 tracker subsystem, making camera arrays attractive for fixed capture spaces. The suit and optical-camera entries are hardware-only listings. MANUS publishes no list price, so its entry uses a reseller listing (September 16, 2026) excluding the software license; the camera uses GoPro list prices, USD 399.99 plus 79.99. With the Wuji price converted to USD, subtotals are approximately USD 7,979 (MANUS) and 5,397 (Wuji), trading added wearable hardware for deployment flexibility and direct finger/contact measurements.