Visual Identity Prompting can be utilized for 'Neural Kinematic Retargeting' to effectively transfer large-scale human egocentric manipulation datasets to robot policies without explicit inverse kinematics.

Motivation

Robot data is scarce, but human video data (e.g., Ego4D) is abundant. Traditional kinematic retargeting is brittle and computationally expensive. Using RoboVIP to 're-skin' human hands into robot grippers while maintaining temporal coherence could unlock massive pre-training datasets.

Proposed Method

Fine-tune the RoboVIP model to perform video-to-video translation where the input is a human manipulation video and the 'Visual Identity' prompt is the target robot. Use edge-detection or depth-maps from the human video as structural guidance to preserve the trajectory, while forcing the model to hallucinate the robot's morphology in place of the hand. Train a Behavior Cloning policy on the re-skinned dataset and test on a real robot arm.

Expected Contribution

A pipeline for converting human video datasets into robot training data using generative visual identity transfer, bypassing complex analytical retargeting.

Required Resources

Large human video datasets, target robot image assets, significant inference compute for video-to-video generation.

Source Paper

RoboVIP: Multi-View Video Generation with Visual Identity Prompting Augments Robot Manipulation

View Paper Details →