CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization

What happened

arXiv:2610.12567v1 Announce Type: new Abstract: Prior research on learning from humans often requires real human demonstrations. However, while the generated videos appear to show plausible motions, they lack both physical consistency and robot grounding, and thus the robot cannot directly roll out the trajectory shown by the video.

On the plus side, these synthetic videos take place in the robot's current environment, and do not require actual human teaching. The object trajectories extracted from the synthetic video provide an initial guess at the correct motion, and the robot must refine these trajectories (for instance, within a digital twin) while enforcing physical feasibility and semantic consistency.