WORLDTECHELA Trains Robot Policies From Action-Free Video With 4M-Parameter Decoders
A preprint posted to arXiv describes ELA, a framework that lets robots learn manipulation from unlabelled video by optimising latent actions for both policy learning and action decoding. The authors report that it beats earlier latent-action methods in simulation and on real robots.
The bottleneck ELA addresses
Imitation learning for robots relies on demonstrations that record the actions performed, and such action-labelled data is in short supply. Action-free demonstrations, including ordinary video, are far more abundant, the arXiv abstract notes.
Recent latent-action methods attempt to pull action-relevant representations out of changes between successive observations. The preprint contends that these methods never tune those latent actions for downstream control, leaving representations that policies struggle to model and that are hard to translate accurately into continuous robot motion.
How the framework is assembled
ELA trains three components together: a latent action model, a latent action policy and an action decoder. The authors say this joint training lets policies make use of large amounts of video while requiring only a small number of action-labelled robot demonstrations.
The action decoders are kept lightweight at 4M parameters. According to the authors, they can be inserted into existing imitation learning methods without altering policy architectures or training objectives.
Reported results
The abstract states that experiments on simulated and real-world manipulation tasks show ELA consistently ahead of prior latent-action baselines. These claims come from the preprint itself, and WORLDTECH has seen no independent replication.
Open questions
- How large are the gains over baselines on the real-world tasks?
- How many action-labelled demonstrations are needed in practice?