Robot Learning In The Wild: Continual Improvement by Watching and Practicing

Deepak Pathak

Carnegie Mellon University, USA

Abstract

How can we train a robot that can generalize to perform thousands of tasks in thousands of environments? This poses a chicken and egg problem: to train robots for generalization, we need large amounts of robotic data from diverse environments, but it is impractical to collect such data unless we can deploy robots that generalize. Passive human videos on the internet can help alleviate this issue by providing diverse scenarios to pretrain robotic skills. However, just watching humans is not enough, the robot needs to learn and improve by autonomously practicing in the real world and adapting its learning to new scenarios. We will unify these three mechanisms -- learning by watching others (social learning), practicing by exploration (curiosity), and adapting already learned skills in real-time (adaptation) -- to define a continually adaptive robotic framework. I will demonstrate the potential of this framework for scaling up robot learning via case studies of controlling dextrous robotic hands from monocular vision, dynamic-legged robots walking from vision on unseen challenging hikes, and mobile manipulators performing lots of diverse manipulation tasks in the wild.

Bio