Scaling Robot Learning with Passively-Collected Human Data

Scaling Robot Learning with Passively-Collected Human Data

Abstract: The foundation of modern AI is scalable knowledge transfer from humans to machines. While Computer Vision and NLP can glean such knowledge from exabytes of human-generated data on the Internet, Robot Learning still heavily relies on resource-intensive teleoperation for data collection. Can we capture real-world human interactions as effortlessly as the Internet captures the virtual world? We propose that passive human data collection is a crucial step towards this future. Just as the Internet evolved into an unintentional data repository for AI, an ideal data collection system should capture sensorimotor data from everyday human activities, without humans’ conscious participation. 

In this talk, I will present a series of passive human data collection systems and robot learning algorithms that can learn from such data. I will present (1) MimicPlay, a constructivist effort to investigate robot learning from human video data through hierarchical imitation learning, (2) MimicTouch for learning tactile-based control policies from human hand demonstrations, and (3) EgoMimic, a full-stack hardware-to-algorithm framework that leverage consumer-grade smart glasses to capture and learn bimanual manipulation tasks from egocentric human data. I will conclude my talk by sharing our vision for a human-centric future of robot learning, where robots can better understand and interact with human and human environments by learning from broad human data.

Bio: Danfei Xu is an Assistant Professor in the School of Interactive Computing at Georgia Institute of Technology. He earned his Ph.D. in Computer Science from Stanford University in 2021. His research focuses on machine learning methods for robotics, particularly in manipulation planning and imitation learning. His work has received Best Paper nominations at the Conference on Robot Learning (CoRL) 2023 and IEEE Robotics and Automation Letters (RA-L) 2023