Researchers at Carnegie Mellon University and tech company Meta have presented RoboAgent, an advanced artificial intelligence agent. Designed with the learning tendencies of infants in mind, RoboAgent is equipped with manipulation abilities that align with those of a 3-year-old child.
Humans naturally engage in observation, imitation, and repetition, skills central to learning. Building on this observation, the research aimed to replicate these attributes in RoboAgent. “Our objective was to produce an AI entity capable of demonstrating a range of skills in unfamiliar situations,” said Vikash Kumar, an adjunct faculty in the School of Computer Science’s Robotics Institute. “RoboAgent combines extensive passive observations with selected active interactions.”
Distinct from previous models designed for specific tasks, RoboAgent displays proficiency in 12 manipulation skills across various scenes. This research stands out because of its real-world application and the use of less data compared to earlier projects. Associate Professor Abhinav Gupta remarked, “We’ve seen a notable breadth of skills, transitioning our work from simulation to actual environments.”
The unique feature of RoboAgent is its dual learning mechanism, which combines firsthand experiences with passive observations derived from internet data. To provide firsthand experiences, the researchers teleoperated the robot through diverse tasks.
Ph.D. student Homanga Bharadwaj explained, “RoboAgent utilizes a policy framework that allows decisions based on temporal chunks of movements, rather than the more common per-timestep actions.”
Robots traditionally struggle with assimilating learning from passive surroundings. RoboAgent addresses this by incorporating internet videos as a learning medium, similar to the passive observation method used by infants. “Through these videos, RoboAgent gains insights into human-object interactions and identifies essential skills for task execution,” shared Mohit Sharma, another Ph.D. student.
Assistant Professor Shubham Tulsiani observed, “RoboAgent’s design, which allows it to train efficiently while drawing primarily from easily accessible internet data, may enhance robotic applicability in dynamic settings such as homes or hospitals.”
The research team is open-sourcing its materials, including RoboSet, described as a comprehensive public robotics dataset on standard hardware. The intent behind this release is to foster further research and development in the domain of robotic agents.
Carnegie Mellon University and Meta researchers have announced RoboAgent, an AI agent that leverages passive observations and active learning to enable a robot to acquire manipulation abilities on par with a toddler. Credit: Carnegie Mellon University
