Researchers at Columbia Engineering have developed a method enabling robots to learn about their own physical structure and movement by observing themselves through a camera. This approach allows robots to create self-simulations without relying on pre-programmed models, making them more adaptable to changes in their environment and capable of recovering from damage.
The study, led by doctoral student Yuhang Hu at Columbia University’s Creative Machines Lab, introduces “Kinematic Self-Awareness,” where robots use raw video input to build a self-model, similar to how humans refine movements by watching their reflection in a mirror. Traditional methods require robots to first train in digital simulations created by engineers, but this research eliminates that need by allowing robots to generate their own models.
Using deep neural networks, the system translates 2D video footage into a 3D understanding of motion, enabling robots to recognize alterations in their structure—such as a bent limb—and adjust their movements accordingly. This capability could enhance the reliability of robots in various settings. In home environments, for instance, a personal assistant robot could detect damage and adapt without requiring repairs. In manufacturing, a robotic arm knocked out of alignment could recalibrate itself instead of halting production.
The research builds on two decades of work by the Columbia team on self-modeling robots. Earlier versions created stick-figure-like self-representations, and later models used multiple cameras to improve accuracy. This latest development marks the first time a robot has autonomously constructed a high-fidelity self-model using only a single camera.
Hod Lipson, director of the Creative Machines Lab, notes that self-modeling is a critical step toward making robots more independent and resilient. “We humans cannot afford to constantly repair and reprogram robots. If they are going to become truly useful, they need to learn to take care of themselves,” he says. Ultimately, the researchers envision a future where robots can “imagine” their own movements and anticipate the consequences of their actions, increasing their autonomy in real-world applications.
Photo credit: Jane Nisselson/Columbia Engineering
