Enhancing AI-Powered Computer Vision Through Physics-Awareness

Artificial Intelligence has undeniably left an indelible mark on our lives. In a pioneering effort to further enhance AI capabilities, researchers from UCLA and the United States Army Research Laboratory have unveiled a unique approach that marries physics-awareness with data-driven techniques in AI-powered computer vision technologies.

The study, published in Nature Machine Intelligence, proposes a groundbreaking hybrid methodology aimed at refining how AI-based machinery senses, interacts, and reacts to its environment in real-time—critical for autonomous vehicles and precision-action robots.

Hybrid Approach for Physics-Aware AI

Traditionally, computer vision, the field that enables AI to comprehend and infer properties of the physical world from images, has largely focused on data-based machine learning. Concurrently, physics-based research sought to unravel the physical principles underlying many computer vision challenges. However, assimilating the understanding of physics into the realm of neural networks has proved challenging.

In a significant breakthrough, the UCLA study intends to combine the deep understanding from data and the real-world know-how of physics, thereby creating a hybrid AI with augmented capabilities. Achuta Kadambi, the study's corresponding author and an assistant professor of electrical and computer engineering at the UCLA Samueli School of Engineering, explains, “Physics-aware forms of inference can enable cars to drive more safely or surgical robots to be more precise.”

Incorporating Physics into Computer Vision AI

The research team outlines three innovative ways to integrate physics into computer vision AI:

1. Infusing physics into AI data sets: This involves tagging objects with additional information, such as their potential speed or weight, akin to characters in video games.
2. Integrating physics into network architectures: This strategy involves running data through a network filter that codes physical properties into what cameras capture.
3. Incorporating physics into network loss function: Here, knowledge based on physics is leveraged to assist AI in interpreting training data on its observations.

These experimental lines of research have already produced promising results in enhancing computer vision. For instance, the hybrid approach enables AI to track and predict an object's motion more precisely and can produce accurate, high-resolution images from scenes obscured by adverse weather.

The Future of Physics-Aware AI

The researchers are optimistic that continued advancements in this dual modality approach might lead deep learning-based AIs to independently learn the laws of physics. This could mark a new frontier in AI-powered computer vision technologies, paving the way for a safer and more precise AI application in various sectors, including autonomous vehicles and surgical robotics.

The study, supported in part by a grant from the Army Research Laboratory, was co-authored by Army Research Laboratory computer scientist Celso de Melo, and UCLA faculty members Stefano Soatto, Cho-Jui Hsieh, and Mani Srivastava. Additional funding came from grants from the National Science Foundation, the Army Young Investigator Program, the Defense Advanced Research Projects Agency, Intrinsic, an Alphabet company, and Amazon.