Spatial AI at the Frontier of XR, Robotics & AI Assistants

Richard Newcombe

Facebook Reality Labs, US

Abstract

Real-time 3D Machine Perception in the wild is starting to become a reality powering revolutions in Mixed Reality, Robotics and a new generation of contextually aware AI Assistants. In this lecture we will look at the key approaches that form the basis of Spatial AI, from on-device real-time visual SLAM and localization technologies through to photo-realistic reconstruction, fine grained scene understanding and real-time object tracking.

We will look at the systems science and engineering level implications of achieving machine perception in the wild taking into account the major power, compute, sensor, security and communications limitations that result from applications/devices running in real-time in mobile or wearable form factors. The lecture will conclude with a view of the key challenges of each of the technologies and an overview of the potential of a future solutions that look at a new generation of Spatial AI research that takes into account solving problems using new device hardware Si and Sensors, through to Edge and Cloud infrastructure.