Always-on Contextual AI: The Future of Wearable Computers and the Potential for a World Model of our Physical Reality

Richard Newcombe

Meta Reality Labs Research, USA

Abstract

AI is here! But how useful can AI be to each of us without the context of our everyday physical realities? Within this decade we have the unique chance to tackle the core research issues that will enable the emergence of a new generation of contextually aware, highly personalized AI. In this talk, I will cover the types of context necessary for AI to become significantly more valuable to us, facilitated by a new era of sensor-rich wearable computers. These range from AR/AI glasses to wristbands and pins, introducing contextually aware computing into our daily lives for the first time. I will outline the essential challenges in achieving continuous, real-time, always-on, low-power context sensing and inference to solving the problem of enabling a boundless context length that can span years and how to interface such rich context with AI today. This involves advanced multimodal algorithms, from SLAM and 3D scene segmentation to eye-gaze and body-pose estimation, to understanding when a person is communicating and referencing things in reality, both to the machine and to others -- all while navigating the limits and constraints of wearable technology in terms of computing power, sensor capabilities, and design for comfort and ubiquity. As we uncover the limits of sensing and context inference, we'll also discover the opportunity to build a new kind of 3D world model, one which is shared between devices and users, and that can dramatically enhance what we can achieve with contextual AI.