Motion and sounds: Using video to reveal invisible motions and to suppress unwanted sounds.

William T. Freeman

Computer Science and Artificial Intelligence Laboratory, MIT, USA

Abstract

This lecture will cover several projects related to motion and sound. Simple wavelet processing allows using a conventional camera as a "motion microscope" to amplify and reveal small motions. Small vibrations may be caused by audio signals, allowing measuring sounds visually. Some motions that are not even in the line of sight of a camera but can be revealed using a vertical edge as an imaging element in a "corner camera"

Using a machine learning approach, we can use video to suppress unwanted audio in a project we call Looking to Listen. This has been a multi-year project in my research team at Google, and I'll describe some of that journey from academic paper to deployed product.