Learning digital humans that act and interact

Michael J. Black

Max Planck Institute for Intelligent Systems and Meshcapade GmbH, DEU

Abstract

Augmented and virtual reality will require artificial humans that interact with real humans as well as with real and virtual 3D worlds. This requires a real-time understanding of humans and scenes as well as the generation of natural and appropriate behavior. We approach the problem of creating such embodied human behavior through capture, modeling, and synthesis. First, we learn realistic and expressive 3D human avatars from 3D scans. We then train neural networks to estimate human pose and shape from images and video. Specifically, we focus on humans interacting with each other and the 3D world. By capturing people in action, we are able to train networks to model and generate human movement and human-scene interaction. To validate our models, we synthesize virtual humans in novel 3D scenes. The goal is to produce realistic human avatars that interact with virtual worlds in ways that are indistinguishable from real humans. This course will introduce students to the current state of the art, including methods for estimating 3D humans from video, generative models of human movement driven by text and high-level goals, and large multi-modal foundation models for reasoning about humans and their behavior.

Bio

Michael Black received his B.Sc. from the University of British Columbia (1985), his M.S. from Stanford (1989), and his Ph.D. from Yale University (1992). He has held positions at the University of Toronto, Xerox PARC, and Brown University. He is an Honorarprofessor at the University of Tübingen and one of the founding directors at the Max Planck Institute for Intelligent Systems in Tübingen, Germany, where he leads the Perceiving Systems department and serves as Managing Director. He was a Distinguished Amazon Scholar (2017-2021). He is a recipient of the PAMI Distinguished Researcher Award, the 2022 and 2010 Koenderink Prize, the 2013 Helmholtz Prize, and the 2020 Longuet-Higgins Prize. He is a member of the German National Academy of Sciences Leopoldina and a foreign member of the Royal Swedish Academy of Sciences. In 2013 he co-founded Body Labs Inc., which was acquired by Amazon in 2017. He is a co-founder and Chief Scientist of Meshcapade.