Don't get lost in Translation: The Neural Rendering of the Champion

Hao Li

Pinscreen, Inc. & MBZUAI, UAE

Abstract

While Pinscreen continues to innovate in the space of AI-driven avatar digitization and virtual assistant technologies, we will showcase a recent development in neural rendering for AI-based VFX production in collaboration with Adapt Entertainment. The Champion is the first complete feature film to be lip sync'ed using facial reenactment by Pinscreen from German/Polish to English. In this Polish film, set in the second world war, the entire production was filmed and finished in German/Polish. Using advances in neural rendering, the entire film has the actor's faces replaced by an inferred version, visually built from actors re-recording dialogue in a sound studio. The film tells the story of pre-war boxing champion Tadeusz "Teddy" Pietrzykowski, who in 1940 arrives with the first transport of prisoners to the newly created Auschwitz concentration camp. The story was filmed without any consideration of later dialogue replacement. This talk discusses the important production issues involved in doing professional neural rendering on such a large scale involving hundreds of shots. Smaller projects have been shown, but these often involve hours per shot and massive manual intervention. Based on new technology, The Champion used only the footage already edited for the final film, combined with a robust and non-intrusive recording of the actors delivering the lines in English. We will discuss will discuss the innovations in Pinscreen technology and provide insights into further advances that the team are working on.