Perception for humanoid robots

Lior Wolf

Tel Aviv University & Mentee Robotics, ISR

Abstract

This tutorial will cover several aspects of perception for humanoid robotics, addressing the unique challenges of perception on a bipedal platform. We will discuss various technologies that Mentee Robotics has been exploring such as NeRF-based localization, adding semantic information into NeRFs (e.g., LeRFs), open-dictionary detection models, text-to-image grounding models, foundational segmentation models such as SAM, obstacle detection based on stereo, camera-based imitation for controlling the hands, ways to minimize the sim2real gap for obtaining agile locomotion, and path planning with RL.