Robot safety is a nuanced concept. We commonly equate safety with collision-avoidance, but in complex, real-world environments (i.e., the “open world’’) it can be much more: for example, a mobile manipulator should understand when it is not confident about a requested task, that areas roped off by caution tape should never be breached, and that objects should be gently pulled from clutter to prevent falling. However, designing robots that have such a nuanced safety understanding---and can reliably generate appropriate actions---is an outstanding challenge. In this talk, I will describe my group’s work on systematically uniting modern machine learning models (such as large vision-language models and latent world models) with classical formulations of safety in the control literature to generalize safe robot decision-making to increasingly open world interactions. Throughout the talk, I will present experimental instantiations of these ideas in domains like vision-based navigation and robotic manipulation.

Our work is broadly motivated by the emergence of learning-based methods in control theory and robotics, with a specific focus on scenarios that have humans in-the-loop with control systems. For instance, learning algorithms are being deployed in semi-autonomous vehicles, robot assistants, brain-machine interfaces, and exoskeletons, where they interact dynamically with a human partner to complete tasks. When learning algorithms are employed in this way, a dynamic game is created that is played between two intelligent agents (the human and machine learners), requiring new techniques to guarantee safety and performance.

General-purpose robot policies hold immense promise, yet they often struggle to generalize to novel scenarios, particularly struggling with grounding language in the physical world. In this talk, I will first propose a systematic taxonomy of robot generalization, providing a framework for understanding and evaluating current state-of-the-art generalist policies. This taxonomy highlights key limitations and areas for improvement. I will then discuss a simple idea for improving the steerability of these policies by improving language grounding in robotic manipulation and navigation. Finally, I will present our recent effort in applying these principles to scaling up generalist policy learning for dexterous manipulation.

Many of us are collecting large scale multitask teleop demonstration data for manipulation, with the belief that it can enable rapidly deploying robots in novel applications and delivering robustness in the 'open world'. But rigorous evaluation of these models is a bottleneck. In this talk, I'll describe our recent efforts at TRI to quantify some of the key 'multitask hypotheses', and some of the tools that we've built in order to make key decisions about data, architecture, and hyperparameters more quickly and with more confidence. And, of course, I’ll bring some cool robot videos.

TBD

TBD

TBD

TBD

TBD