Milan Ganai is a PhD student in the Department of Computer Science advised by Professors Marco Pavone and Clark Barrett. His research interests lie at the intersection of safe AI and robotics, concentrating on developing generalizable physical reasoning capabilities for autonomous systems to reliably adapt to novel environments. Prior to Stanford, he received his BS in Computer Science, summa cum laude with highest distinction, and MS in Computer Science at UC San Diego, where he was a Jacobs School Scholar and Regents Scholar. He performed research in the intersection of control and reinforcement learning under Professors Sicun Gao and Sylvia Herbert and has interned at Amazon Web Services.
Abstract: Foundation models can provide robust high-level reasoning on appropriate safety interventions in hazardous scenarios beyond a robot’s training data, i.e. out-of-distribution (OOD) failures. However, due to the high inference latency of Large Vision and Language Models, current methods rely on manually defined intervention policies to enact fallbacks, thereby lacking the ability to plan generalizable, semantically safe motions. To overcome these challenges we present FORTRESS, a framework that generates and reasons about semantically safe fallback strategies in real time to prevent OOD failures. At a low frequency in nominal operations, FORTRESS uses multi-modal reasoners to identify goals and anticipate failure modes. When a runtime monitor triggers a fallback response, FORTRESS rapidly synthesizes plans to fallback goals while inferring and avoiding semantically unsafe regions in real time. By bridging open-world, multi-modal reasoning with dynamics-aware planning, we eliminate the need for hard-coded fallbacks and human safety interventions. FORTRESS outperforms on-the-fly prompting of slow reasoning models in safety classification accuracy on synthetic benchmarks and real-world ANYmal robot data, and further improves system safety and planning success in simulation and on quadrotor hardware for urban navigation. Website can be found at https://submfort.github.io/fortress/
@inproceedings{GanaiSinhaEtAl2025, author = {Ganai, M. and Sinha, R. and Agia, C. and Morton, D. and Pavone, M.}, title = {Real-Time Out-of-Distribution Failure Prevention via Multi-Modal Reasoning}, booktitle = {{Conf. on Robot Learning}}, year = {2025}, owner = {mganai}, note = {Submitted}, keywords = {sub}, url = {https://arxiv.org/abs/2505.10547}, timestamp = {2025-06-08} }