Rachel Luo

Contacts:

Email: rsluo at stanford dot edu

Rachel Luo


Rachel Luo is a Ph.D. candidate in the Electrical Engineering department. She received a B.S. in Electrical Engineering and Computer Science from MIT in 2014, and an M.S. in Electrical Engineering from Stanford in 2017. Rachel’s research focuses on uncertainty quantification for problems at the intersection of computer vision and robotics.

In her free time, Rachel enjoys photography, rock climbing, hiking, and commuting by electric longboard.

Awards:

  • Stanford Graduate Fellowship
  • National Science Foundation (NSF) Fellowship

ASL Publications

  1. R. Luo, R. Sinha, A. Hindy, S. Zhao, S. Savarese, E. Schmerling, and M. Pavone, “Online Distribution Shift Detection via Recency Prediction,” in Proc. IEEE Conf. on Robotics and Automation, 2023. (Submitted)

    Abstract: When deploying modern machine learning-enabled robotic systems in high-stakes applications, detecting distributional shift is critical. However, most existing methods for detecting distribution shift are not well-suited to robotics settings, where data often arrives in a streaming fashion and may be very high-dimensional. In this work, we present an online method for detecting distributional shift with guarantees on the false positive rate — i.e., when there is no distribution shift, our system is very unlikely (with probability < ε) to falsely issue an alert; any alerts that are issued should therefore be heeded. Our method is specifically designed for efficient detection even with high dimensional data, and it empirically achieves up to 6x faster detection on realistic robotics settings compared to prior work while maintaining a low false negative rate in practice (whenever there is a distribution shift in our experiments, our method indeed emits an alert).

    @inproceedings{LuoSinhaEtAl2023,
      author = {Luo, R. and Sinha, R. and Hindy, A. and Zhao, S. and Savarese, S. and Schmerling, E. and Pavone, M.},
      booktitle = {{Proc. IEEE Conf. on Robotics and Automation}},
      title = {Online Distribution Shift Detection via Recency Prediction},
      year = {2023},
      note = {Submitted},
      url = {https://arxiv.org/abs/2211.09916},
      keywords = {sub},
      owner = {rdyro},
      timestamp = {2022-09-21}
    }
    
  2. R. Luo, S. Zhao, J. Kuck, B. Ivanovic, S. Savarese, E. Schmerling, and M. Pavone, “Sample-Efficient Safety Assurances using Conformal Prediction,” in Workshop on Algorithmic Foundations of Robotics, 2022.

    Abstract: When deploying machine learning models in high-stakes robotics applications, the ability to detect unsafe situations is crucial. Early warning systems can provide alerts when an unsafe situation is imminent (in the absence of corrective action). To reliably improve safety, these warning systems should have a provable false negative rate; i.e., of the situations than are unsafe, fewer than epsilon will occur without an alert. In this work, we present a framework that combines a statistical inference technique known as conformal prediction with a simulator of robot/environment dynamics, in order to tune warning systems to provably achieve an epsilon false negative rate using as few as 1/epsilon data points. We apply our framework to a driver warning system and a robotic grasping application, and empirically demonstrate guaranteed false negative rate and low false detection (positive) rate using very little data.

    @inproceedings{LuoZhaoEtAl2022,
      author = {Luo, R. and Zhao, S. and Kuck, J. and Ivanovic, B. and Savarese, S. and Schmerling, E. and Pavone, M.},
      title = {Sample-Efficient Safety Assurances using Conformal Prediction},
      booktitle = {{Workshop on Algorithmic Foundations of Robotics}},
      year = {2022},
      month = may,
      owner = {rsluo},
      timestamp = {2021-09-20},
      url = {https://arxiv.org/abs/2109.14082}
    }
    
  3. R. Luo, A. Bhatnagar, H. Wang, C. Xiong, S. Savarese, Y. Bai, S. Zhao, S. Ermon, E. Schmerling, and M. Pavone, “Local Calibration: Metrics and Recalibration,” in Proc. Conf. on Uncertainty in Artificial Intelligence, 2022.

    Abstract: Probabilistic classifiers output confidence scores along with their predictions, and these confidence scores should be calibrated, i.e., they should reflect the reliability of the prediction. Confidence scores that minimize standard metrics such as the expected calibration error (ECE) accurately measure the reliability on average across the entire population. However, it is in general impossible to measure the reliability of an individual prediction. In this work, we propose the local calibration error (LCE) to span the gap between average and individual reliability. For each individual prediction, the LCE measures the average reliability of a set of similar predictions, where similarity is quantified by a kernel function on a pretrained feature space and by a binning scheme over predicted model confidences. We show theoretically that the LCE can be estimated sample-efficiently from data, and empirically find that it reveals miscalibration modes that are more fine-grained than the ECE can detect. Our key result is a novel local recalibration method LoRe, to improve confidence scores for individual predictions and decrease the LCE. Experimentally, we show that our recalibration method produces more accurate confidence scores, which improves downstream fairness and decision making on classification tasks with both image and tabular data.

    @inproceedings{LuoEtAl2022,
      author = {Luo, R. and Bhatnagar, A. and Wang, H. and Xiong, C. and Savarese, S. and Bai, Y. and Zhao, S. and Ermon, S. and Schmerling, E. and Pavone, M.},
      title = {Local Calibration: Metrics and Recalibration},
      booktitle = {{Proc. Conf. on Uncertainty in Artificial Intelligence}},
      year = {2022},
      keywords = {pub},
      owner = {rdyro},
      timestamp = {2022-01-26},
      url = {https://arxiv.org/abs/2102.10809}
    }