Rachel Luo

Rachel Luo


Rachel Luo is a Ph.D. candidate in the Electrical Engineering department. She received a B.S. in Electrical Engineering and Computer Science from MIT in 2014, and an M.S. in Electrical Engineering from Stanford in 2017. Rachel’s research focuses on uncertainty quantification for problems at the intersection of computer vision and robotics.

In her free time, Rachel enjoys photography, rock climbing, hiking, and commuting by electric longboard.

Awards:

  • Stanford Graduate Fellowship
  • National Science Foundation (NSF) Fellowship

ASL Publications

  1. R. Luo, S. Zhao, J. Kuck, B. Ivanovic, S. Savarese, E. Schmerling, and M. Pavone, “Sample-Efficient Safety Assurances using Conformal Prediction,” 2022. (Submitted)

    Abstract: When deploying machine learning models in high-stakes robotics applications, the ability to detect unsafe situations is crucial. Early warning systems can provide alerts when an unsafe situation is imminent (in the absence of corrective action). To reliably improve safety, these warning systems should have a provable false negative rate; i.e., of the situations than are unsafe, fewer than epsilon will occur without an alert. In this work, we present a framework that combines a statistical inference technique known as conformal prediction with a simulator of robot/environment dynamics, in order to tune warning systems to provably achieve an epsilon false negative rate using as few as 1/epsilon data points. We apply our framework to a driver warning system and a robotic grasping application, and empirically demonstrate guaranteed false negative rate and low false detection (positive) rate using very little data.

    @inproceedings{LuoZhaoEtAl2022,
      author = {Luo, R. and Zhao, S. and Kuck, J. and Ivanovic, B. and Savarese, S. and Schmerling, E. and Pavone, M.},
      title = {Sample-Efficient Safety Assurances using Conformal Prediction},
      year = {2022},
      month = may,
      note = {Submitted},
      keywords = {sub},
      owner = {rsluo},
      timestamp = {2021-09-20},
      url = {https://arxiv.org/abs/2109.14082}
    }
    
  2. R. Luo, A. Bhatnagar, H. Wang, C. Xiong, S. Savarese, Y. Bai, S. Zhao, S. Ermon, E. Schmerling, and M. Pavone, “Local Calibration: Metrics and Recalibration,” 2022. (Submitted)

    Abstract: Probabilistic classifiers output confidence scores along with their predictions, and these confidence scores should be calibrated, i.e., they should reflect the reliability of the prediction. Confidence scores that minimize standard metrics such as the expected calibration error (ECE) accurately measure the reliability on average across the entire population. However, it is in general impossible to measure the reliability of an individual prediction. In this work, we propose the local calibration error (LCE) to span the gap between average and individual reliability. For each individual prediction, the LCE measures the average reliability of a set of similar predictions, where similarity is quantified by a kernel function on a pretrained feature space and by a binning scheme over predicted model confidences. We show theoretically that the LCE can be estimated sample-efficiently from data, and empirically find that it reveals miscalibration modes that are more fine-grained than the ECE can detect. Our key result is a novel local recalibration method LoRe, to improve confidence scores for individual predictions and decrease the LCE. Experimentally, we show that our recalibration method produces more accurate confidence scores, which improves downstream fairness and decision making on classification tasks with both image and tabular data.

    @article{LuoEtAl2022,
      author = {Luo, R. and Bhatnagar, A. and Wang, H. and Xiong, C. and Savarese, S. and Bai, Y. and Zhao, S. and Ermon, S. and Schmerling, E. and Pavone, M.},
      title = {Local Calibration: Metrics and Recalibration},
      year = {2022},
      note = {Submitted},
      keywords = {sub},
      owner = {rdyro},
      timestamp = {2022-01-26},
      url = {https://arxiv.org/abs/2102.10809}
    }