Automatically Finding Bugs in Deep Learning Models
Deep Neural Networks (DNNs) are increasingly used in a wide range of safety-critical applications (e.g., autonomous driving or medical diagnosis systems). However, like traditional software, DNN-based software has faults too, which manifest as poor model performance. Unfortunately, existing debugging techniques do not support localizing and fixing DNN bugs, as the entire model appears as a black box. Furthermore, Deep Learning (DL) programs are fundamentally different from traditional software due to the nonlinearity of the functions. Current DNN bug localization approaches do not understand model behavior, support silent bugs, and correlate bug symptoms with their fixes. Thus it hinders the developers' ability to ensure the reliability of their systems under design. This dissertation presents three novel techniques for debugging deep neural networks (DNN). These novel techniques are the first to identify, diagnose, and localize faults and provide fix suggestions for DNN models. The first approach, DeepLocalize, converts the DNN into an imperative representation and uses probes to monitor its parameters and layers at training time. DeepLocalize can accurately identify and localize the root causes of numerical bugs, making it more effective than prior works. The second approach, DeepDiagnosis, diagnoses eight types of symptoms while providing actionable fix messages to patch buggy DNN models. DeepDiagnosis can detect silent bugs, which do not result in numerical errors during training. We evaluate DeepDiagnosis on a comprehensive set of 444 models, including 53 real-world examples from GitHub and Stack Overflow and 391 models curated by AUTOTRAINER. Our evaluation shows that DeepDiagnosis has better accuracy and performance than prior works, such as UMLUAT and DeepLocalize. Finally, our third approach, Deep4Deep, is a data-driven technique that leverages semantic features from DNN models to detect and diagnose faults. This technique extracts the faults' semantic information during DNN training while leveraging it as a training dataset to learn and infer fault patterns. This approach automatically links bug symptoms to their root causes. Thus, differently from prior works, Deep4Deep does not rely on manually crafted symptoms to root cause mappings. We evaluated our approach on benchmarks with real-world and mutated models. We observed that it effectively detects and diagnoses various bug types. In particular, our evaluation shows that Deep4Deep outperforms prior works for mutated models in terms of accuracy, precision, and recall while achieving comparable results for real-world models. Overall, our approaches exhibit superior fault detection, bug localization, and symptom identification capabilities, thus helping developers efficiently and effectively patch DNN models. Lastly, this dissertation provides a comprehensive investigation and evaluation of our proposed techniques, shedding light on their effectiveness and limitations while providing several future work research directions.
Committee: Hridesh Rajan (major professor), Myra Cohen, Wei Le, Wensheng Zhang and Simanta Mitra
Join on Zoom: https://iastate.zoom.us/j/95559177810