DeepLocalize: Fault Localization for Deep Neural Networks
Deep neural networks (DNNs) are becoming an integral part of most software systems. Although previous work has shown that DNNs have bugs, existing debugging techniques do not support localizing bugs because of the lack of understanding of models’ behavior. The entire DNN model appears as a black box. To address these problems, we propose an approach and a tool that automatically determines whether the model is buggy or not, and identifies its root causes. Our key insight is that historic trends in values propagated between layers can be analyzed to identify and localize faults. To that end, we first enable a dynamic analysis of deep learning applications by converting it into an imperative representation or alternatively using a callback mechanism. Both mechanisms allow us to insert probes that enable dynamic analysis over the traces produced by the DNN while it is being trained. Then, we conduct a dynamic analysis over the traces to identify the faulty layer or hyperparameter that causes the error. We propose an algorithm that identifies root causes by capturing any numerical error. It also finds the relevance of every layer/parameter on the DNN outcome by monitoring the training process. We have also collected a benchmark containing 40 buggy models and patches that contain real errors in deep learning applications from Stack Overflow and GitHub. Moreover, we shed light on the process of constructing a patch. Our benchmark can be used to evaluate automated debugging tools and repair techniques. We have evaluated our approach using this DNN bug-and-patch benchmark. The result shows that our approach is more efficient than the existing debugging approaches used in the state-of-the-practice Keras library. Our approach was able to detect 40 out of 40 faults, whereas the best debugging approach provided by Keras detects 34 out of 40 faults. Our approach was able to localize 30 out of 40 bugs, whereas Keras was able to localize 0 out of 29 bugs.
Committee: Hridesh Rajan (major professor), Myra Cohen, Wei Le, Simanta Mitra, and Wensheng Zhang
Email email@example.com for the videoconference link if you wish to view this exam.