Fault Localization for Deep Neural Networks
Deep neural networks (DNNs) are becoming an integral part of most software systems. Previous work has shown that DNNs have bugs. Unfortunately, existing debugging techniques don’t support localizing and fixing DNN bugs because of the lack of understanding of model behaviors, inexplicit errors, and several options to fix it. To address these problems, we propose a different approach and a tool (called DeepLocalize) that automatically determines whether the model is buggy or not, and identifies the root causes. Our key insight is that historic trends in values propagated between layers can be analyzed to identify and localize faults. To that end, we conduct dynamic analysis over the traces to identify the faulty layer or hyperparameter that causes the error. We propose an algorithm for identifying root causes by capturing any numerical error and monitoring the model during training, and finding the relevance of every layer/parameter on the DNN outcome. We have collected a benchmark and patches that contain real errors from Stack Overflow and GitHub. Our benchmark can be used to evaluate automated debugging tools and repair techniques. We have evaluated our approach using this benchmark, and the results showed that our approach is much more effective than the existing debugging approach used in the state-of-the-practice Keras library.
Building on this preliminary work, we propose to explore the following next steps: First, a novel model debugging technique that localizes faults, reports error symptoms, and provides suggestions to fix structural bugs. Second, we propose a powerful representation-learning algorithm to automatically learn a semantic representation of model behavior during training. Third, we propose a novel technique inspired by Spectrum-based technique that identifies and localizes bugs in DNNs. Finally, we propose a novel instrumentation framework for Deep learning applications, which maps and instruments statistics data for bug detection and fixing DNN models. It also reduces time and efforts without required an expert. We will leverage the methodology used to evaluate DeepLocalize tool to evaluate the proposed work.
Committee: Hridesh Rajan (major professor), Myra Cohen, Wei Le, Simanta Mitra, and Wensheng Zhang.
Join on Zoom: Please click this URL to start or join. https://iastate.zoom.us/j/92985877376?pwd=K3VtaC9TMGx0cFVmRGxYb3dHT3ZaZz09 Or, go to https://iastate.zoom.us/join and enter meeting ID: 929 8587 7376 and password: 128281 Join from dial-in phone line: Dial: +1 312 626 6799 or +1 646 876 9923 Meeting ID: 929 8587 7376 Participant ID: Shown after joining the meeting International numbers available: https://iastate.zoom.us/u/acbmGR1Vn