Modularity-inspired Techniques for Patching the Weak Components in Deep Learning
Modularity, a core principle in software engineering, involves breaking down software into independent modules that can be reused, replaced, or repaired in isolation. This approach ensures that weak components can be effectively repaired without impacting the rest of the system, thereby maintaining system reliability and reducing maintenance costs through module reusability. The concept has been adapted to deep learning models, offering similar benefits such as reusability and replaceability. However, existing modular approaches are limited in their applicability, particularly to natural language processing models, and struggle with composing modules trained on different data distributions. To address these concerns, our first approach demonstrates that recurrent neural network-based (RNN) NLP models can be effectively decomposed into modules for direct replacement and repair of weaker components, with only a 0.6% average loss in accuracy. We introduce a temporal-order preserving technique by unrolling the feedback loop within RNNs, which enhances modular benefits such as reusability. Our demonstration illustrates that RNNs can be augmented with new natural language capabilities, and issues such as faulty behavior can be addressed through module replacement. In the second approach, we address the limitations of state-of-the-art methods for composing modules across different datasets. Existing voting-based techniques often perform poorly in inter-dataset scenarios. We propose three novel methods that significantly improve module reusability and performance, as shown in our evaluation of 92 convolutional neural network-based modules. This work shows that composition is key to facilitating the effective reuse of DL modules, which is essential for successfully repairing weak models by integrating replaced modules with existing ones. Finally, we present a modularity-inspired technique for repairing weak components in large language models. By isolating and targeting the most error-prone modules within transformer architectures, our method improves repair effectiveness by 43.6% and reduces performance disruption by 46%. Specifically, we treat each layer block within the transformer architecture as a distinct module and identify the weakest module for focused repair. This targeted approach effectively addresses biases and inaccuracies in large language models, as demonstrated in our results. Lastly, this dissertation presents an in-depth analysis and assessment of our proposed techniques, highlighting their effectiveness, efficiency, and limitations. It also offers suggestions for future research in the evolving field of modularity for deep learning.
Committee: Hridesh Rajan (major professor), Wei Le, Myra Cohen, Hongyang Gao, and Pavan Aduri
Join on Zoom: https://iastate.zoom.us/j/96839034774