Speaker:Eliska Kloberdanz

Understanding and Improving Numerical Stability of Deep Learning Algorithms

Deep learning (DL) has become an integral part of solutions to various important problems, which is why ensuring the quality of DL systems is essential. One of the challenges of achieving reliability and robustness of DL software is to ensure that algorithm implementations are numerically stable. Numerical stability is a property of numerical algorithms, which governs how changes or errors introduced through inputs or during execution affect the accuracy of algorithm outputs. In numerically unstable algorithms, those errors are magnified and adversely affect the fidelity of algorithm’s outputs via incorrect or inaccurate results. In this thesis we analyze the numerical stability of DL algorithms. First, we identify and analyze unstable numerical methods and their solutions in DL. We find that quantization of neural networks can cause numerical stability issues and to this end, we propose a new quantization algorithm that optimizes the trade-off between low-bit representation and loss of precision. Finally, we study numerical stability and its impact on robustness of residual networks leveraging ordinary differential equations.

Committee: Wei Le (major professor), Hongyang Gao, Chris Quinn, Hailiang Liu, Zhengdao Wang

Join on WebEx: https://iastate.webex.com/iastate/j.php?MTID=md42d8c5304d332a6c4df84442879f537
Meeting number: 2623 580 0187
Password: AyTWasAx367
Access code: 262 358 00187