Speaker:Waqwoya Abebe

Addressing challenges in decentralized learning and adaption of foundation models

This thesis explores two main topics: Chapters 2 and 3 address data heterogeneity in decentralized and federated learning, while Chapters 4 and 5 focus on the adaptation and optimization of the Segment Anything Model (SAM), a foundation model for image segmentation. We begin by tackling the challenges associated with data heterogeneity in Federated Learning (FL) and Fully Decentralized Learning (FDL). We introduce an innovative privacy-aware client clustering mechanism that facilitates stratified sampling in the FL context and helps optimize peer topology in the FDL setting. Our empirical findings indicate this approach enhances the representativeness of sampled clients in FL and local topology in FDL, significantly boosting the convergence and accuracy of the global model (or peer models). As a result, the proposed method substantially reduces communication overhead among participating devices, making it an advantageous solution. Shifting our focus to foundation models, we propose a means of adapting SAM for the materials science domain. Specifically, we propose a technique called semantic boosting to enhance zero-shot semantic segmentation of micrographs. To accomplish this, we develop a post-processing engine, SAM-I-Am, which markedly improves SAM’s zero-shot segmentation results. Finally, we address the challenge of model size by streamlining SAM via one-shot Neural Architecture Search (NAS). In particular, we develop a novel search space design involving structured pruning and parameter prioritization to transform SAM into a weight-sharing supernetwork. This enables the discovery of efficient subnetworks that perform comparably to the pre-trained SAM model.

Committee: Ali Jannesari (major professor), Qi Li, Wallapak Tavanapong, Ying Cai, and Wensheng Zhang