Efficient Representation Learning for Heterogeneous Systems and Continual Task Adaptation

This dissertation addresses the fundamental challenges of system heterogeneity and continual task adaptation by advancing efficient fine-tuning and representation-learning strategies. It develops adaptive frameworks that allow distributed models to continuously acquire new knowledge and align knowledge without succumbing to communication bottlenecks or catastrophic forgetting.

First, we introduce a dual-distilled federated learning framework designed to address system heterogeneity and data heterogeneity across edge devices. By leveraging trainable global prototypes with adaptive margins, this approach aligns local representations and logits without requiring clients to share uniform model architectures, ensuring robust knowledge transfer and stable global convergence.

Next, we propose a highly communication-efficient federated framework for Large Language Models (LLMs) utilizing Representation Fine-Tuning. By intervening directly on hidden states rather than model weights and employing an "All-But-Me" aggregation strategy coupled with dynamic Test-Time Computing, this method drastically reduces communication payloads while optimally balancing local personalization and global generalization.

Finally, we present a mixture of Sparse Experts for Task-Agnostic Continual Learning (SETA) to resolve the plasticity-stability dilemma in continual learning for LLMs. This approach utilizes a dynamic Mixture of Sparse Experts to structurally decompose the parameter space, protecting shared representations from semantic drift while freezing task-specific features, enabling models to seamlessly acquire new skills without discarding previously learned knowledge.

Together, these contributions chart a cohesive path toward scalable, resource-aware, and highly adaptive machine learning systems that seamlessly bridge the gap between decentralized efficiency and continual real-world deployment.

Committee: Ali Jannesari (major professor), Wensheng Zhang, Meisam Mohammady, Ying Cai and Xiaoqiu Huang