Agentic Mixture-of-Experts for Continual Learning

Neural networks trained sequentially on multiple tasks tend to overwrite previously learned knowledge, a phenomenon known as catastrophic forgetting. This project designs and evaluates a continual learning framework that mitigates forgetting through task-conditioned Mixture-of-Experts (MoE) modularity combined with Orthogonal Gradient Projection (OGP). A compact SwiGLU transformer is converted from a dense architecture into an MoE model, where a controller assigns each expert one of three structural actions Share, Clone, or Freeze, at every task transition, and OGP mathematically removes gradient components that would otherwise interfere with previously learned representations. The system is evaluated across a sequential learning setting (PIQA, SIQA, BoolQ) using Backward Transfer as the primary retention metric, with results validated across both single-run and multi-seed (35-run) experiments. Dense sequential fine-tuning produced consistently negative backward transfer, while every MoE configuration improved retention relative to this baseline; the strongest configuration achieved positive backward transfer with low variance across seeds. OGP was independently validated through a formal unit-test suite confirming its mathematical correctness. Comparing PPO-based, heuristic, and random controllers showed that, with only two task transitions, PPO did not clearly outperform the simpler baselines a limitation attributed to insufficient experience for policy convergence rather than a flaw in the underlying architecture. This work contributes a reproducible, empirically grounded framework for structural continual learning at research scale.

Major Professor: Simanta Mitra