Efficient Scaling of LLMs via Optimization-Aware Architecture Design

The success of large language models (LLMs) is driven in large part by their scale. However, continued scaling is increasingly constrained by compute, data, and deployment costs. This talk targets at efficiently scaling by making neural networks wider and deeper through an optimization-aware architecture design approach. For width scaling, we show that imposing appropriate manifold structures on linear layers can provably alleviate compute and data bottlenecks. We then demonstrate how these ideas translate into practice for LLM fine-tuning, and briefly discuss recent extensions toward pretraining. For depth scaling, we show that the topology of residual connections can make an exponential difference in terms of convergence. Building on this insight, we develop principled residual-connection designs that improve performance across LLM pretraining, diffusion transformers, and reinforcement learning tasks.

About Dr. Li

Bingcong Li is a postdoctoral researcher at ETH Zurich. He received his B.Eng. (highest honors) in Information Science and Engineering from Fudan University (2017) and his Ph.D. in Electrical and Computer Engineering from the University of Minnesota (2022). His research focuses on the foundations of deep learning, with an emphasis on efficient pretraining, fine-tuning, and deployment of large models. He will present a tutorial on LLM fine-tuning at ICASSP 2026. He has received multiple fellowship awards from the University of Minnesota and Fudan University.