Toward Efficient and Reliable Large Language Models

Large language model inference becomes slower and more memory-intensive when the model is large or the input is long. The key aim of this thesis is to enhance LLM inference by improving throughput and reducing memory consumption. To achieve this, we investigate inference efficiency from two perspectives: the model and the context.

From the model perspective, we have developed a low-rank compression method that reduces model size by selecting and pruning bases in the model’s weight matrices. The method relearns the singular value associated with each basis and uses the magnitude of the relearned singular value to determine which bases to retain and which to prune. Building on this foundation, we plan to link basis selection directly to task performance by estimating the loss change caused by removing each basis rather than relying solely on magnitude. Since low-rank compression can change model behavior in ways that are not captured by task performance, we have also studied how low-rank compression affects privacy, adversarial robustness, ethics, and fairness, showing that these properties respond differently to compression. Beyond trustworthiness, low-rank compression may also alter the capabilities required by LLM agents, including tool use, workflow execution, reasoning, and instruction following. We therefore plan to study how these capabilities change under compression and investigate ways to preserve them.

From the context perspective, we plan to improve inference efficiency by reducing the cost of processing long inputs. The proposed context compression technique will leverage semantic trajectories and relationships among context spans to retain the information needed for reasoning while removing redundant content.

Committee: Yang Li (major professor), Hongyang Gao, Chenglin Miao, Wensheng Zhang, and Yan-Bin Jia