Improving robustness in large and multimodal language models under variations in input context

Large language models (LLMs) and multimodal language models (MLMs) have demonstrated strong performance across a wide range of reasoning and information extraction tasks. However, strong performance under standard evaluation settings does not necessarily imply that these models behave robustly when the same underlying information is presented differently. This dissertation investigates how variations in information presentation affect the robustness of LLMs and MLMs, with a particular focus on reasoning and chart understanding. Across four studies, we investigate how the presence of irrelevant information in the input context, variations in question formulation, and differences in how numerical information is presented on chart axes affect the robustness of large and multimodal language models. We further examine why multimodal models fail to generate numerical values accurately and explore ways to mitigate these failures.

The first study investigates the effect of irrelevant information on quantitative reasoning. We evaluate multiple prompting methods on mathematical word problems augmented with controlled irrelevant information and find that prompting strategies that perform well on the original problems can degrade substantially when irrelevant information is introduced. To improve robustness, we introduce prompting approaches that explicitly identify and remove irrelevant information before solving the problem, substantially reducing the performance degradation caused by noises.

The second study examines whether model performance changes when the same underlying reasoning problem is asked in different ways. We show that performance depends not only on the reasoning required by a problem but also on how the question is formulated. In particular, question type, answer-option structure, and wording can affect both reasoning accuracy and final answer selection. These results show that evaluation based on a single question formulation may not fully characterize the reasoning capability of an LLM.

The third study extends this investigation to multimodal chart understanding. We introduce a controlled framework for evaluating chart-to-table translation under variations in y-axis representation, including digit length, axis range, and tick-label format. The results reveal substantial performance differences across visually equivalent charts, with increasing digit length producing particularly severe degradation for several models. These findings demonstrate that numerical information encoded in a chart can be extracted differently depending on how the same values are visually represented.

Building on this observation, the fourth study investigates why multimodal models struggle with longer numerical values and how these failures can be mitigated. We find that models may predict the leading digits correctly while generating an incorrect digit length. Fine-tuning improves performance on digit lengths seen during training but transfers poorly to unseen lengths, while occluding individual y-axis tick labels highlights the importance of visual axis references.

Linear probing further reveals that digit-length information is encoded in the models' internal visual representations, even when the final generated value has an incorrect digit length. This exposes a gap between the information available in a model's representations and its numerical output. Finally, we investigate a relative representation of chart values based on y-axis ticks, which reduces digit-length sensitivity and improves generalization to unseen digit lengths in our evaluations.

Overall, this dissertation shows that the performance of language and multimodal language models can depend strongly on how input context is presented, even when the underlying task or data remain unchanged. The results highlight the importance of evaluating robustness across variations in context, question formulation, and visual and numerical representation, and show that both diagnostic analysis and alternative representations can help identify and reduce these sources of instability.

Committee: Wallapak Tavanapong (major professor), Mengdi Huai, Qi Li, Ying Cai, and Julie Dickerson