Beyond single plots: a benchmark for question answering on scientific multi-charts

Charts are widely used in many fields, where interpreting them correctly is key to informed decisions. In many real-world scenarios, a single chart is insufficient; instead, multiple relevant charts must be understood together to extract meaningful insight and make decisions. Research on understanding multi-chart images has not been extensively explored. In this work, we introduce PolyChartQA, a large-scale dataset specifically designed for question answering over multi-chart figures. PolyChartQA comprises 534 multi-chart images (with a total of 2,297 sub-charts) sourced from peer-reviewed computer science research publications and 2,565 question-answer pairs annotated across diverse question types and difficulty levels. We evaluate the performance of eight state-of-the-art Multimodal Language Models (MLMs) on PolyChartQA across question type, difficulty, question source, and key structural characteristics of multi-charts. Our findings reveal the following insights. (1) Our proposed prompting method improves L-accuracy (LLM-based accuracy) up to 11.03% for multi-chart question answering. (2) Significant gaps remain for MLMs to understand multi-charts correctly, as evidenced by a 27.7% L-accuracy drop on human-authored vs. MLM-generated questions.

Committee: Wallapak Tavanapong (major professor), Qi Li, and Clay Stevens