Spatial + textual learning for charts
1 University of Wisconsin–Madison
2 Amazon AGI 3 MIT
Learning to understand both the visual structure of a chart and the data behind it.
01 / Motivation
Reading a chart means understanding where things are and what they represent.
Charts combine a structured visual layout with an underlying data representation. A model must locate the right subplot, axis, or legend before it can reliably interpret the values and answer a question. START brings these two properties into training through chart-element grounding and chart-to-code generation, alongside chart question answering.
We introduce START-Dataset to support this learning and CS-Bench to evaluate how well models understand chart structure. This page highlights the publicly released, reproduced dataset, benchmark, and checkpoint.
Chart images paired with code, element locations, and question-answer supervision.
Direct grounding and question answering that requires locating chart elements.
Spatial and textual learning improves the released Qwen2.5-VL-7B checkpoint.
02 / Training resource
Real chart complexity. Executable code. Spatial supervision.
The data-generation pipeline starts from real chart images and reconstructs them as Python code. It then evolves that code to extract chart-element locations during rendering. Together with question-answer pairs, these annotations connect visual layout to the data represented in each chart.
Translate a real chart into executable Python, recovering its data representation and visual layout.
Evolve the rendering code to expose positions of visual elements and produce spatial annotations.
Combine chart QA, element grounding, and chart-to-code supervision for SFT and RL.
| Question type | Count | Share |
|---|---|---|
| Number-in-General | 145,676 | 36.43% |
| Chart to code (with QA) | 111,032 | 27.77% |
| Text-in-General | 68,917 | 17.23% |
| Text-in-Chart | 27,262 | 6.82% |
| Number-in-Chart | 23,284 | 5.82% |
| Multiple Round Agentic | 19,789 | 4.95% |
| Other | 3,937 | 0.99% |
03 / Evaluation resource
Can a model locate the evidence behind its answer?
The Chart Spatial understanding Benchmark tests a capability that answer-only evaluation does not directly measure: identifying and localizing chart structure. It includes titles, axis labels and ticks, legends, subplots, and inset subplots.
Locate a specified visual element and return its bounding-box coordinates.
“Locate the name of the y-axis of subplot in row 1 and column 1, output its bbox coordinates using JSON format.”
Example from the released prediction records.
Evaluates localizationAnswer a question about a chart element and identify the relevant region. This tests whether the answer is connected to the correct visual structure.
Questions link textual content with its location in the chart, so answer correctness and localization can be assessed separately.
Evaluates answers + localization
Recall@0.3 measures localization at an intersection-over-union (IoU) threshold of 0.3. Answer accuracy measures correctness on QA-grounding questions. The provided evaluator also reports Recall@0.5.
Download the benchmark, run the supplied inference script, and score the model outputs with the repository’s evaluator. Compare against the reported release baselines.
Evaluation instructions04 / Learning framework
Teach the model to answer, locate, and reconstruct.
START combines chart question answering with two complementary training tasks. The framework studies both supervised fine-tuning and reinforcement learning to strengthen chart understanding.
Predict the location of the element described by an instruction. Explicit spatial supervision connects language to chart layout.
chart + instruction → coordinatesGenerate executable Python that reconstructs the chart. Code represents data values and how those values are visualized.
chart + instruction → PythonUse both spatial structure and textual details to answer questions about the chart.
chart + question → answerSee Section 3.1 of the paper for the learning objectives and reward design.
05 / Reported release results
Better spatial grounding, with gains across chart tasks.
The public START-RL-7B checkpoint is initialized from Qwen2.5-VL-7B and trained on the reproduced START dataset. The table below reports the repository’s released checkpoint results, rather than the original paper’s experimental table.
| Model | CharXiv Descriptive | CharXiv Reasoning | ChartQAPro | ChartMimic | CS-Bench Recall@0.3 | CS-Bench Accuracy |
|---|---|---|---|---|---|---|
| Qwen2.5-VL-7B | 73.0 | 43.2 | 41.9 | 47.1 | 23.1 | 50.8 |
| START-RL-7B | 78.5 | 47.6 | 47.2 | 69.9 | 48.6 | 65.5 |
Source: released checkpoint table. Bold marks the better of the two reported scores. Improvements are absolute score differences. Tables scroll horizontally on small screens.
06 / Inside the data
A chart is more than an image and an answer.
The paper’s dataset illustration shows how one chart can support complementary supervision: recovering Python code, locating visual elements, and answering questions about their content.
The reproduced SFT dataset also includes conversations in which a chart-analyzing agent requests visual operations before answering. In the example below, it highlights four subplots to compare the direction of their trends.
How many subplots show both lines increasing from All Rp to Rp/Re < 10?
Draw bounding boxes around the Quiescent, Star Forming, AGN, and Composite subplots to inspect both lines.
Only the AGN subplot shows both lines increasing. The final answer is 1.
Condensed from the released SFT conversation example; this is an annotation example, not a live model demo.
07 / Explore & reproduce
Start with the data. Evaluate with CS-Bench.
Prepared SFT and RL images and annotations.
BenchmarkChart-element grounding and QA-grounding evaluation.
Data generationGenerated and evolved Python code for chart rendering.
ModelThe publicly released Qwen2.5-VL-7B-based checkpoint.
RepositoryData preprocessing, evaluation scripts, and saved outputs.
PaperMethod, original experiments, ablations, and appendices.
08 / Reference
Please cite START if you use the method, dataset, or benchmark in your research.
@misc{liu2025startspatialtextuallearning,
title={START: Spatial and Textual Learning for Chart Understanding},
author={Zhuoming Liu and Xiaofeng Gao and Feiyang Niu and Qiaozi Gao and Liu Liu and Robinson Piramuthu},
year={2025},
eprint={2512.07186},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2512.07186},
}Contact: Zhuoming Liu