Русская версия | English

Visual Math VLM Evaluation

A reproducible pipeline for evaluating open vision-language models on charts, geometry, coordinate systems, and matrix questions. The repository runs real local multimodal inference, records every prediction, and reports quality together with latency and parsing reliability.

Results

The committed benchmark uses the 40-example synthetic dev split and deterministic greedy decoding.

Model	Parameters	Accuracy	Parse rate	Median latency
Uniform random baseline	—	25.0% expected	—	—
Majority-answer baseline	—	47.5%	—	—
SmolVLM-500M-Instruct	0.5B	40.0%	100%	5.06 s
Qwen3-VL-2B-Instruct	2B	50.0%	100%	1.24 s

The Qwen3-VL baseline solves all 10 chart questions and reaches 60% on the five linear-algebra examples. Coordinate geometry remains the clearest failure mode at 0/5. The majority baseline is unusually strong because 19 of 40 gold answers are C; it is reported alongside the uniform random baseline to avoid overstating a small evaluation set.

Machine-readable artifacts:

• Qwen3-VL metrics and predictions;
• SmolVLM metrics and predictions.

What the project demonstrates

• manifest-backed visual-question datasets with explicit train/dev/test splits;
• real local image-text generation through Hugging Face Transformers;
• model-agnostic CPU, Apple MPS, and CUDA device selection;
• strict multiple-choice parsing and preservation of raw model output;
• accuracy, parse rate, per-subject metrics, median latency, and p95 latency;
• reproducible prediction and metrics artifacts rather than handwritten results;
• reusable visual-to-text adapter, visual-token merge, processor, and training-smoke components;
• 15 unit tests and a Python 3.11–3.13 CI matrix.

Quick start

The project uses Python 3.10+ and uv.

git clone https://github.com/erstcl/vlm-math.git
cd vlm-math
uv sync --extra dev --extra inference

Run the default Qwen3-VL evaluation:

uv run visual-math-vlm \
  --manifest assets/math_vqa_medium/manifest.jsonl \
  --split dev \
  --device auto \
  --output-dir reports/qwen3-vl-2b-dev

Compare a different compatible model:

uv run visual-math-vlm \
  --manifest assets/math_vqa_medium/manifest.jsonl \
  --split dev \
  --model HuggingFaceTB/SmolVLM-500M-Instruct \
  --output-dir reports/smolvlm-500m-dev

Model weights are downloaded from their official Hugging Face repositories and are never committed here.

Evaluation protocol

Setting	Value
Dataset	math_vqa_medium, synthetic dev split
Samples	40
Subjects	algebra, coordinate geometry, geometry, linear algebra, plots
Decoding	greedy, do_sample=False
Generation budget	64 new tokens
Primary hardware	Apple M1, 16 GB unified memory
Runtime	Python 3.14, PyTorch 2.13, Transformers 5.14, MPS

Latency covers model.generate for one image-question pair after model loading. It excludes checkpoint download and initialization. The benchmark is an engineering comparison on a small synthetic set, not a claim about MathVista or broad mathematical reasoning quality.

Architecture

JSONL manifest + images
          │
    MathVQADataset
          │
 strict visual-math prompt
          │
 TransformersVLMBackend ── Qwen / SmolVLM / compatible model
          │
 raw generation + answer parser + latency
          │
 predictions.jsonl + metrics.json

The pretrained evaluation path is deliberately separate from the small adapter components in vlm_math/model.py. This keeps model benchmarking real while retaining the independently testable visual-token and adapter implementation.

Adapter smoke path

The repository also includes a tiny deterministic training path for validating the processor, label masking, backward pass, and checkpoint plumbing without claiming model quality:

uv run python -m vlm_math.train \
  --config configs/adapter_smoke.yaml \
  --fast-train

The verified smoke run completes two CPU optimization steps and reduces loss from 8.4342 to 7.6608. This is a technical contract test, not the reported VLM benchmark.

Project structure

vlm_math/                    # dataset, evaluator, adapter, processor, smoke training
tests/                       # 15 unit tests
assets/toy_math_vqa/         # interface smoke fixtures
assets/math_vqa_medium/      # 260-example synthetic visual-math dataset
reports/                     # committed metrics and prediction records
configs/adapter_smoke.yaml   # deterministic training smoke configuration
.github/workflows/ci.yml     # lint, format, and tests

My contribution

• designed the manifest and preprocessing contracts for visual-math examples;
• implemented and tested the visual adapter and visual-token merge;
• built the model-agnostic Transformers inference backend;
• designed the evaluation protocol and answer parser;
• benchmarked two open VLMs on Apple MPS and analyzed errors by subject;
• packaged the code, CLI, tests, reports, and CI as a reproducible project.

Data and licenses

The included visual-math images and questions are synthetic. See DATA_SOURCES.md for scope and limitations.

Project code is released under the MIT License. Model weights retain their upstream licenses: Qwen3-VL-2B-Instruct and SmolVLM-500M-Instruct are distributed under Apache 2.0 at the time of this benchmark.