[Feat] Add VLMEvalKit integration for nanoVLM evaluation

[khurramHashmi]

This PR integrates VLMEvalKit into nanoVLM as a second evaluation framework alongside the existing lmms-eval integration, enabling evaluation across a broader set of multimodal benchmarks with minimal setup.

What's added

VLMEvalKit adapter (eval/vlmevalkit_adapter.py): A BaseModel subclass that wires nanoVLM's tokenizer, image processor, and VisionLanguageModel into the VLMEvalKit inference API. Handles per-dataset
prompt formatting (MMBench, MMMU, MathVista, ChartQA, DocVQA, TextVQA, MCQ) and supports both image-splitting and non-splitting model variants.

Setup script (eval/setup_vlmevalkit.sh): Idempotent script that initialises the VLMEvalKit submodule, installs it in editable mode, copies the adapter, and patches VLMEvalKit's __init__.py and
config.py to register nanoVLM-460M-8k and nanoVLM-230M-8k.

Launcher script (eval/eval_vlmevalkit.sh): Convenience wrapper with sensible defaults (nanoVLM-460M-8k on MMStar). Accepts custom model and benchmark arguments, and forwards optional LLM-judge env
vars for benchmarks like MMBench.

Smoke test (eval/test_vlmevalkit.py): Covers text-only, image+text, and MCQ inference paths.

Usage

# One-time setup
bash eval/setup_vlmevalkit.sh

# Run default: nanoVLM-460M-8k on MMStar
bash eval/eval_vlmevalkit.sh

# Custom model + benchmarks
bash eval/eval_vlmevalkit.sh nanoVLM-222M MMStar MME OCRBench

# Smoke test
python eval/test_vlmevalkit.py

Other changes

• models/vision_language_model.py: filters unknown keys from old checkpoint configs for backward compatibility; exposes _saved_config_keys so adapters can detect which optional fields were actually
  persisted.
• README.md: adds --recurse-submodules clone note, VLMEvalKit eval section, and marks the roadmap item as done.
• VLMEvalKit added as a git submodule pointing to a fork with the nanoVLM adapter pre-registered.

Acknowledgments

This integration builds upon the excellent work of the https://github.com/open-compass/VLMEvalKit project from OpenCompass, which provides a unified framework for evaluating large vision-language models
across a wide range of multimodal benchmarks.

Note

This PR maintains nanoVLM's philosophy of simplicity and minimal dependencies while complementing the existing lmms-eval integration. VLMEvalKit and lmms-eval serve overlapping but distinct benchmark ecosystems, and having both lowers friction for researchers who prefer one over the other. The integration is designed to be entirely opt-in and non-intrusive to existing training and inference workflows.