Introducing simple-evals-mm

This post introduces simple-evals-mm, a new VLM evaluation framework we have developed.

What is simple-evals-mm?

simple-evals-mm is a framework for evaluating the performance of vision-language models (VLMs). It follows the design of OpenAI’s LLM evaluation framework simple-evals and extends it to VLM evaluation.

Because it inherits the design of simple-evals, it has a simple, easy-to-use interface, and adding new models and tasks is straightforward. On top of that, it comes with features we added ourselves that are useful in practice, such as plotting evaluation results and analyzing model outputs.

Supported Tasks

The following tasks are currently supported.

Japanese

• Multimodal tasks: JAMMEval collection (CC-OCR, CVQA, Heron-Bench, JA-Multi-Image-VQA, JA-VLM-Bench, JDocQA, JGraphQA), BusinessSlideVQA, JMMMU, MECHA-ja

JAMMEval is a benchmark collection built by manually refining the flawed examples in existing Japanese benchmarks. For details, see this post I published recently. With simple-evals-mm, you can reproduce the JAMMEval evaluation results.

English

• Multimodal tasks: AI2D, BLINK, ChartQA, CountBenchQA, DocVQA, InfoVQA, MMMU, OKVQA, RealWorldQA, ScienceQA, SeedBench-v2, TextVQA

• Text-only tasks: GPQA, MATH, MMLU, SimpleQA

Supported Models

GPT-5, Gemini 3 Pro, InternVL, Qwen-VL, Sarashina, and LLM-jp-VL are currently supported. The table below lists the model name prefixes available for each backend.

Adding tasks and models is easy, so please give it a try.

Handy Features

Plotting Evaluation Results

simple-evals-mm can also output plots of evaluation results, like the one below. You can use it, for example, to create the benchmark result figures for a model release post.

uv run python src/simple_evals_mm/visualize.py --evals heronbench

Analyzing Model Outputs

simple-evals-mm also includes a viewer for analyzing model responses. It lets you inspect model outputs alongside the actual images and questions, giving you a deeper understanding of how the model behaves. You can annotate errors at the same time, so it is also handy for categorizing errors.

uv run python -m simple_evals_mm.viewer.app
# Opens http://localhost:5001

Conclusion

simple-evals-mm, introduced in this post, is publicly available on GitHub and free for anyone to use. If you use it, I would be glad if you starred the repository on GitHub or sent us feedback.

References

• simple-evals-mm
  • The GitHub repository of simple-evals-mm introduced in this post.
• simple-evals
  • OpenAI’s LLM evaluation framework, which served as a reference for the design of simple-evals-mm.
• JAMMEval
  • simple-evals-mm was developed as part of the JAMMEval project.