Developing Japanese CLIP Models Leveraging an Open-weight LLM for Large-scale Dataset Translation

As part of the Multimodal Working Group at LLM-jp, we built a translated Japanese image-text pair dataset of about 2 billion examples and trained Japanese CLIP models from scratch.

The released resources are listed below.

Image-text pair dataset

• https://huggingface.co/datasets/llm-jp/relaion2B-en-research-safe-japanese-translation

Models

• https://huggingface.co/llm-jp/llm-jp-clip-vit-base-patch16
• https://huggingface.co/llm-jp/llm-jp-clip-vit-large-patch14

Japanese CLIP evaluation code

• https://github.com/llm-jp/clip-eval

Introduction

Training CLIP requires a large-scale dataset of (image, text) pairs. However, open Japanese image-text pair datasets are scarce, which has been a barrier to developing Japanese CLIP models.

In fact, the largest open Japanese image-text pair dataset was the Japanese subset of ReLAION-5B (about 120 million examples), which is small compared with the English subset (about 2 billion examples).

In this work, we used an open LLM (here, gemma-2-9b-it), which can translate at scale with high quality, high speed, and low cost, to build a large-scale Japanese image-text pair dataset and develop Japanese CLIP models.

This work expands the amount of open Japanese image-text pairs to 2 billion examples.

Image-Text Pair Dataset

We built the dataset by translating the captions of the English subset of ReLAION-5B (2 billion examples), a large-scale English image-text pair dataset, into Japanese with gemma-2-9b-it.

For the translation, we used text2dataset, a translation tool we developed ourselves.

Models

CLIP uses an image encoder to compute image embeddings and a text encoder to compute text embeddings.

We used $\text{ViT-B/16}$ and $\text{ViT-L/14}$ as image encoders and $\text{RoBERTa}_\text{base}$ as the text encoder. As the tokenizer, we used llm-jp-tokenizer v3, modified for encoder models. Because llm-jp-tokenizer v3 has a large vocabulary of about 100,000 tokens, the total number of model parameters is somewhat larger.

Training

After downloading the images in the dataset, we obtained an image-text pair dataset of about 1.5 billion examples and trained on it for 9 epochs. We used OpenCLIP as the training script.

Evaluation

We compared our models with Japanese and multilingual CLIP models on several benchmark tasks, including the widely used ImageNet.

For datasets whose labels are not in Japanese, we translated the labels into Japanese before evaluation.

We used llm-jp/clip-eval as the evaluation code.

Evaluation results

Overall, llm-jp-clip performed well on many tasks among models of the same size, but it performed poorly on the Recruit dataset, which covers the Japanese culture domain. One possible cause is that the source dataset for translation contains little data from the Japanese culture domain.

Going forward, we hope to develop better Japanese CLIP models by adding more data from the Japanese culture domain.

Conclusion

In this post, we introduced llm-jp-clip. The dataset, models, and evaluation code are all publicly available, so please give them a try!!

For more details, please read our paper with the title below, submitted to the 31st Annual Meeting of the Association for Natural Language Processing (NLP2025), to be held in Nagasaki in March 2025.

オープンLLMによる翻訳を活用した日本語CLIPの開発 (Developing a Japanese CLIP Using Translation by an Open LLM). The 31st Annual Meeting of the Association for Natural Language Processing (NLP2025), March 2025. Issa Sugiura, Shuhei Kurita, Yusuke Oda, Daisuke Kawahara, Naoaki Okazaki.

If you are attending NLP2025, we look forward to seeing you there!

References

• ReLAION-5B
• gemma-2-9b-it
• OpenCLIP