Kosmos-2.5 implementation in transformers

[Natyren]

Model description

Hello everyone,

The Kosmos-2.5 is a multimodal literate model that can be used for tasks such as OCR and text-rich image comprehension. It includes a ViT encoder, a Resampler, and a shared decoder module. To the best of my knowledge, the architecture of this model is similar to Kosmos-2 but has some differences. Due to these differences, using this model in Transformers requires a standalone implementation.

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

Paper: https://arxiv.org/pdf/2309.11419
Code: https://github.com/microsoft/unilm/tree/master/kosmos-2.5
Authors: @Dod-o @wolfshow

[amyeroberts]

cc @ydshieh

[Natyren]

I would like to assist with this implementation. If there are any guidelines on how to do it effectively, I would like to join implementation process

[amyeroberts]

Hi @Natyren, there's a guide in the documentation here: https://huggingface.co/docs/transformers/add_new_model

[ydshieh]

Hi @Natyren, sorry for the late reply. I am thinking to talk to the model author about if they are interested in porting this model into transformers. I will come back to you here for the updates.