Kosmos-2的部署和测试教程（huggingface版）

论文标题：
KOSMOS-2: Grounding Multimodal Large Language Models to the World
GitHub：
https://github.com/microsoft/unilm/tree/master/kosmos-2
huggingface：
https://huggingface.co/microsoft/kosmos-2-patch14-224
论文解读：
http://t.csdnimg.cn/i6Wb5

官方提供了两种使用方式，一种是用github上的，下载pt文件，但配置环境非常繁琐。分别提供了docker，和conda环境两种配置。

笔者失败了，还好huggingface的环境很好配置，我直接用的LLaVA的环境。下面是我的环境配置脚本：

cd /path-to/LLaVA
pip install -e .
pip install -e ".[train]"

pip install ninja
pip install flash_attn-2.5.6+cu122torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl

pip uninstall transformer-engine

下载huggingface的模型参数：

git clone https://huggingface.co/microsoft/kosmos-2-patch14-224

然后就能运行官方提供的demo：

import requests

from PIL import Image
from transformers import AutoProcessor, AutoModelForVision2Seq


model = AutoModelForVision2Seq.from_pretrained("microsoft/kosmos-2-patch14-224")
processor = AutoProcessor.from_pretrained("microsoft/kosmos-2-patch14-224")

prompt = "<grounding>An image of"

url = "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.png"
image = Image.open(requests.get(url, stream=True).raw)

# The original Kosmos-2 demo saves the image first then reload it. For some images, this will give slightly different image input and change the generation outputs.
image.save("new_image.jpg")
image = Image.open("new_image.jpg")

inputs = processor(text=prompt, images=image, return_tensors="pt")

generated_ids = model.generate(
    pixel_values=inputs["pixel_values"],
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    image_embeds=None,
    image_embeds_position_mask=inputs["image_embeds_position_mask"],
    use_cache=True,
    max_new_tokens=128,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

# Specify `cleanup_and_extract=False` in order to see the raw model generation.
processed_text = processor.post_process_generation(generated_text, cleanup_and_extract=False)

print(processed_text)
# `<grounding> An image of<phrase> a snowman</phrase><object><patch_index_0044><patch_index_0863></object> warming himself by<phrase> a fire</phrase><object><patch_index_0005><patch_index_0911></object>.`

# By default, the generated  text is cleanup and the entities are extracted.
processed_text, entities = processor.post_process_generation(generated_text)

print(processed_text)
# `An image of a snowman warming himself by a fire.`

print(entities)
# `[('a snowman', (12, 21), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('a fire', (41, 47), [(0.171875, 0.015625, 0.484375, 0.890625)])]`