第一个化学领域的开源多模态大语言模型：通过ChemVLM架起视觉与化学知识的桥梁

Seeing and Understanding:

Bridging Vision with Chemical Knowledge Via ChemVLM

https://huggingface.co/AI4Chem/ChemVLM-26B

https://arxiv.org/pdf/2408.07246

Abstract

In this technical report, we propose ChemVLM, the first open-source multimodal large language model dedicated to the fields of chemistry, designed to address the incompatibility between chemical image understanding and text analysis. Built upon the VIT-MLP-LLM architecture, we leverage ChemLLM-20B as the foundational large model, endowing our model with robust capabilities in understanding and utilizing chemical text knowledge. Additionally, we employ InternVIT-6B as a powerful image encoder. We have curated high-quality data from the chemical domain, including molecules, reaction formulas, and chemistry examination data, and compiled these into a bilingual multimodal question-answering dataset. We test the performance of our model on multiple open-source benchmarks and three custom evaluation sets. Experimental results demonstrate that our model achieves excellent performance, securing state-of-the-art results in five out of six involved tasks.

本文提出了ChemVLM，这是首个面向化学领域的开源多模态大型语言模型，旨在解决化学图像理解与文本分析之间的不兼容问题。

该模型基于VIT-MLP-LLM架构，采用ChemLLM-20B作为基础大型模型，使模型在理解和利用化学文本知识方面具备了强大的能力。

此外，还采用了InternVIT-6B作为强大的图像编码器。从化学领域精心挑选了高质量数据，包括分子、反应式以及化学考试数据，并将这些数据编译成一个双语多模态问答数据集。

本文在多个开源基准测试集和三个自定义评估集上测试了模型的性能。实验结果表明，本文的模型表现优异，在六个相关任务中的五个任务中均达到了最先进水平。

Introduction

需求动机：

大型语言模型（LLMs）因其加速科学理解和发现的巨大潜力，已被广泛应用于各种科学领域[1]。尽管这些模型在促进研究方面具有令人兴奋的可能性，但：</