论文阅读 CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Problem: 前人指标需要参考文本的问题
Solution: 采用CLIP来解决需要参考的问题
Contribution: 提出一种不需要参考文本的评价指标;提出一个reference增强指标RefCLIPScore;verify that CLIP-S is sensitive to adversarially constructed image captions, where one noun-phrase has been swapped for a plausible (but incorrect) distractor; construct a corpus of images that have never been posted publicly online to verify that CLIP-S is able to reconstruct human judgments on never-before-seen images.
Related work
Reference-only image caption evaluation
- BLEU-4:which measures a version of precision between a candidate and the references
- ROUGE-L:which mea- sures a version of recall
- METEOR:which computes a word-level alignment
- CIDEr:which com- bines n-gram tf-idf weighting and stemming
- SPICE:which applies a semantic parser to a set of references, and com- putes similarity using the predicted scene graph
- BERT-S++:Yi et al. (2020) give a method for re-weighting BERTScore (Zhang et al., 2020) specifically tuned to the image caption generation domain
指标介绍相关文章:
https://blog.csdn.net/sophicchen/article/details/114978410
Reference+image caption evaluation:incorporate image-text grounding features in addition to references
- TIGEr (Jiang et al., 2019) uses a pretrained SCAN model (Lee et al., 2018)
- ViLBERTScore-F (Lee et al., 2020) uses a pre- trained ViLBERT model (Lu et al., 2019) that is also fine-tuned on 12 downstream vision and lan- guage tasks (Lu et al., 2020)
Self-retrieval for image captioning
Prior works have proposed incorporating a self-retrieval loss into caption generation, with the intuition that good captions should be able to uniquely identify their images with high accuracy (Dai and Lin, 2017; Luo et al., 2018; Liu et al., 2018); monitoring this type of loss can provide insight into how distinctive the captions are according to the model itself.
Reference-free evaluation
For image captioning, a version of VIFIDEL (Madhyastha et al., 2019) was proposed for reference-free eval- uation; however, VIFIDEL, computed based on a list of detected objects in the image from a fixed ob- ject vocabulary, generally produces less correlation with human ratings vs. reference-based metrics.
CLIPScore
Evaluating Caption Generations with CLIP
输入: image和caption
计算: 将输入的image和caption放入到特征提取器(CLIP)中获取embedding,计算两者的cosine距离