CLAIR: A (surprisingly) simple semantic text metric with large language models.
OTHER License
Official implementation of the paper: "CLAIR: Evaluating Image Captions with Large Language Models".
The evaluation of machine-generated image captions poses an interesting yet persistent challenge. Effective evaluation measures must consider numerous dimensions of similarity, including semantic relevance, visual structure, object interactions, caption diversity, and specificity. Existing highly-engineered measures attempt to capture specific aspects, but fall short in providing a holistic score that aligns closely with human judgments. Here, we propose CLAIR, a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) to evaluate candidate captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Notably, on Flickr8K-Expert, CLAIR achieves relative correlation improvements over SPICE of 39.6% and over image-augmented methods such as RefCLIP-S of 18.3%. Moreover, CLAIR provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score.
pip install git+https://github.com/DavidMChan/clair.git
import os
from clair import clair
os.environ['OPENAI_API_KEY'] = 'YOUR_API_KEY'
candidates = ['A photo of a cat sitting on a couch', 'A cat sitting on a couch']
references = ['A cat sitting on a couch', 'A cat is sitting on a couch', 'A cat is sitting on a sofa']
clair_score = clair(candidates, references, model='chat-gpt')
print(clair_score)
CLAIR relies on the Grazier library to inteface with large language models. The following models are currently supported (see the linked repository for more information on configuring each model):
From OpenAI:
From Anthropic:
From Google/GCP:
From Huggingface
From Facebook (via Huggingface)
From Stanford (via Huggingface)
From Berkeley (via Huggingface)
From StabilityAI (via Huggingface)
From AllenAI (via Huggingface)
From AI21
If you find this repository useful, please cite our paper:
@inproceedings{chan-etal-2023-clair-evaluating,
title = "CLAIR: Evaluating Image Captions with Large Language Models",
author = "Chan, David M and
Petryk, Suzanne and
Gonzalez, Joseph E and
Darrell, Trevor and
Canny, John",
booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2023",
address = "Singapore, Singapore",
publisher = "Association for Computational Linguistics",
}