Low CLIP Score with Pretrained MedCLIP on MIMIC-CXR Dataset

[junzhin]

Hi authors,

I’m using the pretrained MedCLIP model to compute the CLIP score between chest X-rays from the MIMIC-CXR dataset and their associated descriptions, but I’m observing very low scores.

Example:

File path: files/p10/p10000032/s53911762/68b5c4b1-227d0485-9cc38c3f-7b84ab51-4b472714.jpg
Description:
"Single frontal view of the chest provided. There is no focal consolidation, effusion, or pneumothorax. The cardiomediastinal silhouette is normal. Again seen are multiple clips projecting over the left breast and remote left-sided rib fractures. No free air below the right hemidiaphragm is seen. No acute intrathoracic process."

This results in a low CLIP score with MedCLIP:

However, when using the OpenAI CLIP base model, the score is significantly higher

This behavior has been consistently observed across multiple cases. Is this expected? If I plan to use CLIP models for other downstream tasks, would it be accurate to conclude that the OpenAI model might be even more suitable for medical CXR-related applications? Howcome?

BTW, I’m using the implementation from the TorchMetrics CLIP score and TorchMetrics CLIP score doc .

[learningMalanya]

@junzhin Maybe I am blind, but why are you multiplying "100*" to the img_features*txt_features for the results using OpenAI's CLIP model? I can see the "100*" does appear in line 89 of TorchMetrics CLIP score, but why not for the results from MedCLIP also?

[junzhin]

@junzhin Maybe I am blind, but why are you multiplying "100*" to the img_features*txt_features for the results using OpenAI's CLIP model? I can see the "100*" does appear in line 89 of TorchMetrics CLIP score, but why not for the results from MedCLIP also?

Thank you for pointing that out. It seems there was indeed a typo in "medclip," and you're correct that it actually refers to the results after multiplying by 100. I might have accidentally taken screenshots of both methods at different times.

Anyways, the message remains unchanged: "scores are very low"

[CurryaNa]

您好，请问您计算相似性得分的代码可以分享一下吗，以及您MedCLIP用的是哪个模型呢