Accuracy mismatch with comparing papers

[debarshigit]

Thank you for the good work. I have a doubt regarding the values reported in the paper. In Table 4 you have reported 54.87 for EuroSAT 4-shot setting for MaPLe and 75.02 for PromptSRC. However, in Table 13 of PromptSRC paper, 4-shot accuracy of MaPLe is reported as 84.50 and 86.30 for PromptSRC. Could you please explain such a huge variance in results? The backbone is also same, i.e., ViT-B/16.

Thank you.

[htyao89]

Thank you for the good work. I have a doubt regarding the values reported in the paper. In Table 4 you have reported 54.87 for EuroSAT 4-shot setting for MaPLe and 75.02 for PromptSRC. However, in Table 13 of PromptSRC paper, 4-shot accuracy of MaPLe is reported as 84.50 and 86.30 for PromptSRC. Could you please explain such a huge variance in results? The backbone is also same, i.e., ViT-B/16.

Thank you.

The results in Table 4 are all re-implemented by ourselves.

[debarshigit]

Okay, that makes sense. Thank you for the quick reply.

[debarshigit]

Could you please provide the training script for few shot experiments. Only base2new_train.sh is available here.

Thank you.

[htyao89]

Could you please provide the training script for few shot experiments. Only base2new_train.sh is available here.

Thank you.

Commented or removed the following code during training: DATASET.SUBSAMPLE_CLASSES base.

Examples:

cd ..

DATA=XXXXX
TRAINER=TCP
WEIGHT=1.0

CFG=vit_b16_ep100_ctxv1
CTP=end # class token position (end or middle)
NCTX=4 # number of context tokens
SHOTS=4 # number of shots (1, 2, 4, 8, 16)
CSC=False # class-specific context (False or True)
FOLDER=output_1108

for DATASET in caltech101 dtd eurosat fgvc_aircraft food101 oxford_flowers oxford_pets stanford_cars ucf101
do
for SEED in 1 2 3
do
DIR=${FOLDER}${NCTX}/base2new/train_base/${DATASET}/shots${SHOTS}_${WEIGHT}/${TRAINER}/${CFG}/seed${SEED}
if [ -d "$DIR" ]; then
echo "Results are available in ${DIR}. Skip this job"
else
echo "Run this job and save the output to ${DIR}"
python train.py
--root ${DATA}
--seed ${SEED}
--trainer ${TRAINER}
--dataset-config-file configs/datasets/${DATASET}.yaml
--config-file configs/trainers/${TRAINER}/${CFG}.yaml
--output-dir ${DIR}
TRAINER.COOP.N_CTX ${NCTX}
TRAINER.COOP.CSC ${CSC}
TRAINER.COOP.W ${WEIGHT}
TRAINER.COOP.CLASS_TOKEN_POSITION ${CTP}
DATASET.NUM_SHOTS ${SHOTS}
fi
done
done

[debarshigit]

Thank you.