Correction of the accuracy of FastV

[Osilly]

We have evaluated the performacne of FastV on LLaVA-1.5-7B, and we find that the true results are significantly higher than the results reported in your paper.
Specifically, we set this line in the official repo of FastV to K=2, ratio = 0.25 (i.e., keep 576*0.25=144 tokens), and we observed the results below:

	Token number	VQAv2	SQA
True FastV (using official code)	144	74.06	69.06
FastV in your paper	128	61.8	67.3
SparseVLM	128	73.8	69.1

You seem to have significantly depressed the FastV results?

[Gumpest]

Thank you very much for your attention to our work!

First, it is necessary to calculate the equivalent token amount correctly.
For example, T = {(L1-L0) * T0 + (L2-L1) * T1} / L2.

Secondly, we correctly calculate the token amount of FastV: T = (576 * 3 + 144 * 29) / 32 = 184.5. Please check again.

[Osilly]

Thanks for your resopnse! we use T = {(L1-L0) * T0 + (L2-L1) * T1} / L2 to calculate the equivalent token, and the detailed process as follows:

T = (576 * 2 + 144 * 30) / 32 = 171 when we set K=2, ratio = 0.25.

And the results as follows:

	Token number	VQAv2	GQA	SQA
True FastV (using official code)	171	74.06	56.58	69.06
FastV in your paper	192	67.1	52.7	67.3
SparseVLM	192	75.6	57.6	69.1

Can you provide the detailed setting you used?

[ylnxxts]

Thank you again for your attention to our work.

Have you followed the official settings of Fastv? In the Fastv (Section 4.1), the symbol "R" represents the proportion of tokens to be deleted, rather than the proportion of tokens to be retained. As stated in the paper: "The last R% tokens after ranking would be pruned out in successive layers."

Therefore, if K = 2 and R = 0.25, the number of tokens should be calculated as

$$T = (576 * 2 + (576 * 0.75) * 30) / 32 = 441.$$

[Osilly]

@ylnxxts @Gumpest In this line in the official repo of FastV, ratio = 0.25 means keep 25% tokens, equivalent to ''R = 0.75'' in FastV paper.

[Gumpest]

@Osilly Thank you for your correction. We have re-evaluated the accuracy of Fastv and updated it to the latest arXiv.

[Osilly]

The exact results can be found in Table 1 of our Dynamic-LLaVA paper ([ICLR2025] Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification
). We have configured LLaVA-1.5 with the optimal settings and reported the corresponding results.

[Osilly]

Recently, I have observed a substantial number of papers adopting incorrect baseline metrics (specifically for FastV), as their FastV comparisons do not follow the original paper's experimental settings—even though the correct settings for FastV are explicitly stated in the title. I strongly believe that accurate benchmarking metrics are crucial for promoting healthy development in the field. Papers should rigorously validate their baselines to prevent potential misunderstandings and avoid misleading future researchers. We provide the correct results reported in [ICLR2025] Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification.

	Flops	Token	VQAv2	GQA	SciQA	TextVQA	POPE	MME	MMBench	SEED
LLaVA-1.5-7B-FastV$_{k=3,r=0.75}$	-68%	185	75.1	57.5	68.7	56.2	81.0	1458.9	63.5	62.8
Dynamic-LLaVA-7B	-75%	144	77.9	61.3	68.6	56.5	85.9	1501.0	64.1	65.0
LLaVA-1.5-13B-FastV$_{k=3,r=0.75}$	-69%	176	77.0	60.1	72.8	59.0	83.2	1470.3	66.9	65.4
Dynamic-LLaVA-13B	-76%	138	78.8	62.5	72.4	59.6	86.5	1563.3	66.9	66.5

[Man-PaperRejected]

@Gumpest @ylnxxts Thank you for your outstanding work. I have a question to ask: how is the average token setting implemented in FastV? Could you provide specific parameters, such as at which layer and how much to trim? I would be very grateful if I could receive a prompt reply.