Work with PyLate models?

[davidmezzetti]

Hello,

First off, great work on this project!

I noticed in the benchmarks script you included lightonai/GTE-ModernColBERT-v1 but it wasn't scored.

I scored it using this repo and the benchmark scripts and get the following.

Metric	ModernColBERT	FDE(20,5,16) ModernColBERT
ndcg_at_1	0.35294	0.05882
ndcg_at_3	0.31416	0.05294
ndcg_at_5	0.28701	0.05297
ndcg_at_10	0.26125	0.05245
ndcg_at_20	0.24186	0.05263
ndcg_at_100	0.247	0.0671
ndcg_at_1000	0.33668	0.16921
map_at_1	0.03499	0.00202
map_at_3	0.06291	0.00427
map_at_5	0.0735	0.00554
map_at_10	0.08719	0.00736
map_at_20	0.09664	0.00914
map_at_100	0.11057	0.01303
map_at_1000	0.12253	0.02048
recall_at_1	0.03499	0.00202
recall_at_3	0.07216	0.00647
recall_at_5	0.09391	0.01138
recall_at_10	0.1267	0.01995
recall_at_20	0.15834	0.03348
recall_at_100	0.26555	0.09853
recall_at_1000	0.57746	0.4513

I've been working on a similar implementation as part of a new feature in txtai. and have been getting similar results with any PyLate model. Muvera seems to work well ONLY with models trained with the standard ColBERT codebase.

Did you find any parameters in your testing that worked well with this model?

Thank you!

[viig99]

It's seems to be very hyper-parameter sensitive, i had to tune it quite a bit to get decent results. And i couldn't find hyper-params which matched the maxsim score for late interaction models.

[davidmezzetti]

Thank you for getting back.

Did you try lightonai/GTE-ModernColBERT-v1 in your tests?

[davidmezzetti]

Going to close this. Tune into lightonai/pylate#142 for more on this.