Customizable tokenizer for RULER

[changlan]

Adding an optional environment variable TOKENIZER_MODEL which controls the tokenizer model to use for RULER data generation. With this option, the dataset length will be more precise when we evaluate models that do not use gpt-4 tokenizer.

[MaiziXiao]

https://github.com/open-compass/opencompass/blob/main/configs/eval_ruler.py
We have provided the way to use model's own tokenizer to build model specific datasets, you can have a look at the config.

On the other hand, the configuration (**_gen.py) is standard configurations for general evaluations. You are of course welcome to try your own configurations.

[changlan]

Thanks for the review. The general workflow we use opencompass is via the CLI: opencompass --models [custom_model_config] --datasets ruler_4k_gen.py ... However, it seems that it is not possible to specify tokenizer for --datasets. Do you think this is a reasonable use case?

[MaiziXiao]

Thanks for the review. The general workflow we use opencompass is via the CLI: opencompass --models [custom_model_config] --datasets ruler_4k_gen.py ... However, it seems that it is not possible to specify tokenizer for --datasets. Do you think this is a reasonable use case?
That sounds like a reasonable usecase.

[MaiziXiao]

LGTM