Is there a way to get vocabulary size of an initialized tokenizer? #47
Description
Is there a way to get the vocabulary size of an initialized tokenizer from var encoding = GptEncoding.GetEncoding(encodingName);?
{{ message }}
Is there a way to get the vocabulary size of an initialized tokenizer from var encoding = GptEncoding.GetEncoding(encodingName);?