Skip to content

allow option for using the slow tokenizer#2

Merged
snova-jonathanl merged 1 commit intomainfrom
jonathanl/allow_slow_tokenizer
Jul 12, 2024
Merged

allow option for using the slow tokenizer#2
snova-jonathanl merged 1 commit intomainfrom
jonathanl/allow_slow_tokenizer

Conversation

@snova-jonathanl
Copy link
Collaborator

@snova-jonathanl snova-jonathanl commented Jul 12, 2024

  • Allow for specifying a slow tokenizer for llava_hf models.
  • Default will remain the same (use fast tokenizer).
  • Edit the llava run script template to add fast_tokenizer=False to the pretrained args

@snova-jonathanl
allow option for using the slow tokenizer
afda5b9 
Signed-off-by: Jonathan Li <jonathan.li@sambanovasystems.com>
snova-guhae
snova-guhae approved these changes Jul 12, 2024
View reviewed changes
Copy link
Owner

@snova-guhae snova-guhae left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@snova-jonathanl snova-jonathanl merged commit 13a07bf into main Jul 12, 2024
@snova-jonathanl snova-jonathanl deleted the jonathanl/allow_slow_tokenizer branch July 12, 2024 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@snova-guhae snova-guhae snova-guhae approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@snova-jonathanl @snova-guhae