You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
afiaka87 edited this page Apr 15, 2021
·
1 revision
Custom Tokenizer
This repository supports Huggingface Tokenizers if you wish to use it instead of the default simple tokenizer. Simply pass in an extra --bpe_path when invoking train_dalle.py and generate.py, with the path to your BPE json file.
The only requirement is that you use 0 as the padding during tokenization