void tokenizer_init failed

[Bing1002]

allocated 474 MiB for model parameters
train_gpt2fp32cu: train_gpt2_fp32.cu:1815: void tokenizer_init(Tokenizer*, const char*): Assertion `header[1] == 1' failed.
[1] 2229854 abort (core dumped) ./train_gpt2fp32cu

[azret]

Try to rebuild your data files with the train_gpt3.py. The tokenizer headers have changed.