[BUG] Failed to build tokenizers

[dongqqcom]

Describe the bug
Failed to build tokenizers when executing the command "pip install -e .".
ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

To Reproduce
Steps to reproduce the behavior:

1. pip install -e .

Expected behavior
Installing the dependencies without errors.

Additional context
System: macos
Python: 3.6

[dongqqcom]

The same error occurs in both macos and Linux.

[zqwerty]

It seems that the newest version of tokenizer need rust installed. you can install the version we used tokenizers==0.8.0. Or install rust.

[dongqqcom]

你好，我已收到你的邮件，会尽快给你回复。