ArabBert_Tokenizer

• ArabBERT_Tokenizer:

Goal:-

• Writing a sample tokenizer Code and testing it, Using a provided sample code on GitHub and Google Colab.

Steps:-

1. Installing arabert and transformers modules.
2. Using from transformers import AutoTokenizer, AutoModel to import the tokenizer and the model builder.
3. Using from arabert.preprocess import ArabertPreprocessor to import the text preprocessing tool.
4. Calling the Model model_name = "aubmindlab/bert-base-arabertv2".
5. Testing the tokenizer and the preprocessor:-

• Tested with Different forms of Arabic text:
  • العربية الفصحى
  • الْعَرَبِيَّةِ الْفُصْحَى
    Using Shakkala.
  • Egyptian Arabic text.