Tokenizer attribute .tokens_from_list deprecated

[fishcakebaker]

The tokeniser attribute .tokens_from_list has been deprecated in SpaCy.

This is used in Chapter 7, Section 7.8 "Advanced Tokenisation, Stemming and Lemmatization" in block In[39].

I'm using SpaCy version 3.0.6 - Which I am guessing is several versions higher than the book, I just can't find where it is in my copy.

Any suggestions on getting around this function? I'm a bit of a newbie, but the searches online have led to rabbit holes thus far.

[Tanvi09Garg]

Instead using old_tokenizer.tokens_from_list, you can substitute any custom tokenizer that does the correct input -> Doc conversion with the correct vocab for nlp.tokenizer:

from spacy.tokens import Doc
class _PretokenizedTokenizer:
"""Custom tokenizer to be used in spaCy when the text is already pretokenized."""
def init(self, vocab: en_nlp):
"""Initialize tokenizer with a given vocab
:param vocab: an existing vocabulary
"""
self.vocab = vocab
for i in range(0,len(List)):
def call(self, inp: [List[i], str]) -> Doc:
"""Call the tokenizer on input inp.
:param inp: either a string to be split on whitespace, or a list of tokens
:return: the created Doc object
"""
if isinstance(inp, str):
words = inp.split()
spaces = [True] * (len(words) - 1) + ([True] if inp[-1].isspace() else [False])
return Doc(self.vocab, words=words, spaces=spaces)
elif isinstance(inp, list):
return Doc(self.vocab, words=inp)
else:
raise ValueError("Unexpected input format. Expected string to be split on whitespace, or list of tokens.")

[Tanvi09Garg]

'List' is used to store input string/text

[ypauchard]

Probably similar to @Tanvi09Garg here is what works for me:

import re
import spacy
from spacy.tokens import Doc

# regexp used in CountVectorizer
# (?u) sets unicode flag, i.e. patterns are unicode
# \\b word boundary: the end of a word is indicated by whitespace or a non-alphanumeric character
# \\w alphanumeric: [0-9a-zA-Z_]

class RegexTokenizer:
    """Spacy custom tokenizer
        Reference https://spacy.io/usage/linguistic-features#custom-tokenizer
    """
    def __init__(self, vocab, regex_pattern='(?u)\\b\\w\\w+\\b'):
        self.vocab = vocab
        self.regexp = re.compile(regex_pattern)

    def __call__(self, text):
        words = self.regexp.findall(text)
        spaces = [True] * len(words)
        spaces[-1] = False #no space after last word

        return Doc(self.vocab, words=words, spaces=spaces)

nlp = spacy.load('en_core_web_sm', disable=['parser', 'ner'])
nlp.tokenizer = RegexTokenizer(nlp.vocab)

def custom_tokenizer(document):
    doc_spacy = nlp(document)
    return [token.lemma_ for token in doc_spacy]

from sklearn.feature_extraction.text import CountVectorizer
vect = CountVectorizer(tokenizer=custom_tokenizer)

It runs a bit slow, any suggestions to speed this up?