Incorrect detection of sentence boundaries, if last sentence missing eos symbol for trf model

[koder-ua]

How to reproduce the behaviour

In [69]: len(list(spacy.load("en_core_web_trf")("The first sentence. The second sentence. The last one").sents))
Out[69]: 1   <<<<<<<<<<<<<<<<<<<<<< WRONG

In [70]: len(list(spacy.load("en_core_web_trf")("The first sentence. The second sentence. The last one.").sents))
Out[70]: 3

In [71]: len(list(spacy.load("en_core_web_sm")("The first sentence. The second sentence. The last one").sents))
Out[71]: 3

In [72]: len(list(spacy.load("en_core_web_sm")("The first sentence. The second sentence. The last one.").sents))
Out[72]: 3

Your Environment

• Operating System: max os x 10.3
• Python Version Used: 3.11
• spaCy Version Used: 3.7.4
• Environment Information:

en_core_web_trf.__version__  >> '3.7.3'
en_core_web_sm.__version__ >> '3.7.1'

[svlandeg]

Hi!

In this pretrained pipeline, the sentence segmentation is actually done by the parser, and the model was mostly trained on texts with correct punctuation. So unfortunately this type of occassional error is unavoidable.

If you'd like to have more predictable behaviour, you can use the sentencizer instead, which is a more simple rule-based component that splits sentences on punctuation like ., ! or ?.