Set boundary_property with complex character and EOT #4903
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aethanyc
requested changes
May 17, 2024
|
|
||
| assert_eq!(iter.next(), Some(42), "after 4th word and next is EOT"); | ||
| assert_eq!(iter.word_type(), WordType::Letter, "letter"); | ||
| assert!(iter.is_word_like(), "Letter(Thai) is true"); | ||
| } |
There was a problem hiding this comment.
Suggestion: I think it's worth adding extra assertions at the end of the test
assert_eq!(iter.next(), None, "EOT");
assert_eq!(iter.word_type(), WordType::None, "none");
assert!(!iter.is_word_like(), "None is false");
to make sure we fully test every word in the testcase. Same for other testcase in this file.
However, iter.word_type() returns WordType::Letter in rule_status_th(). I've tried added the same three assertions in rule_status_numeric_eof(), and it matches the expectations.
@makotokato Could you check if this is a bug?
There was a problem hiding this comment.
EOT seems to return previous word_type. I didn't define whether we should return any value...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #4446.
If EOT after SA, we should mark as Letter (SA).