[BUG] unexpected exact query result with Chinese

[authurguo]

Describe the bug
There is no results when exact query Chinese word.

To Reproduce

FT.CREATE idx:test ON hash PREFIX 1 "test:" LANGUAGE "chinese" SCHEMA phrase TEXT tags TAG

HSET test:001 phrase "一个眼大" tags "开心" 
HSET test:002 phrase "一个眼大一个眼小" tags "悲伤"

FT.SEARCH idx:test "@phrase:一个眼大" LANGUAGE "chinese" # test:001 test:002
FT.SEARCH idx:test "@phrase:\"一个眼大\"" LANGUAGE "chinese" # NO RESULTS FOUND.

Expected behavior

FT.SEARCH idx:test "@phrase:\"一个眼大\"" LANGUAGE "chinese" 

Command returns test:001 and test:002.

Environment (please complete the following information):

• docker / amd64
• redis/redis-stack:7.2.0-v6

Additional context
default config without any customization.

[adrianoamaral]

@authurguo did you try using TAGs for exact matching queries?

I made a quick test, abusing of your index schema (adding the values in the tags field) and it works for me:

HSET test:003 phrase "一个眼大" tags "一个眼大"
HSET test:004 phrase "一个眼大" tags "一个眼大一个眼小"

FT.SEARCH idx:test "@tags:{一个眼大}" LANGUAGE "Chinese"

1) "1"
2) "test:003"
3) 1) "phrase"
   2) "\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xa4\xa7"
   3) "tags"
   4) "\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xa4\xa7"

FT.SEARCH idx:test "@tags:{一个眼大*}" LANGUAGE "Chinese"

1) "2"
2) "test:003"
3) 1) "phrase"
   2) "\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xa4\xa7"
   3) "tags"
   4) "\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xa4\xa7"
4) "test:004"
5) 1) "phrase"
   2) "\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xa4\xa7"
   3) "tags"
   4) "\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xa4\xa7\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xb0\x8f"

[authurguo]

@authurguo did you try using TAGs for exact matching queries?

I made a quick test, abusing of your index schema (adding the values in the tags field) and it works for me:

HSET test:003 phrase "一个眼大" tags "一个眼大"
HSET test:004 phrase "一个眼大" tags "一个眼大一个眼小"

FT.SEARCH idx:test "@tags:{一个眼大}" LANGUAGE "Chinese"

1) "1"
2) "test:003"
3) 1) "phrase"
   2) "\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xa4\xa7"
   3) "tags"
   4) "\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xa4\xa7"

FT.SEARCH idx:test "@tags:{一个眼大*}" LANGUAGE "Chinese"

1) "2"
2) "test:003"
3) 1) "phrase"
   2) "\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xa4\xa7"
   3) "tags"
   4) "\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xa4\xa7"
4) "test:004"
5) 1) "phrase"
   2) "\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xa4\xa7"
   3) "tags"
   4) "\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xa4\xa7\xe4\xb8\x80\xe4\xb8\xaa\xe7\x9c\xbc\xe5\xb0\x8f"

Thanks!
I have tried exact matching by TAG, and it works.
I also want to confirm whether the exact matching by the Chinese TEXT is effective.

[adrianoamaral]

I still need to review it, but for TEXT it's tokenising using each characters as a term/word. Is that correct, considering the Chinese vocabulary? It means for each one of the characters 一, 个, 眼, 大 the index it's taking them as tokens. For matching this query using TEXT it requires to AND/INTERSECTION ( 一 AND 个 AND 眼 AND 大). TAG it's appropriate for the use cases where you need exact match as shared by you.

[github-actions]

This issue is stale because it has been open for 60 days with no activity.