README.md

elasticsearch-remove-token-filter

This is a customized stop filter plugin, compatible with ES 5.6.5. This stop filter completely removes stop word's token position from the token stream whereas the ES build-in stop filter only removes the stop word token but preserves the stop word token position. The difference can be best illustrated with the following use case.

Example

Text to be indexed is "Hotels pres de la tour eiffel", and both "de" and "la" are listed as stop words for French

ES build-in stop word filter will generate a token stream as followings:

{
 "tokens": [
   {
     "token": "Hotels",
     "start_offset": 0,
     "end_offset": 6,
     "type": "word",
     "position": 0
   },
   {
     "token": "pres",
     "start_offset": 7,
     "end_offset": 11,
     "type": "word",
     "position": 1
   },
   {
     "token": "tour",
     "start_offset": 18,
     "end_offset": 22,
     "type": "word",
     "position": 4
   },
   {
     "token": "eiffel",
     "start_offset": 23,
     "end_offset": 29,
     "type": "word",
     "position": 5
   }
 ]
}

Customized stop word filter will generate a token stream as followings:

{
  "tokens": [
    {
      "token": "Hotels",
      "start_offset": 0,
      "end_offset": 6,
      "type": "word",
      "position": 0
    },
    {
      "token": "pres",
      "start_offset": 7,
      "end_offset": 11,
      "type": "word",
      "position": 1
    },
    {
      "token": "tour",
      "start_offset": 18,
      "end_offset": 22,
      "type": "word",
      "position": 2
    },
    {
      "token": "eiffel",
      "start_offset": 23,
      "end_offset": 29,
      "type": "word",
      "position": 3
    }
  ]
} 

The tokens generated by both filters are the same, but the position offsets are different.

Assuming we are applying the same stop filter at both indexing and search time, then we would see following behaviours for strict phrase match (slop = 0), 'X' means matching:

Query	Build-in Stop Filter	Customized Stop Filter
Hotels pres de la tour eiffel	X	X
Hotels pres de le tour eiffel	X	X
Hotels pres des le tour eiffel	X	X
Hotels pres de tour eiffel		X
Hotels pres tour eiffel		X