[elasticsearch笔记] Analysis - Token Filters

note

• Token filters details
• Token filters accept a stream of tokens from a tokenizer and can modify tokens (eg lowercasing), delete　tokens (eg remove stopwords) or add tokens (eg synonyms).

demo

asciifolding

• A token filter of type asciifolding that converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the “Basic Latin” Unicode block) into their ASCII equivalents, if one exists.
• preserve_original : keep asciifolding char and the original char

PUT /asciifold_example
{
    "settings" : {
        "analysis" : {
            "analyzer" : {
                "default" : {
                    "tokenizer" : "standard",
                    "filter" : ["asciifolding"]
                }
            }
        }
    }
}

PUT /asciifold_example
{
    "settings" : {
        "analysis" : {
            "analyzer" : {
                "default" : {
                    "tokenizer" : "standard",
                    "filter" : ["my_ascii_folding"]
                }
            },
            "filter" : {
                "my_ascii_folding" : {
                    "type" : "asciifolding",
                    "preserve_original" : true
                }
            }
        }
    }
}

lowercase

• Lowercase token filter supports Greek, Irish, and Turkish lowercase token filters through the language parameter.

PUT /lowercase_example
{
  "settings": {
    "analysis": {
      "analyzer": {
        "standard_lowercase_example": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase"]
        },
        "greek_lowercase_example": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["greek_lowercase"]
        }
      },
      "filter": {
        "greek_lowercase": {
          "type": "lowercase",
          "language": "greek"
        }
      }
    }
  }
}

stop

• Elasticsearch provides the following predefined list of languages: arabi