es - elasticsearch自定义分析器 - 内建分词过滤器 - 4

世界上并没有完美的程序，但是我们并不因此而沮丧，因为写程序就是一个不断追求完美的过程。

自定义分析器 :

1. Character filters :
       1. 作用 : 字符的增、删、改转换
       2. 数量限制 : 可以有0个或多个
       3. 内建字符过滤器 :
           1. HTML Strip Character filter : 去除html标签
           2. Mapping Character filter : 映射替换
           3. Pattern Replace Character filter : 正则替换
2. Tokenizer :
       1. 作用 :
           1. 分词
           2. 记录词的顺序和位置（短语查询）
           3. 记录词的开头和结尾位置（高亮）
           4. 记录词的类型（分类）
       2. 数量限制 : 有且只能有一个
       3. 分类 :
           1. 完整分词 :
               1. Standard
               2. Letter
               3. Lowercase
               4. whitespace
               5. UAX URL Email
               6. Classic
               7. Thai
           2. 切词 :
               1. N-Gram
               2. Edge N-Gram
           3. 文本 :
               1. Keyword
               2. Pattern
               3. Simple Pattern
               4. Char Group
               5. Simple Pattern split
               6. Path
3. Token filters :
       1. 作用 : 分词的增、删、改转换
       2. 数量限制 : 可以有0个或多个
       3. 分类 :
           1. apostrophe
           2. asciifolding
           3. cjk bigram
           4. cjk width
           5. classic
           6. common grams
           7. conditional
           8. decimal digit
           9. delimited payload
           10. dictionary decompounder
           11. edge ngram
           12. elision
           13. fingerprint
           14. flatten_graph
           15. hunspell

今天演示12-15
需要注意的是 : 词干提取过滤器 hunspell token filter

# elision token filter
# 元音省略 音节省略
# 适用于法语、爱尔兰语、意大利语、加泰罗尼亚语的分析器
GET /_analyze
{
  "tokenizer": "standard",
  "filter": ["elision"],
  "text": ["j’examine près du wharf"]
}

# 结果
{
  "tokens" : [
    {
      "token" : "examine",
      "start_offset" : 0,
      "end_offset" : 9,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "près",
      "start_offset" : 10,
      "end_offset" : 14,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "du",
      "start_offset" : 15,
      "end_offset" : 17,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "wharf",
      "start_offset" : 18,
      "end_offset" : 23,
      "type" : "<ALPHANUM>",
      "position" : 3
    }
  ]
}

# fingerprint token filter
# 作用   : 将token按字母排序并去重拼接后输出
# 配置项 :
#   1. max_output_size : 输出词的最大长度，默认255
#   2. separator       : 拼接分隔符
GET /_analyze
{
  "tokenizer": "standard",
  "filter": [{
    "type"            : "fingerprint",
    "max_output_size" : 255,
    "separator"       : "+"
  }],
  "text": ["hello good good a me is a", "我是中国人"]
}

# 结果
{
  "tokens" : [
    {
      "token" : "a+good+hello+is+me",
      "start_offset" : 0,
      "end_offset" : 25,
      "type" : "fingerprint",
      "position" : 0
    },
    {
      "token" : "中+人+国+我+是",
      "start_offset" : 26,
      "end_offset" : 31,
      "type" : "fingerprint",
      "position" : 102
    }
  ]
}

# flatten graph token filter
# 作用 : 平铺多点图以适应索引 有损 不建议使用
# 实例 : 同义词平铺
GET /_analyze
{
  "tokenizer": "standard",
  "filter": [{
    "type"     : "synonym_graph",
    "synonyms" : ["dns, domain name system"]
  }, "flatten_graph"],
  "text": ["domain name system is fragile"]
}

# 结果
{
  "tokens" : [
    {
      "token" : "dns",
      "start_offset" : 0,
      "end_offset" : 18,
      "type" : "SYNONYM",
      "position" : 0,
      "positionLength" : 3
    },
    {
      "token" : "domain",
      "start_offset" : 0,
      "end_offset" : 6,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "name",
      "start_offset" : 7,
      "end_offset" : 11,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "system",
      "start_offset" : 12,
      "end_offset" : 18,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "is",
      "start_offset" : 19,
      "end_offset" : 21,
      "type" : "<ALPHANUM>",
      "position" : 3
    },
    {
      "token" : "fragile",
      "start_offset" : 22,
      "end_offset" : 29,
      "type" : "<ALPHANUM>",
      "position" : 4
    }
  ]
}

# hunspell token filter
# 作用   : 词干提取
# 配置项 :
#   1. dirctionary
#   2. dedup        : 是否不显示重复的词，默认true
#   3. lang
#   4. language
#   5. locale
#   6. longest_only : 是否只显示最长的词，默认false
# 注意 : 需要自己安装hunspell字典
GET /_analyze
{
  "tokenizer": "standard",
  "filter": [{
    "type"   : "hunspell",
    "lang"   : "en_US"
  }],
  "text": ["the foxes jumping quickly"]
}

# 结果
{
  "tokens" : [
    {
      "token" : "the",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "fox",
      "start_offset" : 4,
      "end_offset" : 9,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "jump",
      "start_offset" : 10,
      "end_offset" : 17,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "quick",
      "start_offset" : 18,
      "end_offset" : 25,
      "type" : "<ALPHANUM>",
      "position" : 3
    }
  ]
}