2.3 ElasticSearch分词之自定义分词

1.简介
当自带的分词器无法满足需求时，就可以通过自定义分词来解决，自定义分词器的组成包括character filters、tokenizer和token filters三个部分。

2.Character Filters
(1).简介
在tokenizer之前对原始文本进行处理，比如增加、删除或者替换字符等，其会影响后续tokenizer解析的位置和偏移量，自带的三个功能为去除html标签和实体的html_strip、进行字符串替换操作的my_mapping(自定义)以及进行正则匹配替换的my_pattern(自定义)。

(2).query

POST /_analyze

{
	"tokenizer": "keyword",
	"char_filter": ["html_strip"],
	"text": "<b>I&apos;m so happy!</b>"
}

{
  "tokens" : [
    {
      "token" : "I'm so happy!",
      "start_offset" : 3,
      "end_offset" : 25,
      "type" : "word",
      "position" : 0
    }
  ]
}

3.Tokenizer
(1).简介
将原始文本按照一定规则切分为单词，自带的功能包括按单词分割的standard、按非字符分割的letter、按空格分割的whitespace以及按文件路径进行分割的path_hierarchy等。

(2).query

POST /_analyze

{
	"tokenizer": "path_hierarchy",
	"text": "/one/two/three"
}

{
  "tokens" : [
    {
      "token" : "/one",
      "start_offset" : 0,
      "end_offset" : 4,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "/one/two",
      "start_offset" : 0,
      "end_offset" : 8,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "/one/two/three",
      "start_offset" : 0,
      "end_offset" : 14,
      "type" : "word",
      "position" : 0
    }
  ]
}

4.Token Filters
(1).简介
token filters对tokenizer的分词结果进行再加工操作，自带的功能包括将所有单词转为小写的lowercase、删除助词的stop以及添加近义词的synonym等。

(2).query

POST /_analyze

{
	"tokenizer": "standard",
	"text": "a Hello,world!",
	"filter": [
		"stop",
		"lowercase",
		{
			"type": "ngram",
			"min_gram": 3,
			"max_gram": 4
		}
	]
}

{
  "tokens" : [
    {
      "token" : "hel",
      "start_offset" : 2,
      "end_offset" : 7,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "hell",
      "start_offset" : 2,
      "end_offset" : 7,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "ell",
      "start_offset" : 2,
      "end_offset" : 7,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "ello",
      "start_offset" : 2,
      "end_offset" : 7,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "llo",
      "start_offset" : 2,
      "end_offset" : 7,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "wor",
      "start_offset" : 8,
      "end_offset" : 13,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "worl",
      "start_offset" : 8,
      "end_offset" : 13,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "orl",
      "start_offset" : 8,
      "end_offset" : 13,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "orld",
      "start_offset" : 8,
      "end_offset" : 13,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "rld",
      "start_offset" : 8,
      "end_offset" : 13,
      "type" : "<ALPHANUM>",
      "position" : 2
    }
  ]
}

5.自定义分词
(1).简介
自定义分词需要在索引的配置中设置，只需要设置三个组成部分character filters、tokenizer和token filters即可。

(2).案例

PUT /my_index

{
	"settings": {
		"analysis": {
			"analyzer": {
				"my_custom_analyzer": {
					"type": "custom",
					"tokenizer": "standard",
					"char_filter": [
						"html_strip"
					],
					"filter": [
						"lowercase",
						"asciifolding"
					]
				}
			}
		}
	}
}

POST /my_index/_analyze

{
	"analyzer": "my_custom_analyzer",
	"text": "Is this <b>a box</b>"
}

{
  "tokens" : [
    {
      "token" : "is",
      "start_offset" : 0,
      "end_offset" : 2,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "this",
      "start_offset" : 3,
      "end_offset" : 7,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "a",
      "start_offset" : 11,
      "end_offset" : 12,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "box",
      "start_offset" : 13,
      "end_offset" : 20,
      "type" : "<ALPHANUM>",
      "position" : 3
    }
  ]
}