Elastic Stack之分词器

Quincy379

已于 2022-05-23 21:48:39 修改

阅读量202

点赞数

分类专栏： Elastic Stack 文章标签： Elastic Stack

于 2022-05-22 21:35:27 首次发布

本文链接：https://blog.csdn.net/qq_33733970/article/details/124916965

版权

Elastic Stack 专栏收录该内容

4 篇文章 0 订阅

订阅专栏

文档正常化：normalization
字符过滤器(character filter)：分词前的预处理

HTML Strip

PUT test_index
{
	"settings":{
		"analysis":{
			"char_filter":{
				"my_html_filter":{
					"type":"html_strip",
					"escaped_tags":["a"]
				}
			},
			"analyzer":{
				"my_analyzer":{
					"tokenizer":"keyword",
					"char_filter":["my_html_filter"]
				}
			}
		}
	}
}

GET test_index/_analyze
{
	"text":"<p>U r an <a>idiot</a>!</p>"
	"analyzer":"my_analyzer"
}

Mapping

PUT test_index
{
	"settings":{
		"analysis":{
			"char_filter":{
				"my_mapping_filter":{
					"type":"mapping",
					"mappings":[
						"小 => ^",
						"阳 => %",
						"人 => &"
					]
				}
			},
			"analyzer":{
				"my_analyzer":{
					"tokenizer":"keyword",
					"char_filter":["my_mapping_filter"]
				}
			}
		}
	}
}
GET test_index/_analyze
{
	"text":"你这个小阳人，哼！",
	"analyzer":"my_analyzer"
	
}

Pattern Replace

PUT test_index
{
	"settings":{
		"analysis":{
			"char_filter":{
				"my_pattern_filter":{
					"type":"pattern_replace",
					"pattern":"(\\d{3})\\d+(\\d{3})",
					"replacement":"$1######$2"
				}
			},
			"analyzer":{
				"my_analyzer":{
					"tokenizer"："keyword",
					"char_filter":["my_pattern_filter"]
				}
			}
		}
	}
}
GET test_index
{
	"text":"My ID Card No is 31098978786798",
	"analyzer":"my_analyzer"
}

令牌过滤器：token filter

PUT test_index
{
	"settings":{
		"analysis":{
			"filter":{
				"my_synonym_filter":{
					"type":"synonym_graph",
					"synonyms_path":"analysis/synonyms.txt"
					
				}
			},
			"analyzer":{
				"my_analyzer":{
					"tokenizer":"ik_max_word",
					"filter":["my_synonym_filter"]
				}
			}
		}
	}
}
GET test_index
{
	"text":"123456789",
	"analyzer":"my_analyzer"
}


```powershell
PUT test_index
{
	"settings":{
		"analysis":{
			"filter":{
				"my_synonym_filter":{
					"type":"synonym",
					"synonyms":["中国移动，新疆联通=>电信运营商","湖南电信=>南方运营商"]
				}
			},
			"analyzer":{
				"my_analyzer":{
					"tokenizer":"standard",
					"filter":["my_synonym_filter"]
				}
			}
		}
	}
}
GET /test_index/_analyze
{
	"text":["中国移动，新疆联通","湖南电信"],
	"analyzer":"my_analyzer"
}

GET test_index
{
	"tokenizer":"standard",
	"filter":["uppercase"],
	"text":""
}

GET test_index
{
	"tokenizer":"standard",
	"filter":["lowercase"],
	"text":""
}

GET test_index
{
	"tokenizer":"standard",
	"filter":{
		"type":"condition",
		"filter":"uppercase",
		"script":{
			"source":"token.getTerm().length() < 5"
		}
	},
	"text":"qazwsx edcrfvgb yhnuj iklopqasde qwert"
}

#停用词
PUT test_index
{
	"settings":{
		"analysis":{
			"analyzer":{
				"my_analyzer":{
					"type":"standard",
					"stopwords":"_english_"
				}
			}
		}
	}
}

GET test_index/_analyze
{
	"analyzer":"my_analyzer",
	"text":" U and me is dating!"
}

PUT test_index
{
	"settings":{
		"analysis":{
			"analyzer":{
				"my_analyzer":{
					"type":"standard",
					"stopwords":["U","me"]
				}
				
			}
		}
	}
}

GET test_index/_analyze
{
	"analyzer":"my_analyzer",
	"text":"U and me is beautiful"
}