Elasticsearch
基于词项和基与全文的搜索
基于Term的查询
- Term的重要性
Term是表达语意的最小单位 。搜索和利用统计语言模型进行自然语言处理都需要处理Term - 特点
-
Term Level Query: Term Query / Range Query / Exists Query / Prefix Query /Wildcard Query
-
在ES中,Term查询,对输入不做分词。会将输入作为一个整体,在倒排索引中查找准确的词项,并
且使用相关度算分公式为每个包含该词项的文档进行相关度算分-例如“Apple Store” -
可以通过Constant Score将查询转换成一个Filtering, 避免算分,并利用缓存,提高性能
-
关于Term查询的例子
多字段Mapping和Term查询
复合查询一Constant Score转为Filter
- 将 Query转成Filter,忽略TF- -IDF计算,避免相关性算分的开销
- Filter 可以有效利用缓存
基于全文的查询
-
基于全文本的查找
- Match Query / Match Phrase Query / Query String Query
-
特点
- 索引和搜索时都会进行分词,查询字符串先传递到一个合适的分词器,然后生成一个供查询的词项列表
- 查询时候,先会对输入的查询进行分词,然后每个词项逐个进行底层的查询,最终将结果进行合并。并为每个文档生成一个算分。-例如查“Matrix reloaded",会查到包括Matrix或者reload的所有结果。
Match Query Result
Operator
Minimum_ should match
Match Phrase Query
Match Query查询过程
API
DELETE products
PUT products
{
"settings": {
"number_of_shards": 1
}
}
POST /products/_bulk
{ "index": { "_id": 1 }}
{ "productID" : "XHDK-A-1293-#fJ3","desc":"iPhone" }
{ "index": { "_id": 2 }}
{ "productID" : "KDKE-B-9947-#kL5","desc":"iPad" }
{ "index": { "_id": 3 }}
{ "productID" : "JODL-X-1937-#pV7","desc":"MBP" }
GET /products
POST /products/_search
{
"query": {
"term": {
"desc": {
//"value": "iPhone"
"value":"iphone"
}
}
}
}
POST /products/_search
{
"query": {
"term": {
"desc.keyword": {
//"value": "iPhone"
//"value":"iphone"
}
}
}
}
POST /products/_search
{
"query": {
"term": {
"productID": {
"value": "XHDK-A-1293-#fJ3"
}
}
}
}
POST /products/_search
{
//"explain": true,
"query": {
"term": {
"productID.keyword": {
"value": "XHDK-A-1293-#fJ3"
}
}
}
}
POST /products/_search
{
"explain": true,
"query": {
"constant_score": {
"filter": {
"term": {
"productID.keyword": "XHDK-A-1293-#fJ3"
}
}
}
}
}
#设置 position_increment_gap
DELETE groups
PUT groups
{
"mappings": {
"properties": {
"names":{
"type": "text",
"position_increment_gap": 0
}
}
}
}
GET groups/_mapping
POST groups/_doc
{
"names": [ "John Water", "Water Smith"]
}
POST groups/_search
{
"query": {
"match_phrase": {
"names": {
"query": "Water Water",
"slop": 100
}
}
}
}
POST groups/_search
{
"query": {
"match_phrase": {
"names": "Water Smith"
}
}
}
结构化搜索
结构化数据
- 结构化搜索 (Structured search)是 指对结构化数据的搜索
+ 日期,布尔类型和数字都是结构化的 - 文本也可以是结构化的。
- 如彩色笔可以有离散的颜色集合: 红(red) 、 绿(green)、 蓝(blue)
- 一个博客可能被标记了标签,例如,分布式(distributed) 和搜索(search)
- 电商网站.上的商品都有UPCs (通 用产品码Universal Product Codes)或其他的唯一标
识,它们都需要遵从严格规定的、结构化的格式。
ES中的结构化搜索
- 布尔, 时间,日期和数字这类结构化数据:有精确的格式,我们可以对这些格式进行逻辑操
作。包括比较数字或时间的范围,或判定两个值的大小。 - 结构化的文本可以做精确匹配或者部分匹配
Term查询/ Prefix前缀查询 - 结构化结果只有 “是”或“否”两个值
根据场景需要,可以决定结构化搜索是否需要打分
布尔值
数字Range
日期Range
处理空值
查找多个精确值
包含而不是相等
API
#结构化搜索,精确匹配
DELETE products
POST /products/_bulk
{ "index": { "_id": 1 }}
{ "price" : 10,"avaliable":true,"date":"2018-01-01", "productID" : "XHDK-A-1293-#fJ3" }
{ "index": { "_id": 2 }}
{ "price" : 20,"avaliable":true,"date":"2019-01-01", "productID" : "KDKE-B-9947-#kL5" }
{ "index": { "_id": 3 }}
{ "price" : 30,"avaliable":true, "productID" : "JODL-X-1937-#pV7" }
{ "index": { "_id": 4 }}
{ "price" : 30,"avaliable":false, "productID" : "QQPX-R-3956-#aD8" }
GET products/_mapping
#对布尔值 match 查询,有算分
POST products/_search
{
"profile": "true",
"explain": true,
"query": {
"term": {
"avaliable": true
}
}
}
#对布尔值,通过constant score 转成 filtering,没有算分
POST products/_search
{
"profile": "true",
"explain": true,
"query": {
"constant_score": {
"filter": {
"term": {
"avaliable": true
}
}
}
}
}
#数字类型 Term
POST products/_search
{
"profile": "true",
"explain": true,
"query": {
"term": {
"price": 30
}
}
}
#数字类型 terms
POST products/_search
{
"query": {
"constant_score": {
"filter": {
"terms": {
"price": [
"20",
"30"
]
}
}
}
}
}
#数字 Range 查询
GET products/_search
{
"query" : {
"constant_score" : {
"filter" : {
"range" : {
"price" : {
"gte" : 20,
"lte" : 30
}
}
}
}
}
}
# 日期 range
POST products/_search
{
"query" : {
"constant_score" : {
"filter" : {
"range" : {
"date" : {
"gte" : "now-1y"
}
}
}
}
}
}
#exists查询
POST products/_search
{
"query": {
"constant_score": {
"filter": {
"exists": {
"field": "date"
}
}
}
}
}
#处理多值字段
POST /movies/_bulk
{ "index": { "_id": 1 }}
{ "title" : "Father of the Bridge Part II","year":1995, "genre":"Comedy"}
{ "index": { "_id": 2 }}
{ "title" : "Dave","year":1993,"genre":["Comedy","Romance"] }
#处理多值字段,term 查询是包含,而不是等于
POST movies/_search
{
"query": {
"constant_score": {
"filter": {
"term": {
"genre.keyword": "Comedy"
}
}
}
}
}
#字符类型 terms
POST products/_search
{
"query": {
"constant_score": {
"filter": {
"terms": {
"productID.keyword": [
"QQPX-R-3956-#aD8",
"JODL-X-1937-#pV7"
]
}
}
}
}
}
POST products/_search
{
"profile": "true",
"explain": true,
"query": {
"match": {
"price": 30
}
}
}
POST products/_search
{
"profile": "true",
"explain": true,
"query": {
"term": {
"date": "2019-01-01"
}
}
}
POST products/_search
{
"profile": "true",
"explain": true,
"query": {
"match": {
"date": "2019-01-01"
}
}
}
POST products/_search
{
"profile": "true",
"explain": true,
"query": {
"constant_score": {
"filter": {
"term": {
"productID.keyword": "XHDK-A-1293-#fJ3"
}
}
}
}
}
POST products/_search
{
"profile": "true",
"explain": true,
"query": {
"term": {
"productID.keyword": "XHDK-A-1293-#fJ3"
}
}
}
#对布尔数值
POST products/_search
{
"query": {
"constant_score": {
"filter": {
"term": {
"avaliable": "false"
}
}
}
}
}
POST products/_search
{
"query": {
"term": {
"avaliable": {
"value": "false"
}
}
}
}
POST products/_search
{
"profile": "true",
"explain": true,
"query": {
"term": {
"price": {
"value": "20"
}
}
}
}
POST products/_search
{
"profile": "true",
"explain": true,
"query": {
"match": {
"price": "20"
}
}
}
}
POST products/_search
{
"query": {
"constant_score": {
"filter": {
"bool": {
"must_not": {
"exists": {
"field": "date"
}
}
}
}
}
}
}
本节知识点回顾
- 结构化数据&结构化搜索
- 如果不需要算分,可以通过Constant Score,将查询转为Filtering
- 范围查询和Date Math
- 使用Exist查询处理非空Null值
- 精确值&多值字段的精确值查找
- Term查询是包含,不是完全相等。针对多值字段查询要尤其注意