es搜索核心与实战 Day02
一、倒排索引
1.搜索引擎
- 正排索引——文档ld到文档内容和单词的关联+
- 倒排索引——单词到文档Id的关系
2。倒排索引的核心组成
倒排索引包含两个部分
- 单词词典 (Term Dictionary)， 记录所有文档的单词，记录单词到倒排列表的关联关系
单词词典一般比较大，可以通过B +树或哈希拉链法实现，以满足高性能的插入与查询
- 倒排列表(Posting List) - 记录了单词对应的文档结合，由倒排索引项组成
- 倒排索引项(Posting)
1.文档ID
2.词频TF-该单词在文档中出现的次数，用于相关性评分
3.位置(Position) -单词在文档中分词的位置。用于语句搜索(phrase query)
4.偏移(Offset) -记录单词的开始结束位置，实现高亮显示
二、通过Analyzer进行分词
GET _analyze
{
//
"analyzer": standard
"text": "2 running Quick brown-foxes leap over lazy dogs in the summer evening."
}
1.standard analyzer
- 默认分词器
- 按词切分
- 小写处理
返回结果
{
"tokens" : [
{
"token" : "2",//返回结果值
"start_offset" : 0,//结果值开始位置
"end_offset" : 1,//结果值结束位置
"type" : "<NUM>",//结果值类型
"position" : 0//第几个
},
{
"token" : "running",
"start_offset" : 2,
"end_offset" : 9,
"type" : "<ALPHANUM>",
"position" : 1
},
{
"token" : "quick",
"start_offset" : 10,
"end_offset" : 15,
"type" : "<ALPHANUM>",
"position" : 2
},
{
"token" : "brown",
"start_offset" : 16,
"end_offset" : 21,
"type" : "<ALPHANUM>",
"position" : 3
},
{
"token" : "foxes",
"start_offset" : 22,
"end_offset" : 27,
"type" : "<ALPHANUM>",
"position" : 4
},
{
"token" : "leap",
"start_offset" : 28,
"end_offset" : 32,
"type" : "<ALPHANUM>",
"position" : 5
},
{
"token" : "over",
"start_offset" : 33,
"end_offset" : 37,
"type" : "<ALPHANUM>",
"position" : 6
},
{
"token" : "lazy",
"start_offset" : 38,
"end_offset" : 42,
"type" : "<ALPHANUM>",
"position" : 7
},
{
"token" : "dogs",
"start_offset" : 43,
"end_offset" : 47,
"type" : "<ALPHANUM>",
"position" : 8
},
{
"token" : "in",
"start_offset" : 48,
"end_offset" : 50,
"type" : "<ALPHANUM>",
"position" : 9
},
{
"token" : "the",
"start_offset" : 51,
"end_offset" : 54,
"type" : "<ALPHANUM>",
"position" : 10
},
{
"token" : "summer",
"start_offset" : 55,
"end_offset" : 61,
"type" : "<ALPHANUM>",
"position" : 11
},
{
"token" : "evening",
"start_offset" : 62,
"end_offset" : 69,
"type" : "<ALPHANUM>",
"position" : 12
}
]
}
2.simple analyzer
- 按照非字母切分
- 非字母的都被去除
- 小写处理
返回结果
{
"tokens" : [
{
"token" : "running",
"start_offset" : 2,
"end_offset" : 9,
"type" : "word",
"position" : 0
},
{
"token" : "quick",
"start_offset" : 10,
"end_offset" : 15,
"type" : "word",
"position" : 1
},
{
"token" : "brown",
"start_offset" : 16,
"end_offset" : 21,
"type" : "word",
"position" : 2
},
{
"token" : "foxes",
"start_offset" : 22,
"end_offset" : 27,
"type" : "word",
"position" : 3
},
{
"token" : "leap",
"start_offset" : 28,
"end_offset" : 32,
"type" : "word",
"position" : 4
},
{
"token" : "over",
"start_offset" : 33,
"end_offset" : 37,
"type" : "word",
"position" : 5
},
{
"token" : "lazy",
"start_offset" : 38,
"end_offset" : 42,
"type" : "word",
"position" : 6
},
{
"token" : "dogs",
"start_offset" : 43,
"end_offset" : 47,
"type" : "word",
"position" : 7
},
{
"token" : "in",
"start_offset" : 48,
"end_offset" : 50,
"type" : "word",
"position" : 8
},
{
"token" : "the",
"start_offset" : 51,
"end_offset" : 54,
"type" : "word",
"position" : 9
},
{
"token" : "summer",
"start_offset" : 55,
"end_offset" : 61,
"type" : "word",
"position" : 10
},
{
"token" : "evening",
"start_offset" : 62,
"end_offset" : 69,
"type" : "word",
"position" : 11
}
]
}
3.whitespace analyzer
- 按空格切分
返回结果
{
"tokens" : [
{
"token" : "2",
"start_offset" : 0,
"end_offset" : 1,
"type" : "word",
"position" : 0
},
{
"token" : "running",
"start_offset" : 2,
"end_offset" : 9,
"type" : "word",
"position" : 1
},
{
"token" : "Quick",
"start_offset" : 10,
"end_offset" : 15,
"type" : "word",
"position" : 2
},
{
"token" : "brown-foxes",
"start_offset" : 16,
"end_offset" : 27,
"type" : "word",
"position" : 3
},
{
"token" : "leap",
"start_offset" : 28,
"end_offset" : 32,
"type" : "word",
"position" : 4
},
{
"token" : "over",
"start_offset" : 33,
"end_offset" : 37,
"type" : "word",
"position" : 5
},
{
"token" : "lazy",
"start_offset" : 38,
"end_offset" : 42,
"type" : "word",
"position" : 6
},
{
"token" : "dogs",
"start_offset" : 43,
"end_offset" : 47,
"type" : "word",
"position" : 7
},
{
"token" : "in",
"start_offset" : 48,
"end_offset" : 50,
"type" : "word",
"position" : 8
},
{
"token" : "the",
"start_offset" : 51,
"end_offset" : 54,
"type" : "word",
"position" : 9
},
{
"token" : "summer",
"start_offset" : 55,
"end_offset" : 61,
"type" : "word",
"position" : 10
},
{
"token" : "evening.",
"start_offset" : 62,
"end_offset" : 70,
"type" : "word",
"position" : 11
}
]
}
4.stop analyzer
-
相比Simple Analyzer
-
多了stop filter
-
会把the，a，is等修饰性词语去除
返回结果
{
"tokens" : [
{
"token" : "running",
"start_offset" : 2,
"end_offset" : 9,
"type" : "word",
"position" : 0
},
{
"token" : "quick",
"start_offset" : 10,
"end_offset" : 15,
"type" : "word",
"position" : 1
},
{
"token" : "brown",
"start_offset" : 16,
"end_offset" : 21,
"type" : "word",
"position" : 2
},
{
"token" : "foxes",
"start_offset" : 22,
"end_offset" : 27,
"type" : "word",
"position" : 3
},
{
"token" : "leap",
"start_offset" : 28,
"end_offset" : 32,
"type" : "word",
"position" : 4
},
{
"token" : "over",
"start_offset" : 33,
"end_offset" : 37,
"type" : "word",
"position" : 5
},
{
"token" : "lazy",
"start_offset" : 38,
"end_offset" : 42,
"type" : "word",
"position" : 6
},
{
"token" : "dogs",
"start_offset" : 43,
"end_offset" : 47,
"type" : "word",
"position" : 7
},
{
"token" : "summer",
"start_offset" : 55,
"end_offset" : 61,
"type" : "word",
"position" : 10
},
{
"token" : "evening",
"start_offset" : 62,
"end_offset" : 69,
"type" : "word",
"position" : 11
}
]
}
5.keyword analyzer
不分词，直接将输入当成一个term输出
6.pattern analyzer
- 通过正则表达式进行分词
- 默认是\W+,非字符的符号进行分隔
7.english analyzer
三、SearchAPI及URISearch详解
1.URI Search
- 在URI中使用查询参数
2.Request Body Search
- 使用Elasticsearch提供的，基于JSON格式的更加完备的Query Domain Specific Language （DSL）
3.搜索的相关性Relevance
- 搜索是用户和搜索引擎的对话
- 用户关心的是搜索结果的相关性
- 是否可以找到所有相关的内容
- 有多少不相关的内容被返回了
- 文档的打分是否合理
- 结合业务需求，平衡结果排名
Page Rank算法
- 不仅仅是内容
- 更重要的是内容的可信度
4.衡量相关性
- Information Retrieval
- Precision (查准率) -尽可能返回较少的无关文档
- Recall (查全率) -尽量返回较多的相关文档
- Ranking -是否能够按照相关度进行排序?
5.URISearch
a.指定字段
查询出指定字段(title)值为2012的数据
GET /movies/_search?q=2012&df=title
{
"profile": "true"
}
b.泛查询
查询出任意字段值为2012的数据
GET /movies/_search?q=2012
{
"profile": "true"
}
c.Term and Phrase
- Beautiful Mind等效于Beautiful OR Mind
- “Beautiful Mind”，等效于Beautiful AND Mind。Phrase 查询，还要求前后顺序保持一致
//使用引号，Phrase查询
GET /movies/_search?q=title:"Beautiful Mind"
{
"profile": "true"
}
d.分组查询
//分组，Bool查询
GET /movies/_search?q=title:(Beautiful Mind)
{
"profile": "true"
}
必须包含Beautiful和Mind
//查找美丽心灵
GET /movies/_search?q=title:(Beautiful AND Mind)
{
"profile": "true"
}
//查找美丽心灵
GET /movies/_search?q=title:(Beautiful %2BMind)
{
"profile": "true"
}
必须包含Beautiful不包含Mind
//查找美丽心灵
GET /movies/_search?q=title:(Beautiful NOT Mind)
{
"profile": "true"
}
e.范围查询
年份大于1980
//范围查询，区间写法/数学写法
GET /movies/_search?q=year:>=1980
{
"profile": "true"
}
f.通配符查询
- ?代表1个字符，*代表0个或多个字符
title:mi?d
title:be*
四、Requestbody与QueryDSL以及QueryString&SimpleQueryString查询
1.Request Body Search
- 将查询语句通过HTTP Requedt Body发送给Elasticsearch
- Query DSL
2.查询表达式——Match
POST /movies/_search
{
"query": {
"match": {
"title": "Last Christmas"
}
}
}
POST /movies/_search
{
"query": {
"match": {
"title":{
"query": "Last Christmas",
"operator": "and"
}
}
}
}
3.短语搜索------Match Phrase
POST /movies/_search
{
"query": {
"match_phrase": {
"title":{
// 字符按照下列顺序出现
"query": "one love",
//中间可以出现其他字符
"slop": 1
}
}
}
}
4.Simple Query String Query
- 类似 Query String，但是会忽略错误的语法,同时只支持部分查询语法
- 不支持AND OR NOT，会当作字符串处理.
- Term 之间默认的关系是OR,可以指定Operator
- 支持部分逻辑
＋替代AND
| 替代OR
－替代NOT
扫一扫
573
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
