es搜索核心与实战Day08
一、分页于遍历
1.From/Size
- 默认情况下,查询按照相关度算分排序,返回前十条记录
- 容易理解的分页方案
-
- From:开始位置
- Size:期望获取文档的总数
2.Search After避免深度分页问题
- 避免深度分页的性能问题,可以实时获取下一页文档信息
-
- 不支持指定页数(From)
- 只能往下翻
- 第一步搜索需要指定sort,并保证值是唯一的(可以通过加入_id保证唯一性)
- 然后使用上一次,最后一个文档的sort值进行查询
POST tmdb/_search
{
"from": 0,
"size": 10,
"query": {
"match_all": {}
}
}
//Search After避免深度分页的问题
DELETE users
POST users/_doc
{"name":"user1","age":10}
POST users/_doc
{"name":"user2","age":11}
POST users/_doc
{"name":"user3","age":12}
POST users/_doc
{"name":"user4","age":13}
POST users/_count
POST users/_search
{
"size": 1,
"query": {
"match_all": {}
},
"sort": [
{"age":"desc"},
{"_id":"asc"}
]
}
POST users/_search
{
"size": 1,
"query": {
"match_all": {}
},
"search_after":
[
12,
"nVJs4HIBJjUASKqbxyMt"
]
,
"sort": [
{"age": "desc"},
{"_id":"asc"}
]
}
3.Scroll API
- 创建一个快照,有新的数据写入以后,无法被查到
- 每次查询后,输入上一次的Scroll Id
//Scroll API
DELETE users
POST users/_doc
{"name":"user1","age":10}
POST users/_doc
{"name":"user2","age":20}
POST users/_doc
{"name":"user3","age":30}
POST users/_doc
{"name":"user4","age":40}
POST users/_count
POST users/_doc
{"name":"user4","age":40}
POST /users/_search?scroll=5m
{
"size": 1,
"query": {
"match_all": {
}
}
}
POST /_search/scroll
{
"scroll":"1m",
"scroll_id":"DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAQDQWbk1XNFdQaE1TZWFneGJLQXAxUDhydw=="
}
4.不同的搜索类型和使用场景
- Regular
-
- 需要实时获取顶部的部分文档。例如查询最新的订单
- Scroll
-
- 需要全部文档,例如导出全部数据
- Pagination
-
- From和Size
- 如果需要深度分页,择选用Search After
二、处理并发读写操作
1.并发控制的必要性
- 两个Wed程序同时更新某个文档,如果缺乏有效的并发,会导致更改数据丢失
- 悲观的并发控制
-
- 假定有变更冲突的可能。会对资源加锁,防止冲突。例如数据库行锁
- 乐观并发控制
-
- 假定冲突是不会发生的,不会阻塞正在尝试的操作。如果数据在读写中被修改,更新将会失败。应用程序决定如何解决冲突,例如重试更新,使用新的数据,或者将错误报告给用户
- ES采用的是乐观并发
2.ES的乐观并发控制
- ES中的文档是不可变更的。如果你更新一个文档,会将旧文档标记为删除,同时增加一个全新的文档。同时文档的version字段加1
- 内部版本控制
-
- If_seq_no+If_primary_term
- 使用外部版本(使用其他数据库作为主要数据存储)
-
- version+version_type=external
//并发读写、并发锁
DELETE products
PUT products
PUT products/_doc/1
{
"title":"iphone",
"count":100
}
GET products/_doc/1
PUT products/_doc/1?if_seq_no=0&if_primary_term=1
{
"title":"iphone",
"count":103
}
GET products/_doc/1
PUT products/_doc/1?version=30001&version_type=external
{
"title":"iphone",
"count":100
}
三、Bucket&Metric聚合分析及嵌套聚合
1.Bucket&Metric Aggregation
- Metric - 一些系列的统计方法
- Bucket - 一组满足条件的文档
- Aggregation的语法
-
- 属于Search的一部分。一般情况下,建议将其Size指定为0
2.Metric Aggregation
- 单值分析:只输出一个分析结果
-
- min,max,,avg,sum
- Cardinality(类似distinct Count)
- 多值分析:输出多个分析结果
-
- stats,extended stats
- percentile,percentile rank
- top hite(排在前面的示例)
Demo
DELETE /employees
//定义员工表
PUT /employees/
{
"mappings" : {
"properties" : {
"age" : {
"type" : "integer"
},
"gender" : {
"type" : "keyword"
},
"job" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 50
}
}
},
"name" : {
"type" : "keyword"
},
"salary" : {
"type" : "integer"
}
}
}
}
//写入员工表
PUT /employees/_bulk
{ "index" : { "_id" : "1" } }
{ "name" : "Emma","age":32,"job":"Product Manager","gender":"female","salary":35000 }
{ "index" : { "_id" : "2" } }
{ "name" : "Underwood","age":41,"job":"Dev Manager","gender":"male","salary": 50000}
{ "index" : { "_id" : "3" } }
{ "name" : "Tran","age":25,"job":"Web Designer","gender":"male","salary":18000 }
{ "index" : { "_id" : "4" } }
{ "name" : "Rivera","age":26,"job":"Web Designer","gender":"female","salary":22000}
{ "index" : { "_id" : "5" } }
{ "name" : "Rose","age":25,"job":"QA","gender":"female","salary":18000 }
{ "index" : { "_id" : "6" } }
{ "name" : "Lucy","age":31,"job":"QA","gender":"female","salary": 25000}
{ "index" : { "_id" : "7" } }
{ "name" : "Byrd","age":27,"job":"QA","gender":"male","salary":20000 }
{ "index" : { "_id" : "8" } }
{ "name" : "Foster","age":27,"job":"Java Programmer","gender":"male","salary": 20000}
{ "index" : { "_id" : "9" } }
{ "name" : "Gregory","age":32,"job":"Java Programmer","gender":"male","salary":22000 }
{ "index" : { "_id" : "10" } }
{ "name" : "Bryant","age":20,"job":"Java Programmer","gender":"male","salary": 9000}
{ "index" : { "_id" : "11" } }
{ "name" : "Jenny","age":36,"job":"Java Programmer","gender":"female","salary":38000 }
{ "index" : { "_id" : "12" } }
{ "name" : "Mcdonald","age":31,"job":"Java Programmer","gender":"male","salary": 32000}
{ "index" : { "_id" : "13" } }
{ "name" : "Jonthna","age":30,"job":"Java Programmer","gender":"female","salary":30000}
{ "index" : { "_id" : "14" } }
{ "name" : "Marshall","age":32,"job":"Javascript Programmer","gender":"male","salary": 25000}
{ "index" : { "_id" : "15" } }
{ "name" : "King","age":33,"job":"Java Programmer","gender":"male","salary":28000 }
{ "index" : { "_id" : "16" } }
{ "name" : "Mccarthy","age":21,"job":"Javascript Programmer","gender":"male","salary": 16000}
{ "index" : { "_id" : "17" } }
{ "name" : "Goodwin","age":25,"job":"Javascript Programmer","gender":"male","salary": 16000}
{ "index" : { "_id" : "18" } }
{ "name" : "Catherine","age":29,"job":"Javascript Programmer","gender":"female","salary": 20000}
{ "index" : { "_id" : "19" } }
{ "name" : "Boone","age":30,"job":"DBA","gender":"male","salary": 30000}
{ "index" : { "_id" : "20" } }
{ "name" : "Kathy","age":29,"job":"DBA","gender":"female","salary": 20000}
# Metric 聚合,找到最低的工资
POST employees/_search
{
"size": 0,
"aggs": {
"min_salary": {
"min": {
"field":"salary"
}
}
}
}
# Metric 聚合,找到最高的工资
POST employees/_search
{
"size": 0,
"aggs": {
"max_salary": {
"max": {
"field":"salary"
}
}
}
}
# 多个 Metric 聚合,找到最低最高和平均工资
POST employees/_search
{
"size": 0,
"aggs": {
"max_salary": {
"max": {
"field": "salary"
}
},
"min_salary": {
"min": {
"field": "salary"
}
},
"avg_salary": {
"avg": {
"field": "salary"
}
}
}
}
# 一个聚合,输出多值
POST employees/_search
{
"size": 0,
"aggs": {
"stats_salary": {
"stats": {
"field":"salary"
}
}
}
}
3.Bucket Aggregation
- 按照一定的规则,将文档分配到不同的桶中,从而达到分类的目的。ES提供的一些常见的Bucket Aggregation
-
- Terms
- 数字类型
-
- Range/Data Range
- Histogram/Data Histogram
- 支持嵌套:也就在桶里再做分桶
4.Terms Aggregation
- 字段需要打开fielddata,才能进行Terms Aggregation
-
- Keyword默认支持fielddata
- Text需要在Mapping中enable。会按照分词后的结果进行分
- Demo
-
- 对job和job.keyword进行聚合
- 对性别进行Terms聚合
- 指定bucket size
# 对keword 进行聚合
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
//keyword不做分词
"field":"job.keyword"
}
}
}
}
# 对 Text 字段进行 terms 聚合查询,失败
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field":"job"
}
}
}
}
# 对 Text 字段打开 fielddata,支持terms aggregation
PUT employees/_mapping
{
"properties" : {
"job":{
"type": "text",
"fielddata": true
}
}
}
# 对job.keyword 和 job 进行 terms 聚合,分桶的总数并不一样
POST employees/_search
{
"size": 0,
"aggs": {
"cardinate": {
"cardinality": {
"field": "job"
}
}
}
}
# 对 性别的 keyword 进行聚合
POST employees/_search
{
"size": 0,
"aggs": {
"gender": {
"terms": {
"field":"gender"
}
}
}
}
#指定 bucket 的 size
POST employees/_search
{
"size": 0,
"aggs": {
"ages_5": {
"terms": {
"field":"age",
"size":3
}
}
}
}
# 指定size,不同工种中,年纪最大的3个员工的具体信息
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field":"job.keyword"
},
"aggs":{
"old_employee":{
"top_hits":{
"size":3,
"sort":[
{
"age":{
"order":"desc"
}
}
]
}
}
}
}
}
}
优化Terms聚合性能
打开配置聚合分析经常打开对性能要求索引有新的文档写入
5.Range&Histogram
- 按照数字的范围,进行分桶
- 在Range Aggregation中,可以自定义Key
- Demo
#Salary Ranges 分桶,可以自己定义 key
POST employees/_search
{
"size": 0,
"aggs": {
"salary_range": {
"range": {
"field":"salary",
"ranges":[
{
"to":10000
},
{
"from":10000,
"to":20000
},
{
"key":">20000",
"from":20000
}
]
}
}
}
}
#Salary Histogram,工资0到10万,以 5000一个区间进行分桶
POST employees/_search
{
"size": 0,
"aggs": {
"salary_histrogram": {
"histogram": {
"field":"salary",
"interval":5000,
"extended_bounds":{
"min":0,
"max":100000
}
}
}
}
}
# 嵌套聚合1,按照工作类型分桶,并统计工资信息
POST employees/_search
{
"size": 0,
"aggs": {
"Job_salary_stats": {
"terms": {
"field": "job.keyword"
},
"aggs": {
"salary": {
"stats": {
"field": "salary"
}
}
}
}
}
}
# 多次嵌套。根据工作类型分桶,然后按照性别分桶,计算工资的统计信息
POST employees/_search
{
"size": 0,
"aggs": {
"Job_gender_stats": {
"terms": {
"field": "job.keyword"
},
"aggs": {
"gender_stats": {
"terms": {
"field": "gender"
},
"aggs": {
"salary_stats": {
"stats": {
"field": "salary"
}
}
}
}
}
}
}
}
四、Pipeline聚合分析(通过bucket_path关键字指定路径)
1.Pipeline
- 管道的概念:支持对聚合分析的结果,再次进行聚合分析
- Pipeline的分析结果会输出到原有结果中,根据位置的不同,分为两类
-
- Sibling-结果和现有分析结果同级
-
- Max,min,Avg&Sum Bucket
- Stats,Extended Status Bucket
- Percentiles Bucket
- Parent-结果内嵌到现有的聚合分析结果之中
-
- Derivative(求导)
- Cumultive Sum(累计求和)
- Moving Function(滑动窗口)
#平均工资最低的工作类型
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword",
"size": 10
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
}
}
},
"min_salary_by_job":{
"min_bucket": {
"buckets_path": "jobs>avg_salary"
}
}
}
}
#平均工资最高的工作类型
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword",
"size": 10
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
}
}
},
"min_salary_by_job":{
"max_bucket": {
"buckets_path": "jobs>avg_salary"
}
}
}
}
#平均工资的平均工资
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword",
"size": 10
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
}
}
},
"avg_salary_by_job":{
"avg_bucket": {
"buckets_path": "jobs>avg_salary"
}
}
}
}
#平均工资的统计分析
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword",
"size": 10
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
}
}
},
"stats_salary_by_job":{
"stats_bucket": {
"buckets_path": "jobs>avg_salary"
}
}
}
}
#平均工资的百分位数
POST employees/_search
{
"size": 0,
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword",
"size": 10
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
}
}
},
"percentiles_salary_by_job":{
"percentiles_bucket": {
"buckets_path": "jobs>avg_salary"
}
}
}
}
#按照年龄对平均工资求导
POST employees/_search
{
"size":0,
"aggs": {
"age": {
"histogram": {
"field": "age",
"min_doc_count": 1,
"interval": 1
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
},
"derivative_avg_salary":{
"derivative": {
"buckets_path": "avg_salary"
}
}
}
}
}
}
#Cumulative_sum(累计求和)
POST employees/_search
{
"size":0,
"aggs": {
"age": {
"histogram": {
"field": "age",
"min_doc_count": 1,
"interval": 1
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
},
"cumulative_salary":{
"cumulative_sum": {
"buckets_path":"avg_salary"
}
}
}
}
}
}
#Moving Function(移动平均)
POST employees/_search
{
"size":0,
"aggs": {
"age": {
"histogram": {
"field": "age",
"min_doc_count": 1,
"interval": 1
},
"aggs": {
"avg_salary": {
"avg": {
"field": "salary"
}
},
"moving_avg_salary":{
"moving_fn": {
"buckets_path":"avg_salary",
"window":10,
"script":"MovingFunctions.min(values)"
}
}
}
}
}
}
五、作用范围与排序
1.聚合的作用范围
- ES聚合分析的默认作用范围是query的查询结果集
- 同时ES还支持以下方式改变聚合的作用范围
-
- Filter
- Post Filter
- Global
#Query
POST employees/_search
{
"size": 0,
"query": {
"range": {
"age": {
//20
"gte": 40
}
}
},
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword"
}
}
}
}
#Filter
POST employees/_search
{
"size": 0,
"aggs": {
"order_person": {
"filter": {
"range": {
"age": {
"from":35
}
}
},
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword"
}
}
}},
"all_jobs":{
"terms":{
"field":"job.keyword"
}
}
}
}
#Post field,一条语句,找出所有的job类型。还能找到聚合后符合条件的结果
POST employees/_search
{
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword"
}
}
},
"post_filter": {
"match":{
"job.keyword":"Dev Manager"
}
}
}
#global
POST employees/_search
{
"size": 0,
"query": {
"range": {
"age": {
"gte": 40
}
}
},
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword"
}
},
"all":{
"global": {},
"aggs": {
"salary_avg": {
"avg": {
"field": "salary"
}
}
}
}
}
}
2.排序
- 指定order,按照count和key进行排序
-
- 默认情况,按照count降序排序
- 指定size,就能返回相应的桶
#排序 order
#count and key
POST employees/_search
{
"size": 0,
"query": {
"range": {
"age": {
"gte": 20
}
}
},
"aggs": {
"jobs": {
"terms": {
"field": "job.keyword",
"order": [
//升序
{"_count":"asc"},
{"_key":"desc"}
]
}
}
}
}
六、聚合分析的原理及精准度问题
1.分布式系统的近似统计算法
2.Min聚合分析的执行流程
3.Terms Aggregation的返回值
- 在Terns Aggregation的返回中有两个特殊的数值
-
- doc_count_error_upper_bound:被遗漏的term分桶,包含的文档,有可能的最大值
- sum_other_doc_count:除了返回结果bucket的terms以外,其他terms的文档总数(总数-返回的总数)
4.Terms聚合分析的执行
5.如何解决Terms不准的问题:提升shard_size的参数
- Terms聚合分析不准的原因,数据分散在多个分片上,CoordinatingNode无法获取数据全貌
- 解决方法1:当数据量不大时,设置OrimaryShard为1;实现准确性
- 方案2:在分布式数据上,设置shard-size参数,提高精准度
-
- 原理:每次从Shard上额外多获取数据,提升准确率
6.shard_size设定
- 调整shard_size大小,降低doc_count_error_upper_bound来提升准确度
-
- 增加整体计算量,提高了准确度,但会降低相应时间
- Shard Size默认大小设定
-
- shard_size=size*1.5+10
POST _reindex
{
"source": {
"index": "kibana_sample_data_flights"
},
"dest": {
"index": "my_flights"
}
}
GET kibana_sample_data_flights/_count
GET my_flights/_count
get kibana_sample_data_flights/_search
GET kibana_sample_data_flights/_search
{
"size": 0,
"aggs": {
"weather": {
"terms": {
"field":"OriginWeather",
"size":5,
"show_term_doc_count_error":true
}
}
}
}
GET my_flights/_search
{
"size": 0,
"aggs": {
"weather": {
"terms": {
"field":"OriginWeather",
"size":1,
"shard_size":1,
"show_term_doc_count_error":true
}
}
}
}