Elasticsearch-24.Pipeline聚合分析he聚合的作⽤范围及排序

最新推荐文章于 2023-06-24 11:36:13 发布
最新推荐文章于 2023-06-24 11:36:13 发布
阅读量374 收藏
点赞数
分类专栏： elasticsearch 文章标签： elasticsearch 搜索引擎 经验分享 数据分析
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/zhougubei/article/details/124127173
版权

Elasticsearch

Pipeline聚合分析

⼀个例⼦：Pipeline：min_bucket

在员⼯数最多的⼯种⾥，找出平均⼯资最低的⼯种
在这里插入图片描述

Pipeline

  • 管道的概念: ⽀持对聚合分析的结果，再次进⾏聚合分析
  • Pipeline 的分析结果会输出到原结果中，根据位置的不同，分为两类
    • Sibling - 结果和现有分析结果同级
      • Max，min，Avg & Sum Bucket
      • Stats，Extended Status Bucket
      • Percentiles Bucket
    • Parent - 结果内嵌到现有的聚合分析结果之中
      • Derivative （求导）
      • Cumultive Sum （累计求和）
      • Moving Function (滑动窗⼝)

Sibling Pipeline 的例⼦

  • 对不同类型⼯作的，平均⼯资
    • 求最⼤
    • 平均
    • 统计信息
    • 百分位数

demoAPI

DELETE employees
PUT /employees/_bulk
{ "index" : {  "_id" : "1" } }
{ "name" : "Emma","age":32,"job":"Product Manager","gender":"female","salary":35000 }
{ "index" : {  "_id" : "2" } }
{ "name" : "Underwood","age":41,"job":"Dev Manager","gender":"male","salary": 50000}
{ "index" : {  "_id" : "3" } }
{ "name" : "Tran","age":25,"job":"Web Designer","gender":"male","salary":18000 }
{ "index" : {  "_id" : "4" } }
{ "name" : "Rivera","age":26,"job":"Web Designer","gender":"female","salary": 22000}
{ "index" : {  "_id" : "5" } }
{ "name" : "Rose","age":25,"job":"QA","gender":"female","salary":18000 }
{ "index" : {  "_id" : "6" } }
{ "name" : "Lucy","age":31,"job":"QA","gender":"female","salary": 25000}
{ "index" : {  "_id" : "7" } }
{ "name" : "Byrd","age":27,"job":"QA","gender":"male","salary":20000 }
{ "index" : {  "_id" : "8" } }
{ "name" : "Foster","age":27,"job":"Java Programmer","gender":"male","salary": 20000}
{ "index" : {  "_id" : "9" } }
{ "name" : "Gregory","age":32,"job":"Java Programmer","gender":"male","salary":22000 }
{ "index" : {  "_id" : "10" } }
{ "name" : "Bryant","age":20,"job":"Java Programmer","gender":"male","salary": 9000}
{ "index" : {  "_id" : "11" } }
{ "name" : "Jenny","age":36,"job":"Java Programmer","gender":"female","salary":38000 }
{ "index" : {  "_id" : "12" } }
{ "name" : "Mcdonald","age":31,"job":"Java Programmer","gender":"male","salary": 32000}
{ "index" : {  "_id" : "13" } }
{ "name" : "Jonthna","age":30,"job":"Java Programmer","gender":"female","salary":30000 }
{ "index" : {  "_id" : "14" } }
{ "name" : "Marshall","age":32,"job":"Javascript Programmer","gender":"male","salary": 25000}
{ "index" : {  "_id" : "15" } }
{ "name" : "King","age":33,"job":"Java Programmer","gender":"male","salary":28000 }
{ "index" : {  "_id" : "16" } }
{ "name" : "Mccarthy","age":21,"job":"Javascript Programmer","gender":"male","salary": 16000}
{ "index" : {  "_id" : "17" } }
{ "name" : "Goodwin","age":25,"job":"Javascript Programmer","gender":"male","salary": 16000}
{ "index" : {  "_id" : "18" } }
{ "name" : "Catherine","age":29,"job":"Javascript Programmer","gender":"female","salary": 20000}
{ "index" : {  "_id" : "19" } }
{ "name" : "Boone","age":30,"job":"DBA","gender":"male","salary": 30000}
{ "index" : {  "_id" : "20" } }
{ "name" : "Kathy","age":29,"job":"DBA","gender":"female","salary": 20000}



# 平均工资最低的工作类型
POST employees/_search
{
  "size": 0,
  "aggs": {
    "jobs": {
      "terms": {
        "field": "job.keyword",
        "size": 10
      },
      "aggs": {
        "avg_salary": {
          "avg": {
            "field": "salary"
          }
        }
      }
    },
    "min_salary_by_job":{
      "min_bucket": {
        "buckets_path": "jobs>avg_salary"
      }
    }
  }
}


# 平均工资最高的工作类型
POST employees/_search
{
  "size": 0,
  "aggs": {
    "jobs": {
      "terms": {
        "field": "job.keyword",
        "size": 10
      },
      "aggs": {
        "avg_salary": {
          "avg": {
            "field": "salary"
          }
        }
      }
    },
    "max_salary_by_job":{
      "max_bucket": {
        "buckets_path": "jobs>avg_salary"
      }
    }
  }
}


# 平均工资的平均工资
POST employees/_search
{
  "size": 0,
  "aggs": {
    "jobs": {
      "terms": {
        "field": "job.keyword",
        "size": 10
      },
      "aggs": {
        "avg_salary": {
          "avg": {
            "field": "salary"
          }
        }
      }
    },
    "avg_salary_by_job":{
      "avg_bucket": {
        "buckets_path": "jobs>avg_salary"
      }
    }
  }
}


# 平均工资的统计分析
POST employees/_search
{
  "size": 0,
  "aggs": {
    "jobs": {
      "terms": {
        "field": "job.keyword",
        "size": 10
      },
      "aggs": {
        "avg_salary": {
          "avg": {
            "field": "salary"
          }
        }
      }
    },
    "stats_salary_by_job":{
      "stats_bucket": {
        "buckets_path": "jobs>avg_salary"
      }
    }
  }
}


# 平均工资的百分位数
POST employees/_search
{
  "size": 0,
  "aggs": {
    "jobs": {
      "terms": {
        "field": "job.keyword",
        "size": 10
      },
      "aggs": {
        "avg_salary": {
          "avg": {
            "field": "salary"
          }
        }
      }
    },
    "percentiles_salary_by_job":{
      "percentiles_bucket": {
        "buckets_path": "jobs>avg_salary"
      }
    }
  }
}



#按照年龄对平均工资求导
POST employees/_search
{
  "size": 0,
  "aggs": {
    "age": {
      "histogram": {
        "field": "age",
        "min_doc_count": 1,
        "interval": 1
      },
      "aggs": {
        "avg_salary": {
          "avg": {
            "field": "salary"
          }
        },
        "derivative_avg_salary":{
          "derivative": {
            "buckets_path": "avg_salary"
          }
        }
      }
    }
  }
}


#Cumulative_sum
POST employees/_search
{
  "size": 0,
  "aggs": {
    "age": {
      "histogram": {
        "field": "age",
        "min_doc_count": 1,
        "interval": 1
      },
      "aggs": {
        "avg_salary": {
          "avg": {
            "field": "salary"
          }
        },
        "cumulative_salary":{
          "cumulative_sum": {
            "buckets_path": "avg_salary"
          }
        }
      }
    }
  }
}

#Moving Function
POST employees/_search
{
  "size": 0,
  "aggs": {
    "age": {
      "histogram": {
        "field": "age",
        "min_doc_count": 1,
        "interval": 1
      },
      "aggs": {
        "avg_salary": {
          "avg": {
            "field": "salary"
          }
        },
        "moving_avg_salary":{
          "moving_fn": {
            "buckets_path": "avg_salary",
            "window":10,
            "script": "MovingFunctions.min(values)"
          }
        }
      }
    }
  }
}

Parent Pipeline：Derivative

按照年龄，对⼯资进⾏求导（看⼯资发展的趋势）
在这里插入图片描述

Parent Pipeline 的例⼦

  • 年龄直⽅图划分的平均⼯资
    • Cumulative Sum
    • Moving Function

聚合的作⽤范围及排序

聚合的作⽤范围

在这里插入图片描述

Filter

在这里插入图片描述

Post_Filter

在这里插入图片描述

Global

在这里插入图片描述

排序

在这里插入图片描述

基于⼦聚合的值排序

在这里插入图片描述

demoAPI

DELETE /employees
PUT /employees/
{
  "mappings" : {
      "properties" : {
        "age" : {
          "type" : "integer"
        },
        "gender" : {
          "type" : "keyword"
        },
        "job" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 50
            }
          }
        },
        "name" : {
          "type" : "keyword"
        },
        "salary" : {
          "type" : "integer"
        }
      }
    }
}

PUT /employees/_bulk
{ "index" : {  "_id" : "1" } }
{ "name" : "Emma","age":32,"job":"Product Manager","gender":"female","salary":35000 }
{ "index" : {  "_id" : "2" } }
{ "name" : "Underwood","age":41,"job":"Dev Manager","gender":"male","salary": 50000}
{ "index" : {  "_id" : "3" } }
{ "name" : "Tran","age":25,"job":"Web Designer","gender":"male","salary":18000 }
{ "index" : {  "_id" : "4" } }
{ "name" : "Rivera","age":26,"job":"Web Designer","gender":"female","salary": 22000}
{ "index" : {  "_id" : "5" } }
{ "name" : "Rose","age":25,"job":"QA","gender":"female","salary":18000 }
{ "index" : {  "_id" : "6" } }
{ "name" : "Lucy","age":31,"job":"QA","gender":"female","salary": 25000}
{ "index" : {  "_id" : "7" } }
{ "name" : "Byrd","age":27,"job":"QA","gender":"male","salary":20000 }
{ "index" : {  "_id" : "8" } }
{ "name" : "Foster","age":27,"job":"Java Programmer","gender":"male","salary": 20000}
{ "index" : {  "_id" : "9" } }
{ "name" : "Gregory","age":32,"job":"Java Programmer","gender":"male","salary":22000 }
{ "index" : {  "_id" : "10" } }
{ "name" : "Bryant","age":20,"job":"Java Programmer","gender":"male","salary": 9000}
{ "index" : {  "_id" : "11" } }
{ "name" : "Jenny","age":36,"job":"Java Programmer","gender":"female","salary":38000 }
{ "index" : {  "_id" : "12" } }
{ "name" : "Mcdonald","age":31,"job":"Java Programmer","gender":"male","salary": 32000}
{ "index" : {  "_id" : "13" } }
{ "name" : "Jonthna","age":30,"job":"Java Programmer","gender":"female","salary":30000 }
{ "index" : {  "_id" : "14" } }
{ "name" : "Marshall","age":32,"job":"Javascript Programmer","gender":"male","salary": 25000}
{ "index" : {  "_id" : "15" } }
{ "name" : "King","age":33,"job":"Java Programmer","gender":"male","salary":28000 }
{ "index" : {  "_id" : "16" } }
{ "name" : "Mccarthy","age":21,"job":"Javascript Programmer","gender":"male","salary": 16000}
{ "index" : {  "_id" : "17" } }
{ "name" : "Goodwin","age":25,"job":"Javascript Programmer","gender":"male","salary": 16000}
{ "index" : {  "_id" : "18" } }
{ "name" : "Catherine","age":29,"job":"Javascript Programmer","gender":"female","salary": 20000}
{ "index" : {  "_id" : "19" } }
{ "name" : "Boone","age":30,"job":"DBA","gender":"male","salary": 30000}
{ "index" : {  "_id" : "20" } }
{ "name" : "Kathy","age":29,"job":"DBA","gender":"female","salary": 20000}



# Query
POST employees/_search
{
  "size": 0,
  "query": {
    "range": {
      "age": {
        "gte": 20
      }
    }
  },
  "aggs": {
    "jobs": {
      "terms": {
        "field":"job.keyword"
        
      }
    }
  }
}


#Filter
POST employees/_search
{
  "size": 0,
  "aggs": {
    "older_person": {
      "filter":{
        "range":{
          "age":{
            "from":35
          }
        }
      },
      "aggs":{
         "jobs":{
           "terms": {
        "field":"job.keyword"
      }
      }
    }},
    "all_jobs": {
      "terms": {
        "field":"job.keyword"
        
      }
    }
  }
}



#Post field. 一条语句，找出所有的job类型。还能找到聚合后符合条件的结果
POST employees/_search
{
  "aggs": {
    "jobs": {
      "terms": {
        "field": "job.keyword"
      }
    }
  },
  "post_filter": {
    "match": {
      "job.keyword": "Dev Manager"
    }
  }
}


#global
POST employees/_search
{
  "size": 0,
  "query": {
    "range": {
      "age": {
        "gte": 40
      }
    }
  },
  "aggs": {
    "jobs": {
      "terms": {
        "field":"job.keyword"
        
      }
    },
    
    "all":{
      "global":{},
      "aggs":{
        "salary_avg":{
          "avg":{
            "field":"salary"
          }
        }
      }
    }
  }
}


#排序 order
#count and key
POST employees/_search
{
  "size": 0,
  "query": {
    "range": {
      "age": {
        "gte": 20
      }
    }
  },
  "aggs": {
    "jobs": {
      "terms": {
        "field":"job.keyword",
        "order":[
          {"_count":"asc"},
          {"_key":"desc"}
          ]
        
      }
    }
  }
}


#排序 order
#count and key
POST employees/_search
{
  "size": 0,
  "aggs": {
    "jobs": {
      "terms": {
        "field":"job.keyword",
        "order":[  {
            "avg_salary":"desc"
          }]
        
        
      },
    "aggs": {
      "avg_salary": {
        "avg": {
          "field":"salary"
        }
      }
    }
    }
  }
}


#排序 order
#count and key
POST employees/_search
{
  "size": 0,
  "aggs": {
    "jobs": {
      "terms": {
        "field":"job.keyword",
        "order":[  {
            "stats_salary.min":"desc"
          }]
        
        
      },
    "aggs": {
      "stats_salary": {
        "stats": {
          "field":"salary"
        }
      }
    }
    }
  }
}
ES Pipeline Aggregation(管道聚合)
weixin_44195108的博客
03-19 2132
管道聚合处理来自其他聚合而不是文档集的输出，将信息添加到输出树中。 注：关于脚本聚合目前在本文中暂时不会涉及。 主要有如下两种管道聚合方式： parent sibling 下面一一介绍ES定义的管道聚合。 Avg Bucket 聚合 同级管道聚合，它计算同级聚合中指定度量的平均值。同级聚合必须是多桶聚合，针对的是度量聚合(metric Aggregation)。 示例如下： 1{ 2 "av...
ElasticSearch | Pipeline 聚合分析
乌鲁木齐001号程序员
05-28 230
Pipeline 支持对聚合分析的结果，再次进行聚合分析； Pipeline 的分析结果会输出到原结果中，根据位置的不同，分为两类： Sibling - 结果和现有分析结果同级 max / min / avg & sum bucket stats / extended status bucket percentiles bucket Parent - 结果内嵌到现有的聚合分析结果...
深入理解Elasticsearch Pipeline聚集(1)
neweastsun的专栏
02-19 855
深入理解Elasticsearch Pipeline聚集(1) 度量聚集和桶聚集一般用于文档中的数值型字段，而本文讨论的管道聚集针对其他聚集产生的输出值，因此管道聚集是针对中间值而不是原始文档数据。对于及时复杂统计和数学度量，如累加和、导数（变化情况）、移动平均等非常有用。 本文讨论管道聚集的两个基本类型，通过示例展示常用的管道聚集，如求和、累加求和、最小值、最大值、平均值以及导数等管道聚集。 1...
5.3 ElasticSearch聚合分析之Pipeline
Steven的博客
09-05 225
1.Sibling 找出所有bucket中值最小的bucket名称和值，如找出平均薪水最低的岗位。 PUT /employee/_mapping { "size": 0, "aggs": { "job_terms": { "terms": { "field": "job", "size": 10 }, "aggs": { "salary_avg": { "avg": { "field": "salary" } }
Elasticsearch聚合分析 Pipeline聚合分析
男儿当自强
02-01 282
初始化数据 DELETE employees PUT /employees/_bulk { "index" : { "_id" : "1" } } { "name" : "Emma","age":32,"job":"Product Manager","gender":"female","salary":35000 } { "index" : { "_id" : "2" } } { "name" : "Underwood","age":41,"job":"Dev Manager","gender":"m.
elasticsearch-7.17.7-windows-x86-64.zip
11-06
7. **聚合分析**：除了基本的搜索，Elasticsearch还提供丰富的聚合功能，可以进行数据的统计分析，如平均值、最大值、最小值计算，以及桶聚合（例如按时间范围、地理位置等分组）。 8. **安全性**：Elasticsearch ...
最新版linux elasticsearch-7.12.0-linux-x86_64.tar.gz
03-30
Elasticsearch是一个开源的全文搜索引擎，它基于Lucene构建，被广泛应用于数据分析、日志聚合、实时搜索等场景。在Linux环境下，Elasticsearch 7.12.0的安装和使用是许多IT专业人士需要掌握的关键技能之一。这个版本...
elasticsearch-6.2.4.tar.gz.zip
08-31
6. **聚合分析**：Elasticsearch 提供丰富的聚合功能，允许用户对数据进行统计分析，例如平均值、总和、最大值等，甚至支持复杂的数据透视和桶聚合。 7. **监控与告警**：内置的监控工具可以收集集群的性能指标，并...
elasticsearch-7.17.6及对应版本IK分词
02-19
elasticsearch-7.17.6及对应版本IK分词 适合人群：elasticsearch初学者 Elasticsearch 是位于 Elastic Stack 核心的分布式搜索和分析引擎。Logstash 和 Beats 有助于收集、聚合和丰富您的数据并将其存储在 Elastic...
elasticsearch-7.14.2-windows-x86_64.zip
11-18
- **聚合分析**：Elasticsearch支持丰富的聚合函数，如术语聚合、范围聚合等，用于数据挖掘和分析。 Elasticsearch的强大之处在于它的灵活性和可扩展性，适用于日志分析、实时监控、全文搜索等多种场景。了解并熟练...
elasticsearch之聚合的作⽤范围及排序
种一颗十年前的树
06-17 440
聚合的作⽤范围及排序 es聚合分析的默认作用范围是query的查询结果集 同时es还支持以下方式改变聚合的作用范围 filter post_filter global DELETE /employees PUT /employees/ { "mappings" : { "properties" : { "age" : { "type" : "integer" }, "gender" : { "type
elasticsearch之Pipeline 聚合分析
种一颗十年前的树
06-16 586
Pipeline 聚合分析 DELETE employees PUT /employees/_bulk { "index" : { "_id" : "1" } } { "name" : "Emma","age":32,"job":"Product Manager","gender":"female","salary":35000 } { "index" : { "_id" : "2" } } { "name" : "Underwood","age":41,"job":"Dev Manager","ge
ES聚合之Pipeline聚合语法讲解
前者已逝，后者未至。
09-12 1565
掌握Pipeline aggregations（管道聚合，相当于MySQL中的having ）的基本语法。它比另外两种类型的聚合用得较少，这里只举了少量案例供大家参考分析。
Elasticsearch核心技术与实战学习笔记 46 | Pipeline聚合分析
bohu83的博客
06-27 659
一 序 本文属于极客时间Elasticsearch核心技术与实战学习笔记系列。 可以结合上一节课来理解，使用了做统计分析的 聚合分析：英文为Aggregation，是es除搜索功能外提供的针对es数据做统计分析的功能。 功能丰富：提供Bucket、Metric、Pipeline等多种分析方式，可以满足大部分的分析需求 实时性高：所有的计算结果都是即时返回的，而hadoop等大数据系统一般都是T+1级别的 例如聚合分析可以回答如下问题： 请告诉我最近1周每天的订单成交量有多少？ 请告诉我最近1个..
【Elasticsearch】 06-管道聚合查询详解及示例
程序员小屋
06-24 835
管道聚合查询是一种在聚合查询结果上进行进一步计算和分析的方式。它可以用于计算新的指标、进行比较、排序和筛选等操作。管道聚合查询通常作为其他聚合查询的后续处理步骤。
ES中pipeline聚合
u012663412的博客
08-19 495
es pipeline aggregation
ElasticSearch学习笔记之二十八 细说Pipeline Aggregations
灵动的艺术的博客
10-26 4155
ElasticSearch学习笔记之二十八 细说Pipeline AggregationsAvg Bucket Aggregation(平均值分组聚合)Syntax(语法)avg_bucket 参数Max Bucket Aggregation(最大分组聚合)Syntax(语法)max_bucket 参数Min Bucket Aggregation(最小分组聚合)Syntax(语法)min_buck...
Elasticsearch Pipeline 详解
热门推荐
chunqiqian1285的博客
07-12 1万+
并非完全翻译,会穿插相关的Java API来来辅助进行说明(但基本会覆盖官方所有翻译) 原文地址:Processors 版本:6.0.0 说明 本文是建立在有一些Elasticsearch基础和了解相关Pipeline概念的人 Ingest Node...
elasticsearch-7.12.0-py2.py3-none-any.whl
最新发布
07-27
elasticsearch-7.12.0-py2.py3-none-any.whl 是 Elasticsearch 的 Python 客户端库的一个安装文件。Elasticsearch 是一个开源的实时分布式搜索和分析引擎，用于处理大规模数据集。它提供了一个简单可扩展的 RESTful ...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值