ES相关概念理解,常用查询

Query 与 Filter

1、query是要相关性评分的，filter不要
2、query结果无法缓存，filter可以（空间满，使用LRU淘汰）

全文搜索、评分排序，使用query
是非过滤、精确匹配，使用filter

GET /nginx-elk-2020.03.22/_search
{
  "query": {
    "bool":{
      "must": [
        {
         "match": {
            "http_host":"product.mlalgo.api"
          }
        }
      ], "filter": {
        "range": {
          "@timestamp": {
            "from": "2020-03-22T10:00:52.000Z",
            "to": "2020-03-22T18:00:52.000Z"
          }
        }
      }
    }
  }
}

bool 过滤

bool 过滤可以用来合并多个过滤条件查询结果的布尔逻辑，它包含一下操作符：
must :: 多个查询条件的完全匹配,相当于 and 。
must_not :: 多个查询条件的相反匹配，相当于 not 。
should :: 至少有一个查询条件匹配, 相当于 or 。

multi_match 查询

GET /nginx-elk-2020.03.22/_search
{
  "size": 10, 
  "query": {
   "multi_match": {
     "query": "product",
     "fields": ["http_host", "request_uri"]
   }
  }
}

使用 from 及 size 参数进行分页

GET /nginx-elk-2020.03.24/_search
{
  "from": 30,
  "size":10
}

terms 过滤

terms 跟 term 有点类似，但 terms 允许指定多个匹配条件

GET /nginx-elk-2020.03.22/_search
{
  "size": 10, 
  "query": {
    "bool": {
      "must": [
        {
          "terms": {
          "http_host": ["product.mlalgo.api", "api"]
          }
        }
        ]
    }
  }
}

排序

GET /nginx-elk-2020.03.22/_search
{
  "size": 10, 
  "query": {
    "bool": {
      "filter": {
        "term": {
          "http_host": "product.mlalgo.api"
        }
      }
    }
  }, "sort": [
    {
      "upstream_response_time": {
        "order": "desc"
      }
    }
  ]
}

多级排序

GET /nginx-elk-2020.03.22/_search
{
  "size": 10, 
  "query": {
    "bool": {
      "filter": {
        "term": {
          "http_host": "product.mlalgo.api"
        }
      }
    }
  }, "sort": [
    {
      "upstream_response_time": {
        "order": "desc"
      }
    },{
      "_score":{
        "order":"desc"
      }
    }
  ]
}

aggregation

聚合查看时间分布

GET /nginx-elk-2020.03.24/_search
{
 "size": 10
 , "query": {
   "bool": {
     "filter": {
       "term": {
         "http_host": "product.mlalgo.api"
       }
     }
   }
 }
 , "aggs": {
   "percentile_over_time": {
     "date_histogram": {
       "field": "@timestamp",
       "interval": "hour"
     }, "aggs": {
       "count_persent": {
         "percentiles": {
           "field": "upstream_response_time",
           "percents": [
             1,
             5,
             25,
             50,
             75,
             95,
             99
           ]
         }
       }
     }
   }
 }
}

聚合查看各项平均指标

GET /nginx-elk-2020.03.22/_search
{
  "size": 10, 
  "query": {
    "bool": {
      "must": [
        {
          "term": {
          "http_host": "product.mlalgo.api"
        }},
        {"term": {
          "status": "200"
        }},
        {
          "range": {
            "@timestamp": {
              "from": "2020-03-22T10:00:52.000Z",
              "to": "2020-03-22T18:00:52.000Z"
            }
          }
        }
      ]
      , "filter": {
        "range": {
            "upstream_response_time": {
              "gte": 0.05,
              "lte": 1
            }
          }
      }
    }
  }
  , "aggs": {
    "rt_stats": {
      "extended_stats": {
        "field": "upstream_response_time"
      }
    }
  }
}

pipeline aggregation

在已有aggregation返回数组数据之后，再对这组数值值做一次运算。比如对响应时间设置如下：周期为7，移动窗口为30，alpha、beta、gamma参数为0.5，holt-winter季节性预测2个未来值：

GET /nginx-elk-2020.03.24/_search
{
 "size": 10
 , "query": {
   "bool": {
     "filter": {
       "term": {
         "http_host": "product.mlalgo.api"
       }
     }
   }
 }
 , "aggs": {
   "response_histogram": {
     "date_histogram": {
       "field": "@timestamp",
       "interval": "hour"
     }, "aggs": {
       "avg_response": {
         "avg": {
           "field": "upstream_response_time"
         }
       },
       "time_movavg":{
         "moving_avg": {
           "buckets_path": "avg_response",
           "window": 30,
           "model": "holt_winters",
           "predict":2,
           "settings": {
             "type": "mult",
             "alpha": 0.5,
             "beta": 0.5,
             "gamma": 0.5
           }
         }
       }
     }
   }
 }
}

桶排序：aggregation sort

排序时用“.”表达桶之间的嵌套关系

GET /nginx-elk-2020.03.22/_search
{
  "size": 10, 
  "query": {
    "bool": {
      "must": [
        {
          "term": {
          "http_host": "product.mlalgo.api"
        }},
        {"term": {
          "status": "200"
        }},
        {
          "range": {
            "@timestamp": {
              "from": "2020-03-22T10:00:52.000Z",
              "to": "2020-03-22T18:00:52.000Z"
            }
          }
        }
      ]
      , "filter": {
        "range": {
            "upstream_response_time": {
              "gte": 0.05,
              "lte": 1
            }
          }
      }
    }
  }, "aggs": {
    "date_histo": {
      "date_histogram": {
        "field": "@timestamp",
        "order": {
          "rt_stats.max": "desc"
        }, 
        "interval": "hour"
      }, "aggs": {
        "rt_stats": {
          "extended_stats": {
            "field": "upstream_response_time"
          }
        }
      }
    }
  }
}

field_stats

接口：The field stats api allows one to find statistical properties of a field without executing a search。比如要查看某索引的timestamp字段情况：则会返回最大值、最小值、以及是否可以聚合统计

GET /nginx-elk-2020.03.24/_field_stats?fields=@timestamp