elasticsearch知识总结

岁月栖迟

已于 2024-03-23 17:23:23 修改

阅读量180

点赞数 4

文章标签： elasticsearch 大数据搜索引擎

于 2024-03-23 16:22:53 首次发布

本文链接：https://blog.csdn.net/weixin_40341786/article/details/136969286

版权

mysql与 elasticsearch对比

字段类型

Text（文本）：用于存储文本数据。文本字段通常会被分词处理，以支持全文搜索和分析。
Keyword（关键字）：用于存储结构化数据，如标签、关键字等。关键字字段通常不会进行分词，而是将整个值视为一个词项。
Numeric（数值型）：包括整数和浮点数类型。可以根据需要选择不同的数值类型，如long、integer、short、byte、double、float等。
Date（日期）：用于存储日期和时间数据。可以支持不同的日期格式，并提供日期范围查询等功能。
Boolean（布尔型）：用于存储布尔值（true或false）。
Binary（二进制）：用于存储二进制数据，如图片、文件等。
Object（对象）：用于存储复杂的结构化数据，可以包含嵌套字段。
Nested（嵌套对象）：类似于对象类型，但支持更复杂的嵌套结构和查询。
Geo-point（地理坐标点）：用于存储地理位置坐标。
Geo-shape（地理形状）：用于存储复杂的地理形状数据，如多边形、线条等。

elasticsearch核心知识

操作命令详解

1.创建索引

test索引名(默认创建空的mapping)

PUT /test

结果
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "test"
}

也可以直接创建索引并创建mapping

PUT /my_index
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text"
      },
      "content": {
        "type": "text"
      },
      "date": {
        "type": "date",
        "format": "yyyy-MM-dd"
      }
    }
  }
}

2.为索引修改mapping

PUT /test/_mapping
{
    "properties": {
      "title": {
        "type": "text"
      }   
  }
}

3.查看索引

get /test

4.删除索引

delete /test

5.数据准备

PUT product
{
  "mappings" : {
      "properties" : {
        "createtime" : {
          "type" : "date"
        },
        "date" : {
          "type" : "date"
        },
        "desc" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          },
          "analyzer":"ik_max_word"
        },
        "lv" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "name" : {
          "type" : "text",
          "analyzer":"ik_max_word",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "price" : {
          "type" : "long"
        },
        "tags" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "type" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
}
PUT /product/_doc/1
{
    "name" : "小米手机",
    "desc" :  "手机中的战斗机",
    "price" :  3999,
    "lv":"旗舰机",
    "type":"手机",
    "createtime":"2020-10-01T08:00:00Z",
    "tags": [ "性价比", "发烧", "不卡顿" ]
}
PUT /product/_doc/2
{
    "name" : "小米NFC手机",
    "desc" :  "支持全功能NFC，手机中的滑翔机",
    "price" :  4999,
        "lv":"旗舰机",
    "type":"手机",
    "createtime":"2020-05-21T08:00:00Z",
    "tags": [ "性价比", "发烧", "公交卡" ]
}
PUT /product/_doc/3
{
    "name" : "NFC手机",
    "desc" :  "手机中的轰炸机",
    "price" :  2999,
        "lv":"高端机",
    "type":"手机",
    "createtime":"2020-06-20",
    "tags": [ "性价比", "快充", "门禁卡" ]
}
PUT /product/_doc/4
{
    "name" : "小米耳机",
    "desc" :  "耳机中的黄焖鸡",
    "price" :  999,
        "lv":"百元机",
    "type":"耳机",
    "createtime":"2020-06-23",
    "tags": [ "降噪", "防水", "蓝牙" ]
}
PUT /product/_doc/5
{
    "name" : "红米耳机",
    "desc" :  "耳机中的肯德基",
    "price" :  399,
    "type":"耳机",
        "lv":"百元机",
    "createtime":"2020-07-20",
    "tags": [ "防火", "低音炮", "听声辨位" ]
}
PUT /product/_doc/6
{
    "name" : "小米手机10",
    "desc" :  "充电贼快掉电更快，超级无敌望远镜，高刷电竞屏",
    "price" :  "",
        "lv":"旗舰机",
    "type":"手机",
    "createtime":"2020-07-27",
    "tags": [ "120HZ刷新率", "120W快充", "120倍变焦" ]
}
PUT /product/_doc/7
{
    "name" : "挨炮 SE2",
    "desc" :  "除了CPU，一无是处",
    "price" :  "3299",
        "lv":"旗舰机",
    "type":"手机",
    "createtime":"2020-07-21",
    "tags": [ "割韭菜", "割韭菜", "割新韭菜" ]
}
PUT /product/_doc/8
{
    "name" : "XS Max",
    "desc" :  "听说要出新款12手机了，终于可以换掉手中的4S了",
    "price" :  4399,
        "lv":"旗舰机",
    "type":"手机",
    "createtime":"2020-08-19",
    "tags": [ "5V1A", "4G全网通", "大" ]
}
PUT /product/_doc/9
{
    "name" : "小米电视",
    "desc" :  "70寸性价比只选，不要一万八，要不要八千八，只要两千九百九十八",
    "price" :  2998,
        "lv":"高端机",
    "type":"耳机",
    "createtime":"2020-08-16",
    "tags": [ "巨馍", "家庭影院", "游戏" ]
}
PUT /product/_doc/10
{
    "name" : "红米电视",
    "desc" :  "我比上边那个更划算，我也2998，我也70寸，但是我更好看",
    "price" :  2999,
    "type":"电视",
        "lv":"高端机",
    "createtime":"2020-08-28",
    "tags": [ "大片", "蓝光8K", "超薄" ]
}
PUT /product/_doc/11
{
  "name": "红米电视",
  "desc": "我比上边那个更划算，我也2998，我也70寸，但是我更好看",
  "price": 2998,
  "type": "电视",
  "lv": "高端机",
  "createtime": "2020-08-28",
  "tags": [
    "大片",
    "蓝光8K",
    "超薄"
  ]
}
PUT /product/_doc/12
{
  "name": "电视",
  "desc": "我比上边那个更划算，我也2998，我也70寸，但是我更好看",
  "price": 2998,
  "type": "电视",
  "lv": "高端机",
  "createtime": "2020-08-28",
  "tags": [
    "大片",
    "蓝光8K",
    "超薄"
  ]
}

6.普通查询

1.查询所有

match_all

GET /product/_search
{
  "query": {
    "match_all": {}
  }
}

2.match

对于查询会进行分词

GET /product/_search
{
  "query": {
    "match": {
      "name": "小米电视"
    }
  }
}

3.term

查询时不会进行分词

GET /product/_search
{
  "query": {
    "term": {
      "name": "电视"
    }
  }
}

4.terms

可以指定多个精准匹配条件

GET /product/_search
{
  "query": {
    "terms": {
      "name": [
        "电视",
        "手机"
      ]
    }
  }
}

5.range

范围查询

GET /product/_search
{
  "query": {
    "range": {
      "price": {
        "gte":1,
        "lt":30000
      }
    }
  }
}

6.复合查询

must必需条件 must_not非必须 should也可以不匹配 filter过滤

GET /product/_search
{
  "query": {
    "bool": {
      "must": {
       "term": {
         "name":{
           "value":"小米"
         }
       }
      }
      , "filter": {
        "term": {
         "name":{
           "value":"手机"
         }
       }
      }
    }
  }
}

7.桶聚合

统计不同标签的商品数量 size显示的数量

GET product/_search
{
  
  "aggs": {
    "tag_bucket": {
      "terms": {
        "field": "tags.keyword",
        "size":1
      }
    }
  }
}
结果
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 12,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "product",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "name" : "小米手机",
          "desc" : "手机中的战斗机",
          "price" : 3999,
          "lv" : "旗舰机",
          "type" : "手机",
          "createtime" : "2020-10-01T08:00:00Z",
          "tags" : [
            "性价比",
            "发烧",
            "不卡顿"
          ]
        }
      },
      {
        "_index" : "product",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "name" : "小米NFC手机",
          "desc" : "支持全功能NFC，手机中的滑翔机",
          "price" : 4999,
          "lv" : "旗舰机",
          "type" : "手机",
          "createtime" : "2020-05-21T08:00:00Z",
          "tags" : [
            "性价比",
            "发烧",
            "公交卡"
          ]
        }
      },
      {
        "_index" : "product",
        "_type" : "_doc",
        "_id" : "3",
        "_score" : 1.0,
        "_source" : {
          "name" : "NFC手机",
          "desc" : "手机中的轰炸机",
          "price" : 2999,
          "lv" : "高端机",
          "type" : "手机",
          "createtime" : "2020-06-20",
          "tags" : [
            "性价比",
            "快充",
            "门禁卡"
          ]
        }
      },
      {
        "_index" : "product",
        "_type" : "_doc",
        "_id" : "4",
        "_score" : 1.0,
        "_source" : {
          "name" : "小米耳机",
          "desc" : "耳机中的黄焖鸡",
          "price" : 999,
          "lv" : "百元机",
          "type" : "耳机",
          "createtime" : "2020-06-23",
          "tags" : [
            "降噪",
            "防水",
            "蓝牙"
          ]
        }
      },
      {
        "_index" : "product",
        "_type" : "_doc",
        "_id" : "5",
        "_score" : 1.0,
        "_source" : {
          "name" : "红米耳机",
          "desc" : "耳机中的肯德基",
          "price" : 399,
          "type" : "耳机",
          "lv" : "百元机",
          "createtime" : "2020-07-20",
          "tags" : [
            "防火",
            "低音炮",
            "听声辨位"
          ]
        }
      },
      {
        "_index" : "product",
        "_type" : "_doc",
        "_id" : "6",
        "_score" : 1.0,
        "_source" : {
          "name" : "小米手机10",
          "desc" : "充电贼快掉电更快，超级无敌望远镜，高刷电竞屏",
          "price" : "",
          "lv" : "旗舰机",
          "type" : "手机",
          "createtime" : "2020-07-27",
          "tags" : [
            "120HZ刷新率",
            "120W快充",
            "120倍变焦"
          ]
        }
      },
      {
        "_index" : "product",
        "_type" : "_doc",
        "_id" : "7",
        "_score" : 1.0,
        "_source" : {
          "name" : "挨炮 SE2",
          "desc" : "除了CPU，一无是处",
          "price" : "3299",
          "lv" : "旗舰机",
          "type" : "手机",
          "createtime" : "2020-07-21",
          "tags" : [
            "割韭菜",
            "割韭菜",
            "割新韭菜"
          ]
        }
      },
      {
        "_index" : "product",
        "_type" : "_doc",
        "_id" : "8",
        "_score" : 1.0,
        "_source" : {
          "name" : "XS Max",
          "desc" : "听说要出新款12手机了，终于可以换掉手中的4S了",
          "price" : 4399,
          "lv" : "旗舰机",
          "type" : "手机",
          "createtime" : "2020-08-19",
          "tags" : [
            "5V1A",
            "4G全网通",
            "大"
          ]
        }
      },
      {
        "_index" : "product",
        "_type" : "_doc",
        "_id" : "9",
        "_score" : 1.0,
        "_source" : {
          "name" : "小米电视",
          "desc" : "70寸性价比只选，不要一万八，要不要八千八，只要两千九百九十八",
          "price" : 2998,
          "lv" : "高端机",
          "type" : "耳机",
          "createtime" : "2020-08-16",
          "tags" : [
            "巨馍",
            "家庭影院",
            "游戏"
          ]
        }
      },
      {
        "_index" : "product",
        "_type" : "_doc",
        "_id" : "10",
        "_score" : 1.0,
        "_source" : {
          "name" : "红米电视",
          "desc" : "我比上边那个更划算，我也2998，我也70寸，但是我更好看",
          "price" : 2999,
          "type" : "电视",
          "lv" : "高端机",
          "createtime" : "2020-08-28",
          "tags" : [
            "大片",
            "蓝光8K",
            "超薄"
          ]
        }
      }
    ]
  },
  "aggregations" : {
    "tag_bucket" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 21,
      "buckets" : [
        {
          "key" : "大片",
          "doc_count" : 3
        },
        {
          "key" : "性价比",
          "doc_count" : 3
        },
        {
          "key" : "蓝光8K",
          "doc_count" : 3
        },
        {
          "key" : "超薄",
          "doc_count" : 3
        },
        {
          "key" : "发烧",
          "doc_count" : 2
        }
      ]
    }
  }
}

不显示hits数据：size:0

GET product/_search
{
  "size": 0, 
  "aggs": {
    "tag_bucket": {
      "terms": {
        "field": "tags.keyword"
      }
    }
  }
}
结果
{
  "took" : 10,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 12,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "tag_bucket" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 16,
      "buckets" : [
        {
          "key" : "大片",
          "doc_count" : 3
        },
        {
          "key" : "性价比",
          "doc_count" : 3
        },
        {
          "key" : "蓝光8K",
          "doc_count" : 3
        },
        {
          "key" : "超薄",
          "doc_count" : 3
        },
        {
          "key" : "发烧",
          "doc_count" : 2
        },
        {
          "key" : "120HZ刷新率",
          "doc_count" : 1
        },
        {
          "key" : "120W快充",
          "doc_count" : 1
        },
        {
          "key" : "120倍变焦",
          "doc_count" : 1
        },
        {
          "key" : "4G全网通",
          "doc_count" : 1
        },
        {
          "key" : "5V1A",
          "doc_count" : 1
        }
      ]
    }
  }
}

排序

GET product/_search
{
  "size": 0, 
  "aggs": {
    "tag_bucket": {
      "terms": {
        "field": "tags.keyword",
        "size": 3,
        "order": {
          "_count": "desc"
        }
      }
    }
  }
}
结果
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 12,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "tag_bucket" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 26,
      "buckets" : [
        {
          "key" : "大片",
          "doc_count" : 3
        },
        {
          "key" : "性价比",
          "doc_count" : 3
        },
        {
          "key" : "蓝光8K",
          "doc_count" : 3
        }
      ]
    }
  }
}

最贵、最便宜和平均价格三个指标

GET product/_search
{
  "size": 0, 
  "aggs": {
    "max_price": {
      "max": {
        "field": "price"
      }
    },
    "min_price": {
      "min": {
        "field": "price"
      }
    },
    "avg_price": {
      "avg": {
        "field": "price"
      }
    }
  }
}
结果
{
  "took" : 14,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 12,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "max_price" : {
      "value" : 4999.0
    },
    "min_price" : {
      "value" : 399.0
    },
    "avg_price" : {
      "value" : 3007.818181818182
    }
  }
}

单个聚合查询所有指标

GET product/_search
{
  "size": 0, 
  "aggs": {
    "price_stats": {
      "stats": {
        "field": "price"
      }
    }
  }
}
结果
{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 12,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "price_stats" : {
      "count" : 11,
      "min" : 399.0,
      "max" : 4999.0,
      "avg" : 3007.818181818182,
      "sum" : 33086.0
    }
  }
}

去重操作cardinality 按照name去重的数量

GET product/_search
{
  "size": 0, 
  "aggs": {
    "type_count": {
      "cardinality": {
        "field": "name.keyword"
      }
    }
  }
}

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 12,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "type_count" : {
      "value" : 11
    }
  }
}

8.嵌套聚合

GET product/_search
{
  "size": 0,
  "aggs": {
    "<agg_name>": {
      "<agg_type>": {
        "field": "<field_name>"
      },
      "aggs": {
        "<agg_name_child>": {
          "<agg_type>": {
            "field": "<field_name>"
          }
        }
      }
    }
  }
}
# 例：统计不同类型商品的不同级别的数量
GET product/_search
{
  "size": 0, 
  "aggs": {
    "type_lv": {
      "terms": {
        "field": "type.keyword"
      },
      "aggs": {
        "lv": {
          "terms": {
            "field": "lv.keyword"
          }
        }
      }
    }
  }
}
#按照lv分桶 输出每个桶的具体价格信息
GET product/_search
{
  "size": 0, 
  "aggs": {
    "lv_price": {
      "terms": {
        "field": "lv.keyword"
      },
      "aggs": {
        "price": {
          "stats": {
            "field": "price"
          }
        }
      }
    }
  }
}

##结合了上面两个例子
##统计不同类型商品 不同档次的 价格信息 标签信息
GET product/_search
{
  "size": 0, 
  "aggs": {
    "type_agg": {
      "terms": {
        "field": "type.keyword"
      },
      "aggs": {
        "lv_agg": {
          "terms": {
            "field": "lv.keyword"
          },
          "aggs": {
            "price_stats": {
              "stats": {
                "field": "price"
              }
            },
            "tags_buckets": {
              "terms": {
                "field": "tags.keyword"
              }
            }
          }
        }
      }
    }
  }
}

## 统计每个商品类型中 不同档次分类商品中 平均价格最低的档次
GET product/_search
{
  "size": 0,
  "aggs": {
    "type_bucket": {
      "terms": {
        "field": "type.keyword"
      },
      "aggs": {
        "lv_bucket": {
          "terms": {
            "field": "lv.keyword"
          },
          "aggs": {
            "price_avg": {
              "avg": {
                "field": "price"
              }
            }
          }
        },
        "min_bucket": {
          "min_bucket": {
            "buckets_path": "lv_bucket>price_avg"
          }
        }
      }
    }
  }
}

#======================================================
#基于查询结果的聚合
GET product/_search
{
  "size": 0, 
  "query": {
    "range": {
      "price": {
        "gte": 5000
      }
    }
  }, 
  "aggs": {
    "tags_bucket": {
      "terms": {
        "field": "tags.keyword"
      }
    }
  }
}

#基于filter的aggs
GET product/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "range": {
          "price": {
            "gte": 5000
          }
        }
      }
    }
  },
  "aggs": {
    "tags_bucket": {
      "terms": {
        "field": "tags.keyword"
      }
    }
  }
}

GET product/_search
{
  "query": {
    "bool": {
      "filter": {
        "range": {
          "price": {
            "gte": 5000
          }
        }
      }
    }
  }, 
  "aggs": {
    "tags_bucket": {
      "terms": {
        "field": "tags.keyword"
      }
    }
  }
}


#基于聚合的查询
GET product/_search
{
  "aggs": {
    "tags_bucket": {
      "terms": {
        "field": "tags.keyword"
      }
    }
  },
  "post_filter": {
    "term": {
      "tags.keyword": "性价比"
    }
  }
}

#取消查询条件&&查询条件嵌套
## 例：最贵、最便宜和平均价格三个指标
GET product/_search
{
  "size": 10,
  "query": {
    "range": {
      "price": {
        "gte": 4000
      }
    }
  },
  "aggs": {
    "max_price": {
      "max": {
        "field": "price"
      }
    },
    "min_price": {
      "min": {
        "field": "price"
      }
    },
    "avg_price": {
      "avg": {
        "field": "price"
      }
    },
    "all_avg_price": {
      "global": {},
      "aggs": {
        "avg_price": {
          "avg": {
            "field": "price"
          }
        }
      }
    },
    "muti_avg_price": {
      "filter": {
        "range": {
          "price": {
            "lte": 4500
          }
        }
      }, 
      "aggs": {
        "avg_price": {
          "avg": {
            "field": "price"
          }
        }
      }
    }
  }
}


#===============================================
#聚合排序_count _key _term
GET product/_search
{
  "size": 0,
  "aggs": {
    "type_agg": {
      "terms": {
        "field": "tags",
        "order": {
          "_count": "desc"
        },
        "size": 10
      }
    }
  }
}
#多级排序
GET product/_search?size=0
{
  "aggs": {
    "first_sort": {
      "terms": {
        "field": "type.keyword",
        "order": {
          "_count": "desc"
        }
      },
      "aggs": {
        "second_sort": {
          "terms": {
            "field": "lv.keyword",
            "order": {
              "_count": "asc"
            }
          }
        }
      }
    }
  }
}


#多层排序
GET product/_search
{
  "size": 0,
  "aggs": {
    "tag_avg_price": {
      "terms": {
        "field": "type.keyword",
        "order": {
          "agg_stats>stats.sum": "desc"
        }
      },
      "aggs": {
        "agg_stats": {
          "filter": {
            "terms": {
              "type.keyword": [
                "耳机","手机","电视"
              ]
            }
          },
          "aggs": {
            "stats": {
              "extended_stats": {
                "field": "price"
              }
            }
          }
        }
      }
    }
  }
}