ES使用记录

最新推荐文章于 2023-07-31 17:34:36 发布

终遇

最新推荐文章于 2023-07-31 17:34:36 发布

阅读量2.3k

点赞数

文章标签： elasticsearch 大数据搜索引擎

本文链接：https://blog.csdn.net/distinySmile/article/details/127630165

版权

官方文档：Elasticsearch Guide [8.4] | Elastic 官方中文：序言 | Elasticsearch: 权威指南 | Elastic 社区文档：Getting Started（入门指南） - elasticsearch中文文档

一. ES安装相关

1. 使用docker安装es 和kibana

docker pull elasticsearch:7.4.2 存储和检索数据 
docker pull kibana:7.4.2 可视化检索数据

2. 创建对应的ES挂载文件

mkdir -p /mydata/elasticsearch/config 
mkdir -p /mydata/elasticsearch/data 
mkdir -p /mydata/elasticsearch/plugins 
echo "http.host: 0.0.0.0" >> /mydata/elasticsearch/config/elasticsearch.yml

3. 添加selinux规则，将要挂载的目录添加到白名单

chcon -Rt svirt_sandbox_file_t /mydata/elasticsearch/config
chcon -Rt svirt_sandbox_file_t /mydata/elasticsearch/data 
chcon -Rt svirt_sandbox_file_t /mydata/elasticsearch/plugins

4. 启动es

docker run --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e ES_JAVA_OPTS="-Xms128m -Xmx256m" -v /mydata/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /mydata/elasticsearch/data:/usr/share/elasticsearch/data -v /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins -d elasticsearch:7.4.2

二 kibana安装相关

1. 启动kibana

#192.168.206.130这个ip地址换成虚拟机的ip地址。
docker run --name kibana -e ELASTICSEARCH_HOSTS=http://192.168.206.130:9200 -p 5601:5601  -d kibana:7.4.2

2. 启动问题

启动后kibana报server is not ready yet，使用docker logs kibana 查看启动日志。排查错误

常见ip地址找不到或链接错误：

docker inspect elasticsearch，查看es容器内部的ip地址，如172.17.0.5

将kibana启动地址改为 -e ELASTICSEARCH_HOSTS=http://172.17.0.5:9200 即可解决

三 ES使用记录

1. es常用查询

GET /_cat/nodes：查看所有节点 
GET /_cat/health：查看 es 健康状况 
GET /_cat/master：查看主节点 
GET /_cat/indices：查看所有索引 类似mysql show databases

2. es更新时使用乐观锁

查出数据对应的版本号


{ "_index": "customer", //在哪个索引
 "_type": "external", //在哪个类型 
 "_id": "1", //记录 id
 "_version": 2, //版本号
 "_seq_no": 1, //并发控制字段，每次更新就会+1，用来做乐观锁
 "_primary_term": 1, //同上，主分片重新分配，如重启，就会变化 
 "found": true,
 "_source": { //真正的内容
     "name": "John Doe" 
    } 
    }   更新

更新时带上?if_seq_no=0&if_primary_term=1

192.168.101.164:9200/gulimall/order/1?if_seq_no=6&if_primary_term=1

更新失败会返回409

3. 查询示例

1.match查询

GET bank/_search
{
  "query": {
    "match": {
      "firstname": "mill lane"  #匹配指定字段包含指定值    包含mill 或lane的
    }
  },
  "sort":{
    "balance":{
      "order":"desc"    #通过余额降序排序
    }
  },
  "from": 0,   #分页  从第一条开始
  "size": 20,  #分页  取20条数据
  "_source": ["balance","firstname"]  #查询指定字段
  
}

2. match_phrase 短语匹配

GET bank/_search
{
  "query": {
    "match_phrase ": {
      "firstname": "mill lane"  #匹配指定字段包含指定值   包含mill lane ，不会将mill lane分词
    }
  },
  "sort":{
    "balance":{
      "order":"desc"    #通过余额降序排序
    }
  },
  "from": 0,   #分页  从第一条开始
  "size": 20,  #分页  取20条数据
  "_source": ["balance","firstname"]  #查询指定字段
  
}

3. text文本检索使用match 数值查询使用term

4. should查询

应该达到 should 列举的条件，如果达到会增加相关文档的评分，并不会改变查询的结果。如果 query 中只有 should 且只有一种匹配规则，那么 should 的条件就会被作为默认匹配条件而去改变查询结果

## 匹配到genger包含M的数据，会提升score值
GET bank/_search
{
  "query": {
    "bool": {
      "must": [
        {"term": {
          "age": "32"
        }},
        {
          "match": {
            "address": "Place"
          }
        }
      ],
      "should": [
        {"match": {
          "gender": "M"
        }}
      ]
    }
  }
}

## 只有should查询参数，则会将该条件作为匹配查询参数
GET bank/_search
{
  "query": {
    "bool": {
      "should": [
        {"match": {
          "gender": "M"
        }}
      ]
    }
  }
}

5. 复杂聚合查询

#聚合查询 查出所有年龄分布，并且这些年龄段中 M 的平均薪资和 F 的平均薪资以及这个年龄 段的总体平均薪资
GET bank/_search
{
  "query": {
    "match_all": {}
  },
  "size": 0, 
  "aggs": {
    "ageAggs": {
      "terms": {
        "field": "age",
        "size": 100
      },
      "aggs": {
        "genderAggs": {
          "terms": {
            "field": "gender.keyword"
          },
          "aggs": {
            "balanceAggs": {
              "avg": {
                "field": "balance"
              }
            }
          }
        },
        "balanceAgeAvg": {
              "avg": {
                "field": "balance"
              }
            }
      }
    }
  }
  
}

6. es数据迁移

#数据迁移
POST _reindex
{
  "source": {
    "index": "源索引库",
    "type": "account(源类型)"
  },
  "dest": {
    "index": "目标索引库"
  }
}

7. 手动创建映射mapping

字段映射类型在官网可以查看

1. 创建索引并指定映射

#创建索引并指定映射
PUT /my-index
{
  "mappings": {
    "properties": {
      "age": {
        "type": "integer"
      },
      "email": {
        "type": "keyword"
      },
      "name": {
        "type": "text"
      }
    }
  }
}

2. 添加新字段映射

PUT /my-index/_mapping
{
  "properties": {
    "employee-id": {
      "type": "keyword",
      "index": false
    }
  }
}

3.更新映射

对于已经存在的映射字段，我们不能更新。更新必须创建新的索引进行数据迁移

4. 查看映射

GET my-index/_mapping

四 ik分词器安装

1. 安装

安转完ik分词器后，修改ik分词器的权限

chmod -R 777 ik/

进入es 内部的bin目录elasticsearch-plugin list查看是否安装成功

elasticsearch-plugin list

2. 测试分词器

1. ik_smart测试

#测试分词器
POST _analyze
{
  "analyzer": "ik_smart", 
  "text": "我们都有一个家"
}

#结果
{
  "tokens" : [
    {
      "token" : "我们",
      "start_offset" : 0,
      "end_offset" : 2,
      "type" : "CN_WORD",
      "position" : 0
    },
    {
      "token" : "都有",
      "start_offset" : 2,
      "end_offset" : 4,
      "type" : "CN_WORD",
      "position" : 1
    },
    {
      "token" : "一个",
      "start_offset" : 4,
      "end_offset" : 6,
      "type" : "CN_WORD",
      "position" : 2
    },
    {
      "token" : "家",
      "start_offset" : 6,
      "end_offset" : 7,
      "type" : "CN_CHAR",
      "position" : 3
    }
  ]
}

2. ik_max_word测试

#测试分词器
POST _analyze
{
  "analyzer": "ik_max_word", 
  "text": "我们都有一个家"
}
#结果
{
  "tokens" : [
    {
      "token" : "我们",
      "start_offset" : 0,
      "end_offset" : 2,
      "type" : "CN_WORD",
      "position" : 0
    },
    {
      "token" : "都有",
      "start_offset" : 2,
      "end_offset" : 4,
      "type" : "CN_WORD",
      "position" : 1
    },
    {
      "token" : "一个",
      "start_offset" : 4,
      "end_offset" : 6,
      "type" : "CN_WORD",
      "position" : 2
    },
    {
      "token" : "一",
      "start_offset" : 4,
      "end_offset" : 5,
      "type" : "TYPE_CNUM",
      "position" : 3
    },
    {
      "token" : "个",
      "start_offset" : 5,
      "end_offset" : 6,
      "type" : "COUNT",
      "position" : 4
    },
    {
      "token" : "家",
      "start_offset" : 6,
      "end_offset" : 7,
      "type" : "CN_CHAR",
      "position" : 5
    }
  ]
}

结果：

能够看出不同的分词器，分词有明显的区别，所以以后定义一个索引不能再使用默认的 mapping 了，要手工建立 mapping, 因为要选择分词器。

五 springboot使用ES

目前最新版本为7.17，此处使用的7.4.2为项目中曾经的最新版本，仅作一个项目记录

1. 依赖添加

        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>7.4.2</version>
        </dependency>

2. 配置es连接客户端

参考ES7.4配置，其他版本的文档可以通过选择java rest client版本进行查看

@Configuration
public class EsSearchClientConfig {

    public static final RequestOptions COMMON_OPTIONS;
	//共享请求配置
    static {
        RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
        builder.addHeader("Authorization", "Bearer " + "token");
        builder.setHttpAsyncResponseConsumerFactory(
                new HttpAsyncResponseConsumerFactory
                        .HeapBufferedResponseConsumerFactory(1024 * 1024 * 1024));
        COMMON_OPTIONS = builder.build();
    }

	//配置高版本客户端链接
    @Bean
    public RestHighLevelClient restHighLevelClient() {
        RestHighLevelClient client = new RestHighLevelClient(
                RestClient.builder(
                    	//地址 端口 连接方式
                        new HttpHost("192.168.101.164", 9200, "http")));
        return client;
    }

}

3. 使用

3.1 添加

如果不存在索引，则会自动创建。如果不存在保存对象的字段，也会自动创建

    //goodsInfoList为需要保存的实体类集合
	public void saveData(List<GoodsInfo> goodsInfoList) {
        //创建请求
        BulkRequest bulkRequest = new BulkRequest();
        if (!CollectionUtils.isEmpty(goodsInfoList)) {
            goodsInfoList.forEach(s -> {
                //指定索引goods
                IndexRequest request = new IndexRequest("goods");
                //将商品信息转为json字符串存入
                request.source(JSON.toJSONString(s), XContentType.JSON);
                bulkRequest.add(request);
            });
        }
        try {
            //保存数据
            BulkResponse bulk = restHighLevelClient.bulk(bulkRequest, EsSearchClientConfig.COMMON_OPTIONS);
            System.out.println(bulk.toString());
        } catch (IOException e) {
            log.error("保存es失败，错误信息==" + e.getMessage());
        }
    }

3.2 查询

		.... //省略业务处理代码
		//构建查询req
        SearchRequest searchRequest = new SearchRequest();
        //Without arguments this runs against all indices. 不传该参数，则会从所有索引中查询
        searchRequest.indices(EsConstant.PRODUCT_INDEX);
        searchRequest.source(getSearchSourceBuilder(param));
        SearchResult searchResult = new SearchResult();
        SearchResponse response = restHighLevelClient.search(searchRequest, GulimallSearchClientConfig.COMMON_OPTIONS);
		.......
            
      /**
     * remark:处理ESL查询语句
     */
    public SearchSourceBuilder getSearchSourceBuilder(SearchParam param){
        SearchSourceBuilder builder = new SearchSourceBuilder();
        //构建boolQuery
        BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
        //1.关键词查询
        if (!StringUtils.isEmpty(param.getKeyword())){
            boolQuery.must(QueryBuilders.matchQuery("skuTitle",param.getKeyword()));
        }
        //2 过滤条件
        //2.1 三级分类id查询
        if (!StringUtils.isEmpty(param.getCatalog3Id())){
            boolQuery.filter(QueryBuilders.termQuery("catalogId",param.getCatalog3Id()));
        }
        //2.2 品牌id查询
        if (!CollectionUtils.isEmpty(param.getBrandId())){
            boolQuery.filter(QueryBuilders.termsQuery("brandId",param.getBrandId()));
        }
        //2.3 品牌参数查询 嵌入式查询
        if (!CollectionUtils.isEmpty(param.getAttrs())){
            for (String attrStr : param.getAttrs()) {
                BoolQueryBuilder nestedBoolQuery = QueryBuilders.boolQuery();
                String[] split = attrStr.split("_");
                nestedBoolQuery.must(QueryBuilders.termQuery("attrs.attrId",split[0]));
                nestedBoolQuery.must(QueryBuilders.termsQuery("attrs.attrValue",split[1].split(":")));
                boolQuery.filter(QueryBuilders.nestedQuery("attrs",nestedBoolQuery , ScoreMode.None));
            }
        }

        //2.4 价格区间
        if (!StringUtils.isEmpty(param.getSkuPrice())){
            String[] split = param.getSkuPrice().split("_");
            RangeQueryBuilder skuPrice = QueryBuilders.rangeQuery("skuPrice");
            if (split.length>1){
                skuPrice.gte(split[0]);
                skuPrice.lte(split[1]);
            }else {
                if (param.getSkuPrice().startsWith("_")){
                    skuPrice.gte(split[0]);
                }else {
                    skuPrice.lte(split[0]);
                }
            }
            boolQuery.filter(skuPrice);
        }
        boolQuery.filter(QueryBuilders.matchQuery("hasStock",param.getHasStock()==1));

        builder.query(boolQuery);

        //3 排序
        if (!StringUtils.isEmpty(param.getSort())){
            String[] split = param.getSort().split("_");
            builder.sort(split[0],split[1].equalsIgnoreCase("asc")? SortOrder.ASC:SortOrder.DESC);
        }
        //4 分页
        builder.from((param.getPageNum()-1)*EsConstant.PRODUCT_PAGESIZE);
        builder.size(EsConstant.PRODUCT_PAGESIZE);
		//高亮字段
        if (!StringUtils.isEmpty(param.getKeyword())){
            HighlightBuilder highlightBuilder = new HighlightBuilder();
            highlightBuilder.field("skuTitle");
            highlightBuilder.preTags("<b style='color:red'>");
            highlightBuilder.postTags("</b>");
            builder.highlighter(highlightBuilder);
        }

		//聚合查询
        TermsAggregationBuilder brandIdAggs = AggregationBuilders.terms("brandAggs").size(10).field("brandId");
        brandIdAggs.subAggregation(AggregationBuilders.terms("brand_name_aggs").size(10).field("brandName"));
        brandIdAggs.subAggregation(AggregationBuilders.terms("brand_img_aggs").size(10).field("brandImg"));
        builder.aggregation(brandIdAggs);

        TermsAggregationBuilder cateLogAggs = AggregationBuilders.terms("cateLogAggs").field("catalogId").size(10);
        cateLogAggs.subAggregation(AggregationBuilders.terms("catalogNameAggs").field("catalogName").size(10));
        builder.aggregation(cateLogAggs);


        NestedAggregationBuilder nestedAggs = AggregationBuilders.nested("attrIdAggs", "attrs");
        nestedAggs.subAggregation(AggregationBuilders.terms("attrIdAggs").field("attrs.attrId").size(10));
        nestedAggs.subAggregation(AggregationBuilders.terms("attrNameAggs").field("attrs.attrName").size(10));
        nestedAggs.subAggregation(AggregationBuilders.terms("attrValueAggs").field("attrs.attrValue").size(10));

        builder.aggregation(nestedAggs);
        System.out.println("DLS--"+builder.toString());

        return builder;
    }

3.3更多

更多查询参考查询api

注意选择自己使用的版本。。。