springboot 的 elasticsearch 版本:
7.15.2
前情提要:
1.首先要理解 elasticsearch 对于【数据类型】很严格,如果字段类型不规范,在 检索/排序/聚合 时候类型不正确就会出现报错或者查不到数据的问题。所以在一般String类型插入结构如下:
这样的结构,不仅可以支持分词查询,也可以进行精准匹配、对该字段排序、对该字段进行聚合。一箭双雕,通常String都建议用这样的结构。
2.如果是数值类型(Integer、Long、Double)更适合进行排序、聚合、精准匹配。如果确定某类的某个属性是【数值】,最好不要使用String接收(并非完全不可使用),用数值类型接收更合理。如下:
3.如果某个类的属性是Object类型,在进行检索、聚合等操作时,es的type类型不必做其它操作,如下:
像 tradeType、map 这两个属性属于Object类型,在es中不必特殊指定类型,如下:
4.es中的【nested】类型,特定用在某属性是数组类型,而且该属性在es结构中必须要加上【nested】类型,否在在检索、聚合时候就会出现报错或查不到数据问题,以下:
es中结构如下:
subjectList 便是数组类型,并且泛型是包装类型,在es中都需要指定该字段是【nested】。
5.非常注意:如果某属性的泛型是基本类型(比如:List< String >、List< Integer > )那es的类型可不是【nested】,而是keyword或Integer 这种。如下:
es中的结构:
当然会有朋友问,es中貌似也支持array类型,但是我用的版本在使用 array 创建索引时会报错:No handler for type [array]。所以我这边会用keyword类型。
具体支持类型,可参考:Elasticsearch数据类型及其属性
准备索引:
通过es控制台或者postman都可以进行索引操作,类型mysql可视化工具对mysql数据库表结构操作类型,只不过,es对于创建后的索引的字段类型不可改变,如果一定要变更字段类型,就需要删除索引重新创建,这就是为什么对于要插入的数据类型严格控制的原因。
1.创建索引:createIndex():
当你制作好一个java类,并且把属性的类型都规范好,这时候可直接通过该方法创建索引,如果该类中有数组类型的属性:
1.如果数组泛型是基本类型 则 传入 arrayField 参数,在es中type=keyword。
2.如果数组泛型是包装类型 则需要指定 nestedField 参数的属性名,否则在插入数据后会发现,该数组类型的es中的type不是 nested类型。
2.删除索引:deleteIndex(String indexName):
不解释了,没啥好说的。
3.是否存在指定索引:existIndex(String indexName):
不解释了,没啥好说的。
完整示例:
工具类用的json格式化工具:
import com.alibaba.fastjson2.JSON;
import com.alibaba.fastjson2.JSONObject;
java实体类:
@Data
public class UserIndexReqDTO implements Serializable {
private static final long serialVersionUID = 1L;
/**
* [描述] 用户id
*/
private Long id;
private String userName;
private Integer sex;
private Integer age;
/** 用户多个手机号码 */
private List<String> phoneList;
/** 用户科目列表信息 */
private List<SubjectDTO> subjectList;
}
@Data
class SubjectDTO {
/** 科目id */
private Integer id;
/** 科目名称 */
private String subjectName;
/** 科目分值 */
private Integer subjectScore;
}
1.创建索引示例:
@Resource
private EsClientUtil2 esClientUtil2;
private String indexName = "user_index";
@Override
public BaseResponse<Object> addMore( @RequestBody List<UserIndexReqDTO> list){
// 如果指定索引不存在,则按要求创建
if(!esClientUtil2.existIndex(indexName)){
esClientUtil2.createIndex(indexName,Arrays.asList("phoneList"),Arrays.asList("subjectList"));
}
Assert.notEmpty(list,"list is empty");
List<JSONObject> collect = list.stream()
.map(item -> JSONObject.parseObject(JSONObject.toJSONString(item), JSONObject.class))
.collect(Collectors.toList());
boolean addDoc = esClientUtil2.multiAddDoc(indexName, collect);
return BaseResponse.ofSuccess(addDoc);
}
2.准备测试数据:
[
{
"id": 48,
"userName": "小明",
"sex": 1,
"age": 17,
"phoneList": [
"15800000001",
"15800000002"
],
"subjectList": [
{
"id": 62,
"subjectName": "语文",
"subjectScore": 68
}
]
},
{
"id": 49,
"userName": "史泰龙小红",
"sex": 0,
"age": 10,
"phoneList": [
"18732000000",
"18732000001"
],
"subjectList": [
{
"id": 62,
"subjectName": "语文",
"subjectScore": 68
},
{
"id": 63,
"subjectName": "数学",
"subjectScore": 30
}
]
},
{
"id": 50,
"userName": "小黑",
"sex": 1,
"age": 14,
"phoneList": [
"13200000000",
"13200000001"
],
"subjectList": [
{
"id": 62,
"subjectName": "语文",
"subjectScore": 68
},
{
"id": 64,
"subjectName": "英语",
"subjectScore": 80
}
]
}
]
3.根据条件查询:
public BaseResponse<Object> search() {
int pageNum = 1;
int pageSize = 100;
Map<String,List<String>> likeMap = new HashMap<>(3);
Map<String,Object> signMap = new HashMap<>(3);
Map<String,List<String>> multipleMap = new HashMap<>(3);
Map<String,List<Long>> rangeMap = new HashMap<>(3);
List<String> idList = new ArrayList<>();
// 根据值模糊查询 匹配多字段
//likeMap.put("小明",Arrays.asList("name","nickName"));
// 根据值精准匹配 匹配单字段
//signMap.put("sex","1");
//signMap.put("phoneList",Arrays.asList("15800000001","18732000001"));
// 根据值精准匹配 匹配多字段
//multipleMap.put("小红",Arrays.asList("name","nickName"));
// 区间匹配 18<= age <= 40
//rangeMap.put("age",Arrays.asList(18L,40L));
// 根据esId进行检索
//idList = Arrays.asList("1","2","3");
//嵌套查询
Map<String,String> key = new HashMap<>(1);
Map<String,List<String>> value = new HashMap<>(1);
//key.put("subjectList", "subjectList.id");
//value.put("subjectList.id",Arrays.asList("63"));
List<NestedQueryBuilder> nestedListQuery = esClientUtil2.getNestedListQuery(key, value);
BoolQueryBuilder builder = esClientUtil2.getQueryBuilderAnd(likeMap, signMap, multipleMap, rangeMap, nestedListQuery, idList);
PageResult<String> search = esClientUtil2.search(EsConstant.ESALES_USER_COLLECT_INDEX,
builder, null,
null, null, null, pageNum, pageSize);
List<UserIndexReqDTO> result = new ArrayList<>();
for (String s : search.getList()) {
result.add(JSONObject.parseObject(s, UserIndexReqDTO.class));
}
return BaseResponse.ofSuccess(result);
}
4.根据条件聚合:
public BaseResponse<Object> group() {
Map<String,List<String>> likeMap = new HashMap<>(3);
Map<String,Object> signMap = new HashMap<>(3);
Map<String,List<String>> multipleMap = new HashMap<>(3);
Map<String,List<Long>> rangeMap = new HashMap<>(3);
List<String> idList = new ArrayList<>();
// 根据值模糊查询 匹配多字段
//likeMap.put("小明",Arrays.asList("name","nickName"));
// 根据值精准匹配 匹配单字段
//signMap.put("sex","1");
//signMap.put("phoneList",Arrays.asList("15800000001","18732000001"));
// 根据值精准匹配 匹配多字段
//multipleMap.put("小红",Arrays.asList("name","nickName"));
// 区间匹配 18<= age <= 40
//rangeMap.put("age",Arrays.asList(18L,40L));
// 根据esId进行检索
//idList = Arrays.asList("1","2","3");
//嵌套查询
Map<String,String> key = new HashMap<>(1);
Map<String,List<String>> value = new HashMap<>(1);
//key.put("subjectList", "subjectList.id");
//value.put("subjectList.id",Arrays.asList("63"));
List<NestedQueryBuilder> nestedListQuery = esClientUtil2.getNestedListQuery(key, value);
BoolQueryBuilder builder = esClientUtil2.getQueryBuilderAnd(likeMap, signMap, multipleMap, rangeMap, nestedListQuery, idList);
// 根据字段进行聚合
List<Map<String, Object>> sex = esClientUtil2.group(EsConstant.ESALES_USER_COLLECT_INDEX,
"sex",null, builder,
null, null, null);
for (Map<String, Object> map : sex) {
System.out.println(map);
}
// 根据 嵌套 字段进行聚合
List<Map<String, Object>> subjectIdList = esClientUtil2.group(EsConstant.ESALES_USER_COLLECT_INDEX,
null,Arrays.asList("subjectList","subjectList.id"), builder,
null, null, null);
for (Map<String, Object> map : subjectIdList) {
System.out.println(map);
}
return BaseResponse.ofSuccess(sex);
}
4-1.after翻页:
public BaseResponse<Object> search() {
Map<String,List<String>> likeMap = new HashMap<>(3);
Map<String,Object> signMap = new HashMap<>(3);
Map<String,List<String>> multipleMap = new HashMap<>(3);
Map<String,List<Long>> rangeMap = new HashMap<>(3);
List<String> idList = new ArrayList<>();
// 根据值模糊查询 匹配多字段
//likeMap.put("小明",Arrays.asList("name","nickName"));
// 根据值精准匹配 匹配单字段
//signMap.put("sex","1");
//signMap.put("phoneList",Arrays.asList("15800000001","18732000001"));
// 根据值精准匹配 匹配多字段
//multipleMap.put("小红",Arrays.asList("name","nickName"));
// 区间匹配 18<= age <= 40
//rangeMap.put("age",Arrays.asList(18L,40L));
// 根据esId进行检索
//idList = Arrays.asList("1","2","3");
//嵌套查询
Map<String,String> key = new HashMap<>(1);
Map<String,List<String>> value = new HashMap<>(1);
//key.put("subjectList", "subjectList.id");
//value.put("subjectList.id",Arrays.asList("63"));
List<NestedQueryBuilder> nestedListQuery = esClientUtil2.getNestedListQuery(key, value);
BoolQueryBuilder builder = esClientUtil2.getQueryBuilderAnd(likeMap, signMap, multipleMap, rangeMap, nestedListQuery, idList);
PageResult<String> search = null;
Object[] array = null;
// 第一次传 null,等拿到 resultSortValues 值以后再传入
//array = new Object[]{};
Map<String, SortOrder> sortOrderMap = new HashMap<>(1);
sortOrderMap.put("id", SortOrder.ASC);
search = esClientUtil2.testSearchAfter(EsConstant.ESALES_USER_COLLECT_INDEX,builder,
null,sortOrderMap,null,null,1,array);
List<UserIndexReqDTO> result = new ArrayList<>();
for (String s : search.getList()) {
result.add(JSONObject.parseObject(s, UserIndexReqDTO.class));
}
return BaseResponse.ofSuccess(result);
}
4-2.滚动翻页:
工具类中还有滚动查询方法,如果数据量不大的话,一般用不到,es默认当数据量超过1万条就会检索报错,这种报错情况就分两种检索方式,一种加大分页数据量(貌似也有极限),另一种就是采用滚动检索查询。我目前场景用不到,但是已经测试过,滚动查询方法也可用。
1.滚动查询:searchScroll()
2.清理滚动值,释放空间:searchScrollClear()
6.正餐,工具类:
import com.alibaba.fastjson2.JSON;
import com.alibaba.fastjson2.JSONObject;
import lombok.extern.slf4j.Slf4j;
import org.apache.lucene.search.join.ScoreMode;
import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.delete.DeleteRequest;
import org.elasticsearch.action.delete.DeleteResponse;
import org.elasticsearch.action.get.GetRequest;
import org.elasticsearch.action.get.GetResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.search.ClearScrollRequest;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.search.SearchScrollRequest;
import org.elasticsearch.action.support.master.AcknowledgedResponse;
import org.elasticsearch.action.update.UpdateRequest;
import org.elasticsearch.action.update.UpdateResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.client.indices.CreateIndexRequest;
import org.elasticsearch.client.indices.CreateIndexResponse;
import org.elasticsearch.client.indices.GetIndexRequest;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.common.xcontent.json.JsonXContent;
import org.elasticsearch.core.TimeValue;
import org.elasticsearch.index.query.*;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.aggregations.AggregationBuilders;
import org.elasticsearch.search.aggregations.Aggregations;
import org.elasticsearch.search.aggregations.bucket.filter.Filter;
import org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder;
import org.elasticsearch.search.aggregations.bucket.nested.Nested;
import org.elasticsearch.search.aggregations.bucket.nested.NestedAggregationBuilder;
import org.elasticsearch.search.aggregations.bucket.terms.Terms;
import org.elasticsearch.search.aggregations.bucket.terms.TermsAggregationBuilder;
import org.elasticsearch.search.aggregations.metrics.TopHits;
import org.elasticsearch.search.aggregations.metrics.TopHitsAggregationBuilder;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
import org.elasticsearch.search.sort.SortBuilder;
import org.elasticsearch.search.sort.SortBuilders;
import org.elasticsearch.search.sort.SortOrder;
import org.springframework.stereotype.Component;
import javax.annotation.Resource;
import java.io.IOException;
import java.util.*;
/**
* @author Da.Pang
*/
@Slf4j
@Component
public class EsClientUtil2 {
/**
* [描述] 查询超时分钟
*/
private final Integer TIME_OUT = 60;
/**
* [描述] es中的自带id字段
*/
private final String ID_ = "_id";
@Resource
private RestHighLevelClient restHighLevelClient;
/**
* 创建索引
* @param arrayField 数组的泛型是基本类型,示例: List<String> nameList 、 List<Integer> typeIdList
* @param nestedField 数组的泛型是包装类型,示例: List<Map<String,Object>> typeList 、List<Subject> subjectList
*/
public boolean createIndex(String indexName,List<String> arrayField, List<String> nestedField ) {
try {
CreateIndexRequest createIndexRequest = new CreateIndexRequest(indexName);
XContentBuilder builder = getContentBuilder(arrayField,nestedField);
createIndexRequest.mapping(builder);
CreateIndexResponse response = restHighLevelClient.indices().create(createIndexRequest, RequestOptions.DEFAULT);
log.info("创建索引 response 值为: {}", response.toString());
return true;
} catch (Exception e) {
e.printStackTrace();
log.error("创建索引失败:{}",e.getMessage(),e);
throw new BusinessException("创建索引失败",e.getMessage());
}
}
/**
* [描述] 构建 array类型 或者 nested类型
*/
private XContentBuilder getContentBuilder( List<String> arrayField, List<String> fieldList ) throws IOException {
// 构建索引映射(mapping)
XContentBuilder mappingBuilder = JsonXContent.contentBuilder()
.startObject()
.startObject("properties");
for (String field : arrayField) {
mappingBuilder.startObject(field)
.field("type", "keyword")
.endObject() ;
}
for (String filed : fieldList) {
// 第二个nested字段
mappingBuilder.startObject(filed)
.field("type", "nested")
.endObject();
}
// 结束properties
mappingBuilder.endObject()
// 结束根对象
.endObject();
return mappingBuilder;
}
/**
* 判断索引是否存在
*/
public boolean existIndex(String indexName) {
try {
GetIndexRequest getIndexRequest = new GetIndexRequest(indexName);
return restHighLevelClient.indices().exists(getIndexRequest, RequestOptions.DEFAULT);
} catch (Exception e) {
e.printStackTrace();
log.error("检查索引失败:{}",e.getMessage(),e);
throw new BusinessException("检查索引失败",e.getMessage());
}
}
/**
* 删除索引
*/
public boolean deleteIndex(String indexName) {
try {
DeleteIndexRequest deleteIndexRequest = new DeleteIndexRequest(indexName);
AcknowledgedResponse delete = restHighLevelClient.indices().delete(deleteIndexRequest, RequestOptions.DEFAULT);
log.info("删除索引{},返回结果为{}", indexName, delete.isAcknowledged());
return delete.isAcknowledged();
} catch (Exception e) {
e.printStackTrace();
log.error("删除索引失败:{}",e.getMessage(),e);
throw new BusinessException("删除索引失败",e.getMessage());
}
}
/**
* 根据id删除文档
*/
public void deleteDocById(String indexName, String id) {
try {
DeleteRequest deleteRequest = new DeleteRequest(indexName, id);
DeleteResponse deleteResponse = restHighLevelClient.delete(deleteRequest, RequestOptions.DEFAULT);
log.info("删除索引{}中id为{}的文档,返回结果为{}", indexName, id, deleteResponse.status().toString());
} catch (Exception e) {
e.printStackTrace();
throw new BusinessException("删除企业ES数据错误");
}
}
/**
* 批量插入数据
* @return boolean
*/
public boolean multiAddDoc(String indexName, List<JSONObject> list) {
try {
BulkRequest bulkRequest = new BulkRequest();
list.forEach(doc -> {
String source = JSON.toJSONString(doc);
IndexRequest indexRequest = new IndexRequest(indexName);
indexRequest
.id(doc.getString("id"))
.source(source, XContentType.JSON);
bulkRequest.add(indexRequest);
});
BulkResponse bulkResponse = restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
log.info("向索引{}中批量插入数据的结果为 {}", indexName, !bulkResponse.hasFailures());
return !bulkResponse.hasFailures();
} catch (Exception e) {
e.printStackTrace();
log.info("向索引{}中批量插入数据错误信息 {}", indexName, e.getMessage());
throw new BusinessException("插入企业ES数据错误");
}
}
/**
* 更新文档
*/
public boolean updateDoc(String indexName, String docId, JSONObject jsonObject) {
try {
UpdateRequest updateRequest = new UpdateRequest(indexName, docId).doc(JSON.toJSONString(jsonObject), XContentType.JSON);
UpdateResponse updateResponse = restHighLevelClient.update(updateRequest, RequestOptions.DEFAULT);
int total = updateResponse.getShardInfo().getTotal();
log.info("更新文档的影响数量为{}", total);
return total > 0;
} catch (Exception e) {
e.printStackTrace();
log.info("向索引{}中批量更新数据错误信息 {}", indexName, e.getMessage());
throw new BusinessException("更新企业ES数据错误");
}
}
/**
* 根据id查询文档
* @param indexName 索引名
* @param docId id
*/
public String queryDocById(String indexName, String docId) {
try {
GetRequest getRequest = new GetRequest(indexName, docId);
GetResponse getResponse = restHighLevelClient.get(getRequest, RequestOptions.DEFAULT);
return JSONObject.toJSONString(getResponse.getSource());
//return JSONObject.parseObject(JSONObject.toJSONString(getResponse.getSource()),JSONObject.class);
} catch (Exception e) {
e.printStackTrace();
throw new BusinessException("查询企业ES数据错误");
}
}
/**
* [描述]
* @param combinedQueryAnd 通过 getQueryBuilderAnd() 方法获取
* @param combinedQueryOr 通过 getQueryBuilderOr() 方法获取
* @param include 指定查询哪些字段
* @param sortOrderMap 按照指定字段 正倒序 示例: { "age":SortOrder.DESC }
* @param pageNum 起始位置(页码从1开始)(不分页不传)
* @param pageSize 每页大小(不分页不传)
*/
public PageResult<String> search(
String indexName, BoolQueryBuilder combinedQueryAnd,BoolQueryBuilder combinedQueryOr,
Map<String,SortOrder> sortOrderMap,List<String> highlightFields,
String[] include,Integer pageNum , Integer pageSize) {
SearchRequest searchRequest = new SearchRequest(indexName);
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
getSearchSourceBuilder(sourceBuilder,
combinedQueryAnd, combinedQueryOr, sortOrderMap,
highlightFields, include);
if (pageNum != null && pageSize != null) {
sourceBuilder.from((pageNum-1)*pageSize);
sourceBuilder.size(pageSize);
}
sourceBuilder.timeout(TimeValue.timeValueSeconds(TIME_OUT));
searchRequest.source(sourceBuilder);
SearchResponse response;
try {
response = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
e.printStackTrace();
log.error("索引{}查询错误:{}",indexName,e.getMessage());
throw new BusinessException("查询数据错误");
}
return getPageResult(pageNum,pageSize,response,highlightFields);
}
private void getSearchSourceBuilder(
SearchSourceBuilder sourceBuilder,
BoolQueryBuilder combinedQueryAnd, BoolQueryBuilder combinedQueryOr,
Map<String, SortOrder> sortOrderMap, List<String> highlightFields, String[] include){
if(combinedQueryAnd != null){
sourceBuilder.query(combinedQueryAnd);
}
if(combinedQueryOr != null){
sourceBuilder.query(combinedQueryOr);
}
if(sortOrderMap != null){
List<SortBuilder<?>> sorts = new ArrayList<>();
for (Map.Entry<String, SortOrder> entry : sortOrderMap.entrySet()) {
sorts.add(SortBuilders.fieldSort(entry.getKey()).order(entry.getValue()));
}
sourceBuilder.sort(sorts);
}else{
sourceBuilder.sort(SortBuilders.scoreSort().order(SortOrder.DESC));
}
if(highlightFields != null){
HighlightBuilder highlightBuilder = new HighlightBuilder();
for (String field : highlightFields) {
highlightBuilder.field(field).preTags("<em>").postTags("</em>");
}
sourceBuilder.highlighter(highlightBuilder);
}
if(include != null){
sourceBuilder.fetchSource(include,null);
}
}
/**
* [描述]
*/
private PageResult<String> getPageResult(Integer pageNum,Integer pageSize,SearchResponse response,List<String> highlightFields ){
List<String> list = new ArrayList<>();
for (SearchHit hit : response.getHits().getHits()) {
Map<String, Object> sourceAsMap = hit.getSourceAsMap();
if(highlightFields != null){
for (String field : highlightFields) {
sourceAsMap.put(field,handleHeightFiled(hit,field));
}
}
list.add(JSONObject.toJSONString(sourceAsMap));
}
PageResult<String> page = new PageResult<>();
page.setList(list);
page.setTotal(response.getHits().getTotalHits().value);
if(pageSize != null){
// 如果不分页 则不传 pageSize
page.setPageNum(pageNum);
page.setPageSize(pageSize);
long totalPage = page.getTotal() % pageSize == 0 ? page.getTotal() / pageSize : (page.getTotal() / pageSize) + 1;
page.setTotalPage((int) totalPage);
}
return page;
}
private String handleHeightFiled( SearchHit hit,String heightFiled ){
if(hit.getHighlightFields() != null
&& hit.getHighlightFields().get(heightFiled) != null
&& hit.getHighlightFields().get(heightFiled).fragments().length > 0){
return hit.getHighlightFields().get(heightFiled).fragments()[0].string();
}
return null;
}
/**
* [描述] 所有条件以 and 的方式进行组合
* @param likeMap 模糊查询 一个值对应 单/多 字段 示例:{"小明":["name","nickName","detail"]}
* @param singleMap 精准匹配 一个值对应 单 字段 示例:{"age":"18"}
* @param multipleMap 精准匹配 一个值对应 多 字段 示例:{"180":["age","height","weight"]}
* @param rangeMap 指定字段 区间查询 示例:{"price":[100,500]} (注意:value值必须要两个元素)
* @param nestedList 根据 getNestedListQuery() 传参生成的嵌套查询
* @param idList 根据 es的idList 查询数据
*@author Da.Pang
*/
public BoolQueryBuilder getQueryBuilderAnd( Map<String,List<String>> likeMap,
Map<String, Object> singleMap,
Map<String, List<String>> multipleMap,
Map<String, List<Long>> rangeMap,
List<NestedQueryBuilder> nestedList,
List<String> idList
) {
if( likeMap == null && singleMap == null && multipleMap == null && nestedList == null){
return null;
}
BoolQueryBuilder combinedQuery = new BoolQueryBuilder();
if(likeMap != null){
for (Map.Entry<String, List<String>> entry : likeMap.entrySet()) {
for (String value : entry.getValue()) {
MultiMatchQueryBuilder multiMatchQueryBuilder = QueryBuilders.multiMatchQuery(entry.getKey(), value);
combinedQuery.must(multiMatchQueryBuilder);
}
}
}
if(singleMap != null){
for (Map.Entry<String, Object> entry : singleMap.entrySet()) {
TermQueryBuilder termMatch = QueryBuilders.termQuery(entry.getKey(),entry.getValue());
combinedQuery.must(termMatch);
}
}
if(multipleMap != null){
for (Map.Entry<String, List<String>> entry : multipleMap.entrySet()) {
TermsQueryBuilder termsMatch = QueryBuilders.termsQuery(entry.getKey(), entry.getValue());
combinedQuery.must(termsMatch);
}
}
if(nestedList != null){
//嵌套精准查询 可用于多字段精准查询
for (NestedQueryBuilder builder : nestedList) {
combinedQuery.must(builder);
}
}
if(rangeMap != null){
for (Map.Entry<String, List<Long>> entry : rangeMap.entrySet()) {
RangeQueryBuilder rangeQuery = QueryBuilders.rangeQuery(entry.getKey());
// 大于等于指定值
rangeQuery.gte(entry.getValue().get(0));
// 小于等于指定值
rangeQuery.lte(entry.getValue().get(1));
}
}
if(idList != null && !idList.isEmpty()){
IdsQueryBuilder idsQuery = QueryBuilders.idsQuery();
idsQuery.addIds(idList.toArray(new String[0]));
combinedQuery.must(idsQuery);
}
return combinedQuery;
}
/**
* [描述] 所有条件以 or 的方式进行组合
* @param likeMap 模糊查询 一个值对应 单/多 字段 示例:{"小明":["name","nickName","detail"]}
* @param singleMap 精准匹配 一个值对应 单 字段 示例:{"age":"18"}
* @param multipleMap 精准匹配 一个值对应 多 字段 示例:{"180":["age","height","weight"]}
* @param nestedList 根据 getNestedListQuery() 传参生成的嵌套查询
*@author Da.Pang
*/
public BoolQueryBuilder getQueryBuilderOr( Map<String,List<String>> likeMap,
Map<String, Object> singleMap,
Map<String, List<String>> multipleMap,
List<NestedQueryBuilder> nestedList
) {
if(likeMap == null && singleMap == null && multipleMap == null && nestedList == null){
return null;
}
BoolQueryBuilder combinedQuery = new BoolQueryBuilder();
if(likeMap != null){
for (Map.Entry<String, List<String>> entry : likeMap.entrySet()) {
for (String value : entry.getValue()) {
MultiMatchQueryBuilder multiMatchQueryBuilder = QueryBuilders.multiMatchQuery(entry.getKey(), value);
combinedQuery.should(multiMatchQueryBuilder);
}
}
}
if(singleMap != null){
for (Map.Entry<String, Object> entry : singleMap.entrySet()) {
TermQueryBuilder termMatch = QueryBuilders.termQuery(entry.getKey(),entry.getValue());
combinedQuery.should(termMatch);
}
}
if(multipleMap != null){
for (Map.Entry<String, List<String>> entry : multipleMap.entrySet()) {
TermsQueryBuilder termsMatch = QueryBuilders.termsQuery(entry.getKey(), entry.getValue());
combinedQuery.should(termsMatch);
}
}
if(nestedList != null){
//嵌套精准查询 可用于多字段精准查询
for (NestedQueryBuilder builder : nestedList) {
combinedQuery.should(builder);
}
}
return combinedQuery;
}
/**
* [描述]
* @param key 格式: key: 嵌套的path value: 嵌套的某个字段.keyword
* 示例: key: typeList value: typeList.code.keyword(如果该字段类型是keyword 可以去掉 .keyword, 比如类型是:Long、Integer 就不必带.keyword)
*/
public List<NestedQueryBuilder> getNestedListQuery( Map<String,String> key , Map<String,List<String>> value ){
if(key == null || key.isEmpty()){
return null;
}
List<NestedQueryBuilder> result = new ArrayList<>();
for (Map.Entry<String, String> entry : key.entrySet()) {
List<String> valList = value.get(entry.getValue());
NestedQueryBuilder nestedQueryBuilder2 = QueryBuilders.nestedQuery(entry.getKey(), QueryBuilders.termsQuery(entry.getValue(), valList), ScoreMode.None);
result.add(nestedQueryBuilder2);
}
return result;
}
/**
* [描述] 聚合操作
* @param fieldName 指定的字段
* @param nestedFiledName 嵌套属性中的某字段,一次只 add 两个值 示例: ["subjectList","subjectList.id"]
* 可根据条件聚合指定字段,并回显一定的数据字段和数据列表
* 类似: select id,name,sex from t_user group by sex
* @param include 示例:new String[]{"id","name", "productType.code1", "productType.name1"}
* @param hitsSize 回显的list有多少条数据 (注意:include、hitsSize 必须同时传参 或 同时不传)
*@author Da.Pang
* @return name:聚合字段名 count:聚合字段值 list:参与聚合时的数据列表
*/
public List<Map<String, Object>> group(
String indexName, String fieldName,List<String> nestedFiledName,
BoolQueryBuilder combinedQueryAnd,BoolQueryBuilder combinedQueryOr,
String[] include,Integer hitsSize) {
SearchRequest searchRequest = new SearchRequest(indexName);
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
FilterAggregationBuilder filterAgg = null;
if(combinedQueryAnd == null && combinedQueryOr == null){
filterAgg = AggregationBuilders.filter("filtered_agg", new BoolQueryBuilder());
}else {
if(combinedQueryAnd != null){
filterAgg = AggregationBuilders.filter("filtered_agg", combinedQueryAnd);
}
if(combinedQueryOr != null){
filterAgg = AggregationBuilders.filter("filtered_agg", combinedQueryOr);
}
}
if(nestedFiledName != null){
NestedAggregationBuilder nestedAgg = AggregationBuilders.nested("nested_agg", nestedFiledName.get(0));
TermsAggregationBuilder termsAgg = AggregationBuilders.terms("agg_field").field(nestedFiledName.get(1));
if(hitsSize != null && include != null){
TopHitsAggregationBuilder topHitsBuilder = AggregationBuilders.topHits("top_hits");
topHitsBuilder.size(hitsSize);
topHitsBuilder.fetchSource(include,null);
termsAgg.subAggregation(topHitsBuilder);
}
filterAgg.subAggregation(termsAgg);
// 将术语聚合添加到嵌套聚合中
nestedAgg.subAggregation(termsAgg);
// 将聚合添加到搜索源构建器中
sourceBuilder.aggregation(nestedAgg);
}else{
TermsAggregationBuilder termsAgg = AggregationBuilders.terms("agg_field").field(fieldName);
if(hitsSize != null && include != null){
TopHitsAggregationBuilder topHitsBuilder = AggregationBuilders.topHits("top_hits");
// 聚合时,一条聚合信息对应几条数据 不设置则默认查全部数据
topHitsBuilder.size(hitsSize);
// 聚合时,需要显示哪些字段 不设置则查全部字段
topHitsBuilder.fetchSource(include,null);
termsAgg.subAggregation(topHitsBuilder);
}
filterAgg.subAggregation(termsAgg);
}
sourceBuilder.aggregation(filterAgg);
searchRequest.source(sourceBuilder);
try {
SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
List<? extends Terms.Bucket> buckets;
if(nestedFiledName != null){
buckets = getTermsByNested(searchResponse);
}else{
Filter filteredAggregation = searchResponse.getAggregations().get("filtered_agg");
Terms yourFieldAggregation = filteredAggregation.getAggregations().get("agg_field");
buckets = yourFieldAggregation.getBuckets();
}
List<Map<String,Object>> result = new ArrayList<>();
for (Terms.Bucket bucket : buckets) {
Map<String, Object> map = new HashMap<>(2);
map.put("name", bucket.getKeyAsString());
map.put("count", bucket.getDocCount());
if(include != null && hitsSize != null){
TopHits topHits = bucket.getAggregations().get("top_hits");
List<Map<String, Object>> hitList = new ArrayList<>();
for (SearchHit hit : topHits.getHits()) {
Map<String, Object> sourceAsMap = hit.getSourceAsMap();
hitList.add(sourceAsMap);
}
map.put("list", hitList);
}
result.add(map);
}
return result;
} catch (IOException e) {
e.printStackTrace();
log.error("索引{}聚合出现错误{}",indexName,e.getMessage());
throw new BusinessException("聚合查询出现错误");
}
}
/**
* [描述]
*/
private List<? extends Terms.Bucket> getTermsByNested(SearchResponse searchResponse ){
// 解析聚合结果
Aggregations aggregations = searchResponse.getAggregations();
Nested nested = aggregations.get("nested_agg");
Terms subjectIds = nested.getAggregations().get("agg_field");
return subjectIds.getBuckets();
}
/**
* [描述] 滚动检索 当不再需要检索时,最好调用 searchScrollClear() 对滚动ID进行清除
* @param scrollId 滚动ID 传null则初始化滚动 翻页则传入指定值
*@author Da.Pang
* @return list:检索数据 scrollId:滚动ID, 每次翻页时候都需要传入
*/
public PageResult<String> searchScroll(
String indexName,BoolQueryBuilder combinedQueryAnd,BoolQueryBuilder combinedQueryOr,
Map<String,SortOrder> sortOrderMap,List<String> highlightFields,
String[] include,int pageSize,String scrollId
) throws IOException {
SearchResponse searchResponse;
if(scrollId == null){
// 初始化搜索请求
SearchRequest searchRequest = new SearchRequest(indexName);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
getSearchSourceBuilder(searchSourceBuilder,
combinedQueryAnd, combinedQueryOr, sortOrderMap,
highlightFields, include);
// 设置滚动参数
searchRequest.scroll(TimeValue.timeValueSeconds(TIME_OUT));
searchRequest.source(searchSourceBuilder.size(pageSize));
// 发起第一次搜索请求
searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
// 获取滚动ID
scrollId = searchResponse.getScrollId();
}
System.out.printf("scrollId: %s %n", scrollId );
// 使用滚动ID发起下一次滚动请求
SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
scrollRequest.scroll(TimeValue.timeValueSeconds(TIME_OUT));
// 执行滚动请求
searchResponse = restHighLevelClient.scroll(scrollRequest, RequestOptions.DEFAULT);
// 处理搜索结果
List<String> list = new ArrayList<>();
for (SearchHit hit : searchResponse.getHits().getHits()) {
list.add(hit.getSourceAsString());
}
// 更新滚动ID,准备下一次滚动
scrollId = searchResponse.getScrollId();
/* if(list.isEmpty()){
// 正常情况 如果检索查不到数据 则清理滚动ID
searchScrollClear(scrollId );
}*/
PageResult<String> result = new PageResult<>();
result.setList(list);
result.setScrollId(scrollId);
result.setTotal(searchResponse.getHits().getTotalHits().value);
return result;
}
/**
* [描述] 清理指定 scrollId 值
* 一般情况下,当滚动查询没有数据的时候,或不再进行查询时候,就需要清理滚动上下文,释放资源
* 如果报错:ElasticsearchStatusException: Unable to parse response body 表示已经清理指定 scrollId 值了
*/
public void searchScrollClear( String scrollId ) {
try{
// 清除滚动上下文,释放资源
ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
clearScrollRequest.addScrollId(scrollId);
restHighLevelClient.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
}catch(Exception e){
e.printStackTrace();
log.error("清理scrollId错误,scrollId:{},message信息:{}",scrollId,e.getMessage());
}
}
/**
* [描述]通过 after_sort_values 进行翻页
* @param sortOrderMap 由于通过 after 翻页 需要默认排序字段 所以该参数必填
* @param lastSortValues 第一次传null 等拿到 resultSortValues 值以后传入可查到下一页数据
*@author Da.Pang
*/
public PageResult<String> testSearchAfter(String indexName, BoolQueryBuilder combinedQueryAnd, BoolQueryBuilder combinedQueryOr,
@NotNull Map<String,SortOrder> sortOrderMap, List<String> highlightFields,
String[] include, int pageSize, Object[] lastSortValues) {
// 初始化搜索请求
SearchRequest searchRequest = new SearchRequest(indexName);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
getSearchSourceBuilder(searchSourceBuilder,
combinedQueryAnd, combinedQueryOr, sortOrderMap,
highlightFields, include);
searchSourceBuilder.timeout(TimeValue.timeValueSeconds(TIME_OUT));
searchSourceBuilder.size(pageSize);
if(lastSortValues != null){
searchSourceBuilder.searchAfter(lastSortValues);
}
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = null;
try {
searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
e.printStackTrace();
log.error("检索报错,lastSortValues:{},message信息:{}",lastSortValues,e.getMessage());
}
SearchHit[] hits = searchResponse.getHits().getHits();
List<String> list = new ArrayList<>();
for (SearchHit hit : hits) {
list.add(hit.getSourceAsString());
}
if(hits.length == 0){
PageResult<String> result = new PageResult<>();
result.setTotalPage(0);
return result;
}
Object[] resultSortValues = hits[hits.length - 1].getSortValues();
PageResult<String> result = new PageResult<>();
result.setList(list);
result.setTotal(searchResponse.getHits().getTotalHits().value);
result.setLastSortValues(resultSortValues);
return result;
}
}
分页类:
/**
* @author Da.Pang
* 2024/2/26 13:30
*/
@Data
public class PageResult<T> {
private Integer pageNum;
private Integer pageSize;
private Long total;
private List<T> list;
private Integer totalPage;
private String scrollId;
private Object[] lastSortValues;
}
P.S:
1.需要排序的字段,在新增数据时都要加上默认值,最好不要为null插入到es中,否则根据该字段进行排序会出现报错情况。
2.只有以下结构在进行精准匹配才需要 字段.keyword 拼接以下,如果是Integer/Long 等类型,不需要拼接,否则会查不到数据。
{
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
- keyword 类型字段最好都有默认值。
- es在插入数据时,可根据 _id 进行判断,如果存在则全量更新,不存在则全量保存。但是注意,如果第二次更新的某字段为null,则会覆盖之前es中的值。
- 部分相关博客:
分词器区别: https://blog.csdn.net/jiayoubing/article/details/105392105
- 解决Ik分词器对于 纯 字母/数字 关键词命中问题:
问题描述: 比如: 【名侦探可能播放10000期】 这个值,如果直接搜【10000】可能会命中不到结果,因为ik分词器没有分出10000。
使用nGram分词器: https://blog.csdn.net/jicanweiCSDN/article/details/127691815