Spring Boot 集成 Elasticsearch 7.x手记

前言

生产环境采部署的7.5.0版本的es集群，且使用自带的xpack验证，不能升级springboot版本，以免影响现有业务，只能手动兼容。

相关的版本

• SpringCloud Greenwich.SR2
• SpringBoot 2.1.7.RELEASE

项目配置

maven依赖

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.data</groupId>
  <artifactId>spring-data-elasticsearch</artifactId>
  <version>3.2.6.RELEASE</version>
</dependency>

上述spring-data-elasticsearch需要重新引入更新的版本，后面需要使用restclient连接es集群

使用intellij或者maven命令查看spring-data-elasticsearch包的依赖关系，可以知晓几个重要包的版本。

es连接加载配置

网上流传一种配置，只需要修改yml配置，就可直接使用，试了一下没有成功。

 elasticsearch:
    rest:
      username: elastic
      password: 123456
      uris: ["*.*.*.*:9200","*.*.*.*:9200"]

于是作出以下配置

• yml

 elasticsearch:
    rest:
      username: elastic
      password: 123456
      url: "*.*.*.*:9200,*.*.*.*:9200"

• config

import lombok.extern.slf4j.Slf4j;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.client.ClientConfiguration;
import org.springframework.data.elasticsearch.client.RestClients;
import org.springframework.data.elasticsearch.config.AbstractElasticsearchConfiguration;

import java.util.List;

@Configuration
@Slf4j
public class RestClientConfig extends AbstractElasticsearchConfiguration {

//    @Value("${spring.elasticsearch.rest.uris}")
    private List<String> uris;

    @Value("${spring.elasticsearch.rest.username}")
    private String username;

    @Value("${spring.elasticsearch.rest.password}")
    private String password;

    @Value("#{'${spring.elasticsearch.rest.url}'.split(',')}")
    private  String[] url;

    @Override
    @Bean
    public RestHighLevelClient elasticsearchClient() {
        log.info("[RestClientConfig] ----------> url={}", url);
        final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
            .connectedTo(url)
            .withBasicAuth(username, password)
            .build();
        return RestClients.create(clientConfiguration).rest();
    }
}

• Document

@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
@Document(indexName = "elslog", type = "_doc")
public class ElsLog {

    @Id
    private String id;

    private String orderId;

    private String content;

    private Date createTime;
}

• Repository

import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
import org.springframework.stereotype.Repository;
@Repository
public interface ElsLogRepository extends ElasticsearchRepository<ElsLog, String> {
}

• Test

 @Autowired
  private ElsLogRepository elsLogRepository;
  
  public void testSave(String... args) throws Exception {
      ElsLog log = ElsLog.builder()
              .context("test")
              .createTime(new Date())
              .build();
      elsLogRepository.save(log);
      System.out.println("------------------------------> save log to els over.");
  }
  
  public void testQuery() {
// NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder();
//        NativeSearchQuery searchQuery = nativeSearchQueryBuilder.withSort(SortBuilders.fieldSort("createTime").order(SortOrder.DESC)).build();
//        List<ElsLog> elsLogs = elasticsearchRestTemplate.queryForList(searchQuery, ElsLog.class);
//        System.out.println(new Gson().toJson(elsLogs));
}

查询报错解决

问题 java.lang.NoSuchMethodError: org.elasticsearch.search.SearchHits.getTotalHits()J

解决

• 重写 ResultMapper
  需要修改方法 mapResults方法中的“response.getHits().getTotalHits()”，具体代码如下：

import com.fasterxml.jackson.core.JsonEncoding;
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonGenerator;
import org.elasticsearch.action.get.GetResponse;
import org.elasticsearch.action.get.MultiGetItemResponse;
import org.elasticsearch.action.get.MultiGetResponse;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.common.document.DocumentField;
import org.elasticsearch.search.SearchHit;
import org.springframework.core.convert.ConversionService;
import org.springframework.core.convert.support.DefaultConversionService;
import org.springframework.data.domain.Pageable;
import org.springframework.data.elasticsearch.ElasticsearchException;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.ScriptedField;
import org.springframework.data.elasticsearch.core.AbstractResultMapper;
import org.springframework.data.elasticsearch.core.DefaultEntityMapper;
import org.springframework.data.elasticsearch.core.EntityMapper;
import org.springframework.data.elasticsearch.core.aggregation.AggregatedPage;
import org.springframework.data.elasticsearch.core.aggregation.impl.AggregatedPageImpl;
import org.springframework.data.elasticsearch.core.mapping.ElasticsearchPersistentEntity;
import org.springframework.data.elasticsearch.core.mapping.ElasticsearchPersistentProperty;
import org.springframework.data.elasticsearch.core.mapping.SimpleElasticsearchMappingContext;
import org.springframework.data.mapping.PersistentPropertyAccessor;
import org.springframework.data.mapping.context.MappingContext;
import org.springframework.data.mapping.model.ConvertingPropertyAccessor;
import org.springframework.lang.Nullable;
import org.springframework.util.Assert;
import org.springframework.util.StringUtils;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.lang.reflect.Field;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;

public class MyResultMapper extends AbstractResultMapper {

    private final MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext;
    private final ConversionService conversionService;

    public MyResultMapper() {
        this((MappingContext)(new SimpleElasticsearchMappingContext()));
    }

    public MyResultMapper(MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext) {
        this(mappingContext, initEntityMapper(mappingContext));
    }

    public MyResultMapper(EntityMapper entityMapper) {
        this(new SimpleElasticsearchMappingContext(), entityMapper);
    }

    public MyResultMapper(MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext, @Nullable EntityMapper entityMapper) {
        super(entityMapper != null ? entityMapper : initEntityMapper(mappingContext));
        this.conversionService = new DefaultConversionService();
        this.mappingContext = mappingContext;
    }

    private static EntityMapper initEntityMapper(MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext) {
        Assert.notNull(mappingContext, "MappingContext must not be null!");
        return new DefaultEntityMapper(mappingContext);
    }

    @Override
    public <T> AggregatedPage<T> mapResults(SearchResponse response, Class<T> clazz, Pageable pageable) {
        long totalHits = response.getHits().getTotalHits().value;
        float maxScore = response.getHits().getMaxScore();
        List<T> results = new ArrayList();
        Iterator var8 = response.getHits().iterator();

        while(var8.hasNext()) {
            SearchHit hit = (SearchHit)var8.next();
            if (hit != null) {
                T result = null;
                String hitSourceAsString = hit.getSourceAsString();
                if (!StringUtils.isEmpty(hitSourceAsString)) {
                    result = this.mapEntity(hitSourceAsString, clazz);
                } else {
                    result = this.mapEntity(hit.getFields().values(), clazz);
                }

                this.setPersistentEntityId(result, hit.getId(), clazz);
                this.setPersistentEntityVersion(result, hit.getVersion(), clazz);
                this.setPersistentEntityScore(result, hit.getScore(), clazz);
                this.populateScriptFields(result, hit);
                results.add(result);
            }
        }

        return new AggregatedPageImpl(results, pageable, totalHits, response.getAggregations(), response.getScrollId(), maxScore);
    }

    private <T> void populateScriptFields(T result, SearchHit hit) {
        if (hit.getFields() != null && !hit.getFields().isEmpty() && result != null) {
            Field[] var3 = result.getClass().getDeclaredFields();
            int var4 = var3.length;

            for(int var5 = 0; var5 < var4; ++var5) {
                Field field = var3[var5];
                ScriptedField scriptedField = (ScriptedField)field.getAnnotation(ScriptedField.class);
                if (scriptedField != null) {
                    String name = scriptedField.name().isEmpty() ? field.getName() : scriptedField.name();
                    DocumentField searchHitField = (DocumentField)hit.getFields().get(name);
                    if (searchHitField != null) {
                        field.setAccessible(true);

                        try {
                            field.set(result, searchHitField.getValue());
                        } catch (IllegalArgumentException var11) {
                            throw new ElasticsearchException("failed to set scripted field: " + name + " with value: " + searchHitField.getValue(), var11);
                        } catch (IllegalAccessException var12) {
                            throw new ElasticsearchException("failed to access scripted field: " + name, var12);
                        }
                    }
                }
            }
        }

    }

    private <T> T mapEntity(Collection<DocumentField> values, Class<T> clazz) {
        return this.mapEntity(this.buildJSONFromFields(values), clazz);
    }

    private String buildJSONFromFields(Collection<DocumentField> values) {
        JsonFactory nodeFactory = new JsonFactory();

        try {
            ByteArrayOutputStream stream = new ByteArrayOutputStream();
            JsonGenerator generator = nodeFactory.createGenerator(stream, JsonEncoding.UTF8);
            generator.writeStartObject();
            Iterator var5 = values.iterator();

            while(true) {
                while(var5.hasNext()) {
                    DocumentField value = (DocumentField)var5.next();
                    if (value.getValues().size() > 1) {
                        generator.writeArrayFieldStart(value.getName());
                        Iterator var7 = value.getValues().iterator();

                        while(var7.hasNext()) {
                            Object val = var7.next();
                            generator.writeObject(val);
                        }

                        generator.writeEndArray();
                    } else {
                        generator.writeObjectField(value.getName(), value.getValue());
                    }
                }

                generator.writeEndObject();
                generator.flush();
                return new String(stream.toByteArray(), Charset.forName("UTF-8"));
            }
        } catch (IOException var9) {
            return null;
        }
    }

    public <T> T mapResult(GetResponse response, Class<T> clazz) {
        T result = this.mapEntity(response.getSourceAsString(), clazz);
        if (result != null) {
            this.setPersistentEntityId(result, response.getId(), clazz);
            this.setPersistentEntityVersion(result, response.getVersion(), clazz);
        }

        return result;
    }

    public <T> List<T> mapResults(MultiGetResponse responses, Class<T> clazz) {
        List<T> list = new ArrayList();
        MultiGetItemResponse[] var4 = responses.getResponses();
        int var5 = var4.length;

        for(int var6 = 0; var6 < var5; ++var6) {
            MultiGetItemResponse response = var4[var6];
            if (!response.isFailed() && response.getResponse().isExists()) {
                T result = this.mapEntity(response.getResponse().getSourceAsString(), clazz);
                this.setPersistentEntityId(result, response.getResponse().getId(), clazz);
                this.setPersistentEntityVersion(result, response.getResponse().getVersion(), clazz);
                list.add(result);
            }
        }

        return list;
    }

    private <T> void setPersistentEntityId(T result, String id, Class<T> clazz) {
        if (clazz.isAnnotationPresent(Document.class)) {
            ElasticsearchPersistentEntity<?> persistentEntity = (ElasticsearchPersistentEntity)this.mappingContext.getRequiredPersistentEntity(clazz);
            ElasticsearchPersistentProperty idProperty = (ElasticsearchPersistentProperty)persistentEntity.getIdProperty();
            PersistentPropertyAccessor<T> accessor = new ConvertingPropertyAccessor(persistentEntity.getPropertyAccessor(result), this.conversionService);
            if (idProperty != null && idProperty.getType().isAssignableFrom(String.class)) {
                accessor.setProperty(idProperty, id);
            }
        }

    }

    private <T> void setPersistentEntityVersion(T result, long version, Class<T> clazz) {
        if (clazz.isAnnotationPresent(Document.class)) {
            ElasticsearchPersistentEntity<?> persistentEntity = (ElasticsearchPersistentEntity)this.mappingContext.getPersistentEntity(clazz);
            ElasticsearchPersistentProperty versionProperty = persistentEntity.getVersionProperty();
            if (versionProperty != null && versionProperty.getType().isAssignableFrom(Long.class)) {
                Assert.isTrue(version != -1L, "Version in response is -1");
                persistentEntity.getPropertyAccessor(result).setProperty(versionProperty, version);
            }
        }

    }

    private <T> void setPersistentEntityScore(T result, float score, Class<T> clazz) {
        if (clazz.isAnnotationPresent(Document.class)) {
            ElasticsearchPersistentEntity<?> entity = (ElasticsearchPersistentEntity)this.mappingContext.getRequiredPersistentEntity(clazz);
            if (!entity.hasScoreProperty()) {
                return;
            }

            entity.getPropertyAccessor(result).setProperty(entity.getScoreProperty(), score);
        }

    }
}

ElasticSearch的几种client

• ES有4种客户端：Jest client、Rest client、Transport client、Node client
• ES支持两种协议 ：
  1. HTTP协议，支持的客户端有Jest client和Rest client；
  2. Native Elasticsearch binary协议，也就是Transport client和Node client
• Jest client和Rest client区别：
  Jest client非官方支持，在ES5.0之前官方提供的客户端只有Transport client、Node client。在5.0之后官方发布Rest client，并大力推荐。
• Transport client和Node client区别：
  Transport client（7.0弃用）和Node client（2.3弃用）区别：最早的两个客户端，Transport client是不需要单独一个节点。Node client需要单独建立一个节点，连接该节点进行操作，ES2.3之前有独立的API，ES2.3之后弃用该API，推荐用户创建一个节点，并用Transport client连接进行操作。

综上，一般选取官方推荐的Rest客户端。