ES批量添加数据-同步-异步

ES学习记录

1. ES-API

1.1 restClient-客户端

连接Es需要创建一个Es的客户端。

添加依赖如下， 最好和我们的es的服务端的版本保持一致。

服务端版本在kibiana输入：

GET / 

既可以获取到版本信息。

maven

        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>${elasticsearch.version}</version>
        </dependency>

java 代码

package com.example.es;

import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;

@SpringBootTest
class EsApplicationTests {

    private RestHighLevelClient restHighLevelClient;

    private String username;
    private String password;

    @Test
    void test() {
        System.out.println(restHighLevelClient);
    }

    @BeforeEach
    void setUp(){
        // 也可以创建多个ES的server, 可变参数
        RestClientBuilder builder = RestClient.builder(HttpHost.create("http://192.168.200.130:9200"));
        // 还可以添加用户名+密码
        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password));
        // 在设置密码+用户名的时候, 1  2 选一个就可以. 
        // 1. 这个里面是采用了匿名内部类的方式来创建了一个对象.  并且使用了lambda 表达式进行了简化
        builder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));

        // 2. 这个写法和1是完全一致的.
        builder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
            @Override
            public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
                return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
            }
        });
        RestHighLevelClient client = new RestHighLevelClient(builder);

        restHighLevelClient=client;
    }
}

1.2 批量添加文档

批量添加文档~

其实感觉有点鸡肋啊, 应该有那种 可以直接放到指定一个 索引, 可以直接放进一个List , 而不是要一个一个转, 好麻烦.

    /**
     * 批量添加数据
     *
     * @throws IOException
     */
    @Test
    void bulkAdd() throws IOException {

        User user1 = new User("张三", "北京");
        User user2 = new User("李四", "西安");
        User user3 = new User("java", "上海");
        String json1 = JSON.toJSONString(user1);
        String json2 = JSON.toJSONString(user2);
        String json3 = JSON.toJSONString(user3);

        List<DocWriteRequest<?>> userArrayList = new ArrayList<>();

        // 这里要创建每一个IndexRequest,真的是麻烦.
        IndexRequest request1 = new IndexRequest("user");
        request1.source(json1,XContentType.JSON);
        IndexRequest request2 = new IndexRequest("user");
        request2.source(json2,XContentType.JSON);
        IndexRequest request3 = new IndexRequest("user");
        request3.source(json3,XContentType.JSON);

        userArrayList.add(request1);
        userArrayList.add(request2);
        userArrayList.add(request3);
        BulkRequest bulkRequest = new BulkRequest();
        bulkRequest.add(userArrayList);
        client.bulk(bulkRequest,RequestOptions.DEFAULT);
    }

1.3批量异步添加文档

    @Test
    void bulkAsyncAdd() {
        User user1 = new User("张三1", "北京");
        User user2 = new User("李四1", "西安");
        User user3 = new User("java1", "上海");
        String json1 = JSON.toJSONString(user1);
        String json2 = JSON.toJSONString(user2);
        String json3 = JSON.toJSONString(user3);

        // 批量添加文档的list
        List<DocWriteRequest<?>> userArrayList = new ArrayList<>();

        IndexRequest request1 = new IndexRequest("user");
        request1.source(json1, XContentType.JSON);
        IndexRequest request2 = new IndexRequest("user");
        request2.source(json2, XContentType.JSON);
        IndexRequest request3 = new IndexRequest("user");
        request3.source(json3, XContentType.JSON);

        userArrayList.add(request1);
        userArrayList.add(request2);
        userArrayList.add(request3);
        BulkRequest bulkRequest = new BulkRequest();
        bulkRequest.add(userArrayList);
        client.bulkAsync(bulkRequest, RequestOptions.DEFAULT, new ActionListener<BulkResponse>() {
            @Override
            public void onResponse(BulkResponse bulkItemResponses) {
                log.info("异步的线程是:{}",Thread.currentThread().getName());
                log.info("打印执行结果: {}",JSON.toJSONString(bulkItemResponses));
            }

            @Override
            public void onFailure(Exception e) {
                log.error("打印失败的结果: {}",e);
            }
        });
        System.out.println("本次入库main线程已经结束");
    }

本来我想的是, 可以在异步执行后, 执行后的结果, 可以打印出来, 但是发现并不能打印, 异步的线程里面无法直接打印出结果.

需要使用异步编程的 一个类. CompletableFuture

 @Test
    void bulkAsyncAdd3() throws InterruptedException, ExecutionException, ExecutionException {
        User user1 = new User("张三1", "北京");
        User user2 = new User("李四1", "西安");
        User user3 = new User("java1", "上海");
        String json1 = JSON.toJSONString(user1);
        String json2 = JSON.toJSONString(user2);
        String json3 = JSON.toJSONString(user3);

        List<DocWriteRequest<?>> userArrayList = new ArrayList<>();

        IndexRequest request1 = new IndexRequest("user");
        request1.source(json1, XContentType.JSON);
        IndexRequest request2 = new IndexRequest("user");
        request2.source(json2, XContentType.JSON);
        IndexRequest request3 = new IndexRequest("user");
        request3.source(json3, XContentType.JSON);

        userArrayList.add(request1);
        userArrayList.add(request2);
        userArrayList.add(request3);
        BulkRequest bulkRequest = new BulkRequest();
        bulkRequest.add(userArrayList);

        CompletableFuture<BulkResponse> future = new CompletableFuture<>();

        client.bulkAsync(bulkRequest, RequestOptions.DEFAULT, new ActionListener<BulkResponse>() {
            @SneakyThrows
            @Override
            public void onResponse(BulkResponse bulkItemResponses) {
                log.info("打印异步线程:{}",Thread.currentThread().getName());
                future.complete(bulkItemResponses);
            }

            @Override
            public void onFailure(Exception e) {
                future.completeExceptionally(e);
            }
        });

        // 等待异步操作完成
        BulkResponse bulkResponse = future.get();

        // 处理异步操作结果
        log.info("异步的线程是: {}", Thread.currentThread().getName());
        log.info("打印执行结果: {}", JSON.toJSONString(bulkResponse));

        // 在这里可以进行断言或其他验证
    }

后面继续补充~
学习效率太低了 ~ _ ~