华为云Elasticsearch(FusionInsight HD)连接和开发教程02-通过LowLevel RestClient操作ES

典型场景说明

通过典型场景,我们可以快速学习和掌握Elasticsearch的开发过程,并且对关键的接口函数有所了解。

场景说明

假定用户开发一个应用程序,用于搜索所有图书信息,并要求提供关键字搜索关键字相关的图书,并打分按分排序,其中搜索的功能就可以用Elasticsearch来实现,搜索流程如下:

  1. 客户端连接集群
  2. 查询集群健康状态
  3. 检查指定索引是否存在
  4. 创建指定分片数目的索引
  5. 写入索引数据
  6. 批量写入数据
  7. 查询索引信息
  8. 删除索引
  9. 删除索引中的文档
  10. 刷新索引
  11. 多线程样例

样例代码

Low Level Rest Client样例代码

客户端连接集群

功能简介

获取客户端,通过设置IP和端口连接到特定Elasticsearch集群,是使用Elasticsearch提供的API之前的必要工作。

在进行完Elasticsearch操作后,需要调用“RestClient.close()”关闭所申请的资源。

    public static void main(String[] args) {
        HwRestClient hwRestClient = new HwRestClient();
        RestClient restClient = hwRestClient.getRestClient();
        try {
             /.../
        } catch (Exception e) {
            LOG.error("There are exceptions occurred.", e);
        } finally {
            if (restClient != null) {
                try {
                    restClient.close();
                    LOG.info("Close the client successful.");
                } catch (Exception e1) {
                    LOG.error("Close the client failed.", e1);
                }
            }
        }
    }
  • HwRestClient 默认从代码运行路径的conf目录下读取配置文件:esParam.properties、krb5.conf 和 user.keytab;
  • 也支持自定义指定,如代码运行在Windows环境,配置文件均在D盘根目录,通过 HwRestClient hwRestClient = new HwRestClient("D:\\"); 来指定配置文件的路径。

查询集群信息

功能简介

位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/cluster”目录下的QueryClusterInfo.java,作用是查询Elasticsearch集群的相关信息。

    /**
     * Query the cluster's information
     */
    private static void queryClusterInfo(RestClient restClientTest) {
        
        Response response;
        try {
            Request request=new Request("GET", "/_cluster/health");
            //对返回结果进行处理,展示方式更直观
            request.addParameter("pretty", "true");
            response = restClientTest.performRequest(request);
            if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
                LOG.info("QueryClusterInfo successful.");
            } else {
                LOG.error("QueryClusterInfo failed.");
            }
            LOG.info("QueryClusterInfo response entity is : " + EntityUtils.toString(response.getEntity()));
        } catch (Exception e) {
            LOG.error("QueryClusterInfo failed, exception occurred.", e);
        }
    }      

检查指定索引是否存在

功能简介

位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/exist”目录下的IndexIsExist.java,作用是检查指定的索引是否存在于Elasticsearch集群中。

    /**
     * Check the existence of the index or not.
     */
    private static void exist(RestClient restClientTest, String index) {
        
        Response response;
        try {
            Request request = new Request("HEAD", "/" + index);
            response = restClientTest.performRequest(request);
            if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
                LOG.info("***** Check index successful,index is exist : " + index + " *****");
            }
            if (HttpStatus.SC_NOT_FOUND == response.getStatusLine().getStatusCode()) {
                LOG.info("***** Index is not exist : " + index + " *****");
            }
        } catch (Exception e) {
            LOG.error("Check index failed, exception occurred.", e);
        }
    }

创建指定分片数目的索引

功能简介

位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/index”目录下的CreateIndex.java,作用是创建指定primary shardreplica数目的索引。

以下代码中的变量shardNum、replicaNum,即需要创建索引的primary shardreplica数目。

    /**
     * Create one index with customized shard number and replica number.
     */
    private static void createIndexWithShardNum(RestClient restClientTest, String index) {
        
        Response rsp;
        int shardNum = 3;
        int replicaNum = 1;
        String jsonString =
            "{" + "\"settings\":{" + "\"number_of_shards\":\"" + shardNum + "\"," + "\"number_of_replicas\":\"" + replicaNum + "\"" + "}}";
        HttpEntity entity = new NStringEntity(jsonString, ContentType.APPLICATION_JSON);
        try {
            Request request = new Request("PUT", "/" + index);
            request.addParameter("pretty", "true");
            request.setEntity(entity);
            rsp = restClientTest.performRequest(request);
            if (HttpStatus.SC_OK == rsp.getStatusLine().getStatusCode()) {
                LOG.info("CreateIndexWithShardNum successful.");
            } else {
                LOG.error("CreateIndexWithShardNum failed.");
            }
            LOG.info("CreateIndexWithShardNum response entity is : " + EntityUtils.toString(rsp.getEntity()));
        } catch (Exception e) {
            LOG.error("CreateIndexWithShardNum failed, exception occurred.", e);
        }
    }

写入索引数据

功能简介

位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/index”目录下的PutData.java,作用是往指定索引中插入数据,更新索引。

String变量jsonString即为插入的数据内容,用户可以自定义数据内容。

    /**
     * Write one document into the index
    
     */
    private static void putData(RestClient restClientTest, String index, String type, String id) {
        
        Gson gson = new Gson();
        Map<String, Object> esMap = new HashMap<>();
        esMap.put("name", "Happy");
        esMap.put("author", "Alex Yang");
        esMap.put("pubinfo", "Beijing,China");
        esMap.put("pubtime", "2020-01-01");
        esMap.put("description", "Elasticsearch is a highly scalable open-source full-text search and analytics engine.");
        String jsonString = gson.toJson(esMap);

        HttpEntity entity = new NStringEntity(jsonString, ContentType.APPLICATION_JSON);
        Response rsp;

        try {
            Request request = new Request("POST", "/" + index + "/" + type + "/" + id);
            request.addParameter("pretty", "true");
            request.setEntity(entity);
            rsp = restClientTest.performRequest(request);
            if (HttpStatus.SC_OK == rsp.getStatusLine().getStatusCode() || HttpStatus.SC_CREATED == rsp.getStatusLine().getStatusCode()) {
                LOG.info("PutData successful.");
            } else {
                LOG.error("PutData failed.");
            }
            LOG.info("PutData response entity is : " + EntityUtils.toString(rsp.getEntity()));
        } catch (Exception e) {
            LOG.error("PutData failed, exception occurred.", e);
        }
    }

批量写入数据

功能简介

位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/bulk”目录下的Bulk.java,作用是批量写入数据到指定的索引中。

totalRecordNumber是需要插入的总document数量,oneCommit是一次循环插入的document数量。esMap中添加的数据即为需要写入索引的数据信息。

    /**
     * Send a bulk request
     */
    private static void bulk(RestClient restClientTest, String index, String type) {
        //需要写入的总文档数
        long totalRecordNum = 10;
        //一次bulk写入的文档数
        long oneCommit = 5;
        long circleNumber = totalRecordNum / oneCommit;
        StringEntity entity;
        Gson gson = new Gson();
        Map<String, Object> esMap = new HashMap<>();
        String str = "{ \"index\" : { \"_index\" : \"" + index + "\", \"_type\" :  \"" + type + "\"} }";
        for (int i = 1; i <= circleNumber; i++) {
            StringBuilder builder = new StringBuilder();
            for (int j = 1; j <= oneCommit; j++) {
                esMap.clear();
                esMap.put("name", "Linda");
                esMap.put("age", ThreadLocalRandom.current().nextInt(18, 100));
                esMap.put("height", (float)ThreadLocalRandom.current().nextInt(140, 220));
                esMap.put("weight", (float)ThreadLocalRandom.current().nextInt(70, 200));
                esMap.put("cur_time", System.currentTimeMillis());
                String strJson = gson.toJson(esMap);
                builder.append(str).append("\n");
                builder.append(strJson).append("\n");
            }
            entity = new StringEntity(builder.toString(), ContentType.APPLICATION_JSON);
            entity.setContentEncoding("UTF-8");
            Response response;
            try {
                Request request = new Request("PUT", "/_bulk");
                request.addParameter("pretty", "true");
                request.setEntity(entity);
                response = restClientTest.performRequest(request);
                if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
                    LOG.info("Already input documents : " + oneCommit * i);
                } else {
                    LOG.error("Bulk failed.");
                }
                LOG.info("Bulk response entity is : " + EntityUtils.toString(response.getEntity()));
            } catch (Exception e) {
                LOG.error("Bulk failed, exception occurred.", e);
            }
        }
    }

查询索引信息

功能简介

位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/search”目录下的QueryData.java,作用是查询指定index、type、id下的文档信息。

    /**
     * Query all data of one index.
     */
    private static void queryData(RestClient restClientTest, String index, String type, String id){
        
        Response response;
        try {
            Request request = new Request("GET", "/" + index + "/" + type + "/" + id);
            request.addParameter("pretty", "true");
            response = restClientTest.performRequest(request);
            if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
                LOG.info("QueryData successful.");
            } else {
                LOG.error("QueryData failed.");
            }
            LOG.info("QueryData response entity is : " + EntityUtils.toString(response.getEntity()));
        } catch (Exception e) {
            LOG.error("QueryData failed, exception occurred.", e);
        }
    }

删除索引

功能简介

位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/delete”目录下的DeleteIndex.java,作用是删除指定索引。

    /**
     * Delete one index
     */
    private static void deleteIndex(RestClient restClientTest, String index) {
        
        Response response;
        try {
            Request request = new Request("DELETE", "/" + index + "?&pretty=true");
            response = restClientTest.performRequest(request);
            if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
                LOG.info("DeleteIndex successful.");
            } else {
                LOG.error("DeleteIndex failed.");
            }
            LOG.info("DeleteIndex response entity is : " + EntityUtils.toString(response.getEntity()));
        } catch (Exception e) {
            LOG.error("DeleteIndex failed, exception occurred.", e);
        }
    }

刷新索引

刷新单个索引

位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/flush”目录下的Flush.java,作用是刷新指定的索引。

    /**
     * Flush one index data to storage and clearing the internal transaction log
     */
    private static void flushOneIndex(RestClient restClientTest, String index) {
        Map<String, String> params = Collections.singletonMap("pretty", "true");
Response flushRsp;
        try {
            Request request=new Request("POST", "/" + index + "/_flush");
            request.addParameter("pretty", "true");
            flushRsp = restClientTest.performRequest(request);
            LOG.info(EntityUtils.toString(flushRsp.getEntity()));

            if (HttpStatus.SC_OK == flushRsp.getStatusLine().getStatusCode()) {
                LOG.info("Flush one index successful.");
            } else {
                LOG.error("Flush one index failed.");
            }
            LOG.info("Flush one index response entity is : " + EntityUtils.toString(flushRsp.getEntity()));
        } catch (Exception e) {
            LOG.error("Flush one index failed, exception occurred.", e);
        }
    }

刷新所有索引

位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/flush”目录下的Flush.java,作用是刷新所有索引。

    /**
     * Flush all indexes data to storage and clearing the internal transaction log
     * The user who performs this operation must belong to the "supergroup" group.
     */
    private static void flushAllIndexes(RestClient restClientTest) {
        Map<String, String> params = Collections.singletonMap("pretty", "true");
        Response flushRsp;
        try {
            flushRsp = restClientTest.performRequest("POST", "/_all/_flush",params);
            if (HttpStatus.SC_OK == flushRsp.getStatusLine().getStatusCode()) {
                LOG.info("Flush all indexes successful.");
            } else {
                LOG.error("Flush all indexes failed.");
            }
            LOG.info("Flush all indexes response entity is : " + EntityUtils.toString(flushRsp.getEntity()));
        } catch (Exception e) {
            LOG.error("Flush all indexes failed, exception occurred.", e);
        }
    }

多线程样例

功能简介

位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/multithread”目录下的MultithreadRequest.java,作用是创建线程类sendRequestThread,重写run方法,实现bulk请求,批量写入数据。

    /**
     * Thread that sends bulk request
     */
    public  static class sendRequestThread implements Runnable {        
        private RestClient restCientTh;
        private HwRestClient hwRestClient;
        @Override
        public void run() {
            LOG.info("Thread begin.");
            try {
               hwRestClient = new HwRestClient();
                estCientTh = hwRestClient.getRestClient();
                LOG.info("Thread name: " + Thread.currentThread().getName());
                bulk(restCientTh, "huawei1", "type1");
            } catch (Exception e) {
                LOG.error("SendRequestThread has exception.", e);
            } finally {
                if (restCientTh != null) {
                    try {
                        restCientTh.close();
                        LOG.info("Close the client successful in thread : " + Thread.currentThread().getName());
                    } catch (IOException e) {
                        LOG.error("Close the cient failed.", e);
                    }
                }
            }
        }
    }

在main方法中,创建了线程池fixedThreadPool,通过指定线程池大小threadPoolSize和任务数jobNumber,实现多线程发送bulk请求,并发地批量写入数据。

    public static void main(String[] args) throws Exception {
       
        LOG.info("Start to do multithread request !");
        ExecutorService fixedThreadPool;
        HwRestClient hwRestClient = new HwRestClient();
        RestClient restCientThNew;
        restCientThNew = hwRestClient.getRestClient();
        int threadPoolSize = 2;
        int jobNumber = 5;
        try {
            fixedThreadPool = Executors.newFixedThreadPool(threadPoolSize);
            for (int i = 0; i < jobNumber; i++) {
                fixedThreadPool.execute(new sendRequestThread());
            }
            fixedThreadPool.shutdown();
        } catch (Exception e) {
            LOG.error("There are exceptions in sendRequestThread.", e);
        } finally {
            if (restCientThNew != null) {
                try {
                    restCientThNew.close();
                    LOG.info("Close the client successful.");
                } catch (Exception e1) {
                    LOG.error("Close the client failed.", e1);
                }
            }
        }
    }

​​​​​​​

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

sunjian286

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值