典型场景说明
通过典型场景,我们可以快速学习和掌握Elasticsearch的开发过程,并且对关键的接口函数有所了解。
场景说明
假定用户开发一个应用程序,用于搜索所有图书信息,并要求提供关键字搜索关键字相关的图书,并打分按分排序,其中搜索的功能就可以用Elasticsearch来实现,搜索流程如下:
- 客户端连接集群
- 查询集群健康状态
- 检查指定索引是否存在
- 创建指定分片数目的索引
- 写入索引数据
- 批量写入数据
- 查询索引信息
- 删除索引
- 删除索引中的文档
- 刷新索引
- 多线程样例
样例代码
Low Level Rest Client样例代码
客户端连接集群
功能简介
获取客户端,通过设置IP和端口连接到特定Elasticsearch集群,是使用Elasticsearch提供的API之前的必要工作。
在进行完Elasticsearch操作后,需要调用“RestClient.close()”关闭所申请的资源。
public static void main(String[] args) {
HwRestClient hwRestClient = new HwRestClient();
RestClient restClient = hwRestClient.getRestClient();
try {
/.../
} catch (Exception e) {
LOG.error("There are exceptions occurred.", e);
} finally {
if (restClient != null) {
try {
restClient.close();
LOG.info("Close the client successful.");
} catch (Exception e1) {
LOG.error("Close the client failed.", e1);
}
}
}
}
- HwRestClient 默认从代码运行路径的conf目录下读取配置文件:esParam.properties、krb5.conf 和 user.keytab;
- 也支持自定义指定,如代码运行在Windows环境,配置文件均在D盘根目录,通过 HwRestClient hwRestClient = new HwRestClient("D:\\"); 来指定配置文件的路径。
查询集群信息
功能简介
位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/cluster”目录下的QueryClusterInfo.java,作用是查询Elasticsearch集群的相关信息。
/**
* Query the cluster's information
*/
private static void queryClusterInfo(RestClient restClientTest) {
Response response;
try {
Request request=new Request("GET", "/_cluster/health");
//对返回结果进行处理,展示方式更直观
request.addParameter("pretty", "true");
response = restClientTest.performRequest(request);
if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
LOG.info("QueryClusterInfo successful.");
} else {
LOG.error("QueryClusterInfo failed.");
}
LOG.info("QueryClusterInfo response entity is : " + EntityUtils.toString(response.getEntity()));
} catch (Exception e) {
LOG.error("QueryClusterInfo failed, exception occurred.", e);
}
}
检查指定索引是否存在
功能简介
位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/exist”目录下的IndexIsExist.java,作用是检查指定的索引是否存在于Elasticsearch集群中。
/**
* Check the existence of the index or not.
*/
private static void exist(RestClient restClientTest, String index) {
Response response;
try {
Request request = new Request("HEAD", "/" + index);
response = restClientTest.performRequest(request);
if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
LOG.info("***** Check index successful,index is exist : " + index + " *****");
}
if (HttpStatus.SC_NOT_FOUND == response.getStatusLine().getStatusCode()) {
LOG.info("***** Index is not exist : " + index + " *****");
}
} catch (Exception e) {
LOG.error("Check index failed, exception occurred.", e);
}
}
创建指定分片数目的索引
功能简介
位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/index”目录下的CreateIndex.java,作用是创建指定primary shard和replica数目的索引。
以下代码中的变量shardNum、replicaNum,即需要创建索引的primary shard和replica数目。
/**
* Create one index with customized shard number and replica number.
*/
private static void createIndexWithShardNum(RestClient restClientTest, String index) {
Response rsp;
int shardNum = 3;
int replicaNum = 1;
String jsonString =
"{" + "\"settings\":{" + "\"number_of_shards\":\"" + shardNum + "\"," + "\"number_of_replicas\":\"" + replicaNum + "\"" + "}}";
HttpEntity entity = new NStringEntity(jsonString, ContentType.APPLICATION_JSON);
try {
Request request = new Request("PUT", "/" + index);
request.addParameter("pretty", "true");
request.setEntity(entity);
rsp = restClientTest.performRequest(request);
if (HttpStatus.SC_OK == rsp.getStatusLine().getStatusCode()) {
LOG.info("CreateIndexWithShardNum successful.");
} else {
LOG.error("CreateIndexWithShardNum failed.");
}
LOG.info("CreateIndexWithShardNum response entity is : " + EntityUtils.toString(rsp.getEntity()));
} catch (Exception e) {
LOG.error("CreateIndexWithShardNum failed, exception occurred.", e);
}
}
写入索引数据
功能简介
位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/index”目录下的PutData.java,作用是往指定索引中插入数据,更新索引。
String变量“jsonString”即为插入的数据内容,用户可以自定义数据内容。
/**
* Write one document into the index
*/
private static void putData(RestClient restClientTest, String index, String type, String id) {
Gson gson = new Gson();
Map<String, Object> esMap = new HashMap<>();
esMap.put("name", "Happy");
esMap.put("author", "Alex Yang");
esMap.put("pubinfo", "Beijing,China");
esMap.put("pubtime", "2020-01-01");
esMap.put("description", "Elasticsearch is a highly scalable open-source full-text search and analytics engine.");
String jsonString = gson.toJson(esMap);
HttpEntity entity = new NStringEntity(jsonString, ContentType.APPLICATION_JSON);
Response rsp;
try {
Request request = new Request("POST", "/" + index + "/" + type + "/" + id);
request.addParameter("pretty", "true");
request.setEntity(entity);
rsp = restClientTest.performRequest(request);
if (HttpStatus.SC_OK == rsp.getStatusLine().getStatusCode() || HttpStatus.SC_CREATED == rsp.getStatusLine().getStatusCode()) {
LOG.info("PutData successful.");
} else {
LOG.error("PutData failed.");
}
LOG.info("PutData response entity is : " + EntityUtils.toString(rsp.getEntity()));
} catch (Exception e) {
LOG.error("PutData failed, exception occurred.", e);
}
}
批量写入数据
功能简介
位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/bulk”目录下的Bulk.java,作用是批量写入数据到指定的索引中。
totalRecordNumber是需要插入的总document数量,oneCommit是一次循环插入的document数量。esMap中添加的数据即为需要写入索引的数据信息。
/**
* Send a bulk request
*/
private static void bulk(RestClient restClientTest, String index, String type) {
//需要写入的总文档数
long totalRecordNum = 10;
//一次bulk写入的文档数
long oneCommit = 5;
long circleNumber = totalRecordNum / oneCommit;
StringEntity entity;
Gson gson = new Gson();
Map<String, Object> esMap = new HashMap<>();
String str = "{ \"index\" : { \"_index\" : \"" + index + "\", \"_type\" : \"" + type + "\"} }";
for (int i = 1; i <= circleNumber; i++) {
StringBuilder builder = new StringBuilder();
for (int j = 1; j <= oneCommit; j++) {
esMap.clear();
esMap.put("name", "Linda");
esMap.put("age", ThreadLocalRandom.current().nextInt(18, 100));
esMap.put("height", (float)ThreadLocalRandom.current().nextInt(140, 220));
esMap.put("weight", (float)ThreadLocalRandom.current().nextInt(70, 200));
esMap.put("cur_time", System.currentTimeMillis());
String strJson = gson.toJson(esMap);
builder.append(str).append("\n");
builder.append(strJson).append("\n");
}
entity = new StringEntity(builder.toString(), ContentType.APPLICATION_JSON);
entity.setContentEncoding("UTF-8");
Response response;
try {
Request request = new Request("PUT", "/_bulk");
request.addParameter("pretty", "true");
request.setEntity(entity);
response = restClientTest.performRequest(request);
if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
LOG.info("Already input documents : " + oneCommit * i);
} else {
LOG.error("Bulk failed.");
}
LOG.info("Bulk response entity is : " + EntityUtils.toString(response.getEntity()));
} catch (Exception e) {
LOG.error("Bulk failed, exception occurred.", e);
}
}
}
查询索引信息
功能简介
位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/search”目录下的QueryData.java,作用是查询指定index、type、id下的文档信息。
/**
* Query all data of one index.
*/
private static void queryData(RestClient restClientTest, String index, String type, String id){
Response response;
try {
Request request = new Request("GET", "/" + index + "/" + type + "/" + id);
request.addParameter("pretty", "true");
response = restClientTest.performRequest(request);
if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
LOG.info("QueryData successful.");
} else {
LOG.error("QueryData failed.");
}
LOG.info("QueryData response entity is : " + EntityUtils.toString(response.getEntity()));
} catch (Exception e) {
LOG.error("QueryData failed, exception occurred.", e);
}
}
删除索引
功能简介
位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/delete”目录下的DeleteIndex.java,作用是删除指定索引。
/**
* Delete one index
*/
private static void deleteIndex(RestClient restClientTest, String index) {
Response response;
try {
Request request = new Request("DELETE", "/" + index + "?&pretty=true");
response = restClientTest.performRequest(request);
if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
LOG.info("DeleteIndex successful.");
} else {
LOG.error("DeleteIndex failed.");
}
LOG.info("DeleteIndex response entity is : " + EntityUtils.toString(response.getEntity()));
} catch (Exception e) {
LOG.error("DeleteIndex failed, exception occurred.", e);
}
}
刷新索引
刷新单个索引
位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/flush”目录下的Flush.java,作用是刷新指定的索引。
/**
* Flush one index data to storage and clearing the internal transaction log
*/
private static void flushOneIndex(RestClient restClientTest, String index) {
Map<String, String> params = Collections.singletonMap("pretty", "true");
Response flushRsp;
try {
Request request=new Request("POST", "/" + index + "/_flush");
request.addParameter("pretty", "true");
flushRsp = restClientTest.performRequest(request);
LOG.info(EntityUtils.toString(flushRsp.getEntity()));
if (HttpStatus.SC_OK == flushRsp.getStatusLine().getStatusCode()) {
LOG.info("Flush one index successful.");
} else {
LOG.error("Flush one index failed.");
}
LOG.info("Flush one index response entity is : " + EntityUtils.toString(flushRsp.getEntity()));
} catch (Exception e) {
LOG.error("Flush one index failed, exception occurred.", e);
}
}
刷新所有索引
位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/flush”目录下的Flush.java,作用是刷新所有索引。
/**
* Flush all indexes data to storage and clearing the internal transaction log
* The user who performs this operation must belong to the "supergroup" group.
*/
private static void flushAllIndexes(RestClient restClientTest) {
Map<String, String> params = Collections.singletonMap("pretty", "true");
Response flushRsp;
try {
flushRsp = restClientTest.performRequest("POST", "/_all/_flush",params);
if (HttpStatus.SC_OK == flushRsp.getStatusLine().getStatusCode()) {
LOG.info("Flush all indexes successful.");
} else {
LOG.error("Flush all indexes failed.");
}
LOG.info("Flush all indexes response entity is : " + EntityUtils.toString(flushRsp.getEntity()));
} catch (Exception e) {
LOG.error("Flush all indexes failed, exception occurred.", e);
}
}
多线程样例
功能简介
位于“elasticsearch-rest-client-example/src/com/huawei/fusioninsight/elasticsearch/example/lowlevel/multithread”目录下的MultithreadRequest.java,作用是创建线程类sendRequestThread,重写run方法,实现bulk请求,批量写入数据。
/**
* Thread that sends bulk request
*/
public static class sendRequestThread implements Runnable {
private RestClient restCientTh;
private HwRestClient hwRestClient;
@Override
public void run() {
LOG.info("Thread begin.");
try {
hwRestClient = new HwRestClient();
estCientTh = hwRestClient.getRestClient();
LOG.info("Thread name: " + Thread.currentThread().getName());
bulk(restCientTh, "huawei1", "type1");
} catch (Exception e) {
LOG.error("SendRequestThread has exception.", e);
} finally {
if (restCientTh != null) {
try {
restCientTh.close();
LOG.info("Close the client successful in thread : " + Thread.currentThread().getName());
} catch (IOException e) {
LOG.error("Close the cient failed.", e);
}
}
}
}
}
在main方法中,创建了线程池fixedThreadPool,通过指定线程池大小threadPoolSize和任务数jobNumber,实现多线程发送bulk请求,并发地批量写入数据。
public static void main(String[] args) throws Exception {
LOG.info("Start to do multithread request !");
ExecutorService fixedThreadPool;
HwRestClient hwRestClient = new HwRestClient();
RestClient restCientThNew;
restCientThNew = hwRestClient.getRestClient();
int threadPoolSize = 2;
int jobNumber = 5;
try {
fixedThreadPool = Executors.newFixedThreadPool(threadPoolSize);
for (int i = 0; i < jobNumber; i++) {
fixedThreadPool.execute(new sendRequestThread());
}
fixedThreadPool.shutdown();
} catch (Exception e) {
LOG.error("There are exceptions in sendRequestThread.", e);
} finally {
if (restCientThNew != null) {
try {
restCientThNew.close();
LOG.info("Close the client successful.");
} catch (Exception e1) {
LOG.error("Close the client failed.", e1);
}
}
}
}