最近在对ES中大量数据做分页展示并导出
一、offset大于一万的时候报错了。。。
原来有一个参数:index.max_result_window控制
估量一下数据量大小改大即可,此值是integer类型,不能无限大,而且但过大会影响ES查询效率:
步骤:
首先关闭索引:
红线处改成索引名提交即可:
{ “max_result_window” : “100000000”}
PUT改成GET可查看是否成功;
二、使用Scroll-Scans接口导出数据
直接上代码:
public static void main(String[] args) {
// 获取ES连接自己搞
TransportClient client = ESUtils.getESClient();
//输出为csv,为什么是csv?自行google
File file = new File("./write.csv");
Writer writer = null;
try {
//设置编码格式,会有乱码问题
writer = new FileWriterWithEncoding(file,"GBK");
} catch (IOException e) {
e.printStackTrace();
}
CSVWriter csvWriter = new CSVWriter(writer);
SearchResponse scrollResp = searchByESAPI();
//接收scrollResp响应结果
do {
for (SearchHit hit : scrollResp.getHits().getHits()) {
//处理时间,自行处理...
long starttime = hit.getField("starttime").getValue();
long endtime = hit.getField("endtime").getValue();
try {
String[] strs = new String[8];
strs[0] = DateUtil.longToDate_minus8(starttime);
strs[1] = DateUtil.longToDate_minus8(endtime);
strs[2] = hit.getField("summary").getValue().toString();
csvWriter.writeNext(strs);
} catch (ParseException e) {
e.printStackTrace();
}
}
System.out.println("-------------------");
scrollResp = client.
prepareSearchScroll(scrollResp.getScrollId())
.setScroll(new TimeValue(60000))
.execute()
.actionGet();
} while (scrollResp.getHits().getHits().length != 0);//遍历
client.close();
try {
csvWriter.close();
} catch (IOException e) {
e.printStackTrace();
}
}
public static SearchResponse searchByESAPI() {
return client.prepareSearch("eventbase_xxx", "eventbase_xxxx")//indx
.setTypes("eventbase_")//type
//SCROLL在这里
.setScroll(new TimeValue(60000))
// .addSort(FieldSortBuilder.DOC_FIELD_NAME, SortOrder.ASC) //属性排序
//查询条件query_string
.setQuery(QueryBuilders.matchQuery("ceptype", "AttackCountry"))
//多条件过滤filters
.setPostFilter(
QueryBuilders.rangeQuery("starttime").from("2017-08-24 00:00:00").to("2017-08-24 23:20:00")
)
//限制输出字段
.addDocValueField("starttime")
.addDocValueField("endtime")
.addDocValueField("summary")
//每次数量
.setSize(10)
.get();
}
大功告成~
数据量太大会运行很长时间