Elasticsearch学习之（一）线上迁移数据方案_elasticsearch 在线迁移

最新推荐文章于 2024-08-08 00:45:00 发布

2401_84160310

最新推荐文章于 2024-08-08 00:45:00 发布

阅读量742

点赞数 18

分类专栏： 2024年程序员学习文章标签： elasticsearch 学习 jenkins

本文链接：https://blog.csdn.net/2401_84160310/article/details/137486181

版权

2024年程序员学习专栏收录该内容

115 篇文章 1 订阅

订阅专栏

throw new IllegalArgumentException(“索引名称前缀为空! prefix:”+indexPrefix);
}

List indexNames = new ArrayList<>();
Calendar startCalendar = Calendar.getInstance();
startCalendar.setTimeInMillis(start);

Calendar endCalendar = Calendar.getInstance();
endCalendar.setTimeInMillis(end);

SimpleDateFormat dateFormat = new SimpleDateFormat(“yyyyMM”);

// 计算开始和结束的年份
int startYear = startCalendar.get(Calendar.YEAR);
int endYear = endCalendar.get(Calendar.YEAR);

// 遍历每个年份的月份，生成索引名称
for (int year = startYear; year <= endYear; year++) {
Calendar tempCalendar = Calendar.getInstance();
tempCalendar.set(Calendar.YEAR, year);
tempCalendar.set(Calendar.MONTH, Calendar.JANUARY); // 一月份

// 对于开始年份，从开始的月份开始遍历
int startMonth = year == startYear ? startCalendar.get(Calendar.MONTH) : Calendar.JANUARY;

// 对于结束年份，到结束的月份结束遍历
int endMonth = year == endYear ? endCalendar.get(Calendar.MONTH) : Calendar.DECEMBER;

for (int month = startMonth; month <= endMonth; month++) {
tempCalendar.set(Calendar.MONTH, month);
indexNames.add( indexPrefix + dateFormat.format(tempCalendar.getTime()));
}
}

return indexNames;
}

2、数据访问

1、每个月的最后几天生成下个月的索引

注意点：
1、因为月份最后几天不确定，所以从28-31 都计算一下
2、创建索引的配置indexConfiguration自己写个json文件然后放到容器中读取即可

@Scheduled(cron = “0 0 10 28-31 * ?”)
public void createIndex() throws IOException {
//判断是否是最后一天
if(!DateUtil.isLastDayOfMonth()){
log.warn(“索引初始化|判断不是本月最后一天|不进行处理”);
return;
}

// 每月最后一天生成下个月的索引
Calendar instance = Calendar.getInstance();
instance.add(Calendar.MONTH,1);
instance.set(Calendar.DAY_OF_MONTH,1);

String indexName = ElasticsearchUtil.getIndexNameByTimeStamp(newIndexPrefix, instance.getTimeInMillis());
GetIndexRequest getIndexRequest = new GetIndexRequest(indexName);
boolean exists = restHighLevelClient.indices().exists(getIndexRequest, RequestOptions.DEFAULT);
if(exists){
log.warn(“索引初始化|索引已存在|index:{}”,indexName);
return;
}
if(StringUtils.isEmpty(indexConfiguration)){
log.error(“索引初始化｜获取索引初始化配置为空|setting:{}”,indexConfiguration);
return;
}
CreateIndexRequest request = new CreateIndexRequest(indexName);
// 初始化索引
request.source(indexConfiguration,XContentType.JSON);
CreateIndexResponse response = restHighLevelClient.indices().create(request, RequestOptions.DEFAULT);
boolean acknowledged = response.isAcknowledged();
log.info(“索引初始化|初始化完成｜所有节点是否都已确认:{}”,acknowledged);

}

index初始化的配置
从resource下读取配置文件

@Bean(“indexConfiguration”)
public String initIndexConfiguration() throws IOException {
String indexConfiguration= initEsIndexSetting(“indexInitialization.json”);
log.info(“索引初始化|初始化索引配置文件:{}”,indexConfiguration);
return indexConfiguration;
}

private String initEsIndexSetting(String resource) throws IOException {
ClassPathResource classPathResource = new ClassPathResource(resource);
try (InputStream in = classPathResource.getInputStream()) {
return StreamUtils.copyToString(in, StandardCharsets.UTF_8);
}catch (IOException e){
throw new IOException(e);
}
}

2、数据查询

很明显根据时间戳确定索引名称然后查询就行了

public List queryLisNewt(int from, int pageSize, BoolQueryBuilder queryBuilder,FeedbackReport entity) {

Listresults=new ArrayList<>();

if( StringUtils.isEmpty(entity.getStartDate()) || StringUtils.isEmpty(entity.getEndDate()) ){
log.error(“queryLisNewt|查询错误|开始或结束时间为空|startDate:{}|endDate:{}”,entity.getStartDate(),entity.getEndDate());
return results;
}
long start=DateUtil.parseDateString(entity.getStartDate()).getTime();
long end=DateUtil.parseDateString(entity.getEndDate()).getTime();
if(start <=0l || end <=0l){
log.error(“queryLisNewt|查询错误|解析日期错误:param:{}”,entity);
return results;
}
List indexNames = ElasticsearchUtil.getAllIndexNameByTimeStampRange(newIndexPrefix, start, end);
if(CollectionUtils.isEmpty(indexNames)){
log.error(“queryLisNewt|查询错误|获取索引名称为空:{}”,entity);
return results;
}
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder
.query(queryBuilder)
.from(from)
.size(pageSize)
.sort(SortBuilders.fieldSort(“createTime”)
.order(SortOrder.DESC));

SearchRequest searchRequest = new SearchRequest(indexNames.toArray(new String[]{}),searchSourceBuilder);
RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
builder.setHttpAsyncResponseConsumerFactory(new HttpAsyncResponseConsumerFactory
.HeapBufferedResponseConsumerFactory(200*1024*1024));
try {
SearchResponse response = restHighLevelClient.search(searchRequest, builder.build());
SearchHits hits = response.getHits();
for (SearchHit hit : hits) {
Map map = hit.getSourceAsMap();

自我介绍一下，小编13年上海交大毕业，曾经在小公司待过，也去过华为、OPPO等大厂，18年进入阿里一直到现在。

深知大多数大数据工程师，想要提升技能，往往是自己摸索成长或者是报班学习，但对于培训机构动则几千的学费，着实压力不小。自己不成体系的自学效果低效又漫长，而且极易碰到天花板技术停滞不前！

因此收集整理了一份《2024年大数据全套学习资料》，初衷也很简单，就是希望能够帮助到想自学提升又不知道该从何学起的朋友。