场景:需要从A服务器es导出json数据,再导入到B服务器的es。
es版本: 7.15
查出数据
windows可以用postman, linux可以用curl 命令。
{
"took": 12,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 41,
"relation": "eq"
},
"max_score": 1.0,
"hits": [
{
"_index": "my_index",
"_type": "_doc",
"_id": "48eabd2bc0338dfd46dcdf361e4ec94d",
"_score": 1.0,
"_source": {
"evtProId": "1acb5772df8f4647b9d5bc622ac8ebc7",
"progress": 0
}
},
{
"_index": "my_index",
"_type": "_doc",
"_id": "899277630026aa088c5a0d3374293d69",
"_score": 1.0,
"_source": {
"evtProId": "1278b4fcb3b4488c94b1e4ec1421d252",
"progress": 0
}
}
]
}
}
批量导入数据
从网上查找到资料可知批量导入可使用以下方式,需对导出来的json进行一定的处理
HTTP Method: POST
URI: /your_index/_bulk
Request body (should end with a new line):
{ "index" : {"_id" : "48eabd2bc0338dfd46dcdf361e4ec94d" } }
{"evtProId":"1acb5772df8f4647b9d5bc622ac8ebc7","progress":0}
{ "index" : {"_id" : "899277630026aa088c5a0d3374293d69" } }
{"evtProId":"1278b4fcb3b4488c94b1e4ec1421d252","progress":0}
以上可以直接调用postman发送请求
由于其他原因,我只能使用curl的方式调用,首先将请求体另存为data.json
上传到linux 服务器
然后执行curl:
curl --user username:password -XPOST 'http://localhost:9200/your_index/_bulk' -H 'Content-Type: application/json' --data-binary @data.json
附:导出的数据json转换成批量导入格式的文本
简单写一个的程序,请自行完善
package luo;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import lombok.extern.slf4j.Slf4j;
import org.springframework.core.io.ClassPathResource;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.Iterator;
/**
*
*
* @author
* @since 2023/3/23
*/
@Slf4j
public class EsJsonSolve {
public static void main(String[] args) {
ClassPathResource res = new ClassPathResource("es.json");
InputStream jsonFileIs = null;
try {
jsonFileIs = res.getInputStream();
} catch (IOException e) {
e.printStackTrace();
}
StringBuilder sb = new StringBuilder();
try {
JSONObject jsonObject = JSON.parseObject(jsonFileIs, JSONObject.class);
JSONObject firstHits = jsonObject.getJSONObject("hits");
JSONArray secondHitsArr = firstHits.getJSONArray("hits");
Iterator<Object> iter = secondHitsArr.iterator();
while (iter.hasNext()){
JSONObject hitJsonObj = (JSONObject) iter.next();
String source = hitJsonObj.getString("_source");
// log.info(source);
// String insertHead = "{ \"index\" : { \"_index\" : \"ypa_event_form\", \"_id\" : \"%s\" } }";
String insertHead = "{ \"index\" : {\"_id\" : \"%s\" } }";
insertHead = String.format(insertHead, hitJsonObj.getString("_id"));
sb.append(insertHead).append("\n");
sb.append(source).append("\n");
}
sb.append("\n");
System.out.println(sb.toString());
FileOutputStream fileOutputStream = new FileOutputStream("E:\\insetData.json");
fileOutputStream.write(sb.toString().getBytes(StandardCharsets.UTF_8));
fileOutputStream.close();
} catch (IOException e) {
log.error("", e);
}
}
}
导出来的数据json放入es.json, 然后控制台输出可批量导入格式的文本,文本较长可自行输出到一个新文件中
参考链接:
- https://stackoverflow.com/questions/33340153/elasticsearch-bulk-index-json-data