elasticsearch批量提交数据两种方式Python和curl
(带用户名和密码)
1.使用Python批量提交数据
import requests
from apscheduler.schedulers.blocking import BlockingScheduler
from elasticsearch import Elasticsearch
from elasticsearch import helpers
def post_dga():
es = Elasticsearch()
actions = []
f = open("文件名",'r')
for line in f:
line = line.strip().split('\t')
action = {
"_index": "dga",
"_source": {
"dga_family": line[0],
"domian": line[1],
"start_valid_time": line[2],
"end_valid_time": line[3]
}
}
actions.append(action)
#凑够五百个发一次
if (len(actions) == 500):
helpers.bulk(es, actions)
#清理已经发送的数据,防止内存溢出
del actions[0:len(actions)]
#最后的数据不一定是500的整数倍,剩下的数据最后再发一次
if (len(actions) > 0):
helpers.bulk(es, actions)
if __name__ == '__main__':
#定时提交数据
scheduler = BlockingScheduler()
scheduler.add_job(post_dga,'cron',hour=23,minute=55)
scheduler.start()
post_dga()
数据展示
nymaim iirkrd.net 2020-08-05 00:00:00 2020-08-05 23:59:59
nymaim islzzlmjw.org 2020-08-05 00:00:00 2020-08-05 23:59:59
nymaim ugnupdsgfbk.com 2020-08-05 00:00:00 2020-08-05 23:59:59
nymaim eflpufxiqi.biz 2020-08-05 00:00:00 2020-08-05 23:59:59
nymaim fhcmfhdd.com 2020-08-05 00:00:00 2020-08-05 23:59:59
nymaim pjotjjnagb.com 2020-08-05 00:00:00 2020-08-05 23:59:59
这种方式可以连接没有密码的elasticsearch,那么有密码应该什么办呢?
只需要将:
es = Elasticsearch()
换成
es = Elasticsearch(hosts="https://用户名:密码@host:9200",timeout=30, use_ssl=False,
verify_certs=False, ssl_show_warn=False)
使用curl批量提交数据
1、将自己的数据处理成标准的json格式
处理前的数据:
nymaim iirkrd.net 2020-08-05 00:00:00 2020-08-05 23:59:59
nymaim islzzlmjw.org 2020-08-05 00:00:00 2020-08-05 23:59:59
nymaim ugnupdsgfbk.com 2020-08-05 00:00:00 2020-08-05 23:59:59
nymaim eflpufxiqi.biz 2020-08-05 00:00:00 2020-08-05 23:59:59
nymaim fhcmfhdd.com 2020-08-05 00:00:00 2020-08-05 23:59:59
nymaim pjotjjnagb.com 2020-08-05 00:00:00 2020-08-05 23:59:59
目标数据
{"index":{"_index":"dga"}}
{"dga_family": "nymaim", "domian": "wyloyr.info", "start_valid_time": "2020-08-06 00:00:00", "end_valid_time": "2020-08-06 23:59:59"}
{"index":{"_index":"ti-dga"}}
{"dga_family": "nymaim", "domian": "acjsrn.net", "start_valid_time": "2020-08-06 00:00:00", "end_valid_time": "2020-08-06 23:59:59"}
{"index":{"_index":"ti-dga"}}
{"dga_family": "nymaim", "domian": "ftbsnmphv.com", "start_valid_time": "2020-08-06 00:00:00", "end_valid_time": "2020-08-06 23:59:59"}
第一行是index,第二行是键值对
f = open("dga.txt",'r')
list1 = '{"index":{"_index":"dga"'
i =1
for line in f:
line = line.strip().split('\t')
action = {
"dga_family": line[0],
"domian": line[1],
"start_valid_time": line[2],
"end_valid_time": line[3]
}
res_1 = list1 + '}}' + '\n'
# print(res_1)
a = open(r"out.json", "a", encoding='UTF-8')
a.write(res_1)
# 第偶数行,写入键值对
res_2 = str(action)+'\n'
#写进去的键值对有单引号,这里全部换成双引号,注意转义字符
res_2 = res_2.replace("\'","\"")
a = open(r"out.json", "a", encoding='UTF-8')
a.write(res_2)
i +=1
print(i)
a.close()
print("处理完毕")
这样第一步就完成了,已经将数据处理完毕
2、使用curl,利用bulk提交数据
curl -XPOST "http://localhost:9200/_bulk?pretty" -H "Content-Type: application/json;charset=UTF-8" --data-binary @out.json
同样的问题,这里的curl可以连接没有密码的es
有密码的情况应该如何处理呢:
curl -XPOST -k "https://用户名:密码@localhost:9200/_bulk?pretty" -H "Content-Type: application/json;charset=UTF-8" --data-binary @out.json
注意: