python elasticsearch bulk_python-如何使用elasticsearch.helpers.streaming_bulk

有人可以建议如何使用函数elasticsearch.helpers.streaming_bulk代替elasticsearch.helpers.bulk将数据索引到elasticsearch中.

如果我只是简单地更改streaming_bulk而不是批量,则不会索引任何内容,因此我猜它需要以其他形式使用.

下面的代码从CSV文件中以500个元素的块创建索引,类型和索引数据,并进入elasticsearch.它工作正常,但是我在徘徊是否可以提高性能.这就是为什么我想尝试streaming_bulk函数的原因.

目前,我需要10分钟才能为200MB的CSV文档索引100万行.我使用两台机器,Centos 6.6,8个CPU,x86_64,CPU MHz：2499.902,内存：15.574G.

不确定它可以更快地进行.

es = elasticsearch.Elasticsearch([{'host': 'uxmachine-test', 'port': 9200}])

index_name = 'new_index'

type_name = 'new_type'

mapping = json.loads(open(config["index_mapping"]).read()) #read mapping from json file

es.indices.create(index_name)

es.indices.put_mapping(index=index_name, doc_type=type_name, body=mapping)

with open(file_to_index, 'rb') as csvfile:

reader = csv.reader(csvfile) #read documents for indexing from CSV file, more than million rows

content = {"_index": index_name, "_type": type_name}

batch_chunks = []

iterator = 0

for row in reader:

var = transform_row_for_indexing(row,fields, index_name, type_name,id_name,id_increment)

id_increment = id_increment + 1

#var = transform_row_for_indexing(row,fields, index_name, type_name)

batch_chunks.append(var)

if iterator % 500 == 0:

helpers.bulk(es,batch_chunks)

del batch_chunks[:]

print "ispucalo batch"

iterator = iterator + 1

# indexing of last batch_chunk

if len(batch_chunks) != 0:

helpers.bulk(es,batch_chunks)