使用logstash导入分片数量为100的index发生错误
[logstash.outputs.elasticsearch] retrying failed action with response code: 429
({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of processing of [12373521]
[indices:data/write/bulk[s][p]]: request:
BulkShardRequest [[suricatadns140][97]] containing [2] requests,
target allocation id: -erjXlg0RjeETfZKXokiiQ,
primary term: 1 on EsThreadPoolExecutor[name = node-1/write, queue capacity = 200,
org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@139538e0[Running,
pool size = 8, active threads = 8, queued tasks = 200, completed tasks = 179152]]"})
解决:
因为导入的数据和ES写入的数据速度不匹配,ES为了保护自己,产生了拒绝
在ES的配置yml文件中添加:
thread_pool.write.queue_size: 10000