es-数据导出 elastice_dump 报错 parsing_exception The field [fields] is no longer supported

需求背景

• 因为原有es负载过高，接口返回速度慢，定位原因如下
  • 可去重数据没有去重
  • 可清理的脏数据在入库时没有进行合理的过滤
  • 没有做历史数据索引清理
  • 冷热数据没有分离
• 故需要对原有数据进行去重清洗，同时导出新版本es
  • 用python直接导出每次查询有10000上限，尽管可以调整配置文件或者用用游标 scroll解决，但是还是想不要造轮子了
  • 故选择用elasticdump

导出语句如下

elasticdump \
  --input=http://username:password@localhost:9200 \
  --input-index=user_index \
  --output=query.json 

报错如下

 Error: {"error":{"root_cause":[{"type":"parsing_exception","reason":"The field [fields] is no longer supported, please use [stored_fields] to retrieve stored fields or _source filtering if the field is not stored","line":1,"col":36}],"type":"parsing_exception","reason":"The field [fields] is no longer supported, please use [stored_fields] to retrieve stored fields or _source filtering if the field is not stored","line":1,"col":36},"status":400}

报错原因

• elasticsearch版本是5.3，然后elasticdump的版本elasticdump 2.4.2

./node_modules/elasticdump/elasticdump.js

# 用这个语句去查es版本=5.3的就会报错提示你让你用stored_fields
self.options.searchBody = {"query": { "match_all": {} }, "fields": ["*"], "_source": true };
to

self.options.searchBody = {"query": { "match_all": {} }, "stored_fields": ["*"], "_source": true };

解决办法

• 1，修改导出语句(建议)

elasticdump \
  --input=http://username:password@localhost:9200 \
  --input-index=user_index \
  --output=query.json 
  --searchBody='{"query": { "match_all": {} }, "stored_fields": ["*"], "_source": true }'

• 2，修改./node_modules/elasticdump/elasticdump.js(不建议)

./node_modules/elasticdump/elasticdump.js
# 用这个语句去查es版本=5.3的就会报错提示你让你用stored_fields
self.options.searchBody = {"query": { "match_all": {} }, "fields": ["*"], "_source": true };
to
# 换成这个就好了
self.options.searchBody = {"query": { "match_all": {} }, "stored_fields": ["*"], "_source": true };

问题解决参考链接

github