elasticsearch 去重

elasticsearch 去重

1、根据多个属性组合查询是否有重复数据

GET /dbl-2022-08-*/_search { "query": { "match_all": {} }, "size": 1, "aggs": { "NAME": { "terms": { "script": "doc['itemid'].value + '_' + doc['clock'].value + '_' +doc['ns']", "min_doc_count": 2, "size": 2 }, "aggs": { "NAME": { "top_hits": { "size": 10 } } } } } }

2、通过pipeline去重

PUT /_ingest/pipeline/ccc_pipeline { "description": "ccc_pipeline", "processors": [ { "script": { "lang": "painless", "source": "ctx._id = ctx.itemid+'_'+ctx.clock+'_'+ctx.ns;" } } ] }