Kibana aggregation with missing / null fields

Background

Kibana is a good tool for aggregations. It looks like the group by in mysql. However Kibana will ignore documents without a given field when computing the aggregation. How can we take those documents with the given field missing into accout?

Let us take an example.

{                                                                                  
  "k": "v",                                                                        
  "k1": "v1"                                                                       
}                                                                                  
{                                                                                  
  "k": "v",                                                                        
  "k1": "v2"                                                                       
}                                                                                  
{                                                                                  
  "k": "v",                                                                        
  "k1": "v2"                                                                       
}                                                                                  
{                                                                                  
  "k": "v",                                                                        
  "k1": "v2"                                                                       
}                                                                                  
{                                                                                  
  "k": "v",                                                                        
}  

We have 5 documents, and when we aggregate over the k1 field in Kibana visualization, then it will return results like the following:

{                                                                                  
  "v1": 1,                                                                         
  "v2": 3,                                                                         
} 

LOOK, it misses the last document which does not contain k1. How can we solve this?

Solution

This is a normal aggregation (will ignore documents with missing fields).

We can configure the aggregation buckets like this. We add the json in the JSON Input

{"missing":"NULL"}

This says if it comes across a document without this field, then this document will be classified into NULL bucket.

If we apply this to the above case, the final results should return like

{                                                                                  
  "v1": 1,                                                                         
  "v2": 3,                                                                         
  "NULL": 1                                                                        
}

Another case

Before using {"missing": "NULL"}, it looks like,

After using {"missing": "NULL"}, it looks like:

Hope this helps. If you got any question, you are welcome to contact me via:

• email: nisxiya@yeah.net
• wechat: nisxiya