Elastic Stack之 分析Elasticsearch查询语句实战。

1、分析Elasticsearch查询语句的功能。

　　1)、首先需要收集Elasticsearch集群的查询语句。
　　2)、然后分析查询语句的常用语句、响应时长等等指标。
2、分析Elasticsearch查询语句的功能，使用方案。

　　1)、应用Packetbeat + Logstash完成数据收集工作。　　2)、使用Kibana + Elasticsearch完成数据分析工作。

3、分析Elasticsearch查询语句的功能，流程分析。

　　1)、Production Cluster(Elasticsearch集群) -> Packetbeat -> Logstash(Monitoring Logstash) -> Elasticsearch(Monitoring cluster) -> kibana(Monitoring kibana)。
　　2)、Production Cluster，可以使用Elasticsearch，地址http://192.168.110.133:9200。kibana，地址http:192.168.110.133:5601。
　　3)、Elasticsearch(Monitoring cluster，用于存储Packetbeat抓取的查询语句。Elasticsearch地址http://192.168.110.133:8200，可以通过bin/elasticsearch -Ecluster.name=sniff_search -Ehttp.port=8200 -Epath.data=sniff快速启动一个节点。kibana，地址http:192.168.110.133:8601。快速启动方式，bin/kibana -e http://192.168.110.133:8200 -p 8601。
　　注意：Production与Monitoring不能是一个集群，否则会进入抓包死循环。

4、关于Logstash的配置方案，文件名称sniff_search.conf，如下所示：

 1 input {
 2     beats { # 在5044端口接收beats的输入
 3         port => 5044 
 4     }
 5 }
 6 filter {
 7     if "search" in [request]{    # 查询语句的过滤，如果请求中包含search才进行处理
 8         grok {    # 从request中提取query_body，即实际的查询语句。
 9             match => { "request" => ".*\n\{(?<query_body>.*)"} 
10         }
11         grok {    # 从path中提取index，即对某个索引的操作。
12             match => { "path" => "\/(?<index>.*)\/_search"}     
13         }
14      if [index] {
15       } else {
16             mutate {
17               add_field  => { "index" => "All" }
18         }
19       }
20 
21       mutate {
22               update  => { "query_body" => "{%{query_body}"}}
23       }
24 
25   #    mutate {
26   #        remove_field => [ "[http][response][body]" ]
27   #    }
28 }
29 
30 output {
31   #stdout{codec=>rubydebug}
32 
33   if "search" in [request]{    # 只对查询做存储，如果存在查询就保存到监控的elasticsearch中。
34         elasticsearch {    
35         hosts => "192.168.110.133:8200"
36         }
37    }
38 }

关于Packetbeat的配置方案，文件名称sniff_search.yml，如下所示：

 1 #################### Packetbeat Configuration Example #########################
 2 
 3 # This file is an example configuration file highlighting only the most common
 4 # options. The packetbeat.full.yml file from the same directory contains all the
 5 # supported options with more comments. You can use it as a reference.
 6 #
 7 # You can find the full configuration reference here:
 8 # https://www.elastic.co/guide/en/beats/packetbeat/index.html
 9 
10 #============================== Network device ================================
11 
12 # Select the network interface to sniff the data. On Linux, you can use the
13 # "any" keyword to sniff on all connected interfaces.
14 packetbeat.interfaces.device: any
15 
16 packetbeat.protocols.http:
17   # Configure the ports where to listen for HTTP traffic. You can disable
18   # the HTTP protocol by commenting out the list of ports.
19   ports: [9200]
20   send_request: true   
21   include_body_for: ["application/json", "x-www-form-urlencoded"]
22 
23 
24 #================================ Outputs =====================================
25 
26 # Configure what outputs to use when sending the data collected by the beat.
27 # Multiple outputs may be used.
28 
29 #-------------------------- Elasticsearch output ------------------------------
30 #output.elasticsearch:
31   # Array of hosts to connect to.
32   #  hosts: ["localhost:9200"]
33 
34   # Optional protocol and basic auth credentials.
35   #protocol: "https"
36   #username: "elastic"
37   #password: "changeme"
38 
39 #output.console:
40 #    pretty: true
41 
42 output.logstash:  # 输出到 logstash中。 
43     hosts: ["192.168.110.133:5044"]
44 
45 
46 #================================ Logging =====================================
47 
48 # Sets log level. The default log level is info.
49 # Available log levels are: critical, error, warning, info, debug
50 #logging.level: debug
51 
52 # At debug level, you can selectively enable logging only for some components.
53 # To enable all selectors use ["*"]. Examples of other select