Elastic stack(基于7.0.0官方文档)-CSDN博客

Lucene

TextField建立索引但是可以不存储
TermQuery不会将查询分词了，把查询条件当成固定的词条
文档更新是把旧的给删了再整一个新的出来(文档ID会变)。更新的代价太大了
布尔语句，BooleanClause

+ 代表 must - 代表 mustnot 代表 should

Beats

开始看filebeat，被官网带到了Getting started with the Elastic Stack.

Getting started with the Elastic Stack

这个小教程是用metricbeat来采集服务器指标数据，然后用kibana做展示。

安装ES `port:9200`

curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.0.1-linux-x86_64.tar.gz
tar -xzvf elasticsearch-7.0.1-linux-x86_64.tar.gz
cd elasticsearch-7.0.1
./bin/elasticsearch

curl http://127.0.0.1:9200
复制代码

安装kibana `port:5601`

kibana是专门用于ES的，对数据进行搜索以及可视化。小教程里建议kibana和ES装同一台机器上。

配置文件里需要配置ES集群的地址

curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-7.0.1-linux-x86_64.tar.gz
tar xzvf kibana-7.0.1-linux-x86_64.tar.gz
cd kibana-7.0.1-linux-x86_64/
./bin/kibana
复制代码

安装Metricbeat 各种beat

beat是做采集用的，装在服务器上的agent。一般输出到ES和logstash，自己本身不能做解析

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.0.1-linux-x86_64.tar.gz
tar xzvf metricbeat-7.0.1-linux-x86_64.tar.gz
复制代码

用系统模块来采集系统日志，例如CPU、内存
开启系统模块./metricbeat modules enable system
kibana配置文件里默认是没有开启

#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:
复制代码

上述配置文件说了，加命令setup 加载仪表板./metricbeat setup -e，-e是将输出打到stderr不是syslog，就是把日志打到控制台能看得见。
./metricbeat -e

Logstash

如果beat采的数需要额外处理那么需要进logstash(其实就是解析)

curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-7.0.1.tar.gz
tar -xzvf logstash-7.0.1.tar.gz
复制代码

创建一个logstash pipeline 一根管子一边接受beat 一边怼到ES
这根管子是一个配置文件例如demo-metrics-pipeline.conf,监听5044端口

input {
  beats {
    port => 5044
  }
}

# The filter part of this file is commented out to indicate that it
# is optional.
# filter {
#
# }

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}
复制代码

启动 ./bin/logstash -f demo-metrics-pipeline.conf

还得配置beat 让其吐出到logstash

filter

metricbeat采集了cmdline完整参数太长了，解析一下它。用grok

filter {
  if [system][process] {
    if [system][process][cmdline] {
      grok {
        match => { 
          "[system][process][cmdline]" => "^%{PATH:[system][process][cmdline_path]}"
        }
        remove_field => "[system][process][cmdline]" 
      }
    }
  }
}
复制代码

解析这块。grok后面会再写。

正篇FileBeat

FileBeat是一个agent，输出到es、logstash
配置输入，可以有多个输入/var/log/*.log
工作原理
有一个或者多个输入，把收集到的所有日志发给libbeat 平台，libbeat再统一输出。

安装filebeat

直接在官网下载tar.gz安装(后续上容器)

配置filebeat

filebeat有很多modules 常见的日志模块Quick start: modules for common log formats.
filebeat.reference.yml参考所有配置项
最基本的定义一个日志的来源

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
复制代码

/var/log/*/*.log从/var/log子文件夹中获取所有.log，现在还不支持获取所有层中的所有文件
通常！输出要么ES 要么Logstash进行额外的处理。也能吐到kafka
还可以使用filebeat提供的Kibana示例仪表盘
吐ES

output.elasticsearch:
  hosts: ["myEShost:9200"]
复制代码

吐Logstash

setup.kibana:
  host: "mykibanahost:5601" 
复制代码

如果ES和kibana设置了安全性 (这里密码应该配置是加密的，不要硬编码)，加密密码
如果没有为Kibana指定凭据，那用ES的凭据
现在可以再kibana中管理beat Beats central management.这个功能还不是正式版。

output.elasticsearch:
  hosts: ["myEShost:9200"]
  username: "filebeat_internal"
  password: "YOUR_PASSWORD" 
setup.kibana:
  host: "mykibanahost:5601"
  username: "my_kibana_user"  
  password: "YOUR_PASSWORD"
复制代码

详细配置相关信息

filebeat的预定义模块

./filebeat modules list

File is inactive: /var/log/boot.log. Closing because close_inactive of 5m0s reached. 说明文件没有新东西

配置filebeat到logstash

配置Logstash接受filebeat传来的消息，对采集的信息做进一步处理。
filebeat读取log会有游标记录，在data/registry/filebeat/data.json

#这是filebeat.yml 输出到5044端口 默认logstash监听的端口
#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["127.0.0.1:5044"]
复制代码

For this configuration, you must load the index template into Elasticsearch manually because the options for auto loading the template are only available for the Elasticsearch output.??索引模板是什么?

加载索引模板

在Elasticsearch中，索引模板用于定义确定如何分析字段的设置和映射。
Filebeat包安装了推荐的Filebeat索引模板文件。

#==================== Elasticsearch template setting ==========================
#默认一个分片  这就是Filebeat在ES中索引只有一个分片原因
setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false
复制代码

目录布局

用rpm和tar安装目录以及日志的位置不一样具体查官方文档。

beat采集日志有的需要用root采

Filebeat command reference

filebeat是如何工作的

理解这些概念会在配置的时候做出明智的选择

filebeat由两个主要的组件，inputs harvesters输入和收割机
harvesters读单个文件，逐行读取发送到输出。每个文件都有一个harvester，收割机负责文件的打开和关闭，收割机工作时文件时打开的状态。最后没有读不到了会出现close_inactive
输入模块用来管理收割机，并且找到要读取的资源。如果输入类型是log,那么会给这个log文件一个收割机。
默认情况下，Filebeat会保持文件处于打开状态，直到close_inactive达到。

logstash

由于有人帮我们采集好了日志，在kafka中，所以先用logstash对接

logstash能力很强，能够收集各种数据，交给下游来处理(es,kibana)
安装logstash需要配置 JAVA_HOME
输入处理输出

# 能够从标准输入中拿数怼到标准输出  -e 是可以直接跟配置 快速测试
cd logstash-7.0.1
bin/logstash -e 'input { stdin { } } output { stdout {} }'

# 结果如下
什么鬼
{
       "message" => "什么鬼",
      "@version" => "1",
    "@timestamp" => 2019-05-07T02:00:39.581Z,
          "host" => "node1"
}
复制代码

用filebeat读取一个示例数据，output设置为logstash

# logstash管道配置如下 先打印到标准输出上查看
input {
  beats {
    port => 5044
  }
}

# rubydebug 这是用ruby的一个打印库 让输出更好看
output {
  stdout { codec => rubydebug }
}
复制代码

bin/logstash -f first-pipeline.conf --config.test_and_exit这个命令可以看配置文件是不是好使
跑一次bin/logstash -f first-pipeline.conf --config.reload.automatic config.reload.automatic这个选项能自动加载新的配置文件，不用重启logstash
通过IP解析地理位置 geoip插件

* logstash支持多输入多输出，可以直接对接twitter 就没法实验了。可以直接输出到文件。


geoip {
    source => "clientip"
}

{
            "ecs" => {
        "version" => "1.0.0"
    },
          "input" => {
        "type" => "log"
    },
          "agent" => {
        "ephemeral_id" => "860d92a1-9fdb-4b41-8898-75021e3edaaf",
             "version" => "7.0.0",
            "hostname" => "node1",
                  "id" => "c389aa98-534d-4f37-ba62-189148baa6a3",
                "type" => "filebeat"
    },
        "request" => "/robots.txt",
           "verb" => "GET",
           "host" => {
             "hostname" => "node1",
        "containerized" => true,
         "architecture" => "x86_64",
                   "os" => {
              "kernel" => "3.10.0-693.el7.x86_64",
            "codename" => "Maipo",
              "family" => "redhat",
            "platform" => "rhel",
             "version" => "7.4 (Maipo)",
                "name" => "Red Hat Enterprise Linux Server"
        },
                   "id" => "b441ff6952f647e7a366c69db8ea6664",
                 "name" => "node1"
    },
          "ident" => "-",
      "timestamp" => "04/Jan/2015:05:27:05 +0000",
           "auth" => "-",
           "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
       "referrer" => "\"-\"",
       "@version" => "1",
       "response" => "200",
    "httpversion" => "1.1",
        "message" => "218.30.103.62 - - [04/Jan/2015:05:27:05 +0000] \"GET /robots.txt HTTP/1.1\" 200 - \"-\" \"Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)\"",
       "clientip" => "218.30.103.62",
          "geoip" => {
           "region_code" => "BJ",
              "latitude" => 39.9288,
                    "ip" => "218.30.103.62",
              "location" => {
            "lat" => 39.9288,
            "lon" => 116.3889
        },
           "region_name" => "Beijing",
             "longitude" => 116.3889,
             "city_name" => "Beijing",
              "timezone" => "Asia/Shanghai",
         "country_code3" => "CN",
         "country_code2" => "CN",
          "country_name" => "China",
        "continent_code" => "AS"
    },
     "@timestamp" => 2019-05-07T03:43:18.368Z,
            "log" => {
          "file" => {
            "path" => "/itoa/elastic-stack/test-cas/logstash-demo/logstash-tutorial.log"
        },
        "offset" => 19301
    }
}
复制代码

logstash工作原理

输入来源

file 类似于 tail -f
syslog: listens on the well-known port 514 for syslog messages and parses according to the RFC3164 format
redis: reads from a redis server, using both redis channels and redis lists. Redis is often used as a "broker" in a centralized Logstash installation, which queues Logstash events from remote Logstash "shippers".类似于消息队列
beats: processes events sent by Beats.