ELK安装步骤记录

几个软件的版本要保持一致

1.安装JAVA 1.8

yum install java

2.安装elasticsearch

# 下载并安装公共签名密钥
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

# 最新版下载地址 https://www.elastic.co/cn/downloads
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.7.0-x86_64.rpm
# 安装
rpm -ivh elasticsearch-7.7.0-x86_64.rpm


# 配置
# 配置文件都在 /etc/elasticsearch/ 目录下
vi /etc/elasticsearch/elasticsearch.yml

# 集群名称
cluster.name: my-application
# 节点名称
node.name: node-1
# 数据文件与日志文件存放目录
path.data: /home/esData/es
path.logs: /home/esData/log
# 网络设置
network.host: 0.0.0.0
http.port: 9200
# 集群设置
cluster.initial_master_nodes: ["node-1"]


# 修改配置中目录的用户与用户组，不然无法启动
chown -R elasticsearch:elasticsearch /home/esData/log/
chown -R elasticsearch:elasticsearch /home/esData/es/


# 启动
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
sudo systemctl start elasticsearch.service
# 查看状态
systemctl status elasticsearch.service
# 浏览器访问 ip + 9200/?pretty 查看状态

3.安装 kibana

# 下载
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.7.0-x86_64.rpm
# 安装
rpm -ivh kibana-7.7.0-x86_64.rpm
#配置
vi /etc/kibana/kibana.yml
# 取消2、7、21行的注释，行号不一定准确，但一定是下面这几项
server.host: "0.0.0.0"
# 不要用 127.0.0.1，可能会提示 Kibana server is not ready yet
elasticsearch.hosts: ["http://192.168.56.101:9200"]
i18n.locale: "zh-CN"

# 刷新服务配置
systemctl daemon-reload
# 开机自启
systemctl enable kibana.service
# 启动
systemctl start kibana.service
# 查看状态
systemctl status kibana.service

# 浏览器访问 http://192.168.56.101:5601

4.安装logstash

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.7.0.rpm
rpm -ivh logstash-7.7.0.rpm
systemctl enable logstash
systemctl start logstash

5.示例

创建一个logstash从系统安全日志/var/log/secure中收集ssh登录失败日志的配置

/etc/logstash/conf.d/
vi sshd.conf

input {
  file {
    type => "seucure_log"
    path => "/var/log/secure"
  }
}
filter {
  grok {
    add_tag => [ "sshd_fail" ]
    match => { "message" => "Failed %{WORD:sshd_auth_type} for %{USERNAME:sshd_invalid_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{GREEDYDATA:sshd_protocol}" }
  }
}
output {
  elasticsearch {
    index => "sshd_fail-%{+YYYY.MM}"
  }
}

# 修改/var/log/secure的组以及文件权限，设置logstash服务开机自启动，启动logstash服务
chgrp logstash /var/log/secure
chmod 640 /var/log/secure

# 模拟sshd登录失败，生成ssh登录失败日志
# 验证elasticsearch对应的sshd登录失败日志的索引是否生成
# 访问 http://192.168.56.101:9200/_cat/indices?v
 health status index                    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .apm-custom-link         Fi2JSAWpTUWD42Y1-lceUA   1   0          0            0       208b           208b
green  open   .kibana_task_manager_1   W1CCMUzVRYycga3LUdcqGA   1   0          5            1     30.5kb         30.5kb
green  open   .apm-agent-configuration Hb2z_-98TiaJ9Oj2jumxwg   1   0          0            0       208b           208b
green  open   .kibana_1                aWd_Kq0mQK6FVCbMhLK3ag   1   0         94            1    144.2kb        144.2kb
yellow open   sshd_fail-2020.05        2jc6RRK1Q6-AdCLO7Kys_w   1   1          9            0     63.3kb         63.3kb

# 访问 http://192.168.56.101:9200/sshd_fail-2020.05/_search?pretty

在Kibana上进行该索引的添加：

1.进入管理界面，点击菜单Kibana下的“索引模式” -> “创建索引模式” -> 填入生成的索引名称

2.选择@timestamp完成

3.到Kibana的Discovery里选择对应的索引查看

Springboot日志采集

input {
  file {
    type => "springboot"
    path => "/app/logs/seller-admin-0.0.1-SNAPSHOT.jar.log"
  }
}
filter {
  grok {
    add_tag => [ "springboot" ]
    # match => { "message" => "2020-05-16 16:48:41.833  INFO 7558 --- [           main] o.s.s.quartz.SchedulerFactoryBean        : Starting Quartz Scheduler now"}
    match => {"message" => "(?<time>[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]+)\s+%{LOGLEVEL:level}\s+%{NUMBER:thread_id} --- \[.+\]\s+(?<class>.+)\s+: (?<content>.+)" }
    }
}
output {
  elasticsearch {
    index => "springboot-%{+YYYY.MM}"
  }
}