本文详细介绍了如何搭建ELK集群，包括环境准备、安装x-pack、配置logstash收集nginx、tomcat和mysql日志，以及使用metricbeat监控系统资源。同时，文章还提到了在kibana中创建索引规则，通过head插件查看日志，以及日志级别的调整，为生产环境提供日志管理和分析方案。
ELK集群部署（二）
ELK 是 elastic 公司旗下三款产品ElasticSearch、Logstash、Kibana的首字母组合，也即Elastic Stack包含ElasticSearch、Logstash、Kibana、Beats。ELK提供了一整套解决方案，并且都是开源软件，之间互相配合使用，完美衔接，高效的满足了很多场合的应用，是目前主流的一种日志系统。
ElasticSearch 一个基于 JSON 的分布式的搜索和分析引擎，作为 ELK 的核心，它集中存储数据，
用来搜索、分析、存储日志。它是分布式的，可以横向扩容，可以自动发现，索引自动分片
Logstash 一个动态数据收集管道，支持以 TCP/UDP/HTTP 多种方式收集数据（也可以接受 Beats 传输来的数据），
并对数据做进一步丰富或提取字段处理。用来采集日志，把日志解析为json格式交给ElasticSearch
Kibana 一个数据可视化组件，将收集的数据进行可视化展示（各种报表、图形化数据），并提供配置、管理 ELK 的界面
Beats 一个轻量型日志采集器，单一用途的数据传输平台，可以将多台机器的数据发送到 Logstash 或 ElasticSearch
X-Pack 一个对Elastic Stack提供了安全、警报、监控、报表、图表于一身的扩展包，不过收费
官网：https://www.elastic.co/cn/ ，中文文档：https://elkguide.elasticsearch.cn/
下载elk各组件的旧版本：
https://www.elastic.co/downloads/past-releases
环境准备
- 角色划分：
系统：CentOS 7
es主节点/es数据节点/kibana/head 192.168.30.128
es主节点/es数据节点/logstash 192.168.30.129
es主节点/es数据节点/filebeat 192.168.30.130
- 全部关闭防火墙和selinux：
# systemctl stop firewalld && systemctl disable firewalld
# sed -i 's/=enforcing/=disabled/g' /etc/selinux/config && setenforce 0
- 全部配置系统环境：
# vim /etc/security/limits.conf
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
# vim /etc/sysctl.conf
vm.max_map_count=655360
# sysctl -p
- 全部安装Java环境：
# tar zxf jdk-8u191-linux-x64.tar.gz && mv jdk1.8.0_191/ /usr/local/jdk
# vim /etc/profile
JAVA_HOME=/usr/local/jdk
PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/jre/lib
export JAVA_HOME PATH CLASSPATH
# source !$
# java -version
# ln -s /usr/local/jdk/bin/java /usr/local/bin/java
x-pack
- 安装x-pack：
# /usr/local/elasticsearch/bin/elasticsearch-plugin install x-pack
ERROR: this distribution of Elasticsearch contains X-Pack by default
ELK 6.3及更新版本已经默认安装x-pack，我们需要做的就是开启x-pack配置。
- 查看是否有x-pack：
# ls /usr/local/elasticsearch/modules/x-pack-core/x-pack-core-6.7.1.jar
/usr/local/elasticsearch/modules/x-pack-core/x-pack-core-6.7.1.jar
- 创建LicenseVerifier.java文件：
# vim LicenseVerifier.java
package org.elasticsearch.license;
import java.nio.*;
import org.elasticsearch.common.bytes.*;
import java.util.*;
import java.security.*;
import org.elasticsearch.common.xcontent.*;
import org.apache.lucene.util.*;
import org.elasticsearch.core.internal.io.*;
import java.io.*;
public class LicenseVerifier
{
public static boolean verifyLicense(final License license, final byte[] encryptedPublicKeyData) {
return true;
}
public static boolean verifyLicense(final License license) {
return true;
}
}
- 创建XPackBuild.java文件：
# vim XPackBuild.java
package org.elasticsearch.xpack.core;
import org.elasticsearch.common.io.*;
import java.net.*;
import org.elasticsearch.common.*;
import java.nio.file.*;
import java.io.*;
import java.util.jar.*;
public class XPackBuild
{
public static final XPackBuild CURRENT;
private String shortHash;
private String date;
@SuppressForbidden(reason = "looks up path of xpack.jar directly")
static Path getElasticsearchCodebase() {
final URL url = XPackBuild.class.getProtectionDomain().getCodeSource().getLocation();
try {
return PathUtils.get(url.toURI());
}
catch (URISyntaxException bogus) {
throw new RuntimeException(bogus);
}
}
XPackBuild(final String shortHash, final String date) {
this.shortHash = shortHash;
this.date = date;
}
public String shortHash() {
return this.shortHash;
}
public String date() {
return this.date;
}
static {
final Path path = getElasticsearchCodebase();
String shortHash = null;
String date = null;
Label_0157: {
shortHash = "Unknown";
date = "Unknown";
}
CURRENT = new XPackBuild(shortHash, date);
}
}
- 编译这两个文件：
# javac -cp "/usr/local/elasticsearch/lib/*:/usr/local/elasticsearch/modules/x-pack-core/x-pack-core-6.7.1.jar" LicenseVerifier.java
# javac -cp "/usr/local/elasticsearch/lib/*:/usr/local/elasticsearch/modules/x-pack-core/x-pack-core-6.7.1.jar" XPackBuild.java
会发现多出来两个class文件：LicenseVerifier.class，XPackBuild.class
- 覆盖这两个class文件：
# mkdir jar && cd jar/
# cp /usr/local/elasticsearch/modules/x-pack-core/x-pack-core-6.7.1.jar .
# jar xf x-pack-core-6.7.1.jar
# ls
logstash-index-template.json monitoring-es.json public.key triggered-watches.json
META-INF monitoring-kibana.json rollup-dynamic-template.json watches.json
monitoring-alerts.json monitoring-logstash.json security_audit_log.json watch-history.json
monitoring-beats.json org security-index-template.json x-pack-core-6.7.1.jar
# find ./ -name LicenseVerifier.class
./org/elasticsearch/license/LicenseVerifier.class
# find ./ -name XPackBuild.class
./org/elasticsearch/xpack/core/XPackBuild.class
# cp ../LicenseVerifier.class ./org/elasticsearch/license/LicenseVerifier.class
# cp ../XPackBuild.class ./org/elasticsearch/xpack/core/XPackBuild.class
- 重新打包，并替换原来的jar包：
# jar cf x-pack-core-6.7.1.jar ./*
# systemctl stop elasticsearch
# cp x-pack-core-6.7.1.jar /usr/local/elasticsearch/modules/x-pack-core/x-pack-core-6.7.1.jar
集群中所有elasticsearch节点都要替换x-pack的jar包，这一步需要注意elasticsearch启动问题。
- 导入授权文件：
先从官网申请basic授权文件，地址，填入邮箱地址，会发邮件给我们进行下载。
将下载的文件重命名为license.json，并做下面修改：
# vim license.json
"type":"platinum" #白金版
"expiry_date_in_millis":2524579200999 #截止日期2050年
# vim /usr/local/elasticsearch/config/elasticsearch.yml #添加
xpack.security.enabled: false #关闭x-pack安全验证
# systemctl start elasticsearch
# curl -XPUT -u elastic 'http://192.168.30.128:9200/_xpack/license' -H "Content-Type: application/json" -d @license.json
Enter host password for user 'elastic': #默认密码是changeme
{"acknowledged":true,"license_status":"valid"}
# curl -XPUT -u elastic 'http://192.168.30.129:9200/_xpack/license' -H "Content-Type: application/json" -d @license.json
# curl -XPUT -u elastic 'http://192.168.30.130:9200/_xpack/license' -H "Content-Type: application/json" -d @license.json
集群所有节点全部替换license。另外，license.json 也可以通过kibana上传。
- 检验是否成功导入license：
# curl -XGET -u elastic:changeme "http://192.168.30.128:9200/_license"
{
"license" : {
"status" : "active",
"uid" : "5676803e-607b-40ee-809b-86edc32d6645",
"type" : "platinum",
"issue_date" : "2019-04-22T00:00:00.000Z",
"issue_date_in_millis" : 1555891200000,
"expiry_date" : "2049-12-31T16:00:00.999Z",
"expiry_date_in_millis" : 2524579200999,
"max_nodes" : 100,
"issued_to" : "l zx (111)",
"issuer" : "Web Form",
"start_date_in_millis" : 1555891200000
}
}
- 生成ssl证书：
# /usr/local/elasticsearch/bin/elasticsearch-certgen
Please enter the desired output file [certificate-bundle.zip]: elk.zip #保存证书的文件名
Enter instance name: elk #集群名
Enter name for directories and files [elk]: elk #创建证书的文件夹及文件名
Enter IP Addresses for instance (comma-separated if more than one) []: 192.168.30.128,192.168.30.129,192.168.30.130 #集群节点ip，逗号分隔
Enter DNS names for instance (comma-separated if more than one) []: elk-128,elk-129,elk-130 #集群节点名，逗号分隔
Would you like to specify another instance? Press 'y' to continue entering instance information: n #是否有其它实例
Certificates written to /software/elk.zip #证书文件路径
#为es集群创建证书颁发机构
# /usr/local/elasticsearch/bin/elasticsearch-certutil ca #直接回车
#为集群中节点生成证书和私钥
# /usr/local/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 #直接回车
# mkdir /usr/local/elasticsearch/config/certs
# cp *.p12 !$
#将证书拷贝到所有es节点
# scp -r !$ 192.168.30.129:!$ ;scp -r !$ 192.168.30.130:!$
如果生成证书设置了密码，需要将密码添加到elasticsearch秘钥库：
# /usr/local/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
# /usr/local/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
# unzip elk.zip
Archive: elk.zip
creating: ca/
inflating: ca/ca.crt
inflating: ca/ca.key
creating: elk/
inflating: elk/elk.crt
inflating: elk/elk.key
# mkdir /usr/local/elasticsearch/config/certs
# mv ca/* !$ && mv elk/* !$
# scp -r !$ 192.168.30.129:/usr/local/elasticsearch/config/; scp -r !$ 192.168.30.130:/usr/local/elasticsearch/config/
- 全部修改配置：
集群内所有elasticsearch机器
# vim /usr/local/elasticsearch/config/elasticsearch.yml
xpack.security.enabled: true #打开x-pack安全验证
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type #增加head
#开启ssl支持
xpack.security.transport.ssl.enabled: true
xpack.ssl.key: certs/elk.key
xpack.ssl.certificate: certs/elk.crt
xpack.ssl.certificate_authorities: certs/ca.crt
# chown -R elk:elk /usr/local/elasticsearch
# systemctl restart elasticsearch
- 生成用户名和密码：
# /usr/local/elasticsearch/bin/elasticsearch-setup-passwords auto #自动生成（二选一）
# /usr/local/elasticsearch/bin/elasticsearch-setup-passwords interactive #手动生成（二选一）
#自定义各个密码
Enter password for [elastic]: elk-2019
Reenter password for [elastic]: elk-2019
Enter password for [apm_system]: elk-2019
Reenter password for [apm_system]: elk-2019
Enter password for [kibana]: elk-2019
Reenter password for [kibana]: elk-2019
Enter password for [logstash_system]: elk-2019
Reenter password for [logstash_system]: elk-2019
Enter password for [beats_system]: elk-2019
Reenter password for [beats_system]: elk-2019
Enter password for [remote_monitoring_user]: elk-2019
Reenter password for [remote_monitoring_user]: elk-2019
- 修改kibana配置：
192.168.30.128
# vim /usr/local/kibana/config/kibana.yml
#elasticsearch账户和密码
elasticsearch.username: elastic
elasticsearch.password: elk-2019
# systemctl restart kibana
刷新kibana网页，可以看到需要登录账号及密码
输入前面配置的账号：elastic、密码：elk-2019，登录
开启x-pack后访问head需要注意加上账号及密码，例如：
http://192.168.30.128:9100/?auth_user=elastic&auth_password=elk-2019
- 修改logstash配置：
192.168.30.129
# vim /usr/local/logstash/config/logstash.yml
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: elk-2019
xpack.monitoring.elasticsearch.hosts: ["http://192.168.30.128:9200","http://192.168.30.129:9200","http://192.168.30.130:9200"]
xpack.monitoring.collection.interval: 10s
# systemctl restart logstash
另外，logstash收集日志时需要在output部分设定账号密码，否则无法登陆elasticsearch。
output {
elasticsearch {
hosts => ["http://192.168.30.129:9200"]
user => "elastic"
password => "elk-2019"
}
}
- 修改filebeat配置：
192.168.30.130
# vim /usr/local/filebeat/filebeat.yml
setup.kibana:
host: "192.168.30.128:5601"
output.elasticsearch:
hosts: ["192.168.30.130:9200"] #可以是集群中其它es节点ip
username: "elastic"
password: "elk-2019"
# systemctl restart filebeat
metricbeat
- 安装metricbeat：
metricbeat用来监控系统资源，如CPU、内存、network等。
# wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-6.7.1-linux-x86_64.tar.gz
# tar zxf metricbeat-6.7.1-linux-x86_64.tar.gz && mv metricbeat-6.7.1-linux-x86_64 /usr/local/metricbeat
# vim /usr/local/metricbeat/metricbeat.yml
setup.kibana:
host: "192.168.30.128:5601"
output.elasticsearch:
hosts: ["192.168.30.128:9200"] #可以是集群中其它es节点ip
username: "elastic"
password: "elk-2019"
# cd /usr/local/metricbeat
# ./metricbeat modules enable system
# ./metricbeat setup
# nohup ./metricbeat -e &
- 配置metricbeat服务：
metricbeat服务文件
# vim /usr/lib/systemd/system/metricbeat.service
[Unit]
Description=Metricbeat is a lightweight shipper for metrics.
Documentation=https://www.elastic.co/products/beats/metricbeat
Wants=network-online.target
After=network-online.target
[Service]
ExecStart=/usr/local/metricbeat/metricbeat -c /usr/local/metricbeat/metricbeat.yml -path.home /usr/local/metricbeat -path.config /usr/local/metricbeat -path.data /usr/local/metricbeat/data -path.logs /usr/local/metricbeat/logs
Restart=always
[Install]
WantedBy=multi-user.target
- 管理服务：
# systemctl daemon-reload
# systemctl enable metricbeat
# systemctl start metricbeat
到kibana页面查看
logstash收集nginx日志
192.168.30.129
logstash收集nginx访问日志
- 示例1：
logstash收集nginx访问日志，以json格式。
配置nginx
# yum install -y nginx
# vim /etc/nginx/nginx.conf
log_format json '{"@timestamp":"$time_iso8601",'
'"@version":"1",'
'"client":"$remote_addr",'
'"url":"$uri",'
'"status":"$status",'
'"domain":"$host",'
'"host":"$server_addr",'
'"size":"$body_bytes_sent",'
'"responsentime":"$request_time",'
'"referer":"$http_referer",'
'"useragent":"$http_user_agent"'
'}';
# vim /etc/nginx/conf.d/elk.conf
server {
listen 80;
server_name elk.test.com;
location / {
proxy_pass http://192.168.30.128:5601;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
access_log /var/log/nginx/elk_access_json.log json;
}
配置logstash
# vim /usr/local/logstash/conf.d/nginx_access.conf
input {
file {
path => "/var/log/nginx/elk_access_json.log"
codec => "json"
start_position => "beginning"
type => "nginx_access"
}
}
filter {
json {
source => "message"
skip_on_invalid_json => true
}
}
output {
if [type] == "nginx_access" {
elasticsearch {
hosts => ["192.168.30.129:9200"]
user => "elastic"
password => "elk-2019"
index => "nginx-access.log-%{+YYYY.MM.dd}"
}
}
}
启动logstash
# systemctl start nginx
# systemctl restart logstash
在kibana页面创建索引规则nginx-access.log-2019.04*
可以看到nginx-access.log-2019.04*日志内容是json格式的，继续在head上查看
logstash收集nginx错误日志
- 示例2：
logstash收集nginx错误日志。
配置nginx
# vim /etc/nginx/nginx.conf
error_log /var/log/nginx/error.log error;
# vim /etc/nginx/conf.d/test.conf
erver { #将server改为erver
listen 80;
server_name elk.test.com;
location / {
proxy_pass http://192.168.30.128:5601;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
access_log /var/log/nginx/elk_access_json.log json;
}
配置logstash
# vim /usr/local/logstash/conf.d/nginx_error.conf
input {
file {
path => "/var/log/nginx/error.log"
type => "nginx_error"
}
}
filter {
grok {
match => [ "message" , "(?<timestamp>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: (?:, client: (?<clientip>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server}?)(?:, request: %{QS:request})?(?:, upstream: (?<upstream>\"%{URI}\"|%{QS}))?(?:, host: %{QS:request_host})?(?:, referrer: \"%{URI:referrer}\")?"]
}
}
output {
if [type] == "nginx_error" {
elasticsearch {
hosts => ["192.168.30.129:9200"]
user => "elastic"
password => "elk-2019"
index => "nginx-error.log-%{+YYYY.MM.dd}"
}
}
}
启动logstash
# /usr/sbin/nginx -t #生成error日志
nginx: [emerg] unknown directive "erver" in /etc/nginx/conf.d/test.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed
# /usr/sbin/nginx -s reload
nginx: [emerg] unknown directive "erver" in /etc/nginx/conf.d/test.conf:1
# systemctl restart logstash
在kibana页面创建索引规则nginx-error.log-2019.04*
在head上查看
通过上面两个例子，就可以很方便的将nginx访问日志和错误日志收集起来，在kibana中展示并做过滤分析，或者在head中进行查询分析。
其实可以将上面logstash关于nginx的日志配置文件写成一个，用type作区分，使用if条件判断分别过滤收集nginx的日志。
logstash收集tomcat日志
192.168.30.129
* catalina开头的日志为Tomcat的综合日志，它记录Tomcat服务相关信息，也会记录错误日志
* host-manager和manager为管理相关的日志，其中host-manager为虚拟主机的管理日志
* localhost和localhost_access为虚拟主机相关日志，其中带access字样的为访问日志，
不带access字样的为默认主机的错误日志（访问日志默认不会生成，需要在server.xml中配置一下）
错误日志会统一记录到catalina.out中，出现问题应该首先查看它
logstash收集catalina日志和localhost_access_log日志（json格式）。
先手动安装tomcat：https://blog.csdn.net/miss1181248983/article/details/81080091 ，不建议yum安装。
- 配置tomcat：
# vim /usr/local/tomcat/conf/server.xml
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
改为
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log" suffix=".log"
pattern="{"client":"%h", "client user":"%l", "authenticated":"%u", "access time":"%t", "method":"%r", "status":"%s", "send bytes":"%b", "Query?string":"%q", "partner":"%{Referer}i", "Agent version":"%{User-Agent}i"}" />
- 配置logstash：
# vim /usr/local/logstash/conf.d/tomcat.conf
input {
file {
path => "/usr/local/tomcat/logs/catalina.out"
start_position => "beginning"
sincedb_path => "/dev/null"
type => "catalina.out"
}
file {
path => "/usr/local/tomcat/logs/localhost_access_log*.log"
start_position => "beginning"
codec => "json"
sincedb_path => "/dev/null"
type => "localhost_access_log"
}
}
filter {
if [type] == "catalina.out" {
grok {
match => { "message" => "(\[\s?%{LOGLEVEL:level}\] %{DATA:class} %{DATA:logtime} - )?%{GREEDYDATA:message}" }
}
geoip {
source => "clientip"
}
}
if [type] == "localhost_access_log" {
grok {
match => { "message" => "(\[\s?%{LOGLEVEL:level}\] %{DATA:class} %{DATA:logtime} - )?%{GREEDYDATA:message}" }
}
json {
source => "message"
skip_on_invalid_json => true
}
}
}
output {
if [type] == "catalina.out" {
elasticsearch {
hosts => ["192.168.30.129:9200"]
user => "elastic"
password => "elk-2019"
index => "catalina.out-%{+YYYY.MM.dd}"
}
}
if [type] == "localhost_access_log" {
elasticsearch {
hosts => ["192.168.30.129:9200"]
user => "elastic"
password => "elk-2019"
index => "localhost_access_log-%{+YYYY.MM.dd}"
}
}
}
- 启动logstash：
# /usr/local/tomcat/bin/startup.sh
# systemctl restart logstash
在kibana页面创建索引规则catalina.out-2019.04*和localhost_access_log-2019.04*
可以看到localhost_access_log-2019.04*日志内容是json格式的，继续在head上查看
生产环境下，可以将catalina.out日志级别设置为WARN或ERROR，也可以在logstash配置文件中设置关键字过滤出报错日志，从而得到tomcat运行过程中的报错信息。
logstash收集mysql日志
192.168.30.129
logstash收集mysql慢查询日志及错误日志。
先手动安装mysql：https://blog.csdn.net/miss1181248983/article/details/86621439 ，不建议yum安装。
- 配置mysql：
如果是按照上面方式安装的mysql，默认没有密码
#临时生效
# mysql
mysql> show variables like '%slow_query_log%';
+---------------------+----------------------------+
| Variable_name | Value |
+---------------------+----------------------------+
| slow_query_log | OFF | #默认慢查询日志是禁用的
| slow_query_log_file | /data/mysql/test2-slow.log |
+---------------------+----------------------------+
2 rows in set (0.08 sec)
mysql> set global slow_query_log=1;
Query OK, 0 rows affected (0.34 sec)
mysql> show variables like '%slow_query_log%';
+---------------------+----------------------------+
| Variable_name | Value |
+---------------------+----------------------------+
| slow_query_log | ON |
| slow_query_log_file | /data/mysql/test2-slow.log |
+---------------------+----------------------------+
2 rows in set (0.00 sec)
#永久生效
# vim /etc/my.cnf
slow_query_log =1
slow_query_log_file=/data/mysql/test2-slow.log
long_query_time=3 #指定SQL语句最大执行时间，超过这个时间将被记录
#log_queries_not_using_indexes=1 #记录没有使用索引的查询语句到慢查询日志中
# systemctl restart mysqld.service
mysql> show variables like '%log_output%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_output | FILE | #log_output='FILE'表示将日志存入文件，默认值是'FILE'。log_output='TABLE'表示将日志存入数据库
+---------------+-------+
1 row in set (0.02 sec)
mysql> show global status like '%slow_queries%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Slow_queries | 0 | #查询当前有多少条慢查询记录
+---------------+-------+
1 row in set (0.47 sec)
- 配置logstash：
# vim /usr/local/logstash/conf.d/mysql.conf
input {
file {
path => "/data/mysql/test2-slow.log"
start_position => "beginning"
type => "mysql_slow_file"
sincedb_path => "/dev/null"
codec => multiline {
pattern => "^# User@Host:"
negate => true
what => previous
}
}
file {
path => "/usr/local/mysql/log/error.log"
start_position => "beginning"
type => "mysql_error"
sincedb_path => "/dev/null"
}
}
filter {
if [type] == "mysql_slow_file" {
grok {
match => { "message" => "SELECT SLEEP" }
add_tag => [ "sleep_drop" ]
tag_on_failure => [] # prevent default _grokparsefailure tag on real records
}
if "sleep_drop" in [tags] {
drop {}
}
grok {
match => [ "message", "(?m)^# User@Host: %{USER:user}\[[^\]]+\] @ (?:(?<clienthost>\S*) )?\[(?:%{IP:clientip})?\]\s*# Query_time: %{NUMBER:query_time:float}\s+Lock_time: %{NUMBER:lock_time:float}\s+Rows_sent: %{NUMBER:rows_sent:int}\s+Rows_examined: %{NUMBER:rows_examined:int}\s*(?:use %{DATA:database};\s*)?SET timestamp=%{NUMBER:timestamp};\s*(?<query>(?<action>\w+)\s+.*)\n# Time:.*$" ]
}
date {
match => [ "timestamp", "UNIX" ]
remove_field => [ "timestamp" ]
}
}
if [type] == "mysql_error" {
grok {
match => [ 'message', "(?m)^%{NUMBER:date} *%{NOTSPACE:time} %{NUMBER:bytes} %{GREEDYDATA:message}" ]
overwrite => [ 'message' ]
add_field => { "timestamp" => "%{date}%{time}" }
}
}
}
output {
if [type] == "mysql_slow_file" {
elasticsearch {
hosts => ["192.168.30.129:9200"]
user => "elastic"
password => "elk-2019"
index => "mysql_slow.log-%{+YYYY.MM.dd}"
}
}
if [type] == "mysql_error" {
elasticsearch {
hosts => ["192.168.30.129:9200"]
user => "elastic"
password => "elk-2019"
index => "mysql_error.log-%{+YYYY.MM.dd}"
}
}
}
- 启动logstash：
# systemctl restart mysqld.service
# systemctl restart logstash
在kibana页面创建索引规则mysql_slow.log-2019.04*和mysql_error.log-2019.04*
# cat /data/mysql/test2-slow.log
/usr/local/mysql/bin/mysqld, Version: 5.7.25-log (MySQL Community Server (GPL)). started with:
Tcp port: 0 Unix socket: /usr/local/mysql/tmp/mysql.sock
Time Id Command Argument
可以看到当前慢查询日志为空，继续在head上查看
通过上面几个例子，我们可以去收集常用应用的相关日志。当然，直接使用logstash是比较耗费资源的，更合适的方法是通过filebeat替代logstash-agent收集日志。下一步需要做的就是，将日志输出到Zabbix，实现错误日志告警。
扫一扫
2845
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
