0029【ELKF-Linux版】总有一款你喜欢的分布式日志系统，就是它了

零 前期工作

0.1 环境

端口

elasticsearch:9200、9300
kibana:5601
logstash:5044
filebeat:

依赖jdk

elasticsearch: jdk11或以上
logstash: jdk11或以上

功能

ELKF 是 Elasticsearch 、 Logstash 、 Kibana 、 Filebeat 的简称。
elasticsearch: 存储、搜索和分析引擎，特点是高可伸缩、高可靠和易管理等。
kibana: 数据分析和可视化平台，通常依赖 elasticsearch 。
logstash: 数据收集引擎，可以对数据进行过滤、分析、丰富、统一格式等操作，存储到用户指定的位置，包含但不限于文件、 elasticsearch 。
filebeat: 轻量级的开源日志文件数据搜集器，负责对服务的日志进行收集。

简单归纳为：FileBeat收集日志、Logstash解析格式化、Elasticsearch存储、Kibana分析。

0.2 文件上传

将相关文件相继上传到目录/home/elkf 下

0.3 文件解压

cd /home/elkf
tar -xzvf elasticsearch-7.17.0-linux-x86_64.tar.gz
tar -xzvf kibana-7.17.0-linux-x86_64.tar.gz
tar -xzvf logstash-7.17.0-linux-x86_64.tar.gz
tar -xzvf filebeat-7.17.0-linux-x86_64.tar.gz

一 elasticsearch

1.1 配置elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: elasticsearch_prod
cluster.routing.allocation.disk.threshold_enabled: false
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: node-001-data
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
path.data: /home/elkf/elasticsearch-7.17.0/datas
#
# Path to log files:
#
#path.logs: /path/to/logs
path.logs: /home/elkf/elasticsearch-7.17.0/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, X-User"

# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
# ---------------------------------- Security ----------------------------------
#
#                                 *** WARNING ***
#
# Elasticsearch security features are not enabled by default.
# These features are free, but require configuration changes to enable them.
# This means that users don’t have to provide credentials and can get full access
# to the cluster. Network connections are also not encrypted.
#
# To protect your data, we strongly encourage you to enable the Elasticsearch security features. 
# Refer to the following documentation for instructions.
#
# https://www.elastic.co/guide/en/elasticsearch/reference/7.16/configuring-stack-security.html

xpack.security.enabled: true

1.2 配置jvm.options（不配置）

## -Xms4g
## -Xmx4g
#-Xms1g
#-Xmx1g

错误：

encountered [2] errors parsing [/home/elkf/elasticsearch-7.17.0/config/jvm.options]
[1]: encountered improperly formatted JVM option in [/home/elkf/elasticsearch-7.17.0/config/jvm.options] on line number [33]: [ -Xms1g]
[2]: encountered improperly formatted JVM option in [/home/elkf/elasticsearch-7.17.0/config/jvm.options] on line number [34]: [ -Xmx1g]

1.3 配置JAVA_HOME

bin/elasticsearch-env

# 2022-02-14（增加ES_JAVA_HOME，es自带jdk 17，需要jdk11以上）
ES_JAVA_HOME=/home/elkf/elasticsearch-7.17.0/jdk

# now set the path to java
if [ ! -z "$ES_JAVA_HOME" ]; then
  JAVA="$ES_JAVA_HOME/bin/java"
  JAVA_TYPE="ES_JAVA_HOME"
elif [ ! -z "$JAVA_HOME" ]; then
  # fallback to JAVA_HOME
  echo "warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME" >&2
  JAVA="$JAVA_HOME/bin/java"
  JAVA_TYPE="JAVA_HOME"

1.4 用户权限

# 增加用户组
groupadd elkf
# 增加用户，-g 指定组 -p 指定密码
useradd elkf -g elkf -p elkf123$
# 授权，-R : 处理指定目录下的所有文件
chown -R elkf:elkf /home/elkf


chown -R elkf:elkf /home/kfq_gov/tomcat7/webapps/logs
chown -R elkf:elkf /root/logs

chown -R elkf:elkf /root/logs/kfq_gov.log
chmod -R 777 /root/logs/kfq_gov.log

tail -f /home/kfq_gov/tomcat7/webapps/logs/kfq_gov.log
tail -f /root/logs/kfq_gov.log

1.5 配置密码

所有用户：elastic、remote_monitoring_user、beats_system、logstash_system、kibana、kibana_system、apm_system

密码：elkf123456

interactiveller_test elasticsearch-7.17.0]# ./bin/elasticsearch-setup-passwords 
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]: elkf123456
Reenter password for [elastic]: elkf123456
Enter password for [apm_system]: elkf123456
Reenter password for [apm_system]: elkf123456
Enter password for [kibana_system]: elkf123456
Reenter password for [kibana_system]: elkf123456
Enter password for [logstash_system]: elkf123456
Reenter password for [logstash_system]: elkf123456 
Enter password for [beats_system]: elkf123456
Reenter password for [beats_system]: elkf123456
Enter password for [remote_monitoring_user]: elkf123456
Reenter password for [remote_monitoring_user]: elkf123456
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

1.6 启动

当使用root用户执行sh /home/elkf/elasticsearch-start.sh后日志错误：

java.lang.RuntimeException: can not run elasticsearch as root

切换用户为elkf，再执行sh /home/elkf/elasticsearch-start.sh

#! /bin/bash
/home/elkf/elasticsearch-7.17.0/bin/elasticsearch -d > /home/elkf/elasticsearch-7.17.0/logs/elasticsearch.log 2>&1 &

#参数：-d # 后台方式启动
#elasticsearch.log日志好像没啥作用，查看日志文件elasticsearch_prod.log即可。

1.7 权限验证

#切换用户
[root@ser elkf]# su elkf
#启动
bash-4.2$ ./elasticsearch-start.sh

# 无密码测试
bash-4.2$ curl localhost:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

# 密码测试
bash-4.2$ curl localhost:9200 -u elastic:elkf123456
{
  "name" : "node-001-data",
  "cluster_name" : "elasticsearch_prod",
  "cluster_uuid" : "fxP0XBjQRnK3i9atOqPkbg",
  "version" : {
    "number" : "7.17.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "bee86328705acaa9a6daede7140defd4d9ec56bd",
    "build_date" : "2022-01-28T08:36:04.875279988Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

1.8 telnet 9300

bash-4.2$ telnet 127.0.0.1 9300
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

1.9 日志

bash-4.2$ tail -f elasticsearch-7.17.0/logs/logs/elasticsearch_prod.log

# tail -f /home/elkf/elasticsearch-7.17.0/logs/logs/elasticsearch_prod.log

二 kibana

2.1 配置kibana.yml

config/kibana.yml

# server.publicBaseUrl 缺失，在生产环境中运行时应配置。某些功能可能运行不正常。
# 这里地址改为你访问kibana的地址，不能以 / 结尾
server.publicBaseUrl: "http://172.23.8.184:5601"

server.name: kibana
# 默认是localhost或者127.0.0.1，只允许本地访问，设置0.0.0.0可允许远程访问
server.host: "0.0.0.0"
server.port: 5601
elasticsearch.hosts: ["http://localhost:9200"]
#配置本地索引
kibana.index: ".kibana"
#2022-02-14-设置elasticsearch账号和密码
elasticsearch.username: "kibana_system"
elasticsearch.password: "elkf123456"
# 设置中文
i18n.locale: "zh-CN"

2.2 启动

sh /home/elkf/kibana-start.sh

#! /bin/bash
nohup /home/elkf/kibana-7.17.0-linux-x86_64/bin/kibana > /home/elkf/kibana-7.17.0-linux-x86_64/logs/kibana.log 2>&1 &

使用root用户启动后，日志错误：

Kibana should not be run as root.  Use --allow-root to continue.

授权：

[root@ser_teller_test elkf]# chown -R elkf:elkf /home/elkf

切换用户：

[root@ser_teller_test elkf]# su elkf

启动：

bash-4.2$ ./kibana-start.sh

2.3 日志

bash-4.2$ tail -f kibana-7.17.0-linux-x86_64/logs/kibana.log

# tail -f /home/elkf/kibana-7.17.0-linux-x86_64/logs/kibana.log

2.4 测试端口5601

[root@ser_teller_test elkf]# netstat -an | grep 5601
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN

[root@ser_teller_test elkf]# telnet localhost 5601
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

# 测试本地访问
[root@ser_teller_test elkf]# curl http://localhost:5601/
[root@ser_teller_test elkf]#

# 测试远程访问
[root@ser_teller_test elkf]# curl http://172.23.8.184:5601/
curl: (7) Failed connect to 172.23.8.184:5601; 拒绝连接

# kibana.yml加入配置 server.host: "0.0.0.0" 即可
bash-4.2$ curl http://172.23.8.184:5601/
bash-4.2$

2.5 浏览器访问

http://172.23.8.184:5601/
# 需要输入账户密码，elastic/elkf123456

三 logstash

3.1 配置logstash.conf

将 config下的logstash-sample.conf拷贝一份命名为logstash.conf，配置内容：

input {
  beats {
    port => 5044  # 设置专用端口用于接收各个来源的日志
    client_inactivity_timeout => 36000
  }
}

filter {
  json {
      source => "message"
  }
}

output {
  if [filetype] == "supervision-info" {
    # 184-上报终端信息
    elasticsearch {
      user => elastic
      password => elkf123456
      hosts => ["127.0.0.1:9200"]
      index => "supervision-info-%{+YYYY.MM.dd}"
    }
  }
}

3.2 配置JAVA_HOME

LOGSTASH_HOME="$(cd `dirname $SOURCEPATH`/..; pwd)"
export LOGSTASH_HOME
export LS_HOME="${LOGSTASH_HOME}"
SINCEDB_DIR="${LOGSTASH_HOME}"
export SINCEDB_DIR
LOGSTASH_JARS=${LOGSTASH_HOME}/logstash-core/lib/jars

# 2022-02-14-设置JAVA_HOME
LS_JAVA_HOME=/home/elkf/logstash-7.17.0/jdk

3.3 启动

sh /home/elkf/logstash-start.sh

logstash-home=/home/elkf/logstash-7.17.0
nohup $logstash-home/bin/logstash --config.reload.automatic -f $logstash-home/config/logstash.conf > $logstash-homelogs/logstash.log 2>&1 &

切换用户elkf，并执行 ./logstash-start.sh

[root@ser_teller_test elkf]# su elkf
bash-4.2$ ./logstash-start.sh

3.4 日志

bash-4.2$ tail -f logstash-7.17.0/logs/logstash.log

或者
tail -f /home/elkf/logstash-7.17.0/logs/logstash.log

3.5 测试端口5044

[root@ser_teller_test elkf]# telnet 172.23.8.184 5044
Trying 172.23.8.184...
Connected to 172.23.8.184.
Escape character is '^]'.

四 filebeat

4.1 配置filebeat.xml

/home/elkf/filebeat-7.17.0-linux-x86_64/filebeat.xml

# ============================== Filebeat inputs ===============================
## 2022-02-14-上送终端信息
- type: log
  enabled: true  #开启监视，不开不采集
  backoff: "1s"
  tail_files: false 
  paths:
    - /home/vtq/logs/supervision-info.log
  fields:
    filetype: supervision-info  # 加这个就是区分不同文件的
  #fields:自定义字段
  #ields_under_root 为true,则自定义字段将为文档中的顶级字段。
  fields_under_root: true # 普通单行文本日志
  
# ---------------------------- Elasticsearch Output ----------------------------
## 关闭Elasticsearch
##output.elasticsearch:
  # Array of hosts to connect to.
##  hosts: ["localhost:9200"]

# ------------------------------ Logstash Output -------------------------------
## 开启Logstash
output.logstash:
  # The Logstash hosts
  hosts: ["172.23.8.184:5044"]

4.2 配置执行文件

创建logs目录：

mkdir -p /home/elkf/filebeat-7.17.0-linux-x86_64/logs

配置启动文件：

vi /home/elkf/filebeat-start.sh

内容：

#! /bin/bash
FILEBEAT_HOME=/home/elkf/filebeat-7.17.0-linux-x86_64
nohup $FILEBEAT_HOME/filebeat -e -c $FILEBEAT_HOME/filebeat.yml -d "publish" > $FILEBEAT_HOME/logs/filebeat.log 2>&1 &

# -d 后台方式启动

授权文件：

[root@ser_teller_test elkf]# chown -R elkf:elkf /home/elkf
[root@ser_teller_test elkf]# chmod -R 777 filebeat-start.sh

4.3 启动

切换用户

su elkf

执行

bash-4.2$ ./filebeat-start.sh

4.4 日志

bash-4.2$ tail -f filebeat-7.17.0-linux-x86_64/logs/filebeat.log

#或 tail -f /home/elkf/filebeat-7.17.0-linux-x86_64/logs/filebeat.log