FileBeat+ELK的搭建(v7.11.2)

最新推荐文章于 2024-07-27 18:24:42 发布

WIGER_52

最新推荐文章于 2024-07-27 18:24:42 发布

阅读量431

点赞数 1

分类专栏： linux学习

本文链接：https://blog.csdn.net/qq_43753178/article/details/115752336

版权

linux学习专栏收录该内容

27 篇文章 2 订阅

订阅专栏

FileBeat+ELK的搭建（v7.11.2）

架构图：

在这里插入图片描述

JDK的安装：

在安装es之前必须安装jdk,我这里安装jdk-11.0.1；

下载地址：https://jdk.java.net/archive/

jdk11与之前版本不同，安装好的文件夹里没有jre文件，环境变量配置更简单了；

我习惯将文件放在/data/dtstack目录下；

在这里插入图片描述

执行安装命令；

[root@ergou dtstack]# tar -zxvf openjdk-11.0.1_linux-x64_bin.tar.gz

在 /etc/profile.d/ 文件夹下创建 java.sh 文件；

[root@ergou dtstack]# vim /etc/profile.d/java.sh
[root@ergou dtstack]# cat /etc/profile.d/java.sh
JAVA_HOME=/data/dtstack/jdk-11.0.1 #jdk文件路径
PATH=$PATH:$JAVA_HOME/bin
export JAVA_HOME PATH

使java.sh文件生效；

[root@ergou dtstack]# source  /etc/profile.d/java.sh

验证jdk；

[root@ergou dtstack]# java -version
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)

ELK的安装之Elasticsearch安装：

下载地址：https://www.elastic.co/cn/downloads/

安装elasticsearch：

[root@ergou dtstack]# tar -zxvf elasticsearch-7.11.2-linux-x86_64.tar.gz
[root@ergou dtstack]# cd elasticsearch-7.11.2/config
[root@ergou ~]mkdir  -p /usr/elk/elasticsearch-7.11.2/data
[root@ergou ~]mkdir  -p /usr/elk/elasticsearch-7.11.2/logs
[root@ergou dtstack]# vim elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
 cluster.name: es-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
 node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
 path.data: /usr/elk/elasticsearch-7.11.2/data
#
# Path to log files:
#
 path.logs: /usr/elk/elasticsearch-7.11.2/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
 network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
 http.port: 9200
#
 http.cors.enabled: true
 http.cors.allow-origin: "*"
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
 cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

配置完之后，因为ElasticSearch使用非root用户启动，所以创建一个用户，这里我创建用户elk密码也是elk，然后授权目录；

[root@ergou ~]#useradd elk
[root@ergou ~]#passwd elk
Changing password for user elk.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.
[root@ergou ~]# chown -R elk:elk /data/dtstack/elasticsearch-7.11.2/
[root@ergou ~]# chown -R elk:elk /usr/elk/elasticsearch-7.11.2/

修改进程文件配置信息：

[root@ergou ~]#vim /etc/security/limits.conf
//在文件末尾添加下面的参数值
* soft nofile 65536
* hard nofile 131072
[root@ergou ~]#vim /etc/sysctl.conf
vm.max_map_count=655360
保存后，执行：
sysctl -p

启动elasticsearch:

[root@ergou ~]# su elk
[elk@ergou ~]$ cd /data/dtstack/elasticsearch-7.11.2/
[elk@ergou elasticsearch-7.11.2]$ ls
bin  config  jdk  lib  LICENSE.txt  logs  modules  NOTICE.txt  plugins  README.asciidoc   
[elk@ergou elasticsearch-7.11.2]$ ./bin/elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[elk@ergou elasticsearch-7.11.2]$ ss -tnl
State      Recv-Q Send-Q        Local Address:Port                       Peer Address:Port
LISTEN     0      128                       *:22                                    *:*
LISTEN     0      100               127.0.0.1:25                                    *:*
LISTEN     0      128                      :::9200                                 :::*
LISTEN     0      128                      :::9300                                 :::*
LISTEN     0      128                      :::22                                   :::*
LISTEN     0      100                     ::1:25                                   :::*

ELK的安装之Filebeat安装:

安装：

[root@ergou dtstack]# tar  -zxvf filebeat-7.11.2-linux-x86_64.tar.gz
[root@ergou dtstack]# cd /data/dtstack/filebeat-7.11.2-linux-x86_64

编辑配置文件：

# ============================== Filebeat inputs ===============================
#将多份log写入一个索引
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /root/nginx/logs/access.log
  fields:
    log_topics: "nginx"   #用于区分是谁的log
  fields_under_root: true
- type: log
  enabled: true
  paths:
    - /home/ELK/elasticsearch-7.11.1/logs/gc.log
  fields:
    log_topics: "elasticsearch"
  fields_under_root: true
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.50.116:9200"]
  indices:
    - index: "test-log-%{+yyyy.MM.dd}"    #指定的索引名称
    
# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
   hosts: ["192.168.255.192:5044"]

启动filebeat：

[root@ergou filebeat-7.11.2-linux-x86_64]# nohup ./filebeat -e -c filebeat.yml -d "publish" &

ELK的安装之Logstash安装:

[root@ergou dtstack]# tar zxvf  logstash-7.11.2-linux-x86_64.tar.gz

找到/config目录下的logstash-sample.conf文件，修改配置：

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
file{
path => ['/var/log/nginx/*.log']
type => 'nginx_log'
start_position => "beginning"
    }
}
output {
elasticsearch {
hosts => ["http://192.168.255.192:9200"]
index => "user-%{+YYYY.MM.dd}"
       }
}

启动Logstash(root用户)

nohup ./bin/logstash -f /data/dtstack/logstash-7.11.2/config/logstash-sample.conf &

input表示输入源，output表示输出，还可以配置filter过滤，架构如下：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-EvcsUH7M-1618543273141)(C:\Users\Administrator\Pictures\elk\logstash架构图.webp)]

ELK的安装之Kibana安装：

[root@ergou dtstack]# tar -zxvf kibana-7.11.2-linux-x86_64.tar.gz
[root@ergou dtstack]# chown -R elk:elk /data/dtstack/kibana-7.11.2-linux-x86_64
[root@ergou dtstack]# cd kibana-7.11.2-linux-x86_64/config/
[root@ergou config]# vim kibana.yml
# Kibana is served by a back end server. This setting specifies the port to use.
  server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
  server.host: "192.168.255.192"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
  elasticsearch.hosts: ["http://192.168.255.192:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid

# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

启动Kibana(用普通用户elk)

[root@ergou config]# su elk
[elk@ergou config]$ cd /data/dtstack/kibana-7.11.2-linux-x86_64
[elk@ergou kibana-7.11.2-linux-x86_64]$ ls
bin     data         node          nohup.out   package.json  README.txt  x-pack
config  LICENSE.txt  node_modules  NOTICE.txt  plugins       src
[elk@ergou kibana-7.11.2-linux-x86_64]$ nohup ./bin/kibana &
[elk@ergou ~]$ ss -tnl
State      Recv-Q Send-Q        Local Address:Port                       Peer Address:Port
LISTEN     0      128                       *:22                                    *:*
LISTEN     0      100               127.0.0.1:25                                    *:*
LISTEN     0      128         192.168.255.192:5601                                  *:*
LISTEN     0      128                      :::9200                                 :::*
LISTEN     0      128                      :::9300                                 :::*
LISTEN     0      128                      :::22                                   :::*
LISTEN     0      100                     ::1:25                                   :::*