ELK日志框架搭建及实现笔记

最新推荐文章于 2024-09-19 17:30:06 发布

jackaroo2020

最新推荐文章于 2024-09-19 17:30:06 发布

阅读量1k

点赞数

分类专栏：后端工程师

本文链接：https://blog.csdn.net/qq3399013670/article/details/82425475

版权

后端工程师专栏收录该内容

103 篇文章 1 订阅

订阅专栏

logstash配置及启动

 进入到安装目录logstash-6.0.0/bin下，测试是否安装成功
./logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'

临时启动：
./logstash -f ../config/logstash.conf
永久启动：
nohup ./logstash -f ../config/logstash.conf &

示例：多文件读入到ElasticSearch中，logstash.conf配置如下

# 日志输入来源定义
input{
     file{
         path=>[
            "/opt/app/test_log/log/ae.log"
         ]
     start_position => "beginning"
         type => "ae.log"
    }
}

input{
     file{
         path=>[
            "/opt/app/test_log/log/pay.log"
         ]
     start_position => "beginning"
         type => "pay.log"
    }
}
# 日志过滤定义
filter {  
       # 将日志中的时间替换为@timestamp
       grok {
           match => ["message", "%{TIMESTAMP_ISO8601:logdate}"] 
        }   

      date {
           match => ["logdate", "yyyy-MM-dd HH:mm:ss,SSS"]
           target => "@timestamp"
      }

       # 去除冗余字段
       mutate {
          remove_field => "_id"
          remove_field => "@version"
          remove_field => "host"
          remove_field => "_score"
       }  
}


# 日志输出到elasticsearch定义
output{

     elasticsearch {
        action => "index"          
    hosts  => "ip:9200" 
    index  => "logstash-%{type}-%{+YYYY.MM.dd}"
    codec => "json"
    }

# dubug
stdout { codec => rubydebug }

 }

ElasticSearch 安装启动说明

问题一：ERROR: bootstrap checks failed

max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536] 
max number of threads [1024] for user [lishang] likely too low, increase to at least [2048]

解决：切换到root用户，编辑limits.conf 添加类似如下内容

vi /etc/security/limits.conf

添加如下内容:

soft nofile 65536

hard nofile 131072

soft nproc 2048

hard nproc 4096
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
原因：
这是在因为Centos6不支持SecComp，而ES5.2.0默认bootstrap.system_call_filter为true进行检测，所以导致检测失败，失败后直接导致ES不能启动。

解决：
在elasticsearch.yml中配置bootstrap.system_call_filter为false，注意要在Memory下面:
bootstrap.memory_lock: false
bootstrap.system_call_filter: false

问题二：max number of threads [1024] for user [lish] likely too low, increase to at least [2048]
解决：切换到root用户，进入limits.d目录下修改配置文件。

vi /etc/security/limits.d/90-nproc.conf

修改如下内容：

soft nproc 1024
修改为
soft nproc 2048

问题三：max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

解决：切换到root用户修改配置sysctl.conf

vi /etc/sysctl.conf

添加下面配置：

vm.max_map_count=655360

并执行命令：

sysctl -p

然后，重新启动elasticsearch，即可启动成功。

问题四：我们在使用elasticsearch的时候，如果是以root权限来执行elasticsearch

Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:94)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:160)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
1

这是出于系统安全考虑设置的条件。由于ElasticSearch可以接收用户输入的脚本并且执行，为了系统安全考虑，
建议创建一个单独的用户用来运行ElasticSearch

创建elsearch用户组及elsearch用户

groupadd elsearch
useradd elsearch -g elsearch -p elasticsearch

更改elasticsearch文件夹及内部文件的所属用户及组为elsearch:elsearch

chown -R elsearch:elsearch  elasticsearch

elasticsearch为你elasticsearch的目录名称,切换到elsearch用户再启动即可。

nohup ./elasticsearch& 或
nohup ./elasticsearch>/dev/null 2>&1 & 永久启动

elasticsearch.yml 部分说明如下：

# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
# add 
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
# end
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 127.0.0.1
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.

kibana

nohup ./kibana& 或
nohup ./kibana >/dev/null 2>&1 & 永久启动

kibana.yml 部分说明

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "127.0.0.1" #(访问ip)

# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
# the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
# to Kibana. This setting cannot end in a slash.
#server.basePath: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://127.0.0.1:9200" #(访问ip)

其它知识点

一般最新的elk环境jdk需要1.8版本，下载解压版做相应配置即可。
如:logstash 可以在logstash.lib.sh中加入以下设置：

export JAVA_CMD="/opt/app/elk_manager_log/jdk1.8.0_77/bin"
export JAVA_HOME="/opt/app/elk_manager_log/jdk1.8.0_77/"

Linux】如何关闭某个被占用的端口？

1）查找被占用的端口：

abloume@ubuntu:~$ netstat -tln | grep 8000
tcp        0      0 192.168.2.106:8000      0.0.0.0:*               LISTEN  

2）查看被占用端口的PID：
abloume@ubuntu:~$ sudo lsof -i:8000
COMMAND PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
nginx   850     root    6u  IPv4  15078      0t0  TCP 192.168.2.106:8000 (LISTEN)
nginx   851 www-data    6u  IPv4  15078      0t0  TCP 192.168.2.106:8000 (LISTEN)
nginx   852 www-data    6u  IPv4  15078      0t0  TCP 192.168.2.106:8000 (LISTEN)

3）kill掉该进程
abloume@ubuntu:~$ sudo kill -9 850