05. 搭建ELK实现项目日志可视化

一 搭建ELK

下载ES

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.tar.gz
tar -xvf elasticsearch-6.3.2.tar.gz -C /usr/app/ELK/

配置

vi config/elasticsearch.yml

network.host: 0.0.0.0 # 设置为四个0即外部所有主机都可以访问，如果不设置默认为只有本机可访问
http.port: 9200 # 占用的本机的端口号

开启ES

./bin/elasticsearch -d

可能遇到的问题

问题1：
can not run elasticsearch as root # 不允许root用户登录
1. 新建用户
adduser es
passws es
chown -R es:es elasticsearch-6.3.2/ # 授权
2. 切换到新用户开启
su es

问题2：
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
解决方法：
#切换到root用户修改
vim /etc/security/limits.conf
# 在最后面追加下面内容
es hard nofile 65536
es soft nofile 65536
修改后重新登录 feiyu用户，使用如下命令查看是否修改成功
ulimit -Hn 65536
 
问题3：
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决方法 提高vm.max_map_count 的大小
# 切换到root用户
vim /etc/sysctl.conf
# 在最后面追加下面内容
vm.max_map_count=262144
# 使用 sysctl -p 查看修改后的结果
sysctl -p
 
问题4：
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error='Cannot allocate memory' (errno=12)
解决方法：
# 由于elasticsearch5.0默认分配jvm空间大小为2g，修改jvm空间分配
# 如果使用虚拟机安装，内存最好不小于2G
# vim config/jvm.options  
-Xms512m
-Xmx512m

二 搭建Logstash

参考文档：

# Logstash的type用法https://www.orchome.com/486

# codec的编码如何选择：https://blog.csdn.net/weixin_33862993/article/details/92048560

# 如何设置multiline: https://blog.csdn.net/wang_zhenwei/article/details/78705230

# 简单示例： https://blog.csdn.net/qq_33371766/article/details/103841183

# 安装ELK遇见index_not_found_exception异常处理：https://www.jianshu.com/p/0f69b0f92918

# Logstash配置总结： https://blog.csdn.net/xfg0218/article/details/52980726

# Beats：使用 Filebeat 传送多行日志 (21条消息) Beats：使用 Filebeat 传送多行日志_Elastic 中国社区官方博客的博客-CSDN博客

# Logstash学习22_Logstash的multiline 插件：匹配多行日志:(21条消息) Logstash学习22_Logstash的multiline 插件：匹配多行日志_Wang_Zhenwei的博客-CSDN博客

# logstash-2.3.1按日产生索引(%{+YYYY.MM.dd})产生时间比预计晚8小时问题：https://blog.csdn.net/arlendai/article/details/84883584

安装

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz
tar -zxvf logstash-6.3.2.tar.gz -C /usr/local/logstash-6.3.2
cd /usr/local/logstash-6.3.2/bin

启动

./logstash -f stdin.conf # 前台启动，新建配置文件stdin.conf，使用该配置

nohup ./logstash -f stdin.cong > logstashlog.file 2>&1 & # 后台启动，启动日志放在当前目录下的logstashlog.file中，否则默认放在系统的哪个犄角哈喇

简单单行配置

input {
  file {
      path =>"/usr/app/ELK/file/erp.log"
      start_position => "beginning"
      type=>"type1"  #类型名称
  }
}
 
output {
  elasticsearch {
      hosts => ["localhost:9200"]
      index => "index1"     #索引名称
    }
  stdout { codec => json_lines }
}

三 搭建Kabana

安装

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz
tar xf kibana-6.3.2-linux-x86_64.tar.gz

配置

cd /usr/app/ELK/kibana-6.3.2-linux-x86_64/bin
vim ./config/kibana.yml
elasticsearch.url: ""  # 只需要修改URL为ElasticSearch的IP地址

四 Logstash配置

简单单行配置

input {
  file {
      path =>"/usr/app/ELK/file/erp.log"
      start_position => "beginning"
      type=>"type1"  #类型名称
  }
}
 
output {
  elasticsearch {
      hosts => ["localhost:9200"]
      index => "index1"     #索引名称
    }
  stdout { codec => json_lines }
}

读取多个数据源

input { 
    stdin { } 
	file {
		path => "/opt/xinyue/logs/erp_alert.log"
		start_position => "beginning" 
		type => "xinyue_dev_log"
		}
	file {
		path => "/opt/uat/logs/erp_alert.log"
		start_position => "beginning" 
		type => "xinyue_uat_log"
		}
}

output {
    if[type] == "xinyue_dev_log"{
            elasticsearch {
                hosts  => "localhost:9200"
                index => "xinyue_dev_log"
        }
    }
    if[type] == "xinyue_uat_log"{
            elasticsearch {
                hosts  => "localhost:9200"
                index => "xinyue_uat_log"
        }
    }
	stdout { 
		codec => json_lines
    }
}

读取多行日志

input{
        file {
		path => "/usr/app/ELK/file/erp.log"  #日志地址
		codec => multiline {
			pattern => "^\d{4}-\d{1,2}-\d{1,2}"
			negate => true
			what => previous
		}
		start_position=>"beginning"
        }
}
output{
	 elasticsearch {
		hosts => ["120.26.63.19:9200"]   #elasticsearch地址
		index => "test1"   #索引
	}
}

多数据源+读取多行日志文件

input { 
    stdin { } 
	file {
		path => "/opt/xinyue/logs/erp_alert.log"
		start_position => "beginning" 
		type => "xinyue_dev_log"
		codec => multiline {
				pattern => "^\d{4}-\d{1,2}-\d{1,2}"
				negate => true
				what => previous
			}
		}
	file {
		path => "/opt/uat/logs/erp_alert.log"
		start_position => "beginning" 
		type => "xinyue_uat_log"
		codec => multiline {
				pattern => "^\d{4}-\d{1,2}-\d{1,2}"
				negate => true
				what => previous
			}
		}
}

output {
    if[type] == "xinyue_dev_log"{
            elasticsearch {
                hosts  => "localhost:9200"
                index => "xinyue_dev_log_%{+YYYYMMdd}"
        }
    }
    if[type] == "xinyue_uat_log"{
            elasticsearch {
                hosts  => "localhost:9200"
                index => "xinyue_uat_log_%{+YYYYMMdd}"
        }
    }
	stdout { 
		codec => json_lines
    }
}