文章目录
一、集群环境
服务名称 | IP地址 |
---|---|
elk1 | 192.168.4.115 |
elk2 | 192.168.4.116 |
elk3 | 192.168.4.118 |
cat>> /etc/security/limits.conf<<EOF
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
* hard memlock unlimited
* soft memlock unlimited
* - nofile 65536
EOF
cat >> /etc/systemd/system.conf<<EOF
DefaultLimitNOFILE=65536
DefaultLimitNPROC=32000
DefaultLimitMEMLOCK=infinity
EOF
cat >> /etc/sysctl.conf<<EOF
vm.max_map_count=655360
EOF
sysctl -p
#部署java环境
rpm -qa | grep [java][jdk][gcj]
#查看可安装的jdk
yum search java | grep -i --color jdk
#安装JDK
yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
#查看是否生效
java -version
二、部署es
#下载安装ElasticSearch:
- 【官方地址】https://www.elastic.co/cn/downloads/
- 【历史地址】https://www.elastic.co/cn/downloads/past-releases#elasticsearch
- 【下载地址】https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz
#创建数据目录
mkdir -p /data/app/
#上传
rz
#解压
tar -zxvf elasticsearch-6.2.4.tar.gz -C /data/app/
#然后编辑ES的配置文件:
cd /data/app/elasticsearch-6.2.4/
#授权
groupadd es
useradd es -g es -p es
chown es:es -R /data/app/elasticsearch-6.2.4/
#创建数据存放和日志目录
mkdir -p /data/elasticsearch/data
mkdir -p /data/elasticsearch/logs
chown es:es -R /data/elasticsearch/logs
chown es:es -R /data/elasticsearch/data
1)192.168.4.115
cat >>config/elasticsearch.yml<<EOF
cluster.name: mispsearch #组名(同一个组,组名必须一致)
node.name: node-115 #节点名称(建议和主机名一致)
path.data: /data/elasticsearch/data #数据存放路径
path.logs: /data/elasticsearch/logs #日志存放路径
network.host: 0.0.0.0 #网络设置
http.port: 9200 #端口
transport.tcp.port: 9300 #通信端口
discovery.zen.ping.unicast.hosts: ["192.168.4.115:9300", "192.168.4.116:9300","192.168.4.118:9300"]
discovery.zen.minimum_master_nodes: 3
http.cors.enabled: true
http.cors.allow-origin: "*"
EOF
2)192.168.4.116
cat >>config/elasticsearch.yml<<EOF
cluster.name: mispsearch #组名(同一个组,组名必须一致)
node.name: node-115 #节点名称(建议和主机名一致)
path.data: /data/elasticsearch/data #数据存放路径
path.logs: /data/elasticsearch/logs #日志存放路径
network.host: 0.0.0.0 #网络设置
http.port: 9200 #端口
transport.tcp.port: 9300 #通信端口
discovery.zen.ping.unicast.hosts: ["192.168.4.115:9300", "192.168.4.116:9300","192.168.4.118:9300"]
discovery.zen.minimum_master_nodes: 3
http.cors.enabled: true
http.cors.allow-origin: "*"
EOF
3)192.168.4.118
cat >>config/elasticsearch.yml<<EOF
cluster.name: mispsearch #组名(同一个组,组名必须一致)
node.name: node-118 #节点名称(建议和主机名一致)
path.data: /data/elasticsearch/data #数据存放路径
path.logs: /data/elasticsearch/logs #日志存放路径
network.host: 0.0.0.0 #网络设置
http.port: 9200 #端口
transport.tcp.port: 9300 #通信端口
discovery.zen.ping.unicast.hosts: ["192.168.4.115:9300", "192.168.4.116:9300","192.168.4.118:9300"]
discovery.zen.minimum_master_nodes: 3
http.cors.enabled: true
http.cors.allow-origin: "*"
EOF
4)启动测试
#启动测试
su - es
/data/app/elasticsearch-6.2.4/bin/elasticsearch -d
ps -fe|grep es
netstat -tnlp|grep 9300
[es@k8s-node-01 ~]$ ps -fe|grep es
root 1 0 0 18:40 ? 00:00:33 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
dbus 713 1 0 18:40 ? 00:00:02 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 12494 12455 0 21:31 pts/0 00:00:00 su - es
es 12495 12494 0 21:31 pts/0 00:00:00 -bash
root 12682 12655 0 21:36 pts/0 00:00:00 su - es
es 12683 12682 0 21:36 pts/0 00:00:00 -bash
es 12750 1 6 21:36 pts/0 00:00:17 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.99jg36fl -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/data/app/elasticsearch-6.2.4 -Des.path.conf=/data/app/elasticsearch-6.2.4/config -cp /data/app/elasticsearch-6.2.4/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
es 12797 12683 0 21:41 pts/0 00:00:00 ps -fe
es 12798 12683 0 21:41 pts/0 00:00:00 grep --color=auto es
[es@k8s-node-01 ~]$ netstat -tnlp|grep 9300
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:9300 0.0.0.0:* LISTEN 12750/java
- 可通过IP+端口访问成功页面
- 访问:192.168.4.115:9200
5)加入开启自启动
cat >/etc/init.d/elasticsearch<<EOF
#!/bin/sh
#chkconfig: 2345 80 05
#description: elasticsearch
export JAVA_BIN=/usr/local/jdk1.8.0_171/bin
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME JAVA_BIN PATH CLASSPATH
case "$1" in
start)
su elk<<!
cd /usr/local/elasticsearch-7.3.0/
./bin/elasticsearch -d
!
echo "elasticsearch startup"
;;
stop)
es_pid=`jps | grep Elasticsearch | awk '{print $1}'`
kill -9 $es_pid
echo "elasticsearch stopped"
;;
restart)
es_pid=`jps | grep Elasticsearch | awk '{print $1}'`
kill -9 $es_pid
echo "elasticsearch stopped"
su elk<<!
cd /usr/local/elasticsearch-7.3.0/
./bin/elasticsearch -d
!
echo "elasticsearch startup"
;;
*)
echo "start|stop|restart"
;;
esac
exit $?
6)故障
可能出现的问题
java.lang.RuntimeException: can not run elasticsearch as root
解决办法:登陆其他用户(misp)启动服务
java.nio.file.AccessDeniedException: /mnt/misp/software/elasticsearch-6.1.3/config/elasticsearch.yml
解决办法:登陆root用户执行:chown misp /mnt/misp/software/elasticsearch-6.1.3/ -R
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决办法:vim /etc/sysctl.conf 添加vm.max_map_count=262144,然后执行sysctl -p
max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
解决办法:vim /etc/security/limits.conf修改文件末尾hard nofile 65536 soft nofile 65536
ERROR Unable to locate appender "rolling" for logger config "root"
解决办法:查看数据目录和日志目录是否和配置一致,一致后授权普通用户权限(chown es:es -R /data/elasticsearch/{data,logs})。
三、部署logstash
1)官方地址
#下载地址
- https://www.elastic.co/cn/downloads/past-releases#logstash
- https://www.elastic.co/cn/downloads/past-releases/logstash-6-2-4
- https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.tar.gz
2)解压安装
配置在要输出日志的服务器上:
192.168.4.115、192.168.4.116、192.168.4.118
#下载解压
wget -c https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.tar.gz --no-check-certificate
tar -zxvf logstash-6.2.4.tar.gz -C /data/app/
#编辑logstash的配置文件
在config文件目录下新建配置文件logstash.conf
#创建数据存储
cd /data/app/logstash-6.2.4/
mkdir -p /data/logstash/logs/
3)创建配置文件
#创建logstash.conf
vim /data/app/logstash-6.2.4/config/logstash.conf
input
{
file
{
path =>["/data/logstash/logs/mng-manage-provider-impl-1.0.1-SNAPSHOT*.out"]
start_position => beginning
type => "mng_manage_log"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => true
what => previous
}
}
file
{
path =>["/data/logstash/logs/tsm-manage-provider-impl-1.0.1-SNAPSHOT*.out"]
start_position => beginning
type => "tsm_manage_log"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => true
what => previous
}
}
file
{
path =>["/data/logstash/logs//adapter-bank-provider-impl-1.0.1-SNAPSHOT*.out"]
start_position => beginning
type => "adapter-bank_log"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => true
what => previous
}
}
file
{
path =>["/data/logstash/logs/adapter-school-provider-impl-1.0.1-SNAPSHOT*.out"]
start_position => beginning
type => "adapter-school_log"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => true
what => previous
}
}
}
filter {
grok{
match=>{"message"=>"\s*%{TIMESTAMP_ISO8601:logtime}\s%{LOGLEVEL:loglevel}(?<info>([\s\S]*))"}
}
}
output
{
elasticsearch{
hosts => ["192.168.4.115:9200","192.168.4.116:9200","192.168.4.118:9200"]
index => "log-%{+YYYY-MM-dd}"
}
}
4)配置分析
在logstash中,包括了三个阶段,也就是logstash.conf配置文件中的input{} filter{} output{}:
输入input --> 处理filter(不是必须的) --> 输出output
input{}:
file{}为要监听的文件
Path => 为文件路径
start_position => beginning 表示从文件开始位置
codec => multiline{} 一条日志中如果包含换行,开头不为时间,则归为上一条
filter{}:
grok{}对日志进行拆解
match=> 规则
%{TIMESTAMP_ISO8601:logtime}:日志的时间
%{LOGLEVEL:loglevel}:日志的级别(如:INFO、WARN、ERROR)
(?<info>([\s\S]*)):日志内容
output{}
elasticsearch{} 将日志信息存储到elasticsearch集群
hosts => 集群地址
index => 在ES集群中的索引,可以用来区分不同服务器
5)启动
#加入后台启动
cd /data/app/logstash-6.2.4/bin/
nohup ./logstash -f /data/app/logstash-6.2.4/config/logstash.conf &
#查看
ps -ef|grep logstash
netstat -tnlp|grep 9200
6)常用启动参数
-e:立即执行,使用命令行里的配置参数启动实例。
例如 logstash -e 'input {stdin{}} output {stdout{}}'
-f:指定启动实例的配置文件。
例如logstash -f /usr/local/logstash-7.3.0/config/logstash.conf
-t:测试配置文件的正确性
例如logstash -f /usr/local/logstash-7.3.0/config/logstash.conf -t
7)加入开机启动
#开机加载java环境
echo "source /etc/profile" >> /etc/rc.local
echo "nohup ./logstash -f /data/app/logstash-6.2.4/config/logstash.conf &" >> /etc/rc.local
chmod +x /etc/rc.local
四、Kibana
- 【官方地址】https://www.elastic.co/cn/downloads/past-releases#kibana
- 【下载版本选择】https://www.elastic.co/cn/downloads/past-releases/kibana-6-2-4
- 【下载地址】https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-linux-x86_64.tar.gz
1)192.168.4.115
#一台部署安装Kibana:
wget -c https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-linux-x86_64.tar.gz --no-check-certificate
tar -zxvf kibana-6.2.4-linux-x86_64.tar.gz -C /data/app/
cd /data/app/kibana-6.2.4-linux-x86_64/
2)修改配置
#修改配置文件
vim config/kibana.yml
server.port: 5601 #kibana服务端口号
server.host: "192.168.4.115" #主机IP
elasticsearch.url: "http://192.168.4.115:9200" #elasticsearch.url
kibana.index: ".kibana" #默认索引
3)启动
#启动
./bin/kibana
#后台启动
cd /data/app/kibana-6.2.4-linux-x86_64/bin/
nohup /data/app/kibana-6.2.4-linux-x86_64/bin/kibana &
#./bin/kibana >/dev/null 2>&1 &
ps -ef|grep kibana
netstat -tnlp|grep 5601
4)加入开机启动
#添加开机自启动
[root@elk01 ~]# echo " nohup /data/app/kibana-6.2.4-linux-x86_64/bin/kibana &" >> /etc/rc.local
5)报错
【报错】
FATAL { ValidationError: child "server" fails because [child "host" fails because ["host" must be a valid hostname]]
at Object.exports.process (/data/app/kibana-6.2.4-linux-x86_64/node_modules/joi/lib/errors.js:181:19)
【解决】hostname主机之间存在引号
6)访问
页面访问:http://192.168.4.115:5601/app/kibana
[可参考网址]https://blog.csdn.net/w1206507055/article/details/125297841