Elk日志服务器群集部署

实验名称：	Elk日志服务器群集部署	实验人	XX	日期	2021.7.22
实验目的：	完成elk日志服务器部署
实验环境：	操作系统：CentOS7.3
实验步骤：	运行实验环境 两台centos7，所需安装包 部署elasticsearch群集 (一)部署主节点es 第一台群集节点配置 [root@localhost ~]# vim /etc/hostname elk-n1.wqm.org [root@localhost ~]# vim /etc/hosts 172.16.12.11 elk-n1 elk-n1.wqm.org 172.16.12.12 elk-n2 elk-n2.wqm.org [root@localhost ~]# vim /etc/selinux/config //#关闭SELinux SELINUX=permissive [root@localhost ~]# vim /etc/security/limits.conf    //修改系统限制 * soft nofile 65536         //单用户可打开的文件最大数(软限) * hard nofile 131072        //单用户可打开的文件最大数(硬限) * soft nproc 4096       //单用户可用的最大进程数(软限) * hard nproc 8192       //单用户可用的最大进程数(硬限) * soft memlock unlimited        //最大锁定内存地址空间(kb)（软限） * hard memlock unlimited        //最大锁定内存地址空间(kb)（硬限） [root@localhost ~]# reboot      //重启 2、安装elasticsearch及其依赖环境 #安装包下载地址： https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/yum/7.6.0/ #安装java环境(java环境必须是1.8版本以上的) [root@elk-n1 ~]# tar zxvf jdk-8u161-linux-x64.tar.gz [root@elk-n1 ~]# mv jdk1.8.0_161/ /usr/local/jdk1.8.0 [root@elk-n1 ~]# vim /etc/profile export JAVA_HOME=/usr/local/jdk1.8.0 export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export PATH=$JAVA_HOME/bin:$PATH [root@elk-n1 ~]# /etc/profile [root@elk-n1 ~]# java -version java version "1.8.0_161" Java(TM) SE Runtime Environment (build 1.8.0_161-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode) [root@elk-n1 ~]# rpm -ivh elasticsearch-7.6.0-x86_64.rpm        //安装elasticsearch 3、部署elasticsearch [root@elk-n1 ~]# mkdir -p /var/es-data //创建elasticsearch data的存放目录 [root@elk-n1 ~]# chown -R elasticsearch:elasticsearch /var/es-data      //修改该目录的属主属组 [root@elk-n1 ~]# chown -R elasticsearch:elasticsearch /var/log/elasticsearch/    //修改elasticsearch的日志属主属组 [root@elk-n1 ~]# vim /etc/elasticsearch/elasticsearch.yml           //修改elasticsearch的配置文件 cluster.name: my-elk        //设置集群名称 node.name: elk-n1       //设置节点名称 node.master: true       //主节点 node.data: true         //数据节点 path.data: /var/es-data         //修改data存放的路径 path.logs: /var/log/elasticsearch       //修改logs日志的路径 bootstrap.memory_lock: true         //配置内存使用交换分区 network.host: 0.0.0.0       //监听的网络地址 http.port: 9200         //开启监听的端口 cluster.initial_master_nodes: ["elk-n1"]    //集群初始主节点(7.6版必须配置) discovery.zen.ping.unicast.hosts: ["elk-n1", "elk-n2"]      //节点单播通信 http.cors.enabled: true         //此下两条为新增加的参数， http.cors.allow-origin: "*"         //使head插件可以访问es [root@elk-n1 ~]# systemctl stop firewalld       //关闭防火墙 [root@elk-n1 ~]# vim /usr/lib/systemd/system/elasticsearch.service      //修改elasticsearch的Service配置文件 LimitMEMLOCK=infinity       //添加在[Service]下 [root@elk-n1 ~]# systemctl daemon-reload [root@elk-n1 ~]# systemctl start elasticsearch.service 4、测试是否能进入主页配置 [root@elk-n1 ~]# firefox 127.0.0.1:9200   第二台群集节点配置如上，注意主机名和IP 在elk-n1主机上安装Head插件   1、安装epel源 [root@elk-n1 ~]# yum -y install epel-release #先安装phantomjs [root@elk-n1 ~]# tar jxvf phantomjs-2.1.1-linux-x86_64.tar.bz2 [root@elk-n1 ~]# mv phantomjs-2.1.1-linux-x86_64 /usr/local/phantomjs [root@elk-n1 ~]# ln -s /usr/local/phantomjs/bin/phantomjs /usr/bin #测试一下 [root@elk-n1 ~]# phantomjs --version 2.1.1 安装node。（head插件本质上是一个nodejs工程，因此需要安装node，使用npm来安装依赖的包。） [root@elk-n1 ~]# tar zxvf node-v13.8.0-linux-x64.tar.gz [root@elk-n1 ~]# mv node-v13.8.0-linux-x64/ /usr/local/node13.8 [root@elk-n1 ~]# vim /etc/profile export NODE_HOME=/usr/local/node13.8 export PATH=$PATH:$NODE_HOME/bin [root@elk-n1 ~]# source /etc/profile [root@elk-n1 ~]# yum -y install git #安装git，下载Head插件 [root@elk-n1 ~]# git clone https://github.com/mobz/elasticsearch-head.git  //如果失败，https可以换成git [root@elk-n1 ~]# mv elasticsearch-head/ /opt [root@elk-n1 ~]# cd /opt/elasticsearch-head/ #安装grunt [root@elk-n1 elasticsearch-head]# npm install -g grunt --registry=https://registry.npm.taobao.org [root@elk-n1 elasticsearch-head]# npm install -g cnpm --registry=https://registry.npm.taobao.org [root@elk-n1 elasticsearch-head]# cnpm install #安装插件 3、配置 elasticsearch-head下Gruntfile.js文件 [root@elk-n1 elasticsearch-head]# vim Gruntfile.js connect: {      //修改connect配置节点         server: {                 options: {                         hostname: '172.16.12.11',        //填加主机名                         port: 9100,                         base: '.',                         keepalive: true                 }         } } 4、修改 _site/app.js文件 [root@elk-n1 elasticsearch-head]# vim _site/app.js init: function(parent) {        //查找这个字段 this._super(); this.prefs = services.Preferences.instance();   this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://172.16.12.11:9200";      //修改这里 5、启动head插件服务（后台运行）并验证查看 [root@elk-n1 elasticsearch-head]# grunt server & [root@elk-n1 elasticsearch-head]# firefox 172.16.12.11:9100 安装配置Logstash   1、安装配置logstash（你要采集哪里的日志，就装在哪里） [root@elk-n1 ~]# rpm -ivh logstash-7.6.0.rpm [root@elk-n1 ~]# ln -s /etc/logstash /usr/share/logstash/config [root@elk-n1 ~]# vim /etc/logstash/logstash.yml path.config: /etc/logstash/conf.d/*.conf 2、日志采集配置 [root@elk-n1 ~]# chmod a+r /var/log/messages [root@elk-n1 ~]# vim /etc/logstash/conf.d/system.conf input {             file {                  path => "/var/log/messages"  //收集Linux系统日志                         type => "system"                         start_position => "beginning"             } } output {             elasticsearch {         //输出到elasticsearch                         hosts => ["172.16.12.11:9200"]                         index => "system-%{+YYYY.MM.dd}"             } } 启动Logstash [root@elk-n1 ~]# systemctl enable logstash [root@elk-n1 ~]# systemctl start logstash [root@elk-n1 ~]# /usr/share/logstash/bin/logstash -t 'input{stdin{}}output{stdout{codec=>rubydebug}}' 打开另一个终端验证 [root@elk-n1 ~]# firefox 172.16.12.11:9100​   四、安装kibana #安装 [root@elk-n1 ~]# rpm -ivh kibana-7.6.0-x86_64.rpm #编辑/etc/kibana/kibana.yml [root@elk-n1 ~]# vim /etc/kibana/kibana.yml server.port: 5601 server.host: "0.0.0.0" elasticsearch.hosts: ["http://127.0.0.1:9200"] kibana.index: ".kibana" #启动kibana [root@elk-n1 ~]# systemctl enable kibana [root@elk-n1 ~]# systemctl start kibana 测试是否成功   五、安装配置filebeat #安装 [root@elk-n1 ~]# rpm -ivh filebeat-7.6.0-x86_64.rpm #编辑/etc/filebeat/filebeat.yml [root@elk-n1 ~]# vim /etc/filebeat/filebeat.yml output.elasticsearch:     hosts: ["127.0.0.1:9200"] setup.kibana:     host: "127.0.0.1:5601" [root@elk-n1 ~]# filebeat modules enable elasticsearch [root@elk-n1 ~]# filebeat setup [root@elk-n1 ~]# systemctl start filebeat.service   //启动filebeat 输出apache日志    1、安装启动apache [root@elk-n1 ~]# yum -y install httpd [root@elk-n1 ~]# systemctl enable httpd [root@elk-n1 ~]# systemctl start httpd 2、配置Logstash #编辑配置/etc/logstash/conf.d/apache_access.conf [root@elk-n1 ~]# vim /etc/logstash/conf.d/apache_access.conf input {             file {                         path => "/var/log/httpd/access_log"                         type => "Apache_access"                         start_position => "beginning"             } } output {             elasticsearch {                         action => "index"                         hosts => ["172.16.12.11:9200"]                         index => "apache_access-%{+YYYY.MM.dd}"             } } #编辑配置/etc/logstash/conf.d/apache_error.conf [root@elk-n1 ~]# vim /etc/logstash/conf.d/apache_error.conf input {             file {                         path => "/etc/httpd/logs/error_log"                         type => "Apache_error"                         start_position => "beginning"             } } output {             elasticsearch {                         action => "index"                         hosts => ["172.16.12.11:9200"]                         index => "apache_error-%{+YYYY.MM.dd}"             } } 3、启动logatash [root@elk-n1 ~]# chmod a+r /var/log/httpd/ -R #测试或启动Logstash提前关闭防火墙 [root@elk-n1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/apache_access.conf ... ... [root@elk-n1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/apache_error.conf ... ... [root@elk-n1 ~]# systemctl enable logstash [root@elk-n1 ~]# systemctl start logstash #测试是否启动logstash [root@elk-n1 ~]# Firefox 172.16.12.11:9100       Elk群集节点部署实验成功]
名词解释：
其它资料：