ELK日志分析系统搭建

cd /usr/local/src/

       上传的软件包：

              elasticsearch-7.17.6-x86_64.rpm

              kibana-7.17.6-x86_64.rpm

rpm -ivh elasticsearch-7.17.6-x86_64.rpm

       启动elasticsearch

              sudo systemctl daemon-reload

              sudo systemctl enable elasticsearch.service

   查看elasticsearch所有配置文件的路径

              rpm -qc  elasticsearch

       修改elasticsearch配置文件

       vim /etc/elasticsearch/elasticsearch.yml

              cluster.name: my-elk

              node.name: elk01-10

              path.data: /var/lib/elasticsearch

              path.logs: /var/log/elasticsearch

              bootstrap.memory_lock: false

              network.host: 0.0.0.0

              http.port: 9200

              discovery.seed_hosts: ["elk01-10", "elk02-20"]

              cluster.initial_master_nodes: ["elk01-10", "elk02-20"]

scp /etc/elasticsearch/elasticsearch.yml 192.168.157.20:/etc/elasticsearch/

systemctl start elasticsearch.service

ss -anpt | grep java

访问主机IP地址:9200

部署npm

yum install fontconfig-devel

wget -O /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo

yum -y install  npm nodejs

cd /opt/

指向国内源

git clone https://github.com/mobz/elasticsearch-head.git

cd elasticsearch-head/

换成淘宝镜像

npm config set registry https://registry.npm.taobao.org

查看是否换成淘宝镜像

npm  config get registry -V

安装npm

npm install

cd node_modules/

cd ../

配置elasticsearch主配文件

vim /etc/elasticsearch/elasticsearch.yml

            cluster.name: my-elk

              node.name: elk01-10

              path.data: /var/lib/elasticsearch

              path.logs: /var/log/elasticsearch

              bootstrap.memory_lock: false

              network.host: 0.0.0.0

              http.port: 9200

              discovery.seed_hosts: ["elk01-10", "elk02-20"]

              cluster.initial_master_nodes: ["elk01-10", "elk02-20"]

              http.cors.enabled: true

              http.cors.allow-origin: "*"

启动npm到后台

npm  run start &

访问主机IP地址：9100

在有软件的服务器上部署logstash

部署logstash

rpm -ivh logstash-7.17.6-x86_64.rpm

查看logstash有哪些配置文件

rpm -qc logstash

cd /etc/logstash/conf.d/

查看logstash有哪些命令

rpm -ql logstash  

ln -s /usr/share/logstash/bin/logstash  /usr/local/bin/

curl -XPUT 'http://192.168.157.10:9200/index-demo/test/1?pretty&pretty' -H 'Content-Type: application/json' -d '{"user":"zhangsan","mesg":"helloworld"}'

logstash -e 'input {stdin{}} output {stdout{}}'

使用 rubydebug 显示详细输出，

logstash -e 'input {stdin{}} output {stdout{codec=>rubydebug}}'

使用 logstash 将信息写入到 elasticsearch 中

logstash -e 'input {stdin{}} output {elasticsearch {hosts=>["192.168.157.10:9200"]}}'

测试配置文件

vim system.conf

              input {

                file {

                     path => "/var/log/messages"

                     type => "system"

                     start_position => "beginning"

                }

              }

              output {

                elasticsearch {

                     hosts => ["192.168.157.10:9200"]

                     index => "system-%{+YYYY.MM.dd}"

                }

              }

chmod +r /var/log/messages

chmod +r system.conf

启动logstash

systemctl start logstash.service

监控logstash

tailf /var/log/messages

yum安装nginx

cd /etc/yum.repos.d/

vim nginx.repo

              [nginx-stable]

              name=nginx stable repo

              baseurl=http://nginx.org/packages/centos/$releasever/$basearch/

              gpgcheck=1

              enabled=1

              gpgkey=https://nginx.org/keys/nginx_signing.key

              module_hotfixes=true

安装nginx

yum -y install nginx

如果安装报错解决方法：

vim /etc/yum.conf

              sslverify=false

yum -y install nginx

启动nginx

systemctl  start nginx

查看nginx启动日志

cat /var/log/nginx/access.log

写ngx.conf配置文件

cd /etc/logstash/conf.d/

cp system.conf ngx.conf

vim ngx.conf

       input {

         file {

         path => "/var/log/nginx/access.log"

         type => "nginx"

         start_position => "beginning"

         }

       }

       output {

         elasticsearch {

         hosts => ["192.168.157.10:9200"]

         index => "ngin x-%{+YYYY.MM.dd}"

         }

       }

使用logstash启动ngx.conf

logstash -f ngx.conf

在192.168.157.10主机：

安装phantomjs

cd /tmp/

       上传的软件包：

       phantomjs-2.1.1-linux-x86_64.tar.bz2

解压phantomjs

tar xf phantomjs-2.1.1-linux-x86_64.tar.bz2

cd phantomjs-2.1.1-linux-x86_64

cd bin/

cp phantomjs  /usr/local/bin/

重启elasticsearch

systemctl restart elasticsearch.service

ss -anptl | grep java

cd /opt/

cd elasticsearch-head/

安装kibana

cd /usr/local/src/

       kibana-7.17.6-x86_64.rpm

rpm -ivh kibana-7.17.6-x86_64.rpm

修改kibana主配文件

vim /etc/kibana/kibana.yml

              server.port: 5601

              server.host: "192.168.157.10"

              server.name: "elk01-10"

              elasticsearch.hosts: ["http://192.168.3.10:9200"]

              kibana.index: ".kibana"

              i18n.locale: "zh-CN"

启动kibana

systemctl start kibana.service

ss -anpt | grep 5601

监控kibana日志

tailf /var/log/kibana/kibana.log

ss -anpt | grep 5601

访问192.168.157.10:5601

安装压测工具

yum -y install httpd-tools

压测：

ab -n 100 -c 10 http://192.168.157.20/