ELK 通过 Rsyslog 收集 HaProxy 日志

最新推荐文章于 2024-06-08 07:30:00 发布

LlinCK

最新推荐文章于 2024-06-08 07:30:00 发布

阅读量344

点赞数 1

分类专栏： ELK HAproxy 网络文章标签：运维 linux rsyslog elk haproxy

本文链接：https://blog.csdn.net/qq_42606357/article/details/119975767

版权

ELK 同时被 3 个专栏收录

10 篇文章 1 订阅

订阅专栏

网络

10 篇文章 18 订阅

订阅专栏

HAproxy

7 篇文章 0 订阅

订阅专栏

一、部署 HaProxy

SRC_DIR=/usr/local/src
COLOR="echo -e \\033[01;31m"
END='\033[0m'
CPUS=`lscpu |awk '/^CPU\(s\)/{print $2}'`

LUA_FILE=lua-5.4.3.tar.gz
HAPROXY_FILE=haproxy-2.2.12.tar.gz
HAPROXY_INSTALL_DIR=/apps/haproxy

STATS_AUTH_USER=admin
STATS_AUTH_PASSWORD=123456

NET_NAME=`ip a |awk -F"[: ]" '/^2/{print $3}'`
IP=`ip addr show ${NET_NAME}| awk -F" +|/" '/global/{print $3}'`

os(){
    if grep -Eqi "CentOS" /etc/issue || grep -Eq "CentOS" /etc/*-release;then
        rpm -q redhat-lsb-core &> /dev/null || { ${COLOR}"安装lsb_release工具"${END};yum -y install  redhat-lsb-core &> /dev/null; }
    fi
    OS_ID=`lsb_release -is`
}

check_file (){
    cd ${SRC_DIR}
    ${COLOR}'检查HAPROXY相关源码包'${END}
    if [ ! -e ${LUA_FILE} ];then
        ${COLOR}"缺少${LUA_FILE}文件"${END}
        exit
    elif [ ! -e ${HAPROXY_FILE} ];then
        ${COLOR}"缺少${HAPROXY_FILE}文件"${END}
        exit
    else
        ${COLOR}"相关文件已准备好"${END}
    fi
}

install_haproxy(){
    ${COLOR}"开始安装HAPROXY"${END}
    ${COLOR}"开始安装HAPROXY依赖包"${END}
    if [ ${OS_ID} == "CentOS" ] &> /dev/null;then
        yum -y install gcc make gcc-c++ glibc glibc-devel pcre pcre-devel openssl openssl-devel systemd-devel libtermcap-devel ncurses-devel libevent-devel readline-devel &> /dev/null
    else
        apt update &> /dev/null;apt -y install gcc make openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev  libreadline-dev libsystemd-dev &> /dev/null
    fi
    tar xf ${LUA_FILE}
    LUA_DIR=`echo ${LUA_FILE} | sed -nr 's/^(.*[0-9]).*/\1/p'`
    cd ${LUA_DIR}
    make all test
    cd ${SRC_DIR}
    tar xf ${HAPROXY_FILE}
    HAPROXY_DIR=`echo ${HAPROXY_FILE} | sed -nr 's/^(.*[0-9]).*/\1/p'`
    cd ${HAPROXY_DIR}
    make -j ${CPUS} ARCH=x86_64 TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_CPU_AFFINITY=1 USE_LUA=1 LUA_INC=/usr/local/src/${LUA_DIR}/src/ LUA_LIB=/usr/local/src/${LUA_DIR}/src/ PREFIX=${HAPROXY_INSTALL_DIR}
    make install PREFIX=${HAPROXY_INSTALL_DIR}
    [ $? -eq 0 ] && $COLOR"HAPROXY编译安装成功"$END ||  { $COLOR"HAPROXY编译安装失败,退出!"$END;exit; }
    cat > /lib/systemd/system/haproxy.service <<-EOF
[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target

[Service]
ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/lib/haproxy/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID

[Install]
WantedBy=multi-user.target
EOF
    [ -L /usr/sbin/haproxy ] || ln -s ../../apps/haproxy/sbin/haproxy /usr/sbin/ &> /dev/null
    [ -d /etc/haproxy ] || mkdir /etc/haproxy &> /dev/null  
    [ -d /var/lib/haproxy/ ] || mkdir -p /var/lib/haproxy/ &> /dev/null
    cat > /etc/haproxy/haproxy.cfg <<-EOF
global
maxconn 100000
chroot /apps/haproxy
stats socket /var/lib/haproxy/haproxy.sock mode 600 level admin
uid 99
gid 99
daemon
#nbproc 4
#cpu-map 1 0
#cpu-map 2 1
#cpu-map 3 2
#cpu-map 4 3
pidfile /var/lib/haproxy/haproxy.pid
log 127.0.0.1 local3 info

defaults
option http-keep-alive
option forwardfor
maxconn 100000
mode http
timeout connect 300000ms
timeout client 300000ms
timeout server 300000ms

listen stats
    mode http
    bind 0.0.0.0:9999
    stats enable
    log global
    stats uri /haproxy-status
    stats auth ${STATS_AUTH_USER}:${STATS_AUTH_PASSWORD}

listen web_port
    bind ${IP}:80
    mode http
    log global
server web1 127.0.0.1:8080 check inter 3000 fall 2 rise 5
EOF
    echo "PATH=${HAPROXY_INSTALL_DIR}/sbin:${PATH}" > /etc/profile.d/haproxy.sh
    systemctl daemon-reload
    systemctl enable --now haproxy &> /dev/null
    systemctl is-active haproxy &> /dev/null ||  { ${COLOR}"HAPROXY 启动失败,退出!"${END} ; exit; }
    ${COLOR}"HAPROXY安装完成"${END}
}

main(){
    os
    check_file
    install_haproxy
}

main

二、编辑 HaProxy 配置文件，并启动服务

[root@centos7 src]# vim /etc/haproxy/haproxy.cfg
listen kibana
  bind 10.0.0.20:5601
  server 10.0.0.31 10.0.0.31:5601 check inter 3s fall 3 rise 5
  
[root@centos7 src]# systemctl restart haproxy.service

三、编辑 rsyslog 配置文件，并启动服务

[root@centos7 src]# vim /etc/rsyslog.conf
$ModLoad imudp    # 取消注释
$UDPServerRun 514    # 取消注释
local2.* /var/log/haproxy.log

[root@centos7 src]# systemctl restart rsyslog.service

四、通过负载均衡去访问 kibana ，获取 HaProxy 日志

http://10.0.0.20:5601
在这里插入图片描述

[root@centos7 ~]# cat /var/log/haproxy.log
Aug  7 23:16:36 localhost haproxy[1741]: 10.0.0.1:64283 [07/Aug/2021:23:16:36.292] kibana kibana/10.0.0.31 0/0/0/21/22 200 24165 - - ---- 2/2/0/1/0 0/0 "GET /app/home HTTP/1.1"
Aug  7 23:16:36 localhost haproxy[1741]: 10.0.0.1:64283 [07/Aug/2021:23:16:36.314] kibana kibana/10.0.0.31 10/0/0/12/22 304 353 - - ---- 2/2/0/1/0 0/0 "GET /bootstrap.js HTTP/1.1"
Aug  7 23:16:36 localhost haproxy[1741]: 10.0.0.1:64283 [07/Aug/2021:23:16:36.337] kibana kibana/10.0.0.31 24/0/0/3/27 200 1418 - - ---- 2/2/0/1/0 0/0 "GET /ui/favicons/favicon.svg HTTP/1.1"
Aug  7 23:16:36 localhost haproxy[1741]: 10.0.0.1:64283 [07/Aug/2021:23:16:36.364] kibana kibana/10.0.0.31 17/0/0/2/19 200 5640 - - ---- 2/2/0/1/0 0/0 "GET /ui/favicons/favicon.png HTTP/1.1"
Aug  7 23:16:36 localhost haproxy[1741]: 10.0.0.1:64283 [07/Aug/2021:23:16:36.384] kibana kibana/10.0.0.31 328/0/0/3/331 304 340 - - ---- 3/3/2/3/0 0/0 "GET /translations/zh-cn.json HTTP/1.1"
Aug  7 23:16:36 localhost haproxy[1741]: 10.0.0.1:63647 [07/Aug/2021:23:16:36.292] kibana kibana/10.0.0.31 420/0/0/11/432 200 10310 - - ---- 3/3/1/1/0 0/0 "GET /node_modules/@kbn/ui-framework/dist/kui_light.css HTTP/1.1"

五、编辑 Logstash 配置文件

root@web1:~# vim /etc/logstash/conf.d/rsyslog-to-es.conf
input {
  syslog {
    host => "10.0.0.37"
    port => "1280"
  }
}

output {
  stdout {}
}

六、加载指定配置文件，并验证端口是否起来

# 停止服务
root@web1:~# systemctl stop logstash

# 加载指定配置文件
root@web1:~# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/rsyslog-to-es.conf

# 验证1280端口是否起来

在这里插入图片描述

七、修改 rsyslog 配置文件，向远端传输日志

[root@centos7 ~]# vim /etc/rsyslog.conf
local2.* @@10.0.0.37:1280
[root@centos7 ~]# systemctl restart rsyslog.service

在这里插入图片描述

八、修改配置文件将输出改为 elasticsearch

root@web1:~# vim /etc/logstash/conf.d/rsyslog-to-es.conf
input {
  syslog {
    host => "10.0.0.37"
    port => "1280"
    type => "rsyslog"
  }
}

output {
if [type] == "rsyslog" {
    elasticsearch {
      hosts => ["10.0.0.31:9200"]
      index => "logstash-lck-rsyslog-haproxy-%{+YYYY.MM.dd}"
    }
  }
}

九、检测配置文件语法是否正确

root@web1:~# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/rsyslog-to-es.conf -t

在这里插入图片描述

十、启动服务并验证

# 启动服务
systemctl restart logstash.service

在这里插入图片描述

十一、创建索引方便查询日志

在这里插入图片描述

LlinCK

关注

1
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
ELK 通过 Rsyslog 收集 HaProxy 日志

一、部署 HaProxySRC_DIR=/usr/local/srcCOLOR="echo -e \\033[01;31m"END='\033[0m'CPUS=`lscpu |awk '/^CPU$s$/{print $2}'`LUA_FILE=lua-5.4.3.tar.gzHAPROXY_FILE=haproxy-2.2.12.tar.gzHAPROXY_INSTALL_DIR=/apps/haproxySTATS_AUTH_USER=adminSTATS_AUTH_PASSW
复制链接

扫一扫