大数据学习笔记第2课（续） 通过filebeat收集nginx访问日志到kafka集群
1、说明
本文是接着我的上一篇博文《大数据学习笔记第2课 Zookeeper & Kafka集群搭建》往下写的，请先阅读上一篇文章，不然会感到比较突兀。
2、环境说明
本次搭建的环境是在华为云上
1、服务器：三台华为云ECS（都是4核8G）
| 计算机名 | 内网IP | 公网IP | 别名 | 说明 |
|---|---|---|---|---|
| ecs-ae8a-0001 | 192.168.0.177 | x.x.x.x | hadoop01 | zookeeper/broker01 |
| ecs-ae8a-0002 | 192.168.0.56 | x.x.x.x | hadoop02 | broker02 |
| ecs-ae8a-0003 | 192.168.0.59 | x.x.x.x | hadoop03 | broker03 |
2、系统版本：CentOS Linux release 7.5.1804 (Core)
3、zookeeper与kafka服务都已正常启动，且已创建了1个topic，值为qingdao
3、Nginx安装与配置
1、在CentOS上安装Nginx非常简单，我们再hadoop01服务器上执行以下命令：
yum install nginx
安装日志如下：
[root@ecs-ae8a-0001 ~]# yum install nginx
Loaded plugins: fastestmirror
Determining fastest mirrors
epel/x86_64/metalink | 8.4 kB 00:00:00
* base: mirrors.huaweicloud.com
* epel: mirrors.bfsu.edu.cn
* extras: mirrors.bfsu.edu.cn
* updates: mirrors.huaweicloud.com
base | 3.6 kB 00:00:00
epel | 4.7 kB 00:00:00
extras | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
(1/7): epel/x86_64/group_gz | 95 kB 00:00:00
(2/7): extras/7/x86_64/primary_db | 222 kB 00:00:00
(3/7): base/7/x86_64/primary_db | 6.1 MB 00:00:00
(4/7): epel/x86_64/updateinfo | 1.0 MB 00:00:00
(5/7): epel/x86_64/primary_db | 6.9 MB 00:00:00
(6/7): base/7/x86_64/group_gz | 153 kB 00:00:07
(7/7): updates/7/x86_64/primary_db | 3.7 MB 00:00:08
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.16.1-3.el7 will be installed
--> Processing Dependency: nginx-all-modules = 1:1.16.1-3.el7 for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: nginx-filesystem = 1:1.16.1-3.el7 for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: libcrypto.so.1.1(OPENSSL_1_1_0)(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: libssl.so.1.1(OPENSSL_1_1_0)(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: libssl.so.1.1(OPENSSL_1_1_1)(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: nginx-filesystem for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: redhat-indexhtml for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: libcrypto.so.1.1()(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: libprofiler.so.0()(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: libssl.so.1.1()(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
--> Running transaction check
---> Package centos-indexhtml.noarch 0:7-9.el7.centos will be installed
---> Package gperftools-libs.x86_64 0:2.6.1-1.el7 will be installed
---> Package nginx-all-modules.noarch 1:1.16.1-3.el7 will be installed
--> Processing Dependency: nginx-mod-http-image-filter = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
--> Processing Dependency: nginx-mod-http-perl = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
--> Processing Dependency: nginx-mod-http-xslt-filter = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
--> Processing Dependency: nginx-mod-mail = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
--> Processing Dependency: nginx-mod-stream = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
---> Package nginx-filesystem.noarch 1:1.16.1-3.el7 will be installed
---> Package openssl11-libs.x86_64 1:1.1.1g-1.el7 will be installed
--> Running transaction check
---> Package nginx-mod-http-image-filter.x86_64 1:1.16.1-3.el7 will be installed
--> Processing Dependency: gd for package: 1:nginx-mod-http-image-filter-1.16.1-3.el7.x86_64
--> Processing Dependency: libgd.so.2()(64bit) for package: 1:nginx-mod-http-image-filter-1.16.1-3.el7.x86_64
---> Package nginx-mod-http-perl.x86_64 1:1.16.1-3.el7 will be installed
---> Package nginx-mod-http-xslt-filter.x86_64 1:1.16.1-3.el7 will be installed
--> Processing Dependency: libxslt.so.1(LIBXML2_1.0.11)(64bit) for package: 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64
--> Processing Dependency: libxslt.so.1(LIBXML2_1.0.18)(64bit) for package: 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64
--> Processing Dependency: libexslt.so.0()(64bit) for package: 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64
--> Processing Dependency: libxslt.so.1()(64bit) for package: 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64
---> Package nginx-mod-mail.x86_64 1:1.16.1-3.el7 will be installed
---> Package nginx-mod-stream.x86_64 1:1.16.1-3.el7 will be installed
--> Running transaction check
---> Package gd.x86_64 0:2.0.35-26.el7 will be installed
--> Processing Dependency: libjpeg.so.62(LIBJPEG_6.2)(64bit) for package: gd-2.0.35-26.el7.x86_64
--> Processing Dependency: libjpeg.so.62()(64bit) for package: gd-2.0.35-26.el7.x86_64
--> Processing Dependency: libfontconfig.so.1()(64bit) for package: gd-2.0.35-26.el7.x86_64
--> Processing Dependency: libXpm.so.4()(64bit) for package: gd-2.0.35-26.el7.x86_64
--> Processing Dependency: libX11.so.6()(64bit) for package: gd-2.0.35-26.el7.x86_64
---> Package libxslt.x86_64 0:1.1.28-6.el7 will be installed
--> Running transaction check
---> Package fontconfig.x86_64 0:2.13.0-4.3.el7 will be installed
--> Processing Dependency: fontpackages-filesystem for package: fontconfig-2.13.0-4.3.el7.x86_64
--> Processing Dependency: dejavu-sans-fonts for package: fontconfig-2.13.0-4.3.el7.x86_64
---> Package libX11.x86_64 0:1.6.7-3.el7_9 will be installed
--> Processing Dependency: libX11-common >= 1.6.7-3.el7_9 for package: libX11-1.6.7-3.el7_9.x86_64
--> Processing Dependency: libxcb.so.1()(64bit) for package: libX11-1.6.7-3.el7_9.x86_64
---> Package libXpm.x86_64 0:3.5.12-1.el7 will be installed
---> Package libjpeg-turbo.x86_64 0:1.2.90-8.el7 will be installed
--> Running transaction check
---> Package dejavu-sans-fonts.noarch 0:2.33-6.el7 will be installed
--> Processing Dependency: dejavu-fonts-common = 2.33-6.el7 for package: dejavu-sans-fonts-2.33-6.el7.noarch
---> Package fontpackages-filesystem.noarch 0:1.44-8.el7 will be installed
---> Package libX11-common.noarch 0:1.6.7-3.el7_9 will be installed
---> Package libxcb.x86_64 0:1.13-1.el7 will be installed
--> Processing Dependency: libXau.so.6()(64bit) for package: libxcb-1.13-1.el7.x86_64
--> Running transaction check
---> Package dejavu-fonts-common.noarch 0:2.33-6.el7 will be installed
---> Package libXau.x86_64 0:1.0.8-2.1.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==============================================================================================================================================================================================================================================================================
Package Arch Version Repository Size
==============================================================================================================================================================================================================================================================================
Installing:
nginx x86_64 1:1.16.1-3.el7 epel 563 k
Installing for dependencies:
centos-indexhtml noarch 7-9.el7.centos base 92 k
dejavu-fonts-common noarch 2.33-6.el7 base 64 k
dejavu-sans-fonts noarch 2.33-6.el7 base 1.4 M
fontconfig x86_64 2.13.0-4.3.el7 base 254 k
fontpackages-filesystem noarch 1.44-8.el7 base 9.9 k
gd x86_64 2.0.35-26.el7 base 146 k
gperftools-libs x86_64 2.6.1-1.el7 base 272 k
libX11 x86_64 1.6.7-3.el7_9 updates 607 k
libX11-common noarch 1.6.7-3.el7_9 updates 164 k
libXau x86_64 1.0.8-2.1.el7 base 29 k
libXpm x86_64 3.5.12-1.el7 base 55 k
libjpeg-turbo x86_64 1.2.90-8.el7 base 135 k
libxcb x86_64 1.13-1.el7 base 214 k
libxslt x86_64 1.1.28-6.el7 base 242 k
nginx-all-modules noarch 1:1.16.1-3.el7 epel 20 k
nginx-filesystem noarch 1:1.16.1-3.el7 epel 21 k
nginx-mod-http-image-filter x86_64 1:1.16.1-3.el7 epel 30 k
nginx-mod-http-perl x86_64 1:1.16.1-3.el7 epel 39 k
nginx-mod-http-xslt-filter x86_64 1:1.16.1-3.el7 epel 29 k
nginx-mod-mail x86_64 1:1.16.1-3.el7 epel 57 k
nginx-mod-stream x86_64 1:1.16.1-3.el7 epel 85 k
openssl11-libs x86_64 1:1.1.1g-1.el7 epel 1.4 M
Transaction Summary
==============================================================================================================================================================================================================================================================================
Install 1 Package (+22 Dependent packages)
Total download size: 5.9 M
Installed size: 18 M
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/base/packages/centos-indexhtml-7-9.el7.centos.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for centos-indexhtml-7-9.el7.centos.noarch.rpm is not installed
(1/23): centos-indexhtml-7-9.el7.centos.noarch.rpm | 92 kB 00:00:00
(2/23): dejavu-sans-fonts-2.33-6.el7.noarch.rpm | 1.4 MB 00:00:00
(3/23): fontconfig-2.13.0-4.3.el7.x86_64.rpm | 254 kB 00:00:00
(4/23): fontpackages-filesystem-1.44-8.el7.noarch.rpm | 9.9 kB 00:00:00
(5/23): gd-2.0.35-26.el7.x86_64.rpm | 146 kB 00:00:00
(6/23): gperftools-libs-2.6.1-1.el7.x86_64.rpm | 272 kB 00:00:00
(7/23): libXau-1.0.8-2.1.el7.x86_64.rpm | 29 kB 00:00:00
(8/23): libjpeg-turbo-1.2.90-8.el7.x86_64.rpm | 135 kB 00:00:00
(9/23): dejavu-fonts-common-2.33-6.el7.noarch.rpm | 64 kB 00:00:00
(10/23): libxcb-1.13-1.el7.x86_64.rpm | 214 kB 00:00:00
warning: /var/cache/yum/x86_64/7/epel/packages/nginx-1.16.1-3.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Public key for nginx-1.16.1-3.el7.x86_64.rpm is not installed
(11/23): nginx-1.16.1-3.el7.x86_64.rpm | 563 kB 00:00:00
(12/23): nginx-all-modules-1.16.1-3.el7.noarch.rpm | 20 kB 00:00:00
(13/23): nginx-filesystem-1.16.1-3.el7.noarch.rpm | 21 kB 00:00:00
(14/23): nginx-mod-http-image-filter-1.16.1-3.el7.x86_64.rpm | 30 kB 00:00:00
(15/23): nginx-mod-http-perl-1.16.1-3.el7.x86_64.rpm | 39 kB 00:00:00
(16/23): nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64.rpm | 29 kB 00:00:00
(17/23): nginx-mod-mail-1.16.1-3.el7.x86_64.rpm | 57 kB 00:00:00
(18/23): nginx-mod-stream-1.16.1-3.el7.x86_64.rpm | 85 kB 00:00:00
(19/23): libxslt-1.1.28-6.el7.x86_64.rpm | 242 kB 00:00:00
(20/23): openssl11-libs-1.1.1g-1.el7.x86_64.rpm | 1.4 MB 00:00:00
(21/23): libXpm-3.5.12-1.el7.x86_64.rpm | 55 kB 00:00:00
Public key for libX11-common-1.6.7-3.el7_9.noarch.rpm is not installed
(22/23): libX11-common-1.6.7-3.el7_9.noarch.rpm | 164 kB 00:00:00
(23/23): libX11-1.6.7-3.el7_9.x86_64.rpm | 607 kB 00:00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 14 MB/s | 5.9 MB 00:00:00
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
Userid : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
Package : centos-release-7-5.1804.el7.centos.x86_64 (@anaconda)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Is this ok [y/N]: y
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Importing GPG key 0x352C64E5:
Userid : "Fedora EPEL (7) <epel@fedoraproject.org>"
Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
Package : epel-release-7-12.noarch (@epel)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
Installing : fontpackages-filesystem-1.44-8.el7.noarch 1/23
Installing : dejavu-fonts-common-2.33-6.el7.noarch 2/23
Installing : dejavu-sans-fonts-2.33-6.el7.noarch 3/23
Installing : fontconfig-2.13.0-4.3.el7.x86_64 4/23
Installing : gperftools-libs-2.6.1-1.el7.x86_64 5/23
Installing : 1:nginx-filesystem-1.16.1-3.el7.noarch 6/23
Installing : libxslt-1.1.28-6.el7.x86_64 7/23
Installing : libXau-1.0.8-2.1.el7.x86_64 8/23
Installing : libxcb-1.13-1.el7.x86_64 9/23
Installing : 1:openssl11-libs-1.1.1g-1.el7.x86_64 10/23
Installing : centos-indexhtml-7-9.el7.centos.noarch 11/23
Installing : libX11-common-1.6.7-3.el7_9.noarch 12/23
Installing : libX11-1.6.7-3.el7_9.x86_64 13/23
Installing : libXpm-3.5.12-1.el7.x86_64 14/23
Installing : libjpeg-turbo-1.2.90-8.el7.x86_64 15/23
Installing : gd-2.0.35-26.el7.x86_64 16/23
Installing : 1:nginx-mod-mail-1.16.1-3.el7.x86_64 17/23
Installing : 1:nginx-mod-http-perl-1.16.1-3.el7.x86_64 18/23
Installing : 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64 19/23
Installing : 1:nginx-mod-stream-1.16.1-3.el7.x86_64 20/23
Installing : 1:nginx-1.16.1-3.el7.x86_64 21/23
Installing : 1:nginx-mod-http-image-filter-1.16.1-3.el7.x86_64 22/23
Installing : 1:nginx-all-modules-1.16.1-3.el7.noarch 23/23
Verifying : fontconfig-2.13.0-4.3.el7.x86_64 1/23
Verifying : 1:nginx-mod-mail-1.16.1-3.el7.x86_64 2/23
Verifying : 1:nginx-1.16.1-3.el7.x86_64 3/23
Verifying : fontpackages-filesystem-1.44-8.el7.noarch 4/23
Verifying : dejavu-fonts-common-2.33-6.el7.noarch 5/23
Verifying : 1:nginx-mod-http-perl-1.16.1-3.el7.x86_64 6/23
Verifying : libxcb-1.13-1.el7.x86_64 7/23
Verifying : libXpm-3.5.12-1.el7.x86_64 8/23
Verifying : libjpeg-turbo-1.2.90-8.el7.x86_64 9/23
Verifying : libX11-common-1.6.7-3.el7_9.noarch 10/23
Verifying : libX11-1.6.7-3.el7_9.x86_64 11/23
Verifying : 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64 12/23
Verifying : dejavu-sans-fonts-2.33-6.el7.noarch 13/23
Verifying : gd-2.0.35-26.el7.x86_64 14/23
Verifying : 1:nginx-mod-stream-1.16.1-3.el7.x86_64 15/23
Verifying : centos-indexhtml-7-9.el7.centos.noarch 16/23
Verifying : 1:openssl11-libs-1.1.1g-1.el7.x86_64 17/23
Verifying : libXau-1.0.8-2.1.el7.x86_64 18/23
Verifying : libxslt-1.1.28-6.el7.x86_64 19/23
Verifying : 1:nginx-filesystem-1.16.1-3.el7.noarch 20/23
Verifying : gperftools-libs-2.6.1-1.el7.x86_64 21/23
Verifying : 1:nginx-all-modules-1.16.1-3.el7.noarch 22/23
Verifying : 1:nginx-mod-http-image-filter-1.16.1-3.el7.x86_64 23/23
Installed:
nginx.x86_64 1:1.16.1-3.el7
Dependency Installed:
centos-indexhtml.noarch 0:7-9.el7.centos dejavu-fonts-common.noarch 0:2.33-6.el7 dejavu-sans-fonts.noarch 0:2.33-6.el7 fontconfig.x86_64 0:2.13.0-4.3.el7 fontpackages-filesystem.noarch 0:1.44-8.el7 gd.x86_64 0:2.0.35-26.el7
gperftools-libs.x86_64 0:2.6.1-1.el7 libX11.x86_64 0:1.6.7-3.el7_9 libX11-common.noarch 0:1.6.7-3.el7_9 libXau.x86_64 0:1.0.8-2.1.el7 libXpm.x86_64 0:3.5.12-1.el7 libjpeg-turbo.x86_64 0:1.2.90-8.el7
libxcb.x86_64 0:1.13-1.el7 libxslt.x86_64 0:1.1.28-6.el7 nginx-all-modules.noarch 1:1.16.1-3.el7 nginx-filesystem.noarch 1:1.16.1-3.el7 nginx-mod-http-image-filter.x86_64 1:1.16.1-3.el7 nginx-mod-http-perl.x86_64 1:1.16.1-3.el7
nginx-mod-http-xslt-filter.x86_64 1:1.16.1-3.el7 nginx-mod-mail.x86_64 1:1.16.1-3.el7 nginx-mod-stream.x86_64 1:1.16.1-3.el7 openssl11-libs.x86_64 1:1.1.1g-1.el7
Complete!
[root@ecs-ae8a-0001 ~]# which nginx
/usr/sbin/nginx
中间有几步Is this OK?的提问的地方输入y就行。
2、安装完成后
nginx日志目录为：/var/log/nginx
nginx配置文件目录为：/etc/nginx
3、修改/etc/nginx/nginx.conf，内容如下：
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main escape=none '$http_x_forwarded_for -[$time_local] "$request" '
'$status $body_bytes_sent '
'"#http_user_agent"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /usr/share/nginx/html;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
proxy_pass http://127.0.0.1:5000;
}
error_page 404 /404.html;
location = /404.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
}
上面的配置主要修改了2个内容
- 把80端口接收到的请求转发到5000端口下处理。
- 对访问日志的格式进行格式化处理。
4、启动nginx服务，在终端执行以下命令：
nginx
5、如果中间手动删除了/var/log/nginx目录下的日志文件，则需要让Nginx重新加载一下，否则不会自动生成新的日志文件，如下：
killall -s USR1 nginx
4、启动一个Mini Http Server处理5000端口的请求
1、为了能够处理Nginx转发至5000端口的请求，我们可以通过python开启一个httpserver，如下命令：
nohup phython -m SimpleHTTPServer 5000 &
5、filebeat下载、配置与安装
1、切换当前目录为/opt/soft
cd /opt/soft
2、执行以下命令下载filebeat，如下：
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86_64.tar.gz
3、解压filebeat
tar -xzvf filebeat-6.2.4-linux-x86_64.tar.gz
4、复制filebeat至/program/目录下：
cp -r filebeat-6.2.4-linux-x86_64 /program/
5、切换当前目录至/program/filebeat-6.2.4-linux-x86_64/
cd /program/filebeat-6.2.4-linux-x86_64
6、创建一个空文件kafka.yml
touch kafka.yml
7、通过vim kafka.yml修改kafka.yml文件内容，内容如下：
filebeat:
prospectors:
-
paths:
- /var/log/nginx/access.log
input_type: log
output:
kafka:
hosts: ["hadoop01:9092","hadoop02:9092","hadoop03:9092"]
topic: qingdao
6、启动filebeat开启日志收集
执行以下命令
nohup ./filebeat -e -c kafka.yml &
7、通过java程序发送http请求至nginx，使nginx产生访问日志。
1、运行test01的模拟java程序。
2、在浏览器地址栏输入访问地址，访问格式如下：
http:// 启 动 服 务 的 i p 及 端 口 / t e s t / m o c k d a t a / s t a r t ? u r l = h t t p : / / {启动服务的ip及端口}/test/mockdata/start?url=http:// 启动服务的ip及端口/test/mockdata/start?url=http://{nginx的ip}&thread=1
8、通过java程序模拟kafka消息消费
1、运行kafka消费者程序（完整代码），执行效果如下：
9、结尾
至此使用filebeat收集日志数据至kafka集群的环境就搭建完成了，小伙伴们如果觉得有用就点个赞吧~
扫一扫
616
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
