分布式日志数据采集
1.NGINX配置:
slave2作为分发节点,master和slave1作为日志采集节点
slave2的NGINX配置如下图:
slave1和master的NGINX配置如下图:
2.slave1和master的flume作为客户端的配置:
flume client端配置文件:flume-client.properties:
# 定义这个agent中各组件的名字
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# set channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 描述和配置source组件:r1
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /usr/local/logserver_study/cgi_demo/logs/client.ERROR
# 描述和配置sink组件:k1
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = master
a1.sinks.k1.port = 52020
# 描述和配置source channel sink之间的连接关系
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
3.master作为flume服务端的配置:
flume server端配置文件:flume-server.properties:
# 定义这个agent中各组件的名字
a1.channels = c1
a1.sources = r1
a1.sinks = k1
# set channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.sources.r1.type = avro
a1.sources.r1.bind = master
a1.sources.r1.port = 52020
a1.sinks.k1.type=hbase
a1.sinks.k1.table=user_action_table
a1.sinks.k1.columnFamily=action_log
a1.sinks.k1.serializer=com.badou.FlumeHbaseEventSerializer
a1.sinks.k1.serializer.columns=userid,itemid,type,ip
# 描述和配置source channel sink之间的连接关系
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
4.Java代码目录如下图:
]# ./bin/flume-ng agent --conf conf --conf-file ./conf/logserver_flume_hbase/flume-client.properties --name a1 -Dflume.root.logger=INFO,console
master:
]# ./bin/flume-ng agent --conf conf --conf-file ./conf/logserver_flume_hbase/flume-client.properties --name a1 -Dflume.root.logger=INFO,console
master:
]# ./bin/flume-ng agent --conf conf --conf-file ./conf/logserver_flume_hbase/flume-server.properties --name a1 -Dflume.root.logger=INFO,console
master:
]# hbase shell
]# create ‘user_action_table’,‘action_log’
5.Java具体代码及配置如下
1.pom配置
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>FlumehHase</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<