这里用5台机搭建flume-HA集群,hosts加入以下内容
192.168.1.71 node01
192.168.1.72 node02
192.168.1.73 node03
192.168.1.74 node04
192.168.1.75 node05
案例说明-Consolidation(结合-高可用)
假设node01、node02、node03是我们的部署web应用的服务器,
在每个服务器上flume Agent模式部署,采集web应用产生的日志
并传送给Collector。node04、node05是flume Collector模式部署,
负责接收每个Agent传送过来的日志,并上传到hdfs。
3、下载安装
去http://flume.apache.org/download.html下载,这里下载最新稳定版apache-flume-1.8.0-bin.tar.gz
在node01下,/opt/bigdata/ 路径下,解压
tar -zxvf apache-flume-1.8.0-bin.tar.gz
为了方便,重命名下
mv apache-flume-1.8.0-bin ./flume-1.8.0
4、配置flume
(1)配置环境变量
在/opt/bigdata/ 下,默认flume-env.sh.template,复制一份命名flume-env.sh
cp flume-1.8.0/conf/flume-env.sh.template flume-1.8.0/conf/flume-env.sh
编辑 vim flume-1.8.0/conf/flume-env.sh,指定jdk路径
export JAVA_HOME=$JAVA_HOME 或直接指定jdk路径 export JAVA_HOME=/usr/local/jdk1.8
(2)配置启动参数
在node01,/opt/bigdata/ 下,默认flume-conf.properties.template,复制一份命名flume-conf.properties
cp flume-1.8.0/conf/flume-conf.properties.template flume-1.8.0/conf/flume-conf.properties
编辑 vim flume-1.8.0/conf/flume-conf.properties,设置相关参数
由于node01、node02、node03上的flume是Agent角色,所以这三台flume-conf.properties配置一样
#agent1 name
agent1.channels = c1
agent1.sources = r1
agent1.sinks = k1 k2
#set channel
agent1.channels.c1.type = memory
agent1.channels.c1.capacity = 1000 #channel中存储 events 的最大数量
agent1.channels.c1.transactionCapacity = 100 #事物容量,不能大于capacity,不能小于batchSize
#agent1.sources.r1.type = exec
#agent1.sources.r1.command = tail -F /data/logdfs/f.log #实时采集文件
#agent1.sources.r1.channels = c1
agent1.sources.r1.type = spooldir #spooldir类型
agent1.sources.r1.spoolDir = /data/logdfs #此路径下每产生新文件,flume就会自动采集
agent1.sources.r1.fileHeader = true
agent1.sources.r1.channels = c1
# set sink1
agent1.sinks.k1.channel = c1
agent1.sinks.k1.type = avro
agent1.sinks.k1.hostname = node04
agent1.sinks.k1.port = 52020
# set sink2
agent1.sinks.k2.channel = c1
agent1.sinks.k2.type = avro #协议类型
agent1.sinks.k2.hostname = node05
agent1.sinks.k2.port = 52020
#set gruop
agent1.sinkgroups = g1
#set sink group
agent1.sinkgroups.g1.sinks = k1 k2
#set failover
agent1.sinkgroups.g1.processor.type = failover #故障转移,若node04故障,node05自动接替node04工作
agent1.sinkgroups.g1.processor.priority.k1 = 10 #优先级10
agent1.sinkgroups.g1.processor.priority.k2 = 5 #优先级5
agent1.sinkgroups.g1.processor.maxpenalty = 10000 #最长等待10秒转移故障
分发flume
scp flume-1.8.0 node02:`pwd`
scp flume-1.8.0 node03:`pwd`
scp flume-1.8.0 node04:`pwd`
scp flume-1.8.0 node05:`pwd`
node04、node05上的flume是Collector角色,而node04承担了master角色,node05承担了slave角色
所以配置不一样,清空node04、node05配置,从新配置。
node04如下:
a1.sources = r1
a1.channels = kafka_c1 hdfs_c2
a1.sinks = kafka_k1 hdfs_k2
#properties of avro-AppSrv-source
a1.sources.r1.type = avro
a1.sources.r1.bind = node04
a1.sources.r1.port = 52020
a1.sources.r1.channels=kafka_c1 hdfs_c2 #设置sources的channels
#增加拦截器 所有events,增加头,类似json格式里的"headers":{" key":" value"}
a1.sources.r1.interceptors = i1 #拦截器名字
a1.sources.r1.interceptors.i1.type = static #拦截器类型
a1.sources.r1.interceptors.i1.key = Collector #自定义
a1.sources.r1.interceptors.i1.value = node04 #自定义
#set kafka channel
a1.channels.kafka_c1.type = memory
a1.channels.kafka_c1.capacity = 1000
a1.channels.kafka_c1.transactionCapacity = 100
#set hdfs channel
a1.channels.hdfs_c2.type = memory
a1.channels.hdfs_c2.capacity = 1000
a1.channels.hdfs_c2.transactionCapacity = 100
#set sink to kafka
a1.sinks.kafka_k1.type=org.apache.flume.sink.kafka.KafkaSink
a1.sinks.kafka_k1.channel=kafka_c1 #传输的channel名
a1.sinks.kafka_k1.topic = rwb_topic #kafka中的topic
a1.sinks.kafka_k1.brokerList = node01:9092,node02:9092,node03:9092
a1.sinks.kafka_k1.requiredAcks = 1
a1.sinks.kafka_k1.batchSize = 1000
#set sink to hdfs
a1.sinks.hdfs_k2.type=hdfs #传输到hdfs
a1.sinks.hdfs_k2.channel=hdfs_c2 #传输的channel名
a1.sinks.hdfs_k2.hdfs.path=hdfs://mycluster:8020/flume/logdfs #这里hadoop集群时HA集群,所以这里写的是集群的Namespace
a1.sinks.hdfs_k2.hdfs.fileType=DataStream
a1.sinks.hdfs_k2.hdfs.writeFormat=TEXT #文本格式
a1.sinks.hdfs_k2.hdfs.rollInterval=1 #失败1s回滚
a1.sinks.hdfs_k2.hdfs.filePrefix=%Y-%m-%d #文件名前缀
a1.sinks.hdfs_k2.hdfs.fileSuffix=.txt #文件名后缀
a1.sinks.hdfs_k2.hdfs.useLocalTimeStamp = true
node05如下:
a1.sources = r1
a1.channels = kafka_c1 hdfs_c2
a1.sinks = kafka_k1 hdfs_k2
#properties of avro-AppSrv-source
a1.sources.r1.type = avro
a1.sources.r1.bind = node05
a1.sources.r1.port = 52020
a1.sources.r1.channels=kafka_c1 hdfs_c2 #设置sources的channels
#增加拦截器 所有events,增加头,类似json格式里的"headers":{" key":" value"}
a1.sources.r1.interceptors = i1 #拦截器名字
a1.sources.r1.interceptors.i1.type = static #拦截器类型
a1.sources.r1.interceptors.i1.key = Collector #自定义
a1.sources.r1.interceptors.i1.value = node04 #自定义
#set kafka channel
a1.channels.kafka_c1.type = memory
a1.channels.kafka_c1.capacity = 1000
a1.channels.kafka_c1.transactionCapacity = 100
#set hdfs channel
a1.channels.hdfs_c2.type = memory
a1.channels.hdfs_c2.capacity = 1000
a1.channels.hdfs_c2.transactionCapacity = 100
#set sink to kafka
a1.sinks.kafka_k1.type=org.apache.flume.sink.kafka.KafkaSink
a1.sinks.kafka_k1.channel=kafka_c1 #传输的channel名
a1.sinks.kafka_k1.topic = rwb_topic #kafka中的topic
a1.sinks.kafka_k1.brokerList = node01:9092,node02:9092,node03:9092
a1.sinks.kafka_k1.requiredAcks = 1
a1.sinks.kafka_k1.batchSize = 1000
#set sink to hdfs
a1.sinks.hdfs_k2.type=hdfs #传输到hdfs
a1.sinks.hdfs_k2.channel=hdfs_c2 #传输的channel名
a1.sinks.hdfs_k2.hdfs.path=hdfs://mycluster:8020/flume/logdfs #这里hadoop集群时HA集群,所以这里写的是集群的Namespace
a1.sinks.hdfs_k2.hdfs.fileType=DataStream
a1.sinks.hdfs_k2.hdfs.writeFormat=TEXT #文本格式
a1.sinks.hdfs_k2.hdfs.rollInterval=1 #失败1s回滚
a1.sinks.hdfs_k2.hdfs.filePrefix=%Y-%m-%d #文件名前缀
a1.sinks.hdfs_k2.hdfs.fileSuffix=.txt #文件名后缀
a1.sinks.hdfs_k2.hdfs.useLocalTimeStamp = true
(3)启动flume
启动Collector
在node04、node05/opt/bigdata/下分别如下命令启动
flume-1.8.0/bin/flume-ng agent --conf conf --conf-file flume-1.8.0/conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,console > flume-1.8.0/logs/flume-server.log 2>&1 &
启动Agent
在node01、node02、node03 /opt/bigdata/下分别如下命令启动
flume-1.8.0/bin/flume-ng agent --conf conf --conf-file flume-1.8.0/conf/flume-conf.properties --name agent1 -Dflume.root.logger=DEBUG,console > flume-1.8.0/logs/flume-server.log 2>&1 &
a1、agent1 为各自flume的代理名
5、高可用测试
在node01、node02、node03任一节点 /data/logdfs下,产生新文件,
该节点下的flume就会采集传送的node04的flume。
这里在node01下 复制cp /root/install.log 到 /data/logdfs 下
此时tail -f flume-1.8.0/logs/flume-server.log 查看node04的flume日志,部分日志如下
18/06/17 14:54:22 INFO hdfs.HDFSEventSink: Writer callback called.
18/06/17 14:58:28 INFO ipc.NettyServer: Connection to /192.168.1.73:34166 disconnected.
18/06/17 14:59:21 INFO ipc.NettyServer: [id: 0xeffc35bb, /192.168.1.71:34175 => /192.168.1.74:52020] OPEN
18/06/17 14:59:21 INFO ipc.NettyServer: [id: 0xeffc35bb, /192.168.1.71:34175 => /192.168.1.74:52020] BOUND: /192.168.1.74:52020
18/06/17 14:59:21 INFO ipc.NettyServer: [id: 0xeffc35bb, /192.168.1.71:34175 => /192.168.1.74:52020] CONNECTED: /192.168.1.73:34175
18/06/17 14:59:21 INFO hdfs.HDFSDataStream: Serializer = TEXT, UseRawLocalFileSystem = false
18/06/17 14:59:21 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529261961816.txt.tmp
18/06/17 14:59:21 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529261961816.txt.tmp
18/06/17 14:59:22 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529261961816.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529261961816.txt
18/06/17 14:59:22 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529261961817.txt.tmp
18/06/17 14:59:22 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529261961817.txt.tmp
18/06/17 14:59:22 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529261961817.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529261961817.txt
18/06/17 14:59:22 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529261961818.txt.tmp
18/06/17 14:59:22 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529261961818.txt.tmp
18/06/17 14:59:22 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529261961818.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529261961818.txt
18/06/17 14:59:22 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529261961819.txt.tmp
18/06/17 14:59:22 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529261961819.txt.tmp
18/06/17 14:59:22 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529261961819.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529261961819.txt
与此此时看node05的flume日志是没有任何情况的,也就是node05的flume没有工作
接下啦kill掉node04的flume
在node01下,复制一个新文件到cp /root/install.log.syslog 到 /data/logdfs 下,部分日志如下
18/06/17 14:24:17 INFO hdfs.HDFSEventSink: Writer callback called.
18/06/17 14:39:42 INFO ipc.NettyServer: [id: 0x7079374a, /192.168.1.71:51692 => /192.168.1.75:52020] OPEN
18/06/17 14:39:42 INFO ipc.NettyServer: [id: 0x7079374a, /192.168.1.71:51692 => /192.168.1.75:52020] BOUND: /192.168.1.75:52020
18/06/17 14:39:42 INFO ipc.NettyServer: [id: 0x7079374a, /192.168.1.71:51692 => /192.168.1.75:52020] CONNECTED: /192.168.1.71:51692
18/06/17 14:39:45 INFO hdfs.HDFSDataStream: Serializer = TEXT, UseRawLocalFileSystem = false
18/06/17 14:39:45 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785844.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785844.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785844.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529260785844.txt
18/06/17 14:39:46 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785845.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785845.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785845.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529260785845.txt
18/06/17 14:39:46 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785846.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785846.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785846.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529260785846.txt
18/06/17 14:39:46 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785847.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785847.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785847.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529260785847.txt
18/06/17 14:39:46 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785848.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785848.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785848.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529260785848.txt
可以看到node01与node05建立的连接,node05把采集到的文件开始上传到hdfs
node05-flume接替了node04-flume的工作,故障自动转移了。
此时在重新启动node04的flume,这次我们换台机,在node02 /data/lodfs下产生新文件,
复制cp /root/nohup.out 到 /data/lodfs 下,查看node04的flume日志,部分日志如下
18/06/17 14:24:17 INFO hdfs.HDFSEventSink: Writer callback called.
18/06/17 14:39:42 INFO ipc.NettyServer: [id: 0x7079374a, /192.168.1.72:51692 => /192.168.1.74:52020] OPEN
18/06/17 14:39:42 INFO ipc.NettyServer: [id: 0x7079374a, /192.168.1.72:51692 => /192.168.1.74:52020] BOUND: /192.168.1.74:52020
18/06/17 14:39:42 INFO ipc.NettyServer: [id: 0x7079374a, /192.168.1.72:51692 => /192.168.1.74:52020] CONNECTED: /192.168.1.72:51692
18/06/17 14:39:45 INFO hdfs.HDFSDataStream: Serializer = TEXT, UseRawLocalFileSystem = false
18/06/17 14:39:45 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785844.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785844.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785844.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529260785844.txt
18/06/17 14:39:46 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785845.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785845.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785845.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529260785845.txt
18/06/17 14:39:46 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785846.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785846.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785846.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529260785846.txt
18/06/17 14:39:46 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785847.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785847.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785847.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529260785847.txt
18/06/17 14:39:46 INFO hdfs.BucketWriter: Creating hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785848.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Closing hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785848.txt.tmp
18/06/17 14:39:46 INFO hdfs.BucketWriter: Renaming hdfs://mycluster:8020/flume/logdfs/2018-06-17.1529260785848.txt.tmp to hdfs://myclu
ster:8020/flume/logdfs/2018-06-17.1529260785848.txt
可以看到node02与node04建立的连接,node04的flume把采集到的文件开始上传到hdfs,node04的flume启动后,又重新下恢复正常工作了