flume数据传输到kafka

flume 简单介绍

当你看到这篇文章时，应该对flume有一个大概了解但是为照顾刚入门的同学所以还是会说下flume，刚开始使用flume时不需要理解太多里面的东西，只需要理解下面的图就可以使用flume把日志数据传入kafka中，下图中的hdfs只是有代表性的sink而以，我在实际使用中sink是kafka

flume安装

flume环境准备

• centos 6.5
• JDK 1.7+

flume下载安装

• flume 1.7 下载链接
• 安装flume
  

  1.tar -zxvf apache-flume-1.7.0-bin.tar.gz
  2.mv apache-flume-1.7.0-bin flume
  3.cp conf/flume-conf.properties.template conf/flume-conf.properties # flume-conf.properties 配置source,channel,sink等信息
  4.cp conf/flume-env.sh.template conf/flume-env.sh # flume-env.sh配置agent启动项及JAVA环境变量等

flume配置

• 配置flume-conf.properties

agent.sources=r1
agent.sinks=k1
agent.channels=c1

agent.sources.r1.type=exec
agent.sources.r1.command=tail -F /data/logs/access.log
agent.sources.r1.restart=true
agent.sources.r1.batchSize=1000
agent.sources.r1.batchTimeout=3000
agent.sources.r1.channels=c1

agent.channels.c1.type=memory
agent.channels.c1.capacity=102400
agent.channels.c1.transactionCapacity=1000

agent.channels.c1.byteCapacity=134217728
agent.channels.c1.byteCapacityBufferPercentage=80

agent.sinks.k1.channel=c1
agent.sinks.k1.type=org.apache.flume.sink.kafka.KafkaSink
agent.sinks.k1.kafka.topic=xxxxx-kafka
agent.sinks.k1.kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092
agent.sinks.k1.serializer.class=kafka.serializer.StringEncoder
agent.sinks.k1.flumeBatchSize=1000
agent.sinks.k1.useFlumeEventFormat=true
   
   
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

命令规则是 r1->source k1->sink c1->channels agent名称在你启动时-n的参数值

• 配置flume-env.sh

export JAVA_HOME=/data/java/jdk1.8.0_102/
   
   
1

我这边只配置JAVA_HOME，还有agent启动的一些JMX选项没有加，这些可以根据自己的需要添加
* 启动flume-agent

启动flume-agent
./bin/flume-ng agent -c conf -f conf/flume-conf.properties -n agent -Dflume.root.logger=INFO,console
-c 配置文件目录 -f 指定flume配置文件 -n flume客户端名称 Dflume启动时将INFO级别的LOG打印在控制台

总结

1.flume中可以自己定义source,sink，你可以根据自己的需要去做修改或者重新 git地址，从github中pull代码如你只是修改某模块代码就只需要把之前的jar删除，把编译好的jar丢上去就可以，其他的玩法可以看官方文档
2.在使用中memory channels当agent被杀时数据会丢失不会恢复
3.flume在日聚合中是非常灵活的，可以组成各种玩法如我从某tcp端口拿数据传入别的flume agent中等

flume数据传输到HDFS

agent.sources = r1 agent.sinks = k1 agent.channels = c1

sources config

agent.sources.r1.type = exec
agent.sources.r1.command = tail -F /home/hadoop/access.log

channels config

agent.channels.c1.type = memory
agent.channels.c1.capacity = 1000
agent.channels.c1.transactionCapacity = 1000
agent.channels.c1.byteCapacityBufferPercentage = 20
agent.channels.c1.byteCapacity = 1000000
agent.channels.c1.keep-alive = 60

sinks config

agent.sinks.k1.type = hdfs
agent.sinks.k1.channel = c1
agent.sinks.k1.hdfs.path = hdfs://hadoop05:9000/logs/%m/%d
agent.sinks.k1.hdfs.fileType = DataStream
agent.sinks.k1.hdfs.filePrefix = BC-%H
agent.sinks.k1.hdfs.fileSuffix=.log
agent.sinks.k1.hdfs.minBlockReplicas=1
agent.sinks.k1.hdfs.rollInterval=3600
agent.sinks.k1.hdfs.rollSize=132692539
agent.sinks.k1.hdfs.idleTimeout=10
agent.sinks.k1.hdfs.batchSize = 1
agent.sinks.k1.hdfs.rollCount=0
agent.sinks.k1.hdfs.round = true
agent.sinks.k1.hdfs.roundValue = 2
agent.sinks.k1.hdfs.roundUnit = minute
agent.sinks.k1.hdfs.useLocalTimeStamp = true

bind

agent.sources.r1.channels = c1
agent.sinks.k1.channel = c1