flume1.6.0安装及测试

最新推荐文章于 2022-05-17 17:58:21 发布
最新推荐文章于 2022-05-17 17:58:21 发布
阅读量1.5k 收藏 2
点赞数
分类专栏： 大数据运维之flume日常 文章标签： flume

转载来自：http://www.aboutyun.com/thread-8917-1-1.html


1.下载apache-flume-1.6.0-bin.tar.gz

wget http://www.apache.org/dyn/closer.lua/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz

2.解压

tar -zxvf apache-flume-1.6.0-bin.tar.gz

3.配置

先进入 apache-flume-1.6.0-bin/conf
复制一份 flume-env.sh.template 并改名 flume-env.sh
vim flume-env.sh 加上一句 export JAVA_HOME=/home/wkz/jdk7/ （你自己的安装位置）

cp flume-conf.properties.template  flume.conf （后面的名字可以任意起，其实这块也可以直接vim flume.conf，不用cp）

注：这个flume.conf可以指定多个，相当于配置不同的conf文件，有不同的文件传输方式

案例1：Spool
　　　　Spool监测配置的目录下新增的文件，并将文件中的数据读取出来。需要注意两点：
　　　　1) 拷贝到spool目录下的文件不可以再打开编辑。
　　　　2) spool目录下不可包含相应的子目录

#agent1表示代理名称
agent1.sources=source1
agent1.sinks=sink1
agent1.channels=channel1

#配置source1
# spooldir ---监控新增文件

#spooldir 所指定的路径要手动创建
agent1.sources.source1.type=spooldir
agent1.sources.source1.spoolDir=/home/wkz/flumeinfo
agent1.sources.source1.channels=channel1
agent1.sources.source1.fileHeader = false
agent1.sources.source1.interceptors = i1
agent1.sources.source1.interceptors.i1.type = timestamp


#配置channel1
agent1.channels.channel1.type=file
agent1.channels.channel1.checkpointDir=/home/wkz/flume_tmp_cp
agent1.channels.channel1.dataDirs=/home/wkz/flume_tmp


#配置sink1

#type指定数据传输到哪里，这里是传输到hdfs，也可以传输到logger上；同时上传到hdfs上的路径也要手动创建
agent1.sinks.sink1.type=hdfs 
agent1.sinks.sink1.hdfs.path=hdfs://master:9000/logs
agent1.sinks.sink1.hdfs.fileType=DataStream
agent1.sinks.sink1.hdfs.writeFormat=TEXT
agent1.sinks.sink1.hdfs.rollInterval=1
agent1.sinks.sink1.channel=channel1
agent1.sinks.sink1.hdfs.filePrefix=%Y-%m-%d

#filePrefix是看文件是什么会时候上传到hdfs上的，也是hdfs上的文件名

创建本地文件夹： mkdir /home/wkz/flumeinfo
创建HDFS文件夹： hdfs dfs -mkdir /logs
注：flume把发送到本地flumeinfo下的文件发送到hdfs上的logs下，然后spark streaming 在对上传的文件进行处理。

启动脚本（指定代理名称，和对应conf文件启动）
flume-ng agent -n agent1 -c conf -f /home/wkz/flume-1.6.0/conf/flume-conf.properties -Dflume.root.logger=DEBUG,console


案例2：Avro
　　　　Avro可以发送一个给定的文件给Flume，Avro 源使用AVRO RPC机制。

在conf下创建a1.conf

  1. a1.sources = r1
  2. a1.sinks = k1
  3. a1.channels = c1

  5. # Describe/configure the source
  6. a1.sources.r1.type = avro
  7. a1.sources.r1.channels = c1
  8. a1.sources.r1.bind = 0.0.0.0
  9. a1.sources.r1.port = 4141

  11. # Describe the sink
  12. a1.sinks.k1.type = logger

  14. # Use a channel which buffers events in memory
  15. a1.channels.c1.type = memory
  16. a1.channels.c1.capacity = 1000
  17. a1.channels.c1.transactionCapacity = 100

  19. # Bind the source and sink to the channel
  20. a1.sources.r1.channels = c1
  21. a1.sinks.k1.channel = c1
启动flume agent a1

flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a1.conf -n a1 -Dflume.root.logger=INFO,console

创建指定文件

echo "hello world" > /home/wkz/flume-1.6.0/log.00

使用avro-client发送文件（另开一个xshell窗口，-H指定的ip地址）

flume-ng avro-client -c conf -H master -p 4141 -F /home/wkz/flume-1.6.0/log.00  

然后在启动agent的窗口可以看到如下：

*******（省略）

17/10/25 17:40:56 INFO ipc.NettyServer: [id: 0x0fe441b9, /192.168.208.180:45698 => /192.168.208.180:4141] CONNECTED: /192.168.208.180:45698
17/10/25 17:40:56 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64                hello world }
17/10/25 17:40:56 INFO ipc.NettyServer: [id: 0x0fe441b9, /192.168.208.180:45698 :> /192.168.208.180:4141] DISCONNECTED
17/10/25 17:40:56 INFO ipc.NettyServer: [id: 0x0fe441b9, /192.168.208.180:45698 :> /192.168.208.180:4141] UNBOUND

.....


案例3：Exec
　　　　EXEC执行一个给定的命令获得输出的源,如果要使用tail命令，必选使得file足够大才能看到输出内容

在conf下创建a3.conf

a3.sources = r3
a3.sinks = k3
a3.channels = c3

# Describe/configure the source
a3.sources.r3.type = exec
a3.sources.r3.channels = c3
a3.sources.r3.command = tail -F /home/wkz/flume-1.6.0/log_exec_tail

# Describe the sink
a3.sinks.k3.type = logger

# Use a channel which buffers events in memory
a3.channels.c3.type = memory
a3.channels.c3.capacity = 1000
a3.channels.c3.transactionCapacity = 100

# Bind the source and sink to the channel
a3.sources.r3.channels = c3
a3.sinks.k3.channel = c3


启动flume agent a3

flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a3.conf -n a3 -Dflume.root.logger=INFO,console

生成足够多的内容在文件里

for i in {1..100};do echo "exec tail$i" >> /home/wkz/flume-1.6.0/log_exec_tail; done

然后在启动agent的窗口可以看到如下：

***

17/10/25 17:55:29 INFO sink.LoggerSink: Event: { headers:{} body: 65 78 65 63 20 74 61 69 6C 39 35                exec tail95 }
17/10/25 17:55:29 INFO sink.LoggerSink: Event: { headers:{} body: 65 78 65 63 20 74 61 69 6C 39 36                exec tail96 }
17/10/25 17:55:29 INFO sink.LoggerSink: Event: { headers:{} body: 65 78 65 63 20 74 61 69 6C 39 37                exec tail97 }
17/10/25 17:55:29 INFO sink.LoggerSink: Event: { headers:{} body: 65 78 65 63 20 74 61 69 6C 39 38                exec tail98 }
17/10/25 17:55:29 INFO sink.LoggerSink: Event: { headers:{} body: 65 78 65 63 20 74 61 69 6C 39 39                exec tail99 }
17/10/25 17:55:29 INFO sink.LoggerSink: Event: { headers:{} body: 65 78 65 63 20 74 61 69 6C 31 30 30             exec tail100 }


案例4：Syslogtcp
　　　　Syslogtcp监听TCP的端口做为数据源

在conf下创建a4.conf

a4.sources = r4
a4.sinks = k4
a4.channels = c4

# Describe/configure the source
a4.sources.r4.type = syslogtcp
a4.sources.r4.port = 5140
a4.sources.r4.host = localhost
a4.sources.r4.channels = c4

# Describe the sink
a4.sinks.k4.type = logger

# Use a channel which buffers events in memory
a4.channels.c4.type = memory
a4.channels.c4.capacity = 1000
a4.channels.c4.transactionCapacity = 100

# Bind the source and sink to the channel
a4.sources.r4.channels = c4
a4.sinks.k4.channel = c4


启动flume agent a4

flume-ng agent -c . -f /home/wkz/flume-1.6.0/conf/a4.conf -n a4 -Dflume.root.logger=INFO,console

测试产生syslog(需要 yum install nc)

 echo "hello idoall.org syslog" | nc localhost 5140 

然后在启动agent的窗口可以看到如下：

******

17/10/25 18:46:39 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 68 65 6C 6C 6F 20 69 64 6F 61 6C 6C 2E 6F 72 67 hello idoall.org }


案例5：JSONHandler

在conf下创建a5.conf

a5.sources = r5
a5.sinks = k5
a5.channels = c5

# Describe/configure the source
a5.sources.r5.type = org.apache.flume.source.http.HTTPSource
a5.sources.r5.port = 8888
a5.sources.r5.channels = c5

# Describe the sink
a5.sinks.k5.type = logger

# Use a channel which buffers events in memory
a5.channels.c5.type = memory
a5.channels.c5.capacity = 1000
a5.channels.c5.transactionCapacity = 100

# Bind the source and sink to the channel
a5.sources.r5.channels = c5
a5.sinks.k5.channel = c5


启动flume agent a5

flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a5.conf -n a5 -Dflume.root.logger=INFO,console

生成JSON 格式的POST request

curl -X POST -d '[{ "headers" :{"a" : "a1","b" : "b1"},"body" : "idoall.org_body"}]' http://localhost:8888

然后在启动agent的窗口可以看到如下：

*******

17/10/25 18:56:39 INFO sink.LoggerSink: Event: { headers:{b=b1, a=a1} body: 69 64 6F 61 6C 6C 2E 6F 72 67 5F 62 6F 64 79    idoall.org_body }


案例6：Hadoop sink

a6.sources = r6
a6.sinks = k6
a6.channels = c6

# Describe/configure the source
a6.sources.r6.type = syslogtcp
a6.sources.r6.port = 5140
a6.sources.r6.host = localhost
a6.sources.r6.channels = c6

# Describe the sink
a6.sinks.k6.type = hdfs
a6.sinks.k6.channel = c6
a6.sinks.k6.hdfs.path = hdfs://master:9000/user/flume/syslogtcp
a6.sinks.k6.hdfs.filePrefix = Syslog
a6.sinks.k6.hdfs.round = true
a6.sinks.k6.hdfs.roundValue = 10
a6.sinks.k6.hdfs.roundUnit = minute

# Use a channel which buffers events in memory
a6.channels.c6.type = memory
a6.channels.c6.capacity = 1000
a6.channels.c6.transactionCapacity = 100

# Bind the source and sink to the channel
a6.sources.r6.channels = c6
a6.sinks.k6.channel = c6


启动agent(注：此时a6.sinks.k6.hdfs.path = hdfs://master:9000/user/flume/syslogtcp路径自动创建，不知道之前的不创建会不会出错，懒得试了)

flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a6.conf -n a6 -Dflume.root.logger=INFO,console

测试产生syslog

echo "hello idoall flume -> hadoop testing one" | nc localhost 5140

然后在启动agent的窗口可以看到如下：

17/10/25 19:16:25 INFO hdfs.HDFSSequenceFile: writeFormat = Writable, UseRawLocalFileSystem = false
17/10/25 19:16:26 INFO hdfs.BucketWriter: Creating hdfs://master:9000/user/flume/syslogtcp/Syslog.1508930185400.tmp
17/10/25 19:17:00 INFO hdfs.BucketWriter: Closing hdfs://master:9000/user/flume/syslogtcp/Syslog.1508930185400.tmp
17/10/25 19:17:00 INFO hdfs.BucketWriter: Renaming hdfs://master:9000/user/flume/syslogtcp/Syslog.1508930185400.tmp to hdfs://master:9000/user/flume/syslogtcp/Syslog.1508930185400
17/10/25 19:17:00 INFO hdfs.HDFSEventSink: Writer callback called.


案例7：File Roll Sink

a7.sources = r7
a7.sinks = k7
a7.channels = c7

# Describe/configure the source
a7.sources.r7.type = syslogtcp
a7.sources.r7.port = 5555
a7.sources.r7.host = localhost
a7.sources.r7.channels = c7

# Describe the sink
a7.sinks.k7.type = file_roll
a7.sinks.k7.sink.directory = /home/wkz/flume-1.6.0/logs

# Use a channel which buffers events in memory
a7.channels.c7.type = memory
a7.channels.c7.capacity = 1000
a7.channels.c7.transactionCapacity = 100

# Bind the source and sink to the channel
a7.sources.r7.channels = c7
a7.sinks.k7.channel = c7


启动flume agent a7

flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a7.conf -n a7 -Dflume.root.logger=INFO,console

测试产生log

echo "hello idoall.org syslog" | nc localhost 5555

echo "hello idoall.org syslog 2" | nc localhost 5555

查看/home/wkz/flume-1.6.0/logs下是否生成文件,默认每30秒生成一个新文件

[wkz@master logs]$ ll
total 8
-rw-rw-r-- 1 wkz wkz   0 Oct 25 19:24 1508930684254-1
-rw-rw-r-- 1 wkz wkz  50 Oct 25 19:25 1508930684254-2
-rw-rw-r-- 1 wkz wkz   0 Oct 25 19:25 1508930684254-3
-rw-rw-r-- 1 wkz wkz   0 Oct 25 19:26 1508930684254-4
-rw-rw-r-- 1 wkz wkz 124 Oct 25 17:40 flume.log

[wkz@master logs]$ cat 1508930684254-2
hello idoall.org syslog
hello idoall.org syslog 2


案例8：Replicating Channel Selector

Flume支持Fan out流从一个源到多个通道。有两种模式的Fan out，分别是复制和复用。在复制的情况下，流的事件被发送到所有的配置通道。在复用的情况下，事件被发送到可用的渠道中的一个子集。Fan out流需要指定源和Fan out通道的规则。
这次我们需要用到master,slave两台机器。

在master创建a8_Selector.conf配置文件

a8.sources = r8
a8.sinks = k8 k2
a8.channels = c8 c2

# Describe/configure the source
a8.sources.r8.type = syslogtcp
a8.sources.r8.port = 5140
a8.sources.r8.host = localhost
a8.sources.r8.channels = c8 c2
a8.sources.r8.selector.type = replicating

# Describe the sink
a8.sinks.k8.type = avro
a8.sinks.k8.channel = c8
a8.sinks.k8.hostname = master
a8.sinks.k8.port = 5555

a8.sinks.k2.type = avro
a8.sinks.k2.channel = c2
a8.sinks.k2.hostname = slave
a8.sinks.k2.port = 5555

# Use a channel which buffers events in memory
a8.channels.c8.type = memory
a8.channels.c8.capacity = 1000
a8.channels.c8.transactionCapacity = 100

a8.channels.c2.type = memory
a8.channels.c2.capacity = 1000
a8.channels.c2.transactionCapacity = 100


在master创建a8_Avro.conff配置文件

a8.sources = r8
a8.sinks = k8
a8.channels = c8

# Describe/configure the source
a8.sources.r8.type = avro
a8.sources.r8.channels = c8
a8.sources.r8.bind = 0.0.0.0
a8.sources.r8.port = 5555

# Describe the sink
a8.sinks.k8.type = logger

# Use a channel which buffers events in memory
a8.channels.c8.type = memory
a8.channels.c8.capacity = 1000
a8.channels.c8.transactionCapacity = 100

# Bind the source and sink to the channel
a8.sources.r8.channels = c8
a8.sinks.k8.channel = c8

在master上将这2个配置文件复制到slave上一份；

scp ****

打开4个窗口，在master和slave上同时启动两个flume agent：(刚开始机器之间没通信上可能一直报错，不用管，运行一段时间就好了)

flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a8_Selector.conf -n a8 -Dflume.root.logger=INFO,console
flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a8_Avro.conf -n a8 -Dflume.root.logger=INFO,console

然后在master或slave的任意一台机器上，测试产生syslog

echo "hello idoall.org syslog" | nc localhost 5140

在master和slave的sink窗口，分别可以看到以下信息,这说明信息得到了同步：

14/08/10 14:09:20 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 68 65 6C 6C 6F 20 69 64 6F 61 6C 6C 2E 6F 72 67 hello idoall.org }

注：这一步我没有实现，两个sink窗口都没有出现这个内容，不知道为啥。


案例9：Multiplexing Channel Selector

在master创建a9_Selector.conf

a9.sources = r9
a9.sinks = k9 k2
a9.channels = c9 c2

# Describe/configure the source
a9.sources.r9.type = org.apache.flume.source.http.HTTPSource
a9.sources.r9.port = 5140
a9.sources.r9.channels = c9 c2
a9.sources.r9.selector.type = multiplexing

a9.sources.r9.selector.header = type
#映射允许每个值通道可以重叠。默认值可以包含任意数量的通道。
a9.sources.r9.selector.mapping.baidu = c9
a9.sources.r9.selector.mapping.ali = c2
a9.sources.r9.selector.default = c9

# Describe the sink
a9.sinks.k9.type = avro
a9.sinks.k9.channel = c9
a9.sinks.k9.hostname = master
a9.sinks.k9.port = 5555

a9.sinks.k2.type = avro
a9.sinks.k2.channel = c2
a9.sinks.k2.hostname = slave
a9.sinks.k2.port = 5555

# Use a channel which buffers events in memory
a9.channels.c9.type = memory
a9.channels.c9.capacity = 9000
a9.channels.c9.transactionCapacity = 900

a9.channels.c2.type = memory
a9.channels.c2.capacity = 9000
a9.channels.c2.transactionCapacity = 900

在master创建a9_Avro.conf


a9.sources = r9
a9.sinks = k9
a9.channels = c9

# Describe/configure the source
a9.sources.r9.type = avro
a9.sources.r9.channels = c9
a9.sources.r9.bind = 0.0.0.0
a9.sources.r9.port = 5555

# Describe the sink
a9.sinks.k9.type = logger

# Use a channel which buffers events in memory
a9.channels.c9.type = memory
a9.channels.c9.capacity = 1000
a9.channels.c9.transactionCapacity = 100

# Bind the source and sink to the channel
a9.sources.r9.channels = c9
a9.sinks.k9.channel = c9


分别把这两个配置文件发送给slave

在master和slave同时开启两个agent：

flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a9_Selector.conf -n a9 -Dflume.root.logger=INFO,console
flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a9_Avro.conf -n a9 -Dflume.root.logger=INFO,console

然后在master或slave的任意一台机器上，测试产生syslog

curl -X POST -d '[{ "headers" :{"type" : "baidu"},"body" : "idoall_TEST1"}]' http://localhost:5140 && curl -X POST -d '[{ "headers" :{"type" : "ali"},"body" : "idoall_TEST2"}]' http://localhost:5140 && curl -X POST -d '[{ "headers" :{"type" : "qq"},"body" : "idoall_TEST3"}]' http://localhost:5140

在master的sink窗口，可以看到以下信息：

17/10/26 14:14:52 INFO sink.LoggerSink: Event: { headers:{type=baidu} body: 69 64 6F 61 6C 6C 5F 54 45 53 54 31             idoall_TEST1 }
17/10/26 14:14:52 INFO sink.LoggerSink: Event: { headers:{type=qq} body: 69 64 6F 61 6C 6C 5F 54 45 53 54 33             idoall_TEST3 }

在slave的sink窗口，可以看到以下信息：

17/10/26 14:14:54 INFO sink.LoggerSink: Event: { headers:{type=ali} body: 69 64 6F 61 6C 6C 5F 54 45 53 54 32             idoall_TEST2}

可以看到，根据header中不同的条件分布到不同的channel上。


案例10：Flume Sink Processors
　　　　failover的机器是一直发送给其中一个sink，当这个sink不可用的时候，自动发送到下一个sink。

在master创建a10_Processors.conf


a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1 c2

#这个是配置failover的关键，需要有一个sink group
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
#处理的类型是failover
a1.sinkgroups.g1.processor.type = failover
#优先级，数字越大优先级越高，每个sink的优先级必须不相同
a1.sinkgroups.g1.processor.priority.k1 = 5
a1.sinkgroups.g1.processor.priority.k2 = 10
#设置为10秒，当然可以根据你的实际状况更改成更快或者很慢
a1.sinkgroups.g1.processor.maxpenalty = 10000

# Describe/configure the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.port = 5140
a1.sources.r1.channels = c1 c2
a1.sources.r1.selector.type = replicating

# Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = m1
a1.sinks.k1.port = 5555

a1.sinks.k2.type = avro
a1.sinks.k2.channel = c2
a1.sinks.k2.hostname = m2
a1.sinks.k2.port = 5555

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100


在master创建a10_Avro.conf


a10.sources = r10
a10.sinks = k10
a10.channels = c10

# Describe/configure the source
a10.sources.r10.type = avro
a10.sources.r10.channels = c10
a10.sources.r10.bind = 0.0.0.0
a10.sources.r10.port = 5555

# Describe the sink
a10.sinks.k10.type = logger

# Use a channel which buffers events in memory
a10.channels.c10.type = memory
a10.channels.c10.capacity = 1000
a10.channels.c10.transactionCapacity = 100

# Bind the source and sink to the channel
a10.sources.r10.channels = c10
a10.sinks.k10.channel = c10

将这两个配置文件发送到slave
在master和slave上同时启动两个flume agent
flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a10_Processors.conf -n a10 -Dflume.root.logger=INFO,console
flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a10_Avro.conf -n a10 -Dflume.root.logger=INFO,console

然后在master或slave的任意一台机器上，测试产生log
 echo "idoall.org test1 failover" | nc localhost 5140


因为slave的优先级高，所以在slave的sink窗口，可以看到以下信息，而master没有：
17/10/26 14:43:20 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 31 idoall.orgtest1 }

这时我们停止掉slave机器上的sink(ctrl+c)，再次输出测试数据：

echo "idoall.org test2 failover" | nc localhost 5140

可以在master的sink窗口，看到读取到了刚才发送的两条测试数据：(可以看到接受到了以前没接收的信息和新的信息)

17/10/26 14:43:19 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 31 idoall.orgtest1 }
17/10/26 14:47:01 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 32 idoall.orgtest2 }

我们再在slave的sink窗口中，启动sink：

flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a10_Processors.conf -n a10 -Dflume.root.logger=INFO,console

输入两批测试数据：

echo "idoall.org test3 failover" | nc localhost 5140 && echo "idoall.org test4 failover" | nc localhost 5140

在slave的sink窗口，我们可以看到以下信息，因为优先级的关系，log消息会再次落到slave上：（并且以前未接收的test2也重新接受了）

17/10/26 14:48:36 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 32 idoall.orgtest2 }
17/10/26 14:48:36 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 33 idoall.orgtest3 }
17/10/26 14:48:36 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 34 idoall.orgtest4 }


案例11：Load balancing Sink Processor
　　　　load balance type和failover不同的地方是，load balance有两个配置，一个是轮询，一个是随机。两种情况下如果被选择的sink不可用，就会自动尝试发送到下一个可用的sink上面。


在master创建a11_Processors.conf

a11.sources = r11
a11.sinks = k11 k2
a11.channels = c11

#这个是配置Load balancing的关键，需要有一个sink group
a11.sinkgroups = g11
a11.sinkgroups.g11.sinks = k11 k2
a11.sinkgroups.g11.processor.type = load_balance
a11.sinkgroups.g11.processor.backoff = true
a11.sinkgroups.g11.processor.selector = round_robin

# Describe/configure the source
a11.sources.r11.type = syslogtcp
a11.sources.r11.port = 5140
a11.sources.r11.channels = c11

# Describe the sink
a11.sinks.k11.type = avro
a11.sinks.k11.channel = c11
a11.sinks.k11.hostname = master
a11.sinks.k11.port = 5555

a11.sinks.k2.type = avro
a11.sinks.k2.channel = c11
a11.sinks.k2.hostname = slave
a11.sinks.k2.port = 5555

# Use a channel which buffers events in memory
a11.channels.c11.type = memory
a11.channels.c11.capacity = 1000
a11.channels.c11.transactionCapacity = 100

在master创建a11_Avro.conf
a11.sources = r11
a11.sinks = k11
a11.channels = c11

# Describe/configure the source
a11.sources.r11.type = avro
a11.sources.r11.channels = c11
a11.sources.r11.bind = 0.0.0.0
a11.sources.r11.port = 5555

# Describe the sink
a11.sinks.k11.type = logger

# Use a channel which buffers events in memory
a11.channels.c11.type = memory
a11.channels.c11.capacity = 1000
a11.channels.c11.transactionCapacity = 100

# Bind the source and sink to the channel
a11.sources.r11.channels = c11
a11.sinks.k11.channel = c11

将2个配置文件复制到slave上一份
打开4个窗口，在master和slave上同时启动两个flume agent
flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a11_Processors.conf -n a11 -Dflume.root.logger=INFO,console
flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a11_Avro.conf -n a11 -Dflume.root.logger=INFO,console

然后在master或slave的任意一台机器上，测试产生log，一行一行输入，输入太快，容易落到一台机器上
echo "idoall.org test1" | nc localhost 5140
echo "idoall.org test2" | nc localhost 5140
echo "idoall.org test3" | nc localhost 5140
echo "idoall.org test4" | nc localhost 5140

在master的sink窗口，可以看到以下信息：

17/10/26 15:05:24 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 31 idoall.org test1}
17/10/26 15:06:02 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 34 idoall.orgtest4 }

在slave的sink窗口，可以看到以下信息：

17/10/26 15:05:37 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 32 idoall.org test2}
17/10/26 15:05:46 INFO sink.LoggerSink: Event: { headers:{Severity=0, flume.syslog.status=Invalid, Facility=0} body: 69 64 6F 61 6C 6C 2E 6F 72 67 20 74 65 73 74 33 idoall.orgtest3 }

说明轮询模式起到了作用。


案例12：Hbase sink

         测试之前，将hbase启动

执行：start-hbase.sh 
cd /home/wkz/hbase/lib
然后将以下文件复制到flume中：
cp protobuf-java-2.5.0.jar  hbase-client-1.1.12.jar  hbase-common-1.1.12.jar hbase-protocol-1.1.12.jar  hbase-server-1.1.12.jar hbase-hadoop2-compat-1.1.12.jar  hbase-hadoop-compat-1.1.12.jar  htrace-core-3.1.0-incubating.jar /home/wkz/flume-1.6.0/lib/

执行：hbase shell
hbase(main):001:0> create 'test_idoall_org','name'
0 row(s) in 4.6560 seconds

在master创建hbase_simple配置文件:(用刚才创建的hbase表名和列族名)

在master上创建a12_Hbase.conf

a12.sources = r12
a12.sinks = k12
a12.channels = c12

# Describe/configure the source
a12.sources.r12.type = syslogtcp
a12.sources.r12.port = 5140
a12.sources.r12.host = localhost
a12.sources.r12.channels = c12

# Describe the sink
a12.sinks.k12.type = logger
a12.sinks.k12.type = hbase
a12.sinks.k12.table = test_idoall_org
a12.sinks.k12.columnFamily = name
a12.sinks.k12.column = idoall
a12.sinks.k12.serializer =  org.apache.flume.sink.hbase.RegexHbaseEventSerializer
a12.sinks.k12.channel = memoryChannel

# Use a channel which buffers events in memory
a12.channels.c12.type = memory
a12.channels.c12.capacity = 1000
a12.channels.c12.transactionCapacity = 100

# Bind the source and sink to the channel
a12.sources.r12.channels = c12
a12.sinks.k12.channel = c12

启动flume agent
flume-ng agent -c conf -f /home/wkz/flume-1.6.0/conf/a12_Hbase.conf -n a12 -Dflume.root.logger=INFO,console

测试产生syslog
echo "hello idoall.org from flume" | nc localhost 5140

这时登录到hbase中，可以发现新数据已经插入（我测试失败了，估计是flume和hbase版本不匹配）

  1. hbase(main):003:0> scan "test_idoall_org"
  2. ROW                                                    COLUMN+CELL                                                                                              
  3. 1407658495588-XbQCOZrKK8-0                            column=name:payload, timestamp=1407658498203, value=hello idoall.org from flume                                                                                 
  4. 2 row(s) in 0.0200 seconds

我的错误：

17/10/26 15:52:02 WARN source.SyslogUtils: Event created from Invalid Syslog data.
17/10/26 15:52:09 ERROR hbase.HBaseSink: Failed to commit transaction.Transaction rolled back.
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V
at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:377)
at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:372)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hbase.HBaseSink.putEventsAndCommit(HBaseSink.java:372)
at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:342)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
17/10/26 15:52:09 ERROR hbase.HBaseSink: Failed to commit transaction.Transaction rolled back.
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V
at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:377)
at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:372)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hbase.HBaseSink.putEventsAndCommit(HBaseSink.java:372)
at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:342)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V
at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:377)
at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:372)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hbase.HBaseSink.putEventsAndCommit(HBaseSink.java:372)
at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:342)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)


人蠢多读书
关注
  • 0
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
apache-flume-1.6.0-bin.tar
03-21
Flume 是Cloudera提供的分布式数据收集系统，它能够将不同数据源的海量日志数据进行高效的收集、聚合、移动，最后存储到存储中心。Flume NG支持（故障转移）failover和负载均衡。
flume 1.6.0 安装及配置
会吃猫的鱼的专栏
08-22 3463
flume 是一个日志收集系统，它可以自定义数据的来源和目的地，具体的介绍百度有一大把 flume-ng 的ng 指的是 next generation 就是新一代的意思 对应的就有 flume-og 即老的版本 flume 如图所示，我们可以任意定义flume的节点然后最终将log 写入hdfs 或者kafka 或者随便你怎么搞 我这里演示一下基本的安装与配置 st
九十五、尚硅谷kylin单机版环境——安装flume1.6.0
tiantang2renjian的博客
05-17 421
尚硅谷kylin单机版环境——安装flume1.6.0
Flume1.6.0入门：安装、部署、及flume的案例
01-07
Flume1.6.0入门：安装、部署、及flume的案例
apache-flume-1.6.0-bin.tar.gz 1.6版本的
04-10
flume 1.6 版本 目前配合jdk1.7 使用 apache-flume-1.6.0-bin.tar.gz
flume-ng-1.6.0 cdh5.7.0安装包
03-01
在安装和配置Flume-ng-1.6.0-cdh5.7.0时，用户需要注意以下几点： 1. 确保系统环境满足CDH 5.7.0的硬件和软件要求。 2. 安装Java开发环境，因为Flume依赖Java运行。 3. 解压压缩包`apache-flume-1.6.0-cdh5.7.0-bin`...
flume 安装和使用
05-30
安装 Flume 首先需要确保你的系统上已经安装了 Java 1.7 或更高版本，并且正确配置了 Java 环境变量。你可以通过 `java -version` 命令检查 Java 版本。然后，你可以从 Flume 的官方网站...
flume在虚拟机上安装，
09-17
#### 四、Flume配置与测试 - **配置Agent**: Flume的核心组件是Agent，它包含Source、Channel和Sink三个部分。可以通过修改Flume的配置文件来配置这些组件。 - **启动Agent**: 使用以下命令启动一个简单的Flume ...
Flume-ng在windows环境搭建并测试+log4j日志通过Flume输出到HDFS.docx
06-10
Flume-ng 在 Windows 环境搭建并测试 + Log4j 日志通过 Flume 输出到 HDFS Flume-ng 是一个高可用、可靠、分布式的日志聚合系统，可以实时地从各种数据源（如日志文件、网络 socket、数据库等）中收集数据，并将其...
flume1.6.0.chm
07-03
flume-1.6.0.chm由flume的doc制成，包含User Guide、Javadocs等
flume1.6+es5.5.1
08-22
flume1.6+es5.5.1
apache-flume-1.6.0-bin.tar,nc-1.84-22.el6.x86_64
12-24
apache-flume-1.6.0-bin.tar,nc-1.84-22.el6.x86_64 apache-flume-1.6.0-bin.tar,nc-1.84-22.el6.x86_64
apache-flume-1.8.0-bin.tar.gz
04-06
apache flume 1.8 官网下载很慢，在此提供最新的版本，供大家下载
flume1.6.0安装
u012246905的博客
03-03 530
1、下载apache-flume-1.6.0-bin.tar.gz 解压tar -zxvf apache-flume-1.6.0-bin.tar.gz 2、配置环境变量：vi /etc/profile export?FLUME_HOME=/A/soft/flume-1.6.0?? export?FLUME_CONF_DIR=$FLUME_HOME/conf?? export?PATH=$
Flume-ng 1.6.0安装、配置与使用
BlueSky
03-29 7171
1. 介绍 Flume NG是Cloudera提供的一个分布式、可靠、可用的系统，它能够将不同数据源的海量日志数据进行高效收集、聚合、移动，最后存储到一个中心化数据存储系统中。由原来的Flume OG到现在的Flume NG，进行了架构重构，并且现在NG版本完全不兼容原来的OG版本。经过架构重构后，Flume NG更像是一个轻量的小工具，非常简单，容易适应各种方式日志收集，并支持failov
Apache Flume1.6.0安装与部署
Meiyang1990的博客
06-17 3001
1、官网下载apache-flume-1.6.0-bin.tar.gz   ，解压缩        2、修改apache-flume-1.6.0-bin下面的conf配置目录，新建agent1.properties文件，内容如下：                agent1.channels.ch-1.type = memory                agent1.s
flume-1.6.0 的安装
会飞的猪的 博客
06-27 1659
下载apache-flume-1.6.0-bin.tar.gz 点击打开链接
flume 1.6 安装及配置 日志采集配置
不积跬步，无以至千里；不积小流，无以成江海。
09-29 1963
1.下载flume1.6 https://mirrors.tuna.tsinghua.edu.cn/apache/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz   2.安装jdk和Hadoop 具体参照以前wen'文章 3.flume 配置文件修改 修改conf目录下的flume-env.sh文件   export JAVA_HOME=/etc/...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值