Flume-ng分布式环境的部署和配置

最新推荐文章于 2023-10-27 18:13:00 发布
最新推荐文章于 2023-10-27 18:13:00 发布
阅读量1.5k 收藏 2
点赞数
分类专栏： flume

原文地址：http://www.linuxidc.com/Linux/2013-06/86598.htm


1.实验场景

操作系统：CentOS 5.5

JDK版本：1.7.0_21

Flume版本：1.3.1

Hadoop版本：0.20.2

 

配置1个agent ,2个collector,1个storage

 

2.安装步骤JDK+flume

#下载安装jdk1.7

http://www.Oracle.com/technetwork/java/javase/downloads/index.html

tar zxvf jdk-7u21-linux-x64.gz -C /usr/local/

 

#/etc/profile增加环境变量

pathmunge /usr/local/jdk1.7.0_21/bin

export JAVA_HOME=/usr/local/jdk1.7.0_21/

export JRE_HOME=/usr/local/jdk1.7.0_21/jre

export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar

 

#验证java

java -version

 

#下载安装flume 1.3.1

Flume的下载地址

http://www.apache.org/dyn/closer.cgi/flume/1.3.1/apache-flume-1.3.1-bin.tar.gz

tar zxvf flume-distribution-0.9.4-bin.tar.gz -C /usr/local/

 

#/etc/profile增加环境变量

export FLUME_HOME=/usr/local/apache-flume-1.3.1-bin

export FLUME_CONF_DIR=$FLUME_HOME/conf

export PATH=.:$PATH::$FLUME_HOME/bin

 

#验证 flume

# flume-ng version

Flume 1.3.1

Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git

Revision: 77b5d2885fecb3560a873bd89f49cbac8a010347

Compiled by hshreedharan on Fri Dec 21 22:14:21 PST 2012

From source with checksum 2565bdfd8b6af459dbf85c6960f189a5

 

3.一个简单的例子

#设置配置文件

[root@cc-staging-loginmgr2 conf]# cat example.conf

# example.conf: A single-node Flume configuration

 

# Name the components on this agent

a1.sources = r1

a1.sinks = k1

a1.channels = c1

 

# Describe/configure the source

a1.sources.r1.type = netcat

a1.sources.r1.bind = localhost

a1.sources.r1.port = 44444

 

# Describe the sink

a1.sinks.k1.type = logger

 

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

 

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

 

#命令参数说明

-c conf 指定配置目录为conf

-f conf/example.conf 指定配置文件为conf/example.conf

-n a1 指定agent名字为a1,需要与example.conf中的一致

-Dflume.root.logger=INFO,console 指定DEBUF模式在console输出INFO信息

 

#启动agent

cd /usr/local/apache-flume-1.3.1-bin

flume-ng agent -c conf -f conf/example.conf -n a1 -Dflume.root.logger=INFO,console

 

2013-05-24 00:00:09,288 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:150)] Source starting

2013-05-24 00:00:09,303 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:164)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]

 

#在另一个终端进行测试

[root@cc-staging-loginmgr2 conf]# telnet 127.0.0.1 44444

Trying 127.0.0.1...

Connected to localhost.localdomain (127.0.0.1).

Escape character is '^]'.

hello world!

OK

 

#在启动的终端查看console输出

2013-05-24 00:00:24,306 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 21 0D hello world!. }

 

#测试成功，flume可以正常使用

4. Flume Source测试

测试1:

avro source可以发送一个给定的文件给Flume，Avro 源使用AVRO RPC机制

#设置avro配置文件

[root@cc-staging-loginmgr2 conf]# cat avro.conf

# Name the components on this agent

a1.sources = r1

a1.sinks = k1

a1.channels = c1

 

# Describe/configure the source

a1.sources.r1.type = avro

a1.sources.r1.channels = c1

a1.sources.r1.bind = 0.0.0.0

a1.sources.r1.port = 4141

 

# Describe the sink

a1.sinks.k1.type = logger

 

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

 

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

 

#启动flume agent a1

cd /usr/local/apache-flume-1.3.1-bin/conf

flume-ng agent -c . -f avro.conf -n a1 -Dflume.root.logger=INFO,console

 

#创建指定文件

echo "hello world" > /usr/logs/log.10

 

#使用avro-client发送文件

flume-ng avro-client -c . -H localhost -p 4141 -F /usr/logs/log.10

 

#在启动的终端查看console输出

2013-05-27 01:11:45,852 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 hello world}

 

测试2:

Exec source runs a given Unix command on start-up and expects that process to continuously produce data on standard out

 

#修改的配置文件

[root@cc-staging-loginmgr2 conf]# cat exec.conf

# Describe/configure the source

a1.sources.r1.type = exec

a1.sources.r1.command = cat /usr/logs/log.10

a1.sources.r1.channels = c1

 

 

#启动flume agent a1

cd /usr/local/apache-flume-1.3.1-bin/conf

flume-ng agent -c . -f exec.conf -n a1 -Dflume.root.logger=INFO,console

 

#追加内容到文件

echo "exec test" >> /usr/logs/log.10

 

#在启动的终端查看console输出

2013-05-27 01:50:12,825 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 hello world }

2013-05-27 01:50:12,826 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 65 78 65 63 20 74 65 73 74 exec test }

 

#如果要使用tail命令，必选使得file足够大才能看到输出内容

a1.sources.r1.command = tail -F /usr/logs/log.10

 

#生成足够多的内容在文件里

for i in {1..100};do echo "exec test$i" >> /usr/logs/log.10;echo $i;done

 

#可以在console看到output

2013-05-27 19:17:18,157 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.ExecSource.start(ExecSource.java:155)] Exec source starting with command:tail -n 5 -F /usr/logs/log.10

2013-05-27 19:19:50,334 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 65 78 65 63 20 74 65 73 74 37 exec test7 }

 

测试3:

Spooling directory source

This source lets you ingest data by dropping files in a spooling directory on disk. Unlike other asynchronous sources, this source avoids data loss even if Flume is restarted or fails.

SpoolSource:是监测配置的目录下新增的文件，并将文件中的数据读取出来。需要注意两点：1) 拷贝到spool目录下的文件不可以再打开编辑。

2) spool目录下不可包含相应的子目录

 

#修改的配置文件

[root@cc-staging-loginmgr2 conf]# cat spool.conf

# Describe/configure the source

a1.sources.r1.type = spooldir

a1.sources.r1.spoolDir = /usr/logs/flumeSpool

a1.sources.r1.fileHeader = true

a1.sources.r1.channels = c1

 

#启动flume agent a1

cd /usr/local/apache-flume-1.3.1-bin/conf

flume-ng agent -c . -f spool.conf -n a1 -Dflume.root.logger=INFO,console

 

#追加内容到spool目录

[root@cc-staging-loginmgr2 ~]# echo "spool test1" > /usr/logs/flumeSpool/spool1.log

 

#在启动的终端查看console输出

2013-05-27 22:49:06,098 (pool-4-thread-1) [INFO - org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile(SpoolingFileLineReader.java:229)] Preparing to move file /usr/logs/flumeSpool/spool1.log to /usr/logs/flumeSpool/spool1.log.COMPLETED

2013-05-27 22:49:06,101 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{file=/usr/logs/flumeSpool/spool1.log} body: 73 70 6F 6F 6C 20 74 65 73 74 31 spool test1 }

 

测试4

Netcat source 参见第3部分一个简单的例子

 

测试5

Syslog tcp source

 

#修改的配置文件

[root@cc-staging-loginmgr2 conf]# cat syslog.conf

# Describe/configure the source

a1.sources.r1.type = syslogtcp

a1.sources.r1.port = 5140

a1.sources.r1.host = localhost

a1.sources.r1.channels = c1

 

#启动flume agent a1

cd /usr/local/apache-flume-1.3.1-bin/conf

flume-ng agent -c . -f syslog.conf -n a1 -Dflume.root.logger=INFO,console

 

#测试产生syslog, <37>因为需要wire format数据，否则会报错” Failed to extract syslog wire entry”

echo "<37>hello via syslog" | nc localhost 5140

 

#在启动的终端查看console输出

2013-05-27 23:39:10,755 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 68 65 6C 6C 6F 20 76 69 61 20 73 79 73 6C 6F 67 hello via syslog }

 

#UDP需要修改配置文件

a1.sources.r1.type = syslogudp

a1.sources.r1.port = 5140

a1.sources.r1.host = localhost

a1.sources.r1.channels = c1

 

#测试产生syslog

echo "<37>hello via syslog" | nc -u localhost 5140

 

测试6

HTTP source JSONHandler

 

#修改的配置文件

[root@cc-staging-loginmgr2 conf]# cat post.conf

# Describe/configure the source

a1.sources = r1

a1.channels = c1

a1.sources.r1.type = org.apache.flume.source.http.HTTPSource

a1.sources.r1.port = 5140

a1.sources.r1.channels = c1

 

#启动flume agent a1

cd /usr/local/apache-flume-1.3.1-bin/conf

flume-ng agent -c . -f post.conf -n a1 -Dflume.root.logger=INFO,console

 

#生成JSON 格式的POST request

curl -X POST -d '[{ "headers" :{"namenode" : "namenode.example.com","datanode" : "random_datanode.example.com"},"body" : "really_random_body"}]' http://localhost:5140

 

#在启动的终端查看console输出

2013-05-28 01:17:47,186 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{namenode=namenode.example.com, datanode=random_datanode.example.com} body: 72 65 61 6C 6C 79 5F 72 61 6E 64 6F 6D 5F 62 6F really_random_bo }


1.flume sink 测试

测试1 #hdfs sink

Using this sink requires Hadoop to be installed so that Flume can use the Hadoop jars to communicate with the HDFS cluster

需要安装hadoop

 

 

在/usr/local/apache-flume-1.3.1-bin/conf/flume-env.sh加入

export HADOOP_HOME=/usr/local/hadoop

 

#修改配置文件

a1.sources.r1.type = syslogtcp

a1.sources.r1.bind = 0.0.0.0

a1.sources.r1.port = 5140

a1.sources.r1.channels = c1

 

a1.sinks.k1.type = hdfs

a1.sinks.k1.channel = c1

a1.sinks.k1.hdfs.path = hdfs://master:9000/user/hadoop/flume/collected/

a1.sinks.k1.hdfs.filePrefix = Syslog

a1.sinks.k1.hdfs.round = true

a1.sinks.k1.hdfs.roundValue = 10

a1.sinks.k1.hdfs.roundUnit = minute

 

#启动flume agent a1

cd /usr/local/apache-flume-1.3.1-bin/conf

flume-ng agent -c . -f hdfs.conf -n a1 -Dflume.root.logger=INFO,console

 

#测试产生syslog

echo "<37>hello via syslog to hdfs testing one" | nc -u localhost 5140

 

#在启动的终端查看console输出,文件生成成功

2013-05-29 00:53:58,078 (hdfs-k1-call-runner-0) [INFO - org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:208)] Creating hdfs://master:9000/user/hadoop/flume/collected//Syslog.1369814037714.tmp

2013-05-29 00:54:28,220 (hdfs-k1-roll-timer-0) [INFO - org.apache.flume.sink.hdfs.BucketWriter.renameBucket(BucketWriter.java:427)] Renaming hdfs://master:9000/user/hadoop/flume/collected/Syslog.1369814037714.tmp to hdfs://master:9000/user/hadoop/flume/collected/Syslog.1369814037714

 

#在hadoop上查看文件

./hadoop dfs -cat hdfs://172.25.4.35:9000/user/hadoop/flume/collected/Syslog.1369814037714

SEQ!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable^;>Gv$hello via syslog to hdfs testing one

 

#修改配置文件以时间形式自动生成目录

a1.sources.r1.type = org.apache.flume.source.http.HTTPSource

a1.sources.r1.bind = 0.0.0.0

a1.sources.r1.port = 5140

a1.sources.r1.channels = c1

 

# Describe the sink

a1.sinks.k1.type = hdfs

a1.sinks.k1.channel = c1

a1.sinks.k1.hdfs.path = hdfs://master:9000/user/hadoop/flume/collected/%y-%m-%d/%H%M/%S

a1.sinks.k1.hdfs.filePrefix = Syslog.%{host}

a1.sinks.k1.hdfs.round = true

a1.sinks.k1.hdfs.roundValue = 10

a1.sinks.k1.hdfs.roundUnit = minute

 

#生成JSON 格式的POST request, header的timestamp 参数如果格式不对则无法解析

需要生成13为的timestamp才能解析出正确的时间,包含MilliSec

#linux生成当前时间10位Unix timestamp

date +%s

#linux生成当前时间13位Unix timestamp

date +%s%N|awk '{print substr($0,1,13)}'

 

curl -X POST -d '[{ "headers":{"timestamp":"1369818213654","host":"cc-staging-loginmgr2"},"body": "hello via post"}]' http://localhost:5140

 

#在启动的终端查看console输出,文件生成成功

2013-05-29 02:03:38,646 (hdfs-k1-call-runner-4) [INFO - org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:208)] Creating hdfs://master:9000/user/hadoop/flume/collected/2013-05-29/0203/cc-staging-loginmgr2..1369818218614.tmp

2013-05-29 02:04:08,714 (hdfs-k1-roll-timer-0) [INFO - org.apache.flume.sink.hdfs.BucketWriter.renameBucket(BucketWriter.java:427)] Renaming hdfs://master:9000/user/hadoop/flume/collected/2013-05-29/0203/cc-staging-loginmgr2..1369818218614.tmp to hdfs://master:9000/user/hadoop/flume/collected/2013-05-29/0203/cc-staging-loginmgr2..1369818218614

 

#在hadoop上查看文件

./hadoop dfs -ls hdfs://172.25.4.35:9000/user/hadoop/flume/collected/2013-05-29/0203

Found 1 items

-rw-r--r-- 3 root supergroup 129 2013-05-29 02:04 /user/hadoop/flume/collected/2013-05-29/0203/cc-staging-loginmgr2..1369818218614

 

#测试2 logger sink

Logs event at INFO level. Typically useful for testing/debugging purpose

 

#测试3 Avro sink

Flume events sent to this sink are turned into Avro events and sent to the configured hostname / port pair

 

#Avro Source配置文件

a1.sources.r1.type = avro

a1.sources.r1.channels = c1

a1.sources.r1.bind = 0.0.0.0

a1.sources.r1.port = 4545

 

#Avro Sink配置文件

a1.sinks.k1.type = avro

a1.sinks.k1.channel = c1

a1.sinks.k1.hostname = 172.25.4.23

a1.sinks.k1.port = 4545

 

#先启动Avro的Source,监听端口

cd /usr/local/apache-flume-1.3.1-bin/conf

flume-ng agent -c . -f avro.conf -n a1 -Dflume.root.logger=INFO,console

 

#再启动Avro的Sink

cd /usr/local/apache-flume-1.3.1-bin/conf

flume-ng agent -c . -f avro_sink.conf -n a1 -Dflume.root.logger=INFO,console

 

#可以看到已经建立连接

2013-06-02 19:23:00,237 (pool-5-thread-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x7a0e28bf, /172.25.4.32:14894 => /172.25.4.23:4545] CONNECTED: /172.25.4.32:14894

 

#在Avro Sink上生成测试log

echo "<37>hello via avro sink" | nc localhost 5140

 

#在Avro Source上可以看到log已经生成

2013-06-02 19:24:13,740 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 68 65 6C 6C 6F 20 76 69 61 20 61 76 72 6F 20 73 hello via avro s }

 

#测试4 File Roll Sink

Stores events on the local filesystem

 

#修改配置文件

a1.sinks.k1.type = file_roll

a1.sinks.k1.channel = c1

a1.sinks.k1.sink.directory = /var/log/flume

 

#启动file roll 配置文件

cd /usr/local/apache-flume-1.3.1-bin/conf

flume-ng agent -c . -f file_roll.conf -n a1 -Dflume.root.logger=INFO,console

 

#生成测试log

echo "<37>hello via file roll" | nc localhost 5140

echo "<37>hello via file roll 2" | nc localhost 5140

 

#查看/var/log/flume下是否生成文件,默认每30秒生成一个新文件

-rw-r--r-- 1 root root 20 Jun 2 19:44 1370227443397-1

-rw-r--r-- 1 root root 0 Jun 2 19:44 1370227443397-2

-rw-r--r-- 1 root root 22 Jun 2 19:45 1370227443397-3

 

cat 1370227443397-1 1370227443397-3

hello via file roll

hello via file roll 2


2.Flume Channels测试

#Memory Channel

The events are stored in a an in-memory queue with configurable max size. It’s ideal for flow that needs higher throughput and prepared to lose the staged data in the event of a agent failures

 

#flume channel selectors

# Replicating Channel Selector通道复制测试

#2个channel和2个sink的配置文件

# Name the components on this agent

a1.sources = r1

a1.sinks = k1 k2

a1.channels = c1 c2

 

# Describe/configure the source

a1.sources.r1.type = syslogtcp

a1.sources.r1.port = 5140

a1.sources.r1.host = localhost

a1.sources.r1.selector.type = replicating

a1.sources.r1.channels = c1 c2

 

# Describe the sink

a1.sinks.k1.type = avro

a1.sinks.k1.channel = c1

a1.sinks.k1.hostname = 172.25.4.23

a1.sinks.k1.port = 4545

 

a1.sinks.k2.type = avro

a1.sinks.k2.channel = c2

a1.sinks.k2.hostname = 172.25.4.33

a1.sinks.k2.port = 4545

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

 

a1.channels.c2.type = memory

a1.channels.c2.capacity = 1000

a1.channels.c2.transactionCapacity = 100

 

#查看是否都建立了连接

2013-06-04 00:01:53,467 (pool-5-thread-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x122a0fad, /172.25.4.32:55518 => /172.25.4.23:4545] BOUND: /172.25.4.23:4545

2013-06-04 00:01:53,467 (pool-5-thread-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x122a0fad, /172.25.4.32:55518 => /172.25.4.23:4545] CONNECTED: /172.25.4.32:55518

 

2013-06-04 00:01:53,773 (pool-5-thread-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x021881a7, /172.25.4.32:23731 => /172.25.4.33:4545] BOUND: /172.25.4.33:4545

2013-06-04 00:01:53,773 (pool-5-thread-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x021881a7, /172.25.4.32:23731 => /172.25.4.33:4545] CONNECTED: /172.25.4.32:23731

 

#生成测试log

echo "<37>hello via channel selector" | nc localhost 5140

 

#查看2个sink是否得到数据

2013-06-04 00:02:06,479 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 68 65 6C 6C 6F 20 76 69 61 20 63 68 61 6E 6E 65 hello via channe }

 

2013-06-04 00:02:09,788 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 68 65 6C 6C 6F 20 76 69 61 20 63 68 61 6E 6E 65 hello via channe }

 

#flume channel selectors

# Multiplexing Channel Selector 通道复用测试

#2个channel和2个sink的配置文件

a1.sources = r1

a1.sinks = k1 k2

a1.channels = c1 c2

 

# Describe/configure the source

a1.sources.r1.type = org.apache.flume.source.http.HTTPSource

a1.sources.r1.port = 5140

a1.sources.r1.host = 0.0.0.0

a1.sources.r1.selector.type = multiplexing

a1.sources.r1.channels = c1 c2

 

a1.sources.r1.selector.header = state

a1.sources.r1.selector.mapping.CZ = c1

a1.sources.r1.selector.mapping.US = c2

a1.sources.r1.selector.default = c1

 

# Describe the sink

a1.sinks.k1.type = avro

a1.sinks.k1.channel = c1

a1.sinks.k1.hostname = 172.25.4.23

a1.sinks.k1.port = 4545

 

a1.sinks.k2.type = avro

a1.sinks.k2.channel = c2

a1.sinks.k2.hostname = 172.25.4.33

a1.sinks.k2.port = 4545

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

 

a1.channels.c2.type = memory

a1.channels.c2.capacity = 1000

a1.channels.c2.transactionCapacity = 100

 

#根据配置文件生成测试的header 为state的POST请求

curl -X POST -d '[{ "headers" :{"state" : "CZ"},"body" : "TEST1"}]' http://localhost:5140

curl -X POST -d '[{ "headers" :{"state" : "US"},"body" : "TEST2"}]' http://localhost:5140

curl -X POST -d '[{ "headers" :{"state" : "SH"},"body" : "TEST3"}]' http://localhost:5140

 

#查看2个sink得到数据是否和配置文件一致

Sink1:

2013-06-04 23:45:35,296 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{state=CZ} body: 54 45 53 54 31 TEST1 }

2013-06-04 23:45:50,309 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{state=SH} body: 54 45 53 54 33 TEST3 }

 

Sink2:

2013-06-04 23:45:42,293 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{state=US} body: 54 45 53 54 32 TEST2 }


1.Flume Sink Processors测试

#Failover Sink Processor

Failover Sink Processor maintains a prioritized list of sinks, guaranteeing that so long as one is available events will be processed (delivered)

#配置文件

# Name the components on this agent

a1.sources = r1

a1.sinks = k1 k2

a1.channels = c1 c2

 

a1.sinkgroups = g1

a1.sinkgroups.g1.sinks = k1 k2

a1.sinkgroups.g1.processor.type = failover

a1.sinkgroups.g1.processor.priority.k1 = 5

a1.sinkgroups.g1.processor.priority.k2 = 10

a1.sinkgroups.g1.processor.maxpenalty = 10000

 

# Describe/configure the source

a1.sources.r1.type = syslogtcp

a1.sources.r1.port = 5140

a1.sources.r1.host = localhost

a1.sources.r1.selector.type = replicating

a1.sources.r1.channels = c1 c2

 

# Describe the sink

a1.sinks.k1.type = avro

a1.sinks.k1.channel = c1

a1.sinks.k1.hostname = 172.25.4.23

a1.sinks.k1.port = 4545

 

a1.sinks.k2.type = avro

a1.sinks.k2.channel = c2

a1.sinks.k2.hostname = 172.25.4.33

a1.sinks.k2.port = 4545

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

 

a1.channels.c2.type = memory

a1.channels.c2.capacity = 1000

a1.channels.c2.transactionCapacity = 100

 

#生成测试log

echo "<37>test1 failover" | nc localhost 5140

 

#在sink2上产生log，sink1由于优先级小，没有产生

2013-06-05 00:10:51,194 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 74 65 73 74 31 20 66 61 69 6C 6F 76 65 72 test1 failover }

 

#主动关闭sink2,再次生成测试log

echo "<37>test2 failover" | nc localhost 5140

 

#在sink1上会同时生成test1和test2

2013-06-05 00:11:14,312 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 74 65 73 74 31 20 66 61 69 6C 6F 76 65 72 test1 failover }

2013-06-05 00:11:14,312 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 74 65 73 74 32 20 66 61 69 6C 6F 76 65 72 test2 failover }

 

#再次打开sink2,log会根据优先级再到sink2上

echo "<37>test4 failover" | nc localhost 5140

echo "<37>test5 failover" | nc localhost 5140

 

2013-06-05 00:12:33,071 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 74 65 73 74 34 20 66 61 69 6C 6F 76 65 72 test4 failover }

2013-06-05 00:12:55,088 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 74 65 73 74 35 20 66 61 69 6C 6F 76 65 72 test5 failover }

 

#Load balancing Sink Processor测试

Load balancing sink processor provides the ability to load-balance flow over multiple sinks. It maintains an indexed list of active sinks on which the load must be distributed.

 

#配置文件,注:load balance type下必须指定同一个channel到不同的sinks,否则不生效

# Name the components on this agent

a1.sources = r1

a1.sinks = k1 k2

a1.channels = c1

 

a1.sinkgroups = g1

a1.sinkgroups.g1.sinks = k1 k2

a1.sinkgroups.g1.processor.type = load_balance

a1.sinkgroups.g1.processor.backoff = true

a1.sinkgroups.g1.processor.selector = round_robin

 

# Describe/configure the source

a1.sources.r1.type = syslogtcp

a1.sources.r1.port = 5140

a1.sources.r1.host = localhost

a1.sources.r1.channels = c1

 

# Describe the sink

a1.sinks.k1.type = avro

a1.sinks.k1.channel = c1

a1.sinks.k1.hostname = 172.25.4.23

a1.sinks.k1.port = 4545

 

a1.sinks.k2.type = avro

a1.sinks.k2.channel = c1

a1.sinks.k2.hostname = 172.25.4.33

a1.sinks.k2.port = 4545

 

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

 

#生成4个测试log

[root@cc-staging-loginmgr2 ~]# echo "<37>test2 loadbalance" | nc localhost 5140

[root@cc-staging-loginmgr2 ~]# echo "<37>test3 loadbalance" | nc localhost 5140

[root@cc-staging-loginmgr2 ~]# echo "<37>test4 loadbalance" | nc localhost 5140

[root@cc-staging-loginmgr2 ~]# echo "<37>test5 loadbalance" | nc localhost 5140

 

#查看sink输出结果是否为轮询模式

Sink1:

2013-06-06 01:36:03,516 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 74 65 73 74 32 20 6C 6F 61 64 62 61 6C 61 6E 63 test2 loadbalanc }

2013-06-06 01:36:09,769 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 74 65 73 74 34 20 6C 6F 61 64 62 61 6C 61 6E 63 test4 loadbalanc }

 

Sink2:

2013-06-06 01:36:05,809 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 74 65 73 74 33 20 6C 6F 61 64 62 61 6C 61 6E 63 test3 loadbalanc }

2013-06-06 01:36:37,057 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 74 65 73 74 35 20 6C 6F 61 64 62 61 6C 61 6E 63 test5 loadbalanc }


2. Event Serializers测试

Body Text Serializer

Alias: text. This interceptor writes the body of the event to an output stream without any transformation or modification(把body中的内容变成文本内容)

 

#配置文件

a1.sources.r1.type = org.apache.flume.source.http.HTTPSource

a1.sources.r1.port = 5140

a1.sources.r1.host = localhost

a1.sources.r1.channels = c1

 

# Describe the sink

a1.sinks.k1.type = file_roll

a1.sinks.k1.channel = c1

a1.sinks.k1.sink.directory = /var/log/flume

a1.sinks.k1.sink.serializer = text

a1.sinks.k1.sink.serializer.appendNewline = false

 

#生成测试log

curl -X POST -d '[{ "headers" :{"host":"cc-staging-loginmgr2"},"body" : "TEST1 BODY TEXT"}]' http://localhost:5140

curl -X POST -d '[{ "headers" :{"host":"cc-staging-loginmgr2"},"body" : "TEST2 BODY TEXT"}]' http://localhost:5140

curl -X POST -d '[{ "headers" :{"host":"cc-staging-loginmgr2"},"body" : "TEST3 BODY TEXT"}]' http://localhost:5140

 

#查看file roll 文件中的文本内容

cat /var/log/flume/1370675739270-1

TEST1 BODY TEXT

TEST2 BODY TEXT

TEST3 BODY TEXT

 

#Avro Event Serializer

Alias: avro_event. This interceptor serializes Flume events into an Avro container file

把flume event变成avro 中包含的文件

 

 

1.Flume Interceptors测试

Timestamp Interceptor

This interceptor inserts into the event headers, the time in millis at which it processes the event. This interceptor inserts a header with key timestamp whose value is the relevant timestamp

 

Host Interceptor

This interceptor inserts the hostname or IP address of the host that this agent is running on. It inserts a header with key host or a configured key whose value is the hostname or IP address of the host

 

#配置文件

# Name the components on this agent

a1.sources = r1

a1.sinks = k1

a1.channels = c1

 

# Describe/configure the source

a1.sources.r1.type = syslogtcp

a1.sources.r1.bind = 0.0.0.0

a1.sources.r1.port = 5140

a1.sources.r1.channels = c1

 

a1.sources.r1.interceptors = i1 i2

a1.sources.r1.interceptors.i1.preserveExisting = false

a1.sources.r1.interceptors.i1.type = timestamp

a1.sources.r1.interceptors.i2.type = host

a1.sources.r1.interceptors.i2.hostHeader = hostname

a1.sources.r1.interceptors.i2.useIP = false

 

# Describe the sink

a1.sinks.k1.type = hdfs

a1.sinks.k1.channel = c1

a1.sinks.k1.hdfs.path = hdfs://master:9000/user/Hadoop/flume/collected/%Y-%m-%d/%H%M

a1.sinks.k1.hdfs.filePrefix = %{hostname}.

 

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

 

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

 

#启动agent

cd /usr/local/apache-flume-1.3.1-bin/conf

flume-ng agent -c . -f dynamic_intercept.conf -n a1 -Dflume.root.logger=INFO,console

 

#生成测试log

echo "<37>test dynamic interceptor" | nc localhost 5140

 

#查看hdfs生成的文件,可以看到timestamp和hostname都已经生成在header里面,可以根据自定义的格式生成文件夹

./hadoop dfs -ls hdfs://172.25.4.35:9000/user/hadoop/flume/collected/2013-06-16/2331/

Found 1 items

-rw-r--r-- 3 root supergroup 140 2013-06-16 23:32 /user/hadoop/flume/collected/2013-06-16/2331/cc-staging-loginmgr2..1371450697118

 

Static Interceptor

Static interceptor allows user to append a static header with static value to all events

 

#配置文件

# Name the components on this agent

a1.sources = r1

a1.sinks = k1

a1.channels = c1

 

# Describe/configure the source

a1.sources.r1.type = syslogtcp

a1.sources.r1.port = 5140

a1.sources.r1.host = localhost

a1.sources.r1.channels = c1

a1.sources.r1.interceptors = i1

a1.sources.r1.interceptors.i1.type = static

a1.sources.r1.interceptors.i1.key = datacenter

a1.sources.r1.interceptors.i1.value = NEW_YORK

 

# Describe the sink

a1.sinks.k1.type = logger

 

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

 

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

 

#启动agent

cd /usr/local/apache-flume-1.3.1-bin/conf

flume-ng agent -c . -f dynamic_intercept.conf -n a1 -Dflume.root.logger=INFO,console

 

#生成测试log

echo "<37>test1 static interceptor" | nc localhost 5140

 

#查看console输出结果

2013-06-17 00:15:38,453 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4, datacenter=NEW_YORK} body: 74 65 73 74 31 20 73 74 61 74 69 63 20 69 6E 74 test1 static int }


2. zabbix监控Flume

#JVM性能监控

Young GC counts

sudo /usr/local/jdk1.7.0_21/bin/jstat -gcutil $(pgrep java)|tail -1|awk '{print $6}'

 

Full GC counts

sudo /usr/local/jdk1.7.0_21/bin/jstat -gcutil $(pgrep java)|tail -1|awk '{print $8}'

 

JVM total memory usage

sudo /usr/local/jdk1.7.0_21/bin/jmap -histo $(pgrep java)|grep Total|awk '{print $3}'

 

JVM total instances usage

sudo /usr/local/jdk1.7.0_21/bin/jmap -histo $(pgrep java)|grep Total|awk '{print $2}'

 

#flume应用参数监控

启动时加上JSON repoting参数,这样就可以通过http://localhost:34545/metrics访问

flume-ng agent -c . -f exec.conf -n a1 -Dflume.root.logger=INFO,console -Dflume.monitoring.type=http -Dflume.monitoring.port=34545

 

#生成一些数据

for i in {1..100};do echo "exec test$i" >> /usr/logs/log.10;echo $i;done

 

#通过shell脚本对JSON输出进行排版

[root@cc-staging-loginmgr2 conf]# curl http://localhost:34545/metrics 2>/dev/null|sed -e 's/\([,]\)\s*/\1\n/g' -e 's/[{}]/\n/g' -e 's/[",]//g'

 

CHANNEL.c1:

EventPutSuccessCount:100

ChannelFillPercentage:0.0

Type:CHANNEL

StopTime:0

EventPutAttemptCount:100

ChannelSize:0

StartTime:1371709073310

EventTakeSuccessCount:100

ChannelCapacity:1000

EventTakeAttemptCount:115

 

#配置监控flume的脚本文件

[root@cc-staging-loginmgr2 conf]#cat /opt/scripts/monitor_flume.sh

curl http://localhost:34545/metrics 2>/dev/null|sed -e 's/\([,]\)\s*/\1\n/g' -e 's/[{}]/\n/g' -e 's/[",]//g'|grep $1|awk -F: '{print $2}'

 

#在zabbix agent配置文件进行部署

cat /etc/zabbix/zabbix_agentd/zabbix_agentd.userparams.conf

UserParameter=ygc.counts,sudo /usr/local/jdk1.7.0_21/bin/jstat -gcutil $(pgrep java|head -1)|tail -1|awk '{print $6}'

UserParameter=fgc.counts,sudo /usr/local/jdk1.7.0_21/bin/jstat -gcutil $(pgrep java|head -1)|tail -1|awk '{print $8}'

UserParameter=jvm.memory.usage,sudo /usr/local/jdk1.7.0_21/bin/jmap -histo $(pgrep java|head -1)|grep Total|awk '{print $3}'

UserParameter=jvm.instances.usage,sudo /usr/local/jdk1.7.0_21/bin/jmap -histo $(pgrep java|head -1)|grep Total|awk '{print $2}'

UserParameter=flume.monitor[*],/bin/bash /opt/scripts/monitor_flume.sh $1


qinzl_1
关注
专栏目录
大数据平台分布式搭建 - Flume部署
Santorinisu Blogs
11-15 791
Section I：文件清单 1.apache-flume-1.8.0-bin.tar.gz Section II: 下载链接 [Flume 下载链接]：http://flume.apache.org/releases/index.html Section III: 通信工具Telnet和Flume部署 总览，集群信息： 节点角色 Master Slave1 Slave2 IP 1...
Flume-ng分布式部署和配置
程序员成长之路
04-07 2594
Flume-ng分布式部署总结 1.日志总接收端的配置（负责接收各个节点发送过来的日志数据）,修改flume的安装目录下/conf/flume-conf.properties配置文件（默认没有，cp flumeconf.properties.template flume-conf.properties） agent.sources = avrosrc   agent.channels = m
Hadoop生态圈-flume日志收集工具完全分布式部署
weixin_34414196的博客
06-15 281
　　　　　　　　　　　　　　　　　　　　　　　Hadoop生态圈-flume日志收集工具完全分布式部署 　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰 版权声明：原创作品，谢绝转载！否则将追究法律责任。    　　目前为止，Hadoop的一个主流应用就是对于大规模web日志的分析和处理，因此想要把web服务的日志导入到Hadoo...
Flume-ng分布式环境的部署和配置(一)
weixin_33877885的博客
06-27 222
1.实验场景操作系统：Centos 5.5JDK版本：1.7.0_21Flume版本：1.3.1Hadoop版本：0.20.2配置1个agent ,2个collector,1个storage2.安装步骤JDK+flume#下载安装jdk1.7http://www.oracle.com/technetwork/java/javase/downloads/index.htmlta...
Flume分布式海量日志收集框架搭建部署
何星驰的专栏@17316365004@sina.cn
01-18 1712
Flume官网 Flume部署: 下载安装包并上传服务器解压 /apache-flume-1.6.0-bin/conf目录下,将flume-env.sh.template模板文件改名为flume-env.sh,并配置jdk 配置局部环境变量 source ~/.bash_profilesource命令生效 执行如下命令: yum install telnet -y yum in...
flume的source测试赫赫有名
oO_DoraHe的博客
03-23 153
public class MySource extends AbstractSource implements Configurable, PollableSource { //前缀参数 private String prefix; //后缀参数 private String suffix; //数据发送延迟参数 private Long delay; //数据条数 private int n; ...
Flume-ng在windows环境搭建并测试+log4j日志通过Flume输出到HDFS.docx
06-10
在 conf 文件夹中创建 example.conf 文件，配置 Flume-ng 的源、 sink 和通道。使用 telnet 连接 source 写入数据，产生日志数据输出控制台。启动 Flume-ng，使用 telnet 命令写入一些数据，让 Flume 检测。 三、...
flume-ng-1.6.0 cdh5.7.0安装包
03-01
Apache Flume 是一个分布式、可靠且可用于有效收集、聚合和移动大量日志数据的系统。在大数据处理领域，Flume 是一个至关重要的组件，尤其在数据集成和实时数据流处理中。它以其高可用性、容错性和可扩展性而闻名，...
flume-ng-1.6.0-cdh5.5.0.tar.gz
03-07
用户可以通过 Cloudera Manager 这样的管理工具来部署、配置和监控 Flume 实例，简化了大数据环境中的运维工作。 "apache-flume-1.6.0-cdh5.5.0-bin" 是解压后的 Flume 安装包，包含了所有运行 Flume 所需的二进制...
flume-ng-1.6.0-cdh5.10.1.tar.gz的下载
09-15
5. **集成CDH**：`flume-ng-1.6.0-cdh5.10.1` 版本是专门为CDH 5.10.1设计的，这意味着它已经与CDH中的其他组件（如Hadoop、Hive等）进行了很好的集成和优化，可以直接在CDH环境中部署和运行。 解压`apache-flume-...
flume的sink测试赫赫有名
oO_DoraHe的博客
03-23 242
public class MySink extends AbstractSink implements Configurable { //前缀 private String prefix; //后缀 private String suffix; //创建你logger对象 private static final Logger LOG = LoggerFactory.getLogger(Abstr...
大数据学习5之分布式日志收集框架Flume——Flume环境部署
level_Tiller的博客
03-21 167
Flume环境部署 Flume实战
hadoop-flume完全分布式集群搭建
柒柒柒柒
11-23 2830
本次使用的是flume1.7.0版本 以及hadoop-2.7.7版本 flume下载地址如下： 链接：https://pan.baidu.com/s/193j6s-FNePUShCR5ZPNRWg 提取码：qwer 一、解压spark文件 本次依旧解压在opt文件夹中。 #解压 tar zxvf apache-flume-1.7.0-bin.tar.gz #重命名 mv apache-flume-1.7.0-bin flume-1.7.0 二、修改flume-env.sh文件 #进入文件夹 cd .
flume source、sink、Channels测试
大数据技术杂谈
01-08 1007
3.一个简单的例子 #设置配置文件 [root@cc-staging-loginmgr2 conf]# cat example.conf # example.conf: A single-node Flume configuration # Name the components on this agent a1.sources = r1 a1.sink
Flume sink
mthinking
04-23 292
1.hdfs a1.sources = r1 a1.channels = c1 a1.sinks = k1 a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 8888 a1.sinks.k1.type = hdfs a1....
Flume安装部署及使用
最新发布
xtsheng123456的博客
10-27 567
Aapche Flume是由Cloudera提供的一个高可用的，高可靠的，分布式的海量日志采集、聚合和传输的软件，网址： http://flume.apache.org/Apache Flume的核心是把数据从数据源(source)收集过来，再将收集到的数据送到指定的目的地(sink)。为了保证输送的过程一定成功，在送到目的地(sink)之前，会先缓存数据(channel)，待数据真正到达目的地(sink)后，flume在删除自己缓存的数据。
Flume学习系列(六)---- Logger Sink源码解读与自定义Sink初探
Badme
08-22 1522
前言：接上一篇，我们使用flume自带的logger有些bug，只能显示前16个字节。本文将介绍如何解决这个问题，并分析Logger Sink的源码进而得出自定义Sink的一般流程。 一、解决Logger Sink问题 1.1 尝试使用maxBytesToLog属性 之前我们说过，有问题就去官方文档看看，有关Logger Sink的内容如下： 001.png ...
Flume-ng安装与使用
weixin_33698823的博客
02-25 295
一、 Flume NG核心概念 二、 Flumen NG数据流模型Flume以agent为最小的独立运行单位。一个agent就是一个JVM。单agent由Source、Sink和Channel三大组件构成。 Flume的数据流由事件(Event)贯穿始终。事件是Flume的基本数据单位，它携带日志数据(字节数组形式)并且携带有头信息，这些Event由Agen...
Flume-ng整合HBase实战：从数据源到Hbase存储
整合Flume-ng与Hbase的这种方案适用于实时日志分析场景，特别是在大数据环境下，它能够高效地处理大量流入的数据，并将其有效地存储到分布式数据库中，如Hbase，以支持后续的分析和查询操作。通过合理配置Flume的...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值