常用flume操作

flume官网（官网介绍及使用方式都比较全面）：http://flume.apache.org/index.html

1、netcat–logger

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动flume

flume-ng agent --conf-file /root/flume/tt --name a1 -Dflume.root.logger=INFO,console

在source对应的节点上启动telnet

telnet localhost 44444

在telnet输入相应东西，回车，flume端会显示打印

2、exec–logger

a1.sources=r1
a1.channels=c1
a1.sinks=k1

a1.sources.r1.type=exec
a1.sources.r1.command=tail -F /root/flume/log

a1.sinks.k1.type=logger

a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100

a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

启动flume对应文件

flume-ng agent --conf-file /root/flume/exec --name a1 -Dflume.root.logger=INFO,console

此时会监测/root/flume/log文件，一旦有文件输入，flume会获取并打印展示

3、avro–logger

 a1.sources = r1
 a1.sinks = k1
 a1.channels = c1

 a1.sources.r1.type=avro
 a1.sources.r1.channels = c1
 a1.sources.r1.bind=192.168.30.101
 a1.sources.r1.port=55555

 # Describe the sink
 a1.sinks.k1.type = logger

 # Use a channel which buffers events in memory
 a1.channels.c1.type = memory
 a1.channels.c1.capacity = 1000
 a1.channels.c1.transactionCapacity = 100

 # Bind the source and sink to the channel

 a1.sinks.k1.channel = c1
 a1.sinks.k1.type=logger

启动flume

flume-ng agent --conf-file /root/flume/avro --name a1 -Dflume.root.logger=INFO,console

flume-ng avro-client -H localhost -p 55555 -F /root/flume/log

4、netcat–hdfs

a1.channels=c1
a1.sources=r1
a1.sinks=k1

a1.sources.r1.type=netcat
a1.sources.r1.bind=localhost
a1.sources.r1.port=41416

a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://sx/myflume/%y-%m-%d
a1.sinks.k1.hdfs.useLocalTimeStamp=true

a1.channels.c1.type=memory

a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

启动flume

flume-ng agent --conf-file /root/flume/netcat_hdfs --name a1 -Dflume.root.logger=INFO,console

启动telnet（对应端口）

telnet localhost 41416

5、avro–hdfs

# a1 which ones we want to activate.
a1.channels = c1
a1.sources = r1
a1.sinks = k1

a1.sources.r1.type = avro
a1.sources.r1.bind=localhost
a1.sources.r1.port=55555

a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://sxt/myflume

# Define a memory channel called c1 on a1
a1.channels.c1.type = memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity = 100

# Define an Avro source called r1 on a1 and tell it
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

flume

flume-ng agent --conf-file /root/flume/avro_hdfs --name a1 -Dflume.root.logger=INFO,console

flume-ng avro-client -H localhost -p 55555 -F /root/flume/log

6、exec–hdfs

a1.sources=r1
a1.channels=c1
a1.sinks=k1

a1.sources.r1.type=exec
a1.sources.r1.command=tail -F /root/flume/log

a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path = hdfs://sxt/myflume/qq

a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100

a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

启动flume

flume-ng agent --conf-file /root/flume/exec_hdfs --name a1 -Dflume.root.logger=INFO,console

此时会监测/root/flume/log文件，一旦有文件输入，会写入hdfs