flume练习记录

flume-1监控日志，flume-1的数据传送给flume-2，flume-2将数据追加到本地文件，同时将数据传输到flume-3

flume-work1-1.conf

# Name the components on this agent a1.sources = r1 a1.channels = c1 c2 a1.sinks = k1 k2 # copy duohe a1.sources.r1.selector.type = replicating # Describe/configure the source a1.sources.r1.type = exec a1.sources.r1.command = tail -F /opt/modules/hive/logs/hive.log a1.sources.r1.shell = /bin/bash -c # Describe the sink a1.sinks.k1.type = avro a1.sinks.k1.hostname = liangzelei002 a1.sinks.k1.port = 9527 a1.sinks.k2.type =avro a1.sinks.k2.hostname = liangzelei002 a1.sinks.k2.port = 9528 # Describe the channel a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 a1.channels.c2.type = memory a1.channels.c2.capacity = 1000 a1.channels.c2.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 c2 a1.sinks.k1.channel = c1 a1.sinks.k2.channel = c2

flume-work1-2.conf

# Name the components on this agent a2.sources = r2 a2.channels = c2 a2.sinks = k2 # Describe/configure the source a2.sources.r2.type = avro a2.sources.r2.bind = liangzelei002 a2.sources.r2.port = 9527 # Describe the sink a2.sinks.k2.type = file_roll a2.sinks.k2.sink.rollInterval = 5 a2.sinks.k2.batchSize = 50 a2.sinks.k2.sink.directory = /opt/tmp/flume_work1 # Describe the channel a2.channels.c2.type = memory a2.channels.c2.capacity = 1000 a2.channels.c2.transactionCapacity = 100 # Bind the source and sink to the channel a2.sources.r2.channels = c2 a2.sinks.k2.channel = c2

flume-work1-3.conf

# Name the components on this agent a3.sources = r2 a3.channels = c2 a3.sinks = k2 # Describe/configure the source a3.sources.r2.type = avro a3.sources.r2.bind = liangzelei002 a3.sources.r2.port = 9528 # Describe the sink a3.sinks.k2.type = logger # Describe the channel a3.channels.c2.type = memory a3.channels.c2.capacity = 1000 a3.channels.c2.transactionCapacity = 100 # Bind the source and sink to the channel a3.sources.r2.channels = c2 a3.sinks.k2.channel = c2

启动顺序为
bin/flume-ng agent
  --conf conf/
  --name a2
  --conf-file works/flume-work1-2.conf
  -Dflume.root.logger==INFO,console
bin/flume-ng agent 
  --conf conf/ 
  --name a3 
  --conf-file works/flume-work1-3.conf 
  -Dflume.root.logger==INFO,console
bin/flume-ng agent 
  --conf conf/ 
  --name a1 
  --conf-file works/flume-work1-1.conf 

  -Dflume.root.logger==INFO,console

原因是需要先占用9527和9528的端口，才能向这两个端口发送数据