flume篇

flume介绍

日志数据收集器

flume使用步骤

1. 定义source，channel(通道),sink(转存的位置)
2. 启动agent
3. 如果有数据，就已经开始接受转存了

flume运行机理

flume type介绍

• source type
  
  • Avro, Exec, Jms, Spooling directory, Netcat, Http,
    Syslog, Thrift, twitter等
    高级编写自己的source type
• channel
  
  • 可以存放在memory、jdbc、file中
• sink type
  
  • HDFS, Hbase 或SPARK STREAM也可能是另一个sink

flume demo

安装解压flume：
/home/hadoop/opt/apache-flume-1.8.0-bin/conf
vi spooldir.conf

========================================================
spooldir.sources=sa
spooldir.channels=ma
spooldir.sinks=ha

spooldir.sources.sa.type=spooldir
spooldir.sources.sa.spoolDir=/home/hadoop/firstdemo/flume_spider
spooldir.sources.sa.fileHeader = true

spooldir.channels.ma.type=memory
spooldir.channels.ma.capacity=10000
spooldir.channels.ma.transactioncapacity=1000000

#spooldir.sinks.ha.type=logger
spooldir.sinks.ha.type=hdfs
spooldir.sinks.ha.hdfs.fileType=DataStream
spooldir.sinks.ha.hdfs.path=/user/hadoop/spider
spooldir.sinks.ha.hdfs.writeFormat=Text
spooldir.sinks.ha.hdfs.batchSize=10000
spooldir.sinks.ha.hdfs.rollCount=1000
spooldir.sinks.ha.hdfs.fileSuffix=.csv
spooldir.sinks.ha.hdfs.filePrefix=test
spooldir.sinks.ha.hdfs.rollSize=0
spooldir.sinks.ha.hdfs.rollInterval=0


spooldir.sources.sa.channels=ma
spooldir.sinks.ha.channel=ma 
me
==

启动flume
./bin/flume-ng agent -n spooldir -c conf -f conf/spooldir.conf
重新加载flume
./bin/flume-ng agent -n spooldir -c conf -f conf/spooldir.conf -Dflume.root.logger=INFO,console
“`