hadoop+flume配置

1、安装flume

下载
wget http://mirrors.hust.edu.cn/apache/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz
解压
tar -zxvf apache-flume-1.8.0-bin.tar.gz
配置环境变量
vi /etc/profile

source /etc/profile
验证版本
flume-ng version

2、配置flume

在conf目录下面
mv flume-env.sh.template flume-env.sh
在flume-env.sh中添加以下配置
export JAVA_HOME=/home/hadoop/jdk1.8.0_25

新建avro.conf(名称随便取，后缀名.conf)
vim avro.conf
在avro.conf 加入以下配置

a1.sources = r1
a1.sinks = k1
a1.channels = c1

#sources
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /home/hadoop/flume1
a1.sources.r1.channels = c1
a1.sources.r1.fileHeader = false
#sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://master:9000/flume1
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.writeFormat = TEXT
a1.sinks.k1.hdfs.rollInteval = 4
a1.sinks.k1.channel = c1
#channel
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /home/hadoop/flume1_tmp
a1.channels.c1.dataDirs = /home/hadoop/flume1_data

新建文件夹
mkdir -p /home/hadoop/flume1
mkdir -p /home/hadoop/flume1_tmp
mkdir -p /home/hadoop/flume1_data

3、启动flume

在conf目录下
flume-ng agent --conf conf --conf-file avro.conf --name a1 -Dflume.root.logger=INFO,console
另开一个终端
在/home/hadoop/flume1下面新建文件夹，写入字符串
vim text.txt
在hadoop的dfs中查看
hadoop fs -ls /


继续查看flume1文件夹的内容