flume 安装与配置

flume

安装与配置

0、按照jdk。

官网分别下载jdk-1.8.0和apache-flume二进制包
设定软件路径如下
jdk: /usr/local/jdk-1.8.0
flume: /opt/apache-flume

1、配置flume-env.sh

cd /opt/apache-flume-1.8.0/conf
cp flume-env.sh.template flume-env.sh
// 指向JAVA
echo "export JAVA_HOME=/usr/local/jdk1.8.0_162/" >> flume-env.sh

2、编辑配置文件(vim default.conf)

# 设定配置组log
log.sinks    = sk
log.sources  = sc
log.channels = ch

# 配置sources
log.sources.sc.type = exec  # source类型:命令
log.sources.sc.command = tail -F /data/log/access.log

# 配置sinks
log.sinks.sk.type = org.apache.flume.sink.kafka.KafkaSink
log.sinks.sk.kafka.topic = test
log.sinks.sk.kafka.bootstrap.servers = localhost:9092

# 配置channels。记录checkpoint
log.channels.ch.type = file 
log.channels.ch.checkpointDir= /data/flume/check
log.channels.ch.dataDirs = /data/flume/data

# 将source和sink绑定到channle
log.sinks.sk.channel = ch
log.sources.sc.channel = ch

3、启动flume

cd /opt/apache-flume-1.8.0
bin/flume-ng agent --conf conf --conf-file conf/default.conf -name log

4、启动kafka，监听输出数据