Flume安装

2019独角兽企业重金招聘Python工程师标准>>>

1、下载apache-flume-1.7.0-bin.tar.gz

2、在安装apache-flume-1.7.0之前，确保JDK1.7+已安装完毕

3、解压apache-flume-1.7.0.tar.gz

tar -zxvf apache-flume-1.7.0-bin.tar.gz
mv /app/apache-flume-1.7.0-bin /app/flume

4、配置环境变量，vi /etc/profile

FLUME_HOME=/app/flume
export PATH=$PATH:$FLUME_HOME/bin

5、重命名flume-env.sh.template为flume-env.sh，并修改

export JAVA_HOME=/app/jdk1.8.0_111
export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
export JAVA_OPTS="$JAVA_OPTS -Dorg.apache.flume.log.rawdata=true -Dorg.apache.flume.log.printconfig=true "
FLUME_CLASSPATH="/app/flume/lib"

6、不同数据类型演示，修改flume-conf.properties.template为flume-xxx.properties

    1. avro类型

        1）. 命名为flume-avro.properties

# Describe the source 
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 8888

# Describe the sink
# 将数据输出至日志中
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

        2）. 启动flume

flume-ng agent -n a1 -c ./conf/ -f ./conf/flume-avro.properties -Dflume.root.logger=INFO,console

        3）. 启动flume客户端，创建/tmp/t1.txt，内容为111、222、333各一行

flume-ng avro-client -H localhost -p 8888 -F /tmp/t1.txt

        4）. 服务端输出结果

               

 

 

 

 

7、注意事项

    同一个机器的不同source需要设置不同的fileRecord和trackerDir，（注：如果这两个参数的配置重复会导致该source的配置无效的问题）

转载于:https://my.oschina.net/u/3209870/blog/822446