一 单节点flume部署
1 hadoop准备
在hdfs中创建flume目录,并将flume目录的权限分配给flume用户
hdfs dfs -mkdir flume
hdfs dfs -chown -R flume:flume /flume
2 flume-env.sh
进入${FLUME_HOME}/conf
cp flume-env.sh.template flume-env.sh
vi flume-env.sh,设置JAVA的环境变量
export JAVA_HOME=/usr/java/jdk1.7.0_79
3 flume-conf.properties
下方配置中application和dir由应用系统在logback.xml中配置
agent1.sources=source1
agent1.channels=channel1
agent1.sinks=sink1
agent1.sources.source1.type=avro
agent1.sources.source1.bind=0.0.0.0
agent1.sources.source1.port=44444
agent1.sources.source1.channels=channel1
agent1.channels.channel1.type=memory
agent1.channels.channel1.capacity=10000
agent1.channels.channel1.transactionCapacity=1000
agent1.channels.channel1.keep-alive=30
agent1.sinks.sink1.type=hdfs
agent1.sinks.sink1.channel=channel1
agent1.sinks.sink1.hdfs.path=hdfs://dashuju174:9000/flume/%{application}/%{dir}/%Y%m%d
agent1.sinks.sink1.hdfs.fileType=DataStream
agent1.sinks.sink1.hdfs.writeFormat=Text
agent1.sinks.sink1.hdfs.rollInterval=0
agent1.sinks.sink1.hdfs.rollSize=10240
agent1.sinks.sink1.hdfs.rollCount=0
agent1.sinks.sink1.hdfs.idleTimeout=60
4 logback配置
logback.xml中添加下方日志,专门针对TRACE级别的日志。选择TRACE的原因是因为开发人员对INFO\ERROR\DEBUG比较熟悉,对TRACE应用较为生疏,故我这里要求开发人员统一使用TRACE级别的日志。
<appender name="flume" class="com.gilt.logback.flume.FlumeLogstashV1Appender">
<!-- 此项是连接flume节点的ip和端口 -->
<flumeAgents> 192.168.5.174:44444 </flumeAgents>
<flumeProperties>connect-timeout=4000;request-timeout=8000</flumeProperties>
<batchSize>100</batchSize>
<reportingWindow>1000</reportingWindow>
<!-- 此项配置avro头部信息 -->
<additionalAvroHeaders>myHeader=dengjun</additionalAvroHeaders>
<!-- 此项配置当前应用 -->
<application>bds</application>
<filter class="ch.qos.logback.classic.filter.LevelFilter"><!-- 只打印错误日志 -->
<level>TRACE</level>
<onMatch>ACCEPT</onMatch>
<onMismatch>DENY</onMismatch>
</filter>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%message%n%ex</pattern>
</layout>
</appender>
5 环境变量
flume依赖zookeeper,故安装flume之前需要安装zookeeper
tar zxvf apache-flume-1.6.0-bin.tar.gz
ln -s apache-flume-1.6.0 flume
设置flume的环境变量
export FLUME_HOME=/home/hadoop/application/flume
export PATH=
P
A
T
H
:
PATH:
PATH:FLUME_HOME/bin
二 上线运行宕机记录
按照上述配置后,上线第3天,flume宕机了。我这里把解决过程记录下来,毕竟第一次使用flume。
待续。。。