Zookeeper+flume+kafka日志处理系统的部署

1、安装JDK

JDK8 Download Page 选择下载JDK8

假定工作目录在/home/test下

解压：tar zxvf jdk-8u144-linux-x64.tar.gz

配置环境变量：

echo "export JAVA_HOME=/home/test/jdk1.8.0_144" >> ~/.bashrc

echo "export JRE_HOME=$JAVA_HOME/jre" >> ~/.bashrc

echo "export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib" >> ~/.bashrc

echo "export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin" >> ~/.bashrc

验证安装成功：

$ source .bashrc

$ java -version

java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

安装成功！

2、安装Zookeeper

Zookeeper Download Page 点击进去，找到stable版本下载

解压：tar zxvf zookeeper-3.4.10.tar.tgz

cd zookeeper-3.4.10

cp conf/zoo_sample.cfg conf/zoo.cfg

配置环境变量：

echo "export ZOOKEEPER_HOME=/home/test/zookeeper-3.4.10" >> ~/.bashrc

echo "export PATH=$PATH:$ZOOKEEPER_HOME/bin" >> ~/.bashrc

$ source ~/.bashrc

启动Zookeeper：

sh $ZOOKEEPER_HOME/bin/zkServer.sh start

3、安装kafka

Kafka Download Page 找到最新版本下载

tar zxvf kafka_2.11-0.11.0.1.tgz

config目录是其配置文件所在目录，使用默认配置启动kafka：

bash /home/test/kafka_2.11-0.11.0.1/bin/kafka-server-start.sh -daemon /home/test/kafka_2.11-0.11.0.1/config/server.properties &

4、安装flume

Flume Download Page 下载最新版本

tar zxvf apache-flume-1.7.0-bin.tar.gz

配置其source为一个log文件，channel为memory，sink为kafka：

Agent名称定义为agent

Source:可以理解为输入端，定义名称为s1 

channel：传输频道，定义为c1，设置为内存模式

sinks：可以理解为输出端，定义为sk1

agent.sources = s1 

agent.channels = c1 

agent.sinks = k1

以下workdir即为/home/test

touch ${workdir}/test.log #这里的log文件就是source，在启动flume是需要用到

cp ${workdir}/apache-flume-1.7.0-bin/conf/flume-env.sh.template ${workdir}/apache-flume-1.7.0-bin/conf/flume-env.sh
echo "JAVA_HOME=${workdir}/jdk1.8.0_144" >> ${workdir}/apache-flume-1.7.0-bin/conf/flume-env.sh

touch ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sources = s1"                                             >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.channels = c1"                                            >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sinks = k1"                                               >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sources.s1.type=exec"                                     >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sources.s1.command=tail -f ../../test.log"                >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sources.s1.channels=c1"                                   >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.channels.c1.type=memory"                                  >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.channels.c1.capacity=10000"                               >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.channels.c1.transactionCapacity=100"                      >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sinks.k1.type=org.apache.flume.sink.kafka.KafkaSink"      >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sinks.k1.brokerList=127.0.0.1:9092"                       >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sinks.k1.topic=testKJ1"                                   >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sinks.k1.serializer.class=kafka.serializer.StringEncoder" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sinks.k1.channel=c1"                                      >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf

echo "Start flume:"
${workdir}/apache-flume-1.7.0-bin/bin/flume-ng agent -n agent -c conf -f ${workdir}/apache-flume-1.7.0-bin/conf/test.conf -Dflume.root.logger=INFO,console

echo "Everything is prepared"
echo "now write messages to test.log"
echo "Use ${workdir}/kafka_2.11-0.11.0.1/bin/kafka-console-consumer.sh -zookeeper localhost:2181 --from-beginning --topic testKJ1"
echo "to see all messages"
echo "Enjoy!"

参考：http://www.jianshu.com/p/f0a08bd4f975

           http://www.cnblogs.com/wangyangliuping/p/5546465.html

           http://www.jb51.net/article/53542.htm