1、安装JDK
JDK8 Download Page 选择下载JDK8
假定工作目录在/home/test下
解压:tar zxvf jdk-8u144-linux-x64.tar.gz
配置环境变量:
echo "export JAVA_HOME=/home/test/jdk1.8.0_144" >> ~/.bashrc
echo "export JRE_HOME=$JAVA_HOME/jre" >> ~/.bashrc
echo "export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib" >> ~/.bashrc
echo "export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin" >> ~/.bashrc
验证安装成功:
$ source .bashrc
$ java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
安装成功!
2、安装Zookeeper
Zookeeper Download Page 点击进去,找到stable版本下载
解压:tar zxvf zookeeper-3.4.10.tar.tgz
cd zookeeper-3.4.10
cp conf/zoo_sample.cfg conf/zoo.cfg
配置环境变量:
echo "export ZOOKEEPER_HOME=/home/test/zookeeper-3.4.10" >> ~/.bashrc
echo "export PATH=$PATH:$ZOOKEEPER_HOME/bin" >> ~/.bashrc
$ source ~/.bashrc
启动Zookeeper:
sh $ZOOKEEPER_HOME/bin/zkServer.sh start
3、安装kafka
Kafka Download Page 找到最新版本下载
tar zxvf kafka_2.11-0.11.0.1.tgz
config目录是其配置文件所在目录,使用默认配置启动kafka:
bash /home/test/kafka_2.11-0.11.0.1/bin/kafka-server-start.sh -daemon /home/test/kafka_2.11-0.11.0.1/config/server.properties &
4、安装flume
Flume Download Page 下载最新版本
tar zxvf apache-flume-1.7.0-bin.tar.gz
配置其source为一个log文件,channel为memory,sink为kafka:
echo "JAVA_HOME=${workdir}/jdk1.8.0_144" >> ${workdir}/apache-flume-1.7.0-bin/conf/flume-env.sh
echo "agent.sources = s1" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.channels = c1" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sinks = k1" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sources.s1.type=exec" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sources.s1.command=tail -f ../../test.log" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sources.s1.channels=c1" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.channels.c1.type=memory" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.channels.c1.capacity=10000" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.channels.c1.transactionCapacity=100" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sinks.k1.type=org.apache.flume.sink.kafka.KafkaSink" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sinks.k1.brokerList=127.0.0.1:9092" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sinks.k1.topic=testKJ1" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sinks.k1.serializer.class=kafka.serializer.StringEncoder" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
echo "agent.sinks.k1.channel=c1" >> ${workdir}/apache-flume-1.7.0-bin/conf/test.conf
${workdir}/apache-flume-1.7.0-bin/bin/flume-ng agent -n agent -c conf -f ${workdir}/apache-flume-1.7.0-bin/conf/test.conf -Dflume.root.logger=INFO,console
echo "Everything is prepared"
echo "now write messages to test.log"
echo "Use ${workdir}/kafka_2.11-0.11.0.1/bin/kafka-console-consumer.sh -zookeeper localhost:2181 --from-beginning --topic testKJ1"
echo "to see all messages"
echo "Enjoy!"