kafka+zookeeper+storm+hdfs实现日志处理
硬件需求:
服务器三台:192.168.0.63/64/65
一、安装zookeeper
wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
cp zookeeper-3.4.6.tar.gz /usr/local/
tar -zxvf zookeeper-3.4.6.tar.gz
ln -s zookeeper-3.4.6 zookeeper
mkdir -p /opt/zkdata/data /opt/zkdata/log
cd /usr/local/zookeeper/conf
cp zoo_sample.cfg zoo.cfg
vim zoo.cfg
#下面是编辑内容
tickTime=2000
initLimit=10
syncLimit=4
dataDir=/opt/zkdata/data
dataLogDir=/opt/zkdata/log
clientPort=2181
server.0=192.168.0.63:8800:7700
server.1=192.168.0.64:8801:7701
server.2=192.168.0.65:8802:7702
cd /opt/zkdata/data
echo 0 > myid
上述操作同步到另外2台服务器
执行启动:/usr/local/zookeeper/bin/zkServer.sh start
二、安装kafka
wget http://mirror.bit.edu.cn/apache/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
cp kafka_2.11-0.9.0.0.tgz /usr/local/
tar -zxvf kafka_2.11-0.9.0.0.tgz
ln -s kafka_2.11-0.9.0.0.tgz kafka
cd /usr/local/kafka/config
vim server.properties
#下面是编辑内容
broker.id=0
zookeeper.connect=h1:2181,h2:2181,h3:2181/kafka
advertised.host.name=192.168.0.63(多机器时ip不同,仅配置host.name就会出现Exception in thread "main" kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries)
然后保存
/usr/local/zookeeper/bin/zkCli.sh
create /kafka ''
同样的安装配置工作在另外两台服务器上执行。
然后分别修改两台服务器上的/usr/local/kafka/config/server.properties
broker.id=1,和broker.id=2
然后在三台服务器上分别执行:
bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &
如果要后台启动:
/usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties &
三、安装storm
wget http://apache.opencas.org/storm/apache-storm-0.10.0/apache-storm-0.10.0.tar.gz
cp apache-storm-0.10.0.tar.gz /usr/local
tar -zxvf apache-storm-0.10.0.tar.gz
ln -s apache-storm-0.10.0 storm
#修改/etc/profile文件,
export STORM_PATH="/usr/local/storm"
PATH=$JAVA_HOME/bin:$ZOOKEEPER/bin:$STORM/bin:$PATH
. /etc/profile
mkdir -p /opt/storm/workdir/
vim /usr/local/storm/conf/storm.yaml
#注意:文件中配置项前面须有2个空格
storm.zookeeper.servers:
- "192.168.0.63"
- "192.168.0.64"
- "192.168.0.65"
nimbus.host: "192.168.0.63"
#配置slot的ip地址。配几个地址,就有几个slot(worker)。若提交的topology所声明的worker数超过当前可用的slot,该topology提交会失败。
supervisor.slots.ports:
- 6700
- 6701
- 6702
storm.local.dir: "/opt/storm/workdir"
以上步骤在其他机器上重复执行
启动使用storm
1. 在nimbus机器的Storm的bin目录下执行: storm nimbus
2. 在nimbus机器上启动 Storm UI: storm ui
3. 启动supervisor: storm supervisor
4. 打开Storm UI 页面。http://192.168.0.63:8080
参考文章http://shiyanjun.cn/archives/934.html
硬件需求:
服务器三台:192.168.0.63/64/65
一、安装zookeeper
wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
cp zookeeper-3.4.6.tar.gz /usr/local/
tar -zxvf zookeeper-3.4.6.tar.gz
ln -s zookeeper-3.4.6 zookeeper
mkdir -p /opt/zkdata/data /opt/zkdata/log
cd /usr/local/zookeeper/conf
cp zoo_sample.cfg zoo.cfg
vim zoo.cfg
#下面是编辑内容
tickTime=2000
initLimit=10
syncLimit=4
dataDir=/opt/zkdata/data
dataLogDir=/opt/zkdata/log
clientPort=2181
server.0=192.168.0.63:8800:7700
server.1=192.168.0.64:8801:7701
server.2=192.168.0.65:8802:7702
cd /opt/zkdata/data
echo 0 > myid
上述操作同步到另外2台服务器
执行启动:/usr/local/zookeeper/bin/zkServer.sh start
二、安装kafka
wget http://mirror.bit.edu.cn/apache/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
cp kafka_2.11-0.9.0.0.tgz /usr/local/
tar -zxvf kafka_2.11-0.9.0.0.tgz
ln -s kafka_2.11-0.9.0.0.tgz kafka
cd /usr/local/kafka/config
vim server.properties
#下面是编辑内容
broker.id=0
zookeeper.connect=h1:2181,h2:2181,h3:2181/kafka
advertised.host.name=192.168.0.63(多机器时ip不同,仅配置host.name就会出现Exception in thread "main" kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries)
然后保存
/usr/local/zookeeper/bin/zkCli.sh
create /kafka ''
同样的安装配置工作在另外两台服务器上执行。
然后分别修改两台服务器上的/usr/local/kafka/config/server.properties
broker.id=1,和broker.id=2
然后在三台服务器上分别执行:
bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &
如果要后台启动:
/usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties &
三、安装storm
wget http://apache.opencas.org/storm/apache-storm-0.10.0/apache-storm-0.10.0.tar.gz
cp apache-storm-0.10.0.tar.gz /usr/local
tar -zxvf apache-storm-0.10.0.tar.gz
ln -s apache-storm-0.10.0 storm
#修改/etc/profile文件,
export STORM_PATH="/usr/local/storm"
PATH=$JAVA_HOME/bin:$ZOOKEEPER/bin:$STORM/bin:$PATH
. /etc/profile
mkdir -p /opt/storm/workdir/
vim /usr/local/storm/conf/storm.yaml
#注意:文件中配置项前面须有2个空格
storm.zookeeper.servers:
- "192.168.0.63"
- "192.168.0.64"
- "192.168.0.65"
nimbus.host: "192.168.0.63"
#配置slot的ip地址。配几个地址,就有几个slot(worker)。若提交的topology所声明的worker数超过当前可用的slot,该topology提交会失败。
supervisor.slots.ports:
- 6700
- 6701
- 6702
storm.local.dir: "/opt/storm/workdir"
以上步骤在其他机器上重复执行
启动使用storm
1. 在nimbus机器的Storm的bin目录下执行: storm nimbus
2. 在nimbus机器上启动 Storm UI: storm ui
3. 启动supervisor: storm supervisor
4. 打开Storm UI 页面。http://192.168.0.63:8080
参考文章http://shiyanjun.cn/archives/934.html