zookeeper和kafka的介绍

zookeeper部署

[hadoop@centos-1 zookeeper-3.4.5-cdh4.3.0]$ bin/./zkServer.sh start
JMX enabled by default
Using config: /export/home/tools/zookeeper-3.4.5-cdh4.3.0/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[hadoop@centos-2 zookeeper-3.4.5-cdh4.3.0]$ bin/./zkServer.sh status
JMX enabled by default
Using config: /export/home/tools/zookeeper-3.4.5-cdh4.3.0/bin/../conf/zoo.cfg
Mode: standalone
```Step




<div class="se-preview-section-delimiter"></div>

1：以t账号登陆，创建stream用户
[root@iZ25fnur5jkZ home]# useradd storm
[root@iZ25fnur5jkZ home]# passwd storm
Step2：切换为shaka账号
[shaka@iZ25fnur5jkZ ~]$su - storm Step3：创建zookeeper目录 [shaka@iZ25fnur5jkZ ~]$ mkdir zookeeper
Step4：进入zookeeper目录
[shaka@iZ25fnur5jkZ ~]$cd zookeeper； Step5：拿到老师给的安装包，放到zookeeper目录下 Step6：解压安装包 [shaka@iZ25fnur5jkZ zookeeper]$ tar -xvf zookeeper-3.3.6.deploy.tar.gz
Step7：获取本地IP，方法有多种，下面为常用的命令：
[shaka@iZ25fnur5jkZ conf]$ifconfig 得到IP Step8：修改zookeeper的配置文件（接下页ppt）[shaka@iZ25fnur5jkZ zookeeper]$ vim conf/zoo.cfg
Step10：修改配置文件中的dataDir、dataLogDir、server.1

The number of milliseconds of each tick tickTime=2000
# The number of ticks that the initial # synchronization phase can take initLimit=10

sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. dataDir=/home/shaka/dep/zookeeper-3.3.6/data （改成自己的目录）
the port at which the clients will connect clientPort=2181
# set logs dataLogDir=/home/shaka/dep/zookeeper-3.3.6/logs （改成自己的目录） # set server server.1=hostname:4887:5887 （改成自己的hostname或IP）
#server.2=10.162.219.52:4887:5887 #server.3=10.163.15.119:4887:5887 # add by shaka # set max client connects maxClientCnxns=30Step11：启动zookeeper
修改bin下执行权限： chmod +x *
[shaka@iZ25fnur5jkZ zookeeper]$ bin/zkServer.sh start

Step12：启动后查看状态：
[shaka@iZ25fnur5jkZ zookeeper]$ bin/zkServer.sh status

停止ZK服务: [shaka@iZ25fnur5jkZ zookeeper]$ bin/zkServer.sh stop

重启ZK服务: [shaka@iZ25fnur5jkZ zookeeper]$ bin/zkServer.sh restart

zk的初始化目录为：get /zookeeper/quota

Step13：启动zookeeper的客户端

安装zkui ##首先安装编译工具mvn：
vim ~/.bash_profile export mvn3  export MVN_HOME=/usr/local/streamprocessing/dependence/apache-maven-3.2.1 export PATH=$PATH:$MVN_HOME/bin
进入zkui目录
mvn clean install
ln -s target/zkui-2.0-SNAPSHOT-jar-with-dependencies.jar  zkui-2.0-SNAPSHOT-jar-with-dependencies.jar
vim config.cfg

zkServer=cluster-node-01:2181

启动：nohup java -jar zkui-2.0-SNAPSHOT-jar-with-dependencies.jar
 &

查看：http://123.56.76.213:9090  或者    jps -l
登录账号
"username":"admin" , "password":"manager"
日志：zkui-out.log   nohup.out

创建一个node： 以自己名字命名





<div class="se-preview-section-delimiter"></div>

伪分布式安装

tar -xzf kafka_2.10-0.8.1.1.tgz

启动ZK bin/zookeeper-server-start.sh config/zookeeper.properties

启动服务 bin/kafka-server-start.sh -daemon config/server.properties

创建主题 bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

查看主题 bin/kafka-topics.sh --list --zookeeper localhost:2181
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
创建生产者
 bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 

创建消费者
 bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

伪分布式安装
==

tar -xzf kafka_2.10-0.8.1.1.tgz

启动ZK bin/zookeeper-server-start.sh config/zookeeper.properties

启动服务 bin/kafka-server-start.sh -daemon config/server.properties

创建主题 bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test

查看主题 bin/kafka-topics.sh –list –zookeeper localhost:2181
bin/kafka-topics.sh –describe –zookeeper localhost:2181 –topic test
创建生产者
bin/kafka-console-producer.sh –broker-list localhost:9092 –topic test

创建消费者
bin/kafka-console-consumer.sh –zookeeper localhost:2181 –topic test –from-beginning

“`

kafka部署

step1：拷贝软件包

step2：解压
tar -xvf kafka_2.9.2-0.8.1.1.tgz 或者用 unzip kafka_2.9.2-0.8.1.1.zip
,修改配置文件conf/server.properties：
broker.id=0；
host.name=10.100.5.9；
zookeeper.connect=10.100.5.9:2181可逗号分隔配置多个

step3: 确保有执行权限
chmod +x sbin/*

step 4修改配置文件vim log4j.properties
log4j.appender.D.File = /data1/home/shaka/kafka/kafka_2.9.2-0.8.1.1/logs/debug.log
log4j.appender.E.File = /data1/home/shaka/kafka/kafka_2.9.2-0.8.1.1/logs/error.log

Step5：启动kafka服务
sbin/start-kafka.sh
查看是否启动： jsp -l
step6.创建topic主题：
bin/kafka-create-topic.sh –zookeeper 10.100.5.9:2181 –replica 1 –partition 1 –topic mykafka