文章目录
前言
详解
一、关于zookeeper和kafka
二、Ubuntu下安装步骤
1.下载
https://zookeeper.apache.org/releases.html
https://kafka.apache.org/downloads
2.Ubuntu安装zookeeper
2.1步骤演示(对腾讯文档补充说明)
腾讯文档链接:https://cloud.tencent.com/developer/article/1544293
根据个人的实际文件所在目录更改/mnt/后面地址,这里将此文件解压在/usr/local/目录下
//解压
//查看,更改文件名
//创建数据文件夹
hadoop@LAPTOP-KB9G98PA:~$ sudo tar -zxf /mnt/c/tools/bigdata/apache-zookeeper-3.7.0-bin.tar.gz -C /usr/local
[sudo] password for hadoop:
hadoop@LAPTOP-KB9G98PA:~$ cd /usr/local
hadoop@LAPTOP-KB9G98PA:/usr/local$ ls
apache-zookeeper-3.7.0-bin etc hadoop include man share
bin games hbase lib sbin src
hadoop@LAPTOP-KB9G98PA:/usr/local$ sudo mv ./apache-zookeeper-3.7.0-bin/ ./zookeeper
hadoop@LAPTOP-KB9G98PA:/usr/local$ ls
bin etc games hadoop hbase include lib man sbin share src zookeeper
hadoop@LAPTOP-KB9G98PA:/usr/local$ cd zookeeper
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper$ sudo chmod -R 777 ../zookeeper
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper$ mkdir data
zoo_sample.cfg文件是一个示例配置代码,我们在这里复制一份并修改内容
#查看
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper$ ls
LICENSE.txt NOTICE.txt README.md README_packaging.md bin conf data docs lib
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper$ cd conf
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/conf$ ls
configuration.xsl log4j.properties zoo_sample.cfg
#复制
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/conf$ cp zoo_sample.cfg zoo.cfg
#修改文件
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/conf$ vi zoo.cfg
zoo.cfg文件内容配置
tickTime = 2000
dataDir = /usr/local/zookeeper/data
clientPort = 2181
initLimit = 10
syncLimit = 5
server.1=127.0.0.1:2888:3888
server.2=127.0.0.1:2889:3889
server.3=127.0.0.1:2890:3890
#此部分应可以省略,把zookeeper加入最高权限后无需自己创建下列目录(若提示此目录不存在再自行创建)
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper$ mkdir zk1
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper$ mkdir zk2
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper$ mkdir zk3
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper$ cd zk1
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/zk1$ mkdir data
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/zk1$ cd ../zk2
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/zk2$ mkdir data
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/zk2$ cd ../zk3
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/zk3$ mkdir data
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/zk3$ cd ../conf
*************
#创建三个机器的配置文件并修改配置,主要是clientPort 和dataDir
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/conf$ cp zoo.cfg zoo1.cfg
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/conf$ cp zoo.cfg zoo2.cfg
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/conf$ cp zoo.cfg zoo3.cfg
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/conf$ vi zoo1.cfg
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/conf$ vi zoo2.cfg
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/conf$ vi zoo3.cfg
#添加myid文件,仅需输入一个数字,不能多空格或其他字符
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/conf$ cd ../zk1/data
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/zk1/data$ vi myid
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/zk1/data$ cd ..
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/zk1$ cd ../zk2/data
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/zk2/data$ vi myid
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/zk2/data$ cd ..
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/zk2$ cd ../zk3/data
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/zk3/data$ vi myid
在根端配置环境变量
hadoop@LAPTOP-KB9G98PA:~$ vim ~/.bashrc
hadoop@LAPTOP-KB9G98PA:~$ source ~/.bashrc
#添加如下
# ZOOKEEPER 路径修改
export ZOOKEEPER_HOME=/usr/local/zookeeper
export PATH=${ZOOKEEPER_HOME}/bin:$PATH
启动如下
hadoop@LAPTOP-KB9G98PA:~$ cd /usr/local/zookeeper
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper$ cd bin
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/bin$ zkServer.sh start ../conf/zoo1.cfg
ZooKeeper JMX enabled by default
Using config: ../conf/zoo1.cfg
Starting zookeeper ... STARTED
2.2出现问题及解决
3.Ubuntu安装kafka
3.1安装解压配置
解压方式同zookeeper,同时将kafka加入777权限中。
修改 kafka/config目录下的server.properties
hadoop@LAPTOP-KB9G98PA:/usr/local/kafka/config$ vi server.properties
#只要检查更改
broker.id=0
log.dirs=/usr/local/kafka/logs #自己设置目录
zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
num.partitions=3
#新建
default.replication.factor=3
3.2启动运行
我们学习的消息队列方式,使用了zookeeper的伪集群模式,故我们不需要使用kafka自带的zookeeper功能,提前打开zookeeper配置好的三台机器
可用jps命令检查
hadoop@LAPTOP-KB9G98PA:/usr/local/zookeeper/bin$ jps
4358 Jps
4298 QuorumPeerMain
4219 QuorumPeerMain
4158 QuorumPeerMain
cd到kafka目录下 运行
hadoop@LAPTOP-KB9G98PA:/usr/local/kafka/bin$ cd ..
hadoop@LAPTOP-KB9G98PA:/usr/local/kafka$ ./bin/kafka-server-start.sh config/server.properties &
[1] 5175
hadoop@LAPTOP-KB9G98PA:/usr/local/kafka$ jps
5619 Jps
5175 Kafka
4298 QuorumPeerMain
4219 QuorumPeerMain
4158 QuorumPeerMain
到目前为止,kafka一份已经配置完毕,在https://www.cnblogs.com/zhaoshizi/p/12154518.html
提到了复制三份并使用了软连接(存在疑问,不是很清楚三台kafka具体原理)
之后通过命令分别测试
#首先创建topic,在这里replication-factor 1被我修改为1,为3是总提示broker只启动一个,未发现原因
./bin/kafka-topics.sh --create --topic itcasttopic --partitions 3 --replication-factor 1 --zookeeper localhost:2181,localhost:2182,localhost:2183
#查看新建好的主题
./bin/kafka-topics.sh --list --zookeeper localhost:2181
#发送消息
./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic itcasttopic
#消费---*****此处输入后无反应,使用ctrl+c强制退出(有出现结果的求教)
./bin/kafka-console-consumer.sh --from-beginning --topic itcasttopic --bootstrap-server localhost:9092
#查看特定主题的详细信息
./bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic itcasttopic
#删除
./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic itcasttopic
3.3问题
问题大致出在broker和localhost
留待验证
总结
费劲
费劲
费劲