Hadoop(四)

最新推荐文章于 2024-07-17 14:44:32 发布

AC.WJH

最新推荐文章于 2024-07-17 14:44:32 发布

阅读量262

点赞数 1

文章标签： hadoop 大数据数据库

本文链接：https://blog.csdn.net/weixin_63816398/article/details/126091211

版权

Zookerper实战（开发重点）

分布式安装部署

1）集群规划

在hadoop102、hadoop103、hadoop104三个节点上部署Zookeeper

2)解压安装

1.解压Zookeeper安装包到/opt/module/目录下

2.修改/opt/module/apache-zookeeper-3.5.7-bin 名称为zookeeper-3.5.7

3.同步/opt/module/zookeeper-3.5.7目录分发到hadoop103、hadoop104

3)配置服务器编号

1.在/opt/module/zookeeper-3.5.7/这个目录下创建zkData

2.在/opt/module/zookeeper-3.5.7/zkData目录下创建一个myid的文件

在文件中添加与server对应的编号：2

3.拷贝配置好的zookeeper到其他机器上，并分别在hadoop103、hadoop104内容为3、4

[atguigu@hadoop102 software]$ tar -zxvf zookeeper-3.5.7.tar.gz -C /opt/module/

[atguigu@hadoop102 module]$ mv apache-zookeeper-3.5.7-bin/ zookeeper-3.5.7

[atguigu@hadoop102 module]$ xsync zookeeper-3.5.7/

[atguigu@hadoop102 zookeeper-3.5.7]$ mkdir zkData

[atguigu@hadoop102 zkData]$ vi myid

[atguigu@hadoop102 zkData]$ xsync myid

4）配置zoo.cfg文件

1.重命名/opt/module/zookeeper-3.5.7/conf这个目录下的zoo_sample.cfg为zoo.cfg

2.打开zoo.cfg 修改数据存储路径配置

[atguigu@hadoop102 conf]$ mv zoo_sample.cfg zoo.cfg

[atguigu@hadoop102 conf]$ vim zoo.cfg

dataDir=/opt/module/zookeeper-3.5.7/zkData

增加如下配置

#######################cluster##########################
server.2=hadoop102:2888:3888
server.3=hadoop103:2888:3888
server.4=hadoop104:2888:3888

3.同步zoo.cfg配置文件

[atguigu@hadoop102 conf]$ xsync zoo.cfg

5）集群操作

1.分别启动Zookeeper

[atguigu@hadoop102 zookeeper-3.5.7]$ bin/zkServer.sh start

[atguigu@hadoop103 zookeeper-3.5.7]$ bin/zkServer.sh start

[atguigu@hadoop104 zookeeper-3.5.7]$ bin/zkServer.sh start

2.查看状态

[atguigu@hadoop102 zookeeper-3.5.7]# bin/zkServer.sh status
JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Mode: follower
[atguigu@hadoop103 zookeeper-3.5.7]# bin/zkServer.sh status
JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Mode: leader
[atguigu@hadoop104 zookeeper-3.4.5]# bin/zkServer.sh status
JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Mode: follower

ZK集群启动停止脚本

1.在hadoop102的/home/atguigu/bin目录创建脚本

2.增加脚本权限

3.Zookeeper集群启动脚本

4.Zookeeper集群停止脚本

[atguigu@hadoop102 bin]$ vim zk.sh

[atguigu@hadoop102 bin]$ chmod u+x zk.sh

[atguigu@hadoop102 module]$ zk.sh start

[atguigu@hadoop102 module]$ zk.sh stop

脚本内容

#!/bin/bash

case $1 in
"start"){
	for i in hadoop102 hadoop103 hadoop104
	do
        echo ---------- zookeeper $i 启动 ------------
		ssh $i "/opt/module/zookeeper-3.5.7/bin/zkServer.sh start"
	done
};;
"stop"){
	for i in hadoop102 hadoop103 hadoop104
	do
        echo ---------- zookeeper $i 停止 ------------    
		ssh $i "/opt/module/zookeeper-3.5.7/bin/zkServer.sh stop"
	done
};;
"status"){
	for i in hadoop102 hadoop103 hadoop104
	do
        echo ---------- zookeeper $i 状态 ------------    
		ssh $i "/opt/module/zookeeper-3.5.7/bin/zkServer.sh status"
	done
};;
esac

Kafka安装

集群规划
hadoop102	hadoop103	hadoop104
zk	zk	zk
kafka	kafka	kafka

集群部署

1）解压安装包

2）修改解压后的文件名称

3）在/opt/module/kafka目录下创建logs文件夹

[atguigu@hadoop102 software]$ tar -zxvf kafka_2.11-2.4.1.tgz -C /opt/module/

[atguigu@hadoop102 module]$ mv kafka_2.11-2.4.1/ kafka

[atguigu@hadoop102 kafka]$ mkdir logs

4）修改配置文件

[atguigu@hadoop102 kafka]$ cd config/

[atguigu@hadoop102 config]$ vi server.properties

修改或者增加以下内容：
#broker的全局唯一编号，不能重复
broker.id=0
#删除topic功能使能
delete.topic.enable=true
#kafka运行日志存放的路径
log.dirs=/opt/module/kafka/data
#配置连接Zookeeper集群地址
zookeeper.connect=hadoop102:2181,hadoop103:2181,hadoop104:2181/kafka

5）配置环境变量

[atguigu@hadoop102 module]$ sudo vi /etc/profile.d/my_env.sh

#KAFKA_HOME

export KAFKA_HOME=/opt/module/kafka
export PATH=$PATH:$KAFKA_HOME/bin

[atguigu@hadoop102 module]$ source /etc/profile.d/my_env.sh

6）分发安装包分别在hadoop103和hadoop104上修改配置文件

/opt/module/kafka/config/server.properties中的broker.id=1、broker.id=2 (id不能重复）

7)启动集群

依次在hadoop102、hadoop103、hadoop104节点上启动kafka

[atguigu@hadoop102 kafka]$ bin/kafka-server-start.sh -daemon /opt/module/kafka/config/server.properties

[atguigu@hadoop103 kafka]$ bin/kafka-server-start.sh -daemon  /opt/module/kafka/config/server.properties

[atguigu@hadoop104 kafka]$ bin/kafka-server-start.sh -daemon  /opt/module/kafka/config/server.properties

8）关闭集群

[atguigu@hadoop102 kafka]$ bin/kafka-server-stop.sh
[atguigu@hadoop103 kafka]$ bin/kafka-server-stop.sh
[atguigu@hadoop104 kafka]$ bin/kafka-server-stop.sh

Kafka集群启动停止脚本

1）在/home/atguigu/bin目录下创建脚本kf.sh

2）增加脚本执行权限

3）kf集群启动脚本

4）kf集群停止脚本

[atguigu@hadoop102 bin]$ vim kf.sh

[atguigu@hadoop102 bin]$ chmod u+x kf.sh

[atguigu@hadoop102 module]$ kf.sh start

[atguigu@hadoop102 module]$ kf.sh stop

脚本内容为

#! /bin/bash

case $1 in
"start"){
    for i in hadoop102 hadoop103 hadoop104
    do
        echo " --------启动 $i Kafka-------"
        ssh $i "/opt/module/kafka/bin/kafka-server-start.sh -daemon /opt/module/kafka/config/server.properties"
    done
};;
"stop"){
    for i in hadoop102 hadoop103 hadoop104
    do
        echo " --------停止 $i Kafka-------"
        ssh $i "/opt/module/kafka/bin/kafka-server-stop.sh stop"
    done
};;
esac

Kafka常用命令

1）查看Kafka Topic列表

[atguigu@hadoop102 kafka]$ bin/kafka-topics.sh --zookeeper hadoop102:2181/kafka --list

2）创建Kafka Topic 进入到/opt/module/kafka/目录下创建日志主题

[atguigu@hadoop102 kafka]$ bin/kafka-topics.sh --zookeeper hadoop102:2181,hadoop103:2181,hadoop104:2181/kafka  --create --replication-factor 1 --partitions 1 --topic topic_log

3）删除Kafka Topic

[atguigu@hadoop102 kafka]$ bin/kafka-topics.sh --delete --zookeeper hadoop102:2181,hadoop103:2181,hadoop104:2181/kafka --topic topic_log

4）Kafka生产消息

[atguigu@hadoop102 kafka]$ bin/kafka-console-producer.sh \
--broker-list hadoop102:9092 --topic topic_log
>hello world
>atguigu  atguigu

5）Kafka消费消息

[atguigu@hadoop102 kafka]$ bin/kafka-console-consumer.sh \
--bootstrap-server hadoop102:9092 --from-beginning --topic topic_log

--from-beginning：会把主题中以往所有的数据都读取出来。根据业务场景选择是否增加该配置。

6）查看Kafka Topic详情

[atguigu@hadoop102 kafka]$ bin/kafka-topics.sh --zookeeper hadoop102:2181/kafka \
--describe --topic topic_log