Hadoop（六）

我可以.

已于 2022-08-17 20:55:29 修改

阅读量250

点赞数

文章标签： java 开发语言

于 2022-08-17 19:26:38 首次发布

本文链接：https://blog.csdn.net/qq_61741696/article/details/126323600

版权

hadoop压测环境准备

修改网络适配器

虚拟机-->右击-->设置-->网络适配器-->高级

[doudou@hadoop101 software]$ python -m SimpleHTTPServer

hadoop压测与测试

测试HDFS写性能

[doudou@hadoop101 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.3-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 128MB

测试HDFS读性能

[doudou@hadoop101 hadoop]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.3-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 128MB

Zookeeper安装

[doudou@hadoop101 opt]$ cd software/
[doudou@hadoop101 software]$ rz -E

[doudou@hadoop101 software]$ ll
total 749004
-rw-r--r--. 1 doudou root   9311744 Aug 15 03:40 apache-zookeeper-3.5.7-bin.tar.gz
-rw-r--r--. 1 doudou root 224565455 Aug  7 07:03 bigtable.lzo
-rw-r--r--. 1 doudou root 338075860 Aug  7 07:08 hadoop-3.1.3.tar.gz
-rw-r--r--. 1 doudou root 195013152 Aug  5 08:08 jdk-8u212-linux-x64.tar.gz
-rw-r--r--. 1 doudou root       266 Aug 13 05:02 TestDFSIO_results.log
[doudou@hadoop101 software]$ tar -zxvf apache-zookeeper-3.5.7-bin.tar.gz -C /opt/module/

[doudou@hadoop101 module]$ mv apache-zookeeper-3.5.7-bin/ zookeeper-3.5.7
[doudou@hadoop101 module]$ ll
total 0
drwxr-xr-x.  3 doudou root 117 Aug  7 06:36 applog
drwxr-xr-x. 11 doudou root 173 Aug 12 08:11 hadoop-3.1.3
drwxr-xr-x.  7 doudou root 245 Apr  1  2019 jdk1.8.0_212
drwxr-xr-x.  6 doudou root 134 Aug 15 03:46 zookeeper-3.5.7

[doudou@hadoop101 module]$ cd zookeeper-3.5.7/
[doudou@hadoop101 zookeeper-3.5.7]$ mkdir zkData
[doudou@hadoop101 zookeeper-3.5.7]$ cd zkData/
[doudou@hadoop101 zkData]$ vim myid

myid里面的虚拟机id号应不同

hadoop101：1

hadoop102：2

hadoop103：3

分发
[doudou@hadoop101 module]$ xsync zookeeper-3.5.7/

配置conf
[doudou@hadoop101 zookeeper-3.5.7]$ cd conf/
[doudou@hadoop101 conf]$ mv zoo_sample.cfg zoo.cfg 
[doudou@hadoop101 conf]$ vim zoo.cfg

dataDir=/opt/module/zookeeper-3.5.7/zkData



添加

#######################cluster########################## 
server.1=hadoop101:2888:3888 
server.2=hadoop102:2888:3888 
server.3=hadoop103:2888:3888

[doudou@hadoop101 conf]$ xsync zoo.cfg

开启服务
[doudou@hadoop101 zookeeper-3.5.7]$ bin/zkServer.sh start

查看服务是否开启
[doudou@hadoop101 zookeeper-3.5.7]$ bin/zkServer.sh status

zk集群

[doudou@hadoop102 bin]$ vim zk.sh

#!/bin/bash

case $1 in
"start"){
	for i in hadoop102 hadoop103 hadoop104
	do
        echo ---------- zookeeper $i 启动 ------------
		ssh $i "/opt/module/zookeeper-3.5.7/bin/zkServer.sh start"
	done
};;
"stop"){
	for i in hadoop102 hadoop103 hadoop104
	do
        echo ---------- zookeeper $i 停止 ------------    
		ssh $i "/opt/module/zookeeper-3.5.7/bin/zkServer.sh stop"
	done
};;
"status"){
	for i in hadoop102 hadoop103 hadoop104
	do
        echo ---------- zookeeper $i 状态 ------------    
		ssh $i "/opt/module/zookeeper-3.5.7/bin/zkServer.sh status"
	done
};;
esac

查看

[doudou@hadoop102 bin]$ zk.sh status

停止

[doudou@hadoop102 bin]$ zk.sh stop

[doudou@hadoop102 bin]$ xcall.sh  jps
--------- hadoop102 ----------
55987 DataNode
56326 NodeManager
56503 JobHistoryServer
60012 Jps
55854 NameNode
--------- hadoop103 ----------
54464 ResourceManager
54598 NodeManager
54266 DataNode
56140 Jps
--------- hadoop104 ----------
53476 NodeManager
55272 Jps
53259 DataNode
53371 SecondaryNameNode

开启

[doudou@hadoop102 bin]$ zk.sh start

[doudou@hadoop102 bin]$ xcall.sh jps
--------- hadoop102 ----------
60080 QuorumPeerMain
55987 DataNode
60148 Jps
56326 NodeManager
56503 JobHistoryServer
55854 NameNode
--------- hadoop103 ----------
54464 ResourceManager
54598 NodeManager
56264 Jps
54266 DataNode
56202 QuorumPeerMain
--------- hadoop104 ----------
53476 NodeManager
55399 Jps
53259 DataNode
53371 SecondaryNameNode
55342 QuorumPeerMain

Kafka集群安装

安装
[doudou@hadoop102 software]$ rz -E

[doudou@hadoop102 software]$ ll
total 817516
-rw-r--r--. 1 doudou doudou   9311744 8月  15 03:40 apache-zookeeper-3.5.7-bin.tar.gz
-rw-r--r--. 1 doudou doudou 224565455 8月   7 07:03 bigtable.lzo
-rw-r--r--. 1 doudou doudou 338075860 8月   7 07:08 hadoop-3.1.3.tar.gz
-rw-r--r--. 1 doudou doudou 195013152 8月   5 08:08 jdk-8u212-linux-x64.tar.gz
-rw-r--r--. 1 doudou doudou  70159813 8月  17 02:41 kafka_2.11-2.4.1.tgz

压缩
[doudou@hadoop102 software]$ tar -zxvf kafka_2.11-2.4.1.tgz  -C /opt/module/

修改名字
[doudou@hadoop102 module]$ mv kafka_2.11-2.4.1/ kafka

配置Kafka

[doudou@hadoop102 kafka]$ mkdir logs

[doudou@hadoop102 config]$ vi server.properties

#broker 的全局唯一编号，不能重复
broker.id=0
#删除 topic 功能使能delete.topic.enable=true
#kafka 运行日志存放的路径log.dirs=/opt/module/kafka/logs 
#配置连接Zookeeper 集群地址
zookeeper.connect=hadoop102:2181,hadoop103:2181,hadoop104:2181

分发
[doudou@hadoop102 module]$ xsync kafka/

分别在hadoop103，hadoop104上修改broker.id为1，2.

配置环境变量

[doudou@hadoop102 kafka]$ sudo vim /etc/profile.d/my_env.sh

#KAFKA_HOME
export KAFKA_HOME=/opt/module/kafka 
export PATH=$PATH:$KAFKA_HOME/bin

[doudou@hadoop102 kafka]$ source /etc/profile.d/my_env.sh 
[doudou@hadoop102 kafka]$ sudo /home/doudou/bin/xsync /etc/profile.d/my_env.sh

启动集群

[doudou@hadoop102 kafka]$ bin/kafka-server-start.sh -daemon config/server.properties

Kafka集群启动停止脚本

[doudou@hadoop102 bin]$ vim kf.sh

#! /bin/bash

case $1 in
"start"){
    for i in hadoop102 hadoop103 hadoop104
    do
        echo " --------启动 $i Kafka-------"
        ssh $i "/opt/module/kafka/bin/kafka-server-start.sh -daemon /opt/module/kafka/config/server.properties"
    done
};;
"stop"){
    for i in hadoop102 hadoop103 hadoop104
    do
        echo " --------停止 $i Kafka-------"
        ssh $i "/opt/module/kafka/bin/kafka-server-stop.sh stop"
    done
};;
esac