2024年大数据最新阿里云安装Hadoop全家桶(1)

最新推荐文章于 2024-06-12 12:30:49 发布

2401_84164527

最新推荐文章于 2024-06-12 12:30:49 发布

阅读量929

点赞数 15

分类专栏：程序员文章标签：大数据阿里云 hadoop

本文链接：https://blog.csdn.net/2401_84164527/article/details/138403143

版权

程序员专栏收录该内容

186 篇文章 0 订阅

订阅专栏


检查java


![Untitled](https://img-blog.csdnimg.cn/img_convert/7922e8364e68cc97486d75fbcb61df1f.png)


检查hadoop


![Untitled](https://img-blog.csdnimg.cn/img_convert/f5e4bdf76bc7895088b3dc7ed8e004b7.png)


格式化

cd /opt/module/hadoop/
bin/hdfs namenode -format


启动脚本

vim hdp.sh

#!/bin/bash
if [ $# -lt 1 ]
then
echo “No Args Input…”
exit ;
fi
case $1 in
“start”)
echo " =================== 启动 hadoop集群 ==================="

    echo " --------------- 启动 hdfs ---------------"
    ssh hadoop102 "/opt/module/hadoop/sbin/start-dfs.sh"
    echo " --------------- 启动 yarn ---------------"
    ssh hadoop103 "/opt/module/hadoop/sbin/start-yarn.sh"
    echo " --------------- 启动 historyserver ---------------"
    ssh hadoop102 "/opt/module/hadoop/bin/mapred --daemon start historyserver"

;;
“stop”)
echo " =================== 关闭 hadoop集群 ==================="

    echo " --------------- 关闭 historyserver ---------------"
    ssh hadoop102 "/opt/module/hadoop/bin/mapred --daemon stop historyserver"
    echo " --------------- 关闭 yarn ---------------"
    ssh hadoop103 "/opt/module/hadoop/sbin/stop-yarn.sh"
    echo " --------------- 关闭 hdfs ---------------"
    ssh hadoop102 "/opt/module/hadoop/sbin/stop-dfs.sh"

;;
*)
echo “Input Args Error…”
;;
esac

给权限
chmod 777 hdp.sh


jps查看脚本

vim xcall

#! /bin/bash
for i in hadoop102 hadoop103 hadoop104
do
echo --------- $i ----------
ssh $i “/opt/module/jdk/bin/jps $$*”
done

chmod 777


#### 启动与关闭

hdp.sh start
hdp.sh stop


![Untitled](https://img-blog.csdnimg.cn/img_convert/88643f7b3d6134cf3876eda901605e2b.png)


![Untitled](https://img-blog.csdnimg.cn/img_convert/7156799b8986971a15ebcc4cf038548e.png)


![Untitled](https://img-blog.csdnimg.cn/img_convert/23eff056774879f9d9012607a04d3b86.png)


### Zookeeper


解压修改名字

tar -zxvf apache-zookeeper-3.7.1-bin.tar.gz -C /opt/module/
mv apache-zookeeper-3.7.1-bin/ zookeeper


#### 配置


配置服务器编号

cd zookeeper/
mkdir zkData
cd zkData/
vim myid
2


注意编号是2


修改配置文件

cd conf/
mv zoo_sample.cfg zoo.cfg
xedit zoo.cfg

dataDir=/opt/module/zookeeper/zkData
#######################cluster##########################
server.2=hadoop102:2888:3888
server.3=hadoop103:2888:3888
server.4=hadoop104:2888:3888


[zoo.cfg](%E9%98%BF%E9%87%8C%E4%BA%91%E5%AE%89%E8%A3%85Hadoop%E5%9F%BA%E7%A1%80%20b14fbba39fbf42b5b80ca10cd96fa22e/_opt_module_zookeeper_conf_zoo.cfg)


![Untitled](https://img-blog.csdnimg.cn/img_convert/6b3797d24f0fe6f01e9914503f4db112.png)


或者替换配置文件内容


#### 分发

cd /opt/moudle
xsync zookeeper/


修改hadoop103,hadoop104的myid配置

hadoop103 对应3
hadoop104 对应4


#### 脚本

cd /home/hadoop/bin
vim zk.sh

#!/bin/bash

设置JAVA_HOME和更新PATH环境变量

export JAVA_HOME=/opt/module/jdk
export PATH= $P A T H :$ JAVA_HOME/bin

检查输入参数

if [ $# -ne 1 ]; then
echo “用法: $0 {start|stop|status}”
exit 1
fi

执行操作

case “$1” in
start)
echo “---------- Zookeeper 启动 ------------”
/opt/module/zookeeper/bin/zkServer.sh start
ssh hadoop103 “export JAVA_HOME=/opt/module/jdk; export PATH=$PATH:$JAVA_HOME/bin; /opt/module/zookeeper/bin/zkServer.sh start”
ssh hadoop104 “export JAVA_HOME=/opt/module/jdk; export PATH=$PATH:$JAVA_HOME/bin; /opt/module/zookeeper/bin/zkServer.sh start”
;;
stop)
echo “---------- Zookeeper 停止 ------------”
/opt/module/zookeeper/bin/zkServer.sh stop
ssh hadoop103 “export JAVA_HOME=/opt/module/jdk; export PATH=$PATH:$JAVA_HOME/bin; /opt/module/zookeeper/bin/zkServer.sh stop”
ssh hadoop104 “export JAVA_HOME=/opt/module/jdk; export PATH=$PATH:$JAVA_HOME/bin; /opt/module/zookeeper/bin/zkServer.sh stop”
;;
status)
echo “---------- Zookeeper 状态 ------------”
/opt/module/zookeeper/bin/zkServer.sh status
ssh hadoop103 “export JAVA_HOME=/opt/module/jdk; export PATH=$PATH:$JAVA_HOME/bin; /opt/module/zookeeper/bin/zkServer.sh status”
ssh hadoop104 “export JAVA_HOME=/opt/module/jdk; export PATH=$PATH:$JAVA_HOME/bin; /opt/module/zookeeper/bin/zkServer.sh status”
;;
*)
echo “未知命令: $1”
echo “用法: $0 {start|stop|status}”
exit 2
;;
esac


![Untitled](https://img-blog.csdnimg.cn/img_convert/062541434fb4b67b42d963aac7b5797a.png)


### kafka


解压和修改名

tar -zxvf kafka_2.12-3.3.1.tgz -C /opt/module/
mv kafka_2.12-3.3.1/ kafka


#### 配置

xedit server.properties

添加
advertised.listeners=PLAINTEXT://hadoop102:9092

修改
log.dirs=/opt/module/kafka/datas
zookeeper.connect=hadoop102:2181,hadoop103:2181,hadoop104:2181/kafka


环境变量

#KAFKA_HOME
export KAFKA_HOME=/opt/module/kafka
export PATH= $P A T H :$ KAFKA_HOME/bin


记得刷新


#### 分发

xsync kafka/


修改hadoop103/104的配置文件

[hadoop@hadoop103 module]$ vim kafka/config/server.properties
修改:

The id of the broker. This must be set to a unique integer for each broker.

broker.id=1
#broker对外暴露的IP和端口（每个节点单独配置）
advertised.listeners=PLAINTEXT://hadoop103:9092

[hadoop@hadoop104 module]$ vim kafka/config/server.properties
修改:

The id of the broker. This must be set to a unique integer for each broker.

broker.id=2
#broker对外暴露的IP和端口（每个节点单独配置）
advertised.listeners=PLAINTEXT://hadoop104:9092


#### 脚本


请记住kakfa是在zookeeper启动下才能成功启动

vim kf.sh

#!/bin/bash

Kafka和Zookeeper的配置

KAFKA_HOME=/opt/module/kafka
ZOOKEEPER_HOME=/opt/module/zookeeper
JAVA_HOME=/opt/module/jdk

定义启动Kafka的函数

start_kafka() {
echo “Starting Kafka on hadoop102…”
$KAFKA_HOME/bin/kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties

echo "Starting Kafka on hadoop104..."
ssh hadoop104 "export JAVA\_HOME=$JAVA\_HOME; export KAFKA\_HOME=$KAFKA\_HOME; $KAFKA\_HOME/bin/kafka-server-start.sh -daemon $KAFKA\_HOME/config/server.properties"

echo "Starting Kafka on hadoop103..."
ssh hadoop103 "export JAVA\_HOME=$JAVA\_HOME; export KAFKA\_HOME=$KAFKA\_HOME; $KAFKA\_HOME/bin/kafka-server-start.sh -daemon $KAFKA\_HOME/config/server.properties"

}

定义停止Kafka的函数

stop_kafka() {
echo “Stopping Kafka on hadoop102…”
$KAFKA_HOME/bin/kafka-server-stop.sh

echo "Stopping Kafka on hadoop104..."
ssh hadoop104 "export KAFKA\_HOME=$KAFKA\_HOME; $KAFKA\_HOME/bin/kafka-server-stop.sh"

echo "Stopping Kafka on hadoop103..."
ssh hadoop103 "export KAFKA\_HOME=$KAFKA\_HOME; $KAFKA\_HOME/bin/kafka-server-stop.sh"

}

定义检查Kafka状态的函数

check_status() {
echo “Checking Kafka status on hadoop102…”
ssh hadoop102 “jps | grep -i kafka”

echo "Checking Kafka status on hadoop104..."
ssh hadoop104 "jps | grep -i kafka"

echo "Checking Kafka status on hadoop103..."
ssh hadoop103 "jps | grep -i kafka"

}

处理命令行参数

case “$1” in
start)
start_kafka
;;
stop)
stop_kafka
;;
status)
check_status
;;
*)
echo “Usage: $0 {start|stop|status}”
exit 1
esac


![Untitled](https://img-blog.csdnimg.cn/img_convert/015a0a77a43101b6dfc8662442d02d39.png)


### Flume


解压和修改名

tar -zxvf apache-flume-1.10.1-bin.tar.gz -C /opt/module/
mv apache-flume-1.10.1-bin/ flume


#### 配置


[log4j2.xml](%E9%98%BF%E9%87%8C%E4%BA%91%E5%AE%89%E8%A3%85Hadoop%E5%9F%BA%E7%A1%80%20b14fbba39fbf42b5b80ca10cd96fa22e/log4j2.xml)

vim log4j2.xml

修改
/opt/module/flume/log

添加


分发

xsync flume/


### MySQL


[MySQL下载地址]( )(推荐)


或者用上面的


解压下载的


![Untitled](https://img-blog.csdnimg.cn/img_convert/a56269d043e298ac576324145c3d4144.png)


上传MySQL和hive


![Untitled](https://img-blog.csdnimg.cn/img_convert/d98ad60868eda7678c3481fc2c113a57.png)


#### 安装MySQL

cd /opt/software/MySQL/
sh install_mysql.sh


root 密码是 000000


#### 检查登录

mysql -root -p000000


![Untitled](https://img-blog.csdnimg.cn/img_convert/fad84380cb30b9f734c4a65f5f563b9b.png)


安装成功


### Hive


解压和修改名

tar -zxvf hive-3.1.3.tar.gz -C /opt/module/

mv apache-hive-3.1.3-bin/ hive


#### 配置


**hive-env.sh**

vim hive-env.sh

export HADOOP_HOME=/opt/module/hadoop
export HIVE_CONF_DIR=/opt/module/hive/conf
export HIVE_AUX_JARS_PATH=/opt/module/hive/lib


**hive-site.xml**

vim hive-site.xml

<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> javax.jdo.option.ConnectionURL jdbc:mysql://hadoop102:3306/metastore?useSSL=false&useUnicode=true&characterEncoding=UTF-8&allowPublicKeyRetrieval=true

<!--配置Hive连接MySQL的驱动全类名-->
<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.cj.jdbc.Driver</value>
</property>

<!--配置Hive连接MySQL的用户名 -->
<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
</property>

<!--配置Hive连接MySQL的密码 -->
<property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>000000</value>
</property>

<property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
</property>

<property>
    <name>hive.metastore.schema.verification</name>
    <value>false</value>
</property>

<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>

<property>
    <name>hive.server2.thrift.bind.host</name>
    <value>hadoop102</value>
</property>

<property>
    <name>hive.metastore.event.db.notification.api.auth</name>
    <value>false</value>
</property>

<property>
    <name>hive.cli.print.header</name>
    <value>true</value>
</property>

<property>
    <name>hive.cli.print.current.db</name>
    <value>true</value>
</property>


**环境变量**

#HIVE_HOME
export HIVE_HOME=/opt/module/hive
export PATH= $P A T H :$ HIVE_HOME/bin


**解决日志Jar包冲突**

cd /opt/module/hive/lib/
mv log4j-slf4j-impl-2.17.1.jar log4j-slf4j-impl-2.17.1.jar.bak


**将MySQL的JDBC驱动拷贝到Hive的lib目录**

cp /opt/software/MySQL/mysql-connector-j-8.0.31.jar /opt/module/hive/lib/

检查
ll | grep mysql


![Untitled](https://img-blog.csdnimg.cn/img_convert/a5a95c15048774f81b35862dbfcbaa94.png)


#### 配置元数据库


**mysql操作**

mysql -uroot -p000000
mysql> create database metastore;
mysql> quit;


**hive操作**

hdp.sh start
cd /opt/module/hive
bin/schematool -initSchema -dbType mysql -verbos


![Untitled](https://img-blog.csdnimg.cn/img_convert/9e7993d0008e983c7067adf8add0cf7b.png)


**修改元数据库字符集**

use metastore;
alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8;
alter table TABLE_PARAMS modify column PARAM_VALUE mediumtext character set utf8;


**启动**

hive

show databases;


![Untitled](https://img-blog.csdnimg.cn/img_convert/7112740c2761ecc48070ca1a2817bf79.png)


#### 配置客户端


beeline

在hive目录创建logs文件夹
#先启动metastore服务然后启动hiveserver2服务
nohup bin/hive --service metastore >> logs/metastore.log 2>&1 &
nohup bin/hive --service hiveserver2 >> logs/hiveserver2.log 2>&1 &

bin/beeline
! connect jdbc:hive2://hadoop102:10000
hadoop
回车


![Untitled](https://img-blog.csdnimg.cn/img_convert/4542ab758bd447bb2e8ddbcea06661ac.png)


DG配置


![Untitled](https://img-blog.csdnimg.cn/img_convert/2e25f376e781841fd21012839e489905.png)


### HBase


安装前确保Hadoop集群和zk启动


![Untitled](https://img-blog.csdnimg.cn/img_convert/4003fa66fc387ef1f3f29c218db8220e.png)


解压和修改名

tar -zxvf hbase-2.4.11-bin.tar.gz -C /opt/module/
mv hbase-2.4.11/ hbase


#### 配置


环境变量配置

#HBASE_HOME
export HBASE_HOME=/opt/module/hbase
export PATH= $P A T H :$ HBASE_HOME/bin


记得分发和刷新一下


**hbase-env.sh**

export HBASE_MANAGES_ZK=false


**hbase-site.xml**

<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> hbase.cluster.distributed true hbase.zookeeper.quorum hadoop102,hadoop103,hadoop104 The directory shared by RegionServers. hbase.rootdir hdfs://hadoop102:8020/hbase The directory shared by RegionServers. hbase.wal.provider filesystem


**regionservers**

hadoop102
hadoop103
hadoop104


解决HBase和Hadoop的log4j兼容性问题

cd /opt/module/hbase/lib/client-facing-thirdparty/
mv slf4j-reload4j-1.7.33.jar slf4j-reload4j-1.7.33.jar.bak


#### 分发和启动


成功图,端口号16010


![Untitled](https://img-blog.csdnimg.cn/img_convert/1cf52367b6fe286ce1439f76e73447c1.png)


如果说找不到jdk


自动在hbase-env.sh添加jdk路径


#### 高可用

touch conf/backup-masters
echo hadoop103 > conf/backup-masters
最后分发就行


### Flink


解压和修改名

tar -zxvf flink-1.17.1-bin-scala_2.12.tgz -C /opt/module/
mv flink-1.17.1/ flink


#### 配置


flink-conf.yaml

JobManager节点地址.

jobmanager.rpc.address: hadoop102
jobmanager.bind-host: 0.0.0.0
rest.address: hadoop102
rest.bind-address: 0.0.0.0

TaskManager节点地址.需要配置为当前机器名

taskmanager.bind-host: 0.0.0.0
taskmanager.host: hadoop102


workers

hadoop102
hadoop103
hadoop104


masters




![img](https://img-blog.csdnimg.cn/img_convert/2cc4757be28a08b3385d86a042173dfe.png)
![img](https://img-blog.csdnimg.cn/img_convert/5aa99e0b0b62110239ace97a50f4e689.png)
![img](https://img-blog.csdnimg.cn/img_convert/0367d7b137b8584a01b7e69ade452fd4.png)

**既有适合小白学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，涵盖了95%以上大数据知识点，真正体系化！**

**由于文件比较多，这里只是将部分目录截图出来，全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频，并且后续会持续更新**

**[需要这份系统化资料的朋友，可以戳这里获取](https://bbs.csdn.net/topics/618545628)**

-masters
echo hadoop103 > conf/backup-masters
最后分发就行

Flink

解压和修改名

tar -zxvf flink-1.17.1-bin-scala_2.12.tgz -C /opt/module/
mv flink-1.17.1/ flink

配置

flink-conf.yaml

# JobManager节点地址.
jobmanager.rpc.address: hadoop102
jobmanager.bind-host: 0.0.0.0
rest.address: hadoop102
rest.bind-address: 0.0.0.0
# TaskManager节点地址.需要配置为当前机器名
taskmanager.bind-host: 0.0.0.0
taskmanager.host: hadoop102

workers

hadoop102
hadoop103
hadoop104

masters

[外链图片转存中…(img-E2ZoPEvn-1714664151507)]
[外链图片转存中…(img-zM88V00b-1714664151508)]
[外链图片转存中…(img-d4uVVCii-1714664151508)]

既有适合小白学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，涵盖了95%以上大数据知识点，真正体系化！

由于文件比较多，这里只是将部分目录截图出来，全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频，并且后续会持续更新

需要这份系统化资料的朋友，可以戳这里获取

2401_84164527

关注

15
点赞
踩
29

收藏

觉得还不错? 一键收藏
4
评论
2024年大数据最新阿里云安装Hadoop全家桶(1)

cd /opt/module/hadoop/bin/hdfs namenode -formatvim hdp.sh#!/bin/bashif [ $# -lt 1 ]thenecho “No Args Input…”exit ;ficase $1 in“start”)echo " =================== 启动 hadoop集群 ===================";;“stop”)echo " =================== 关闭 hadoop集群 ==
复制链接

扫一扫