openEuler搭建hadoop 高可用完全分布式集群

李昊哲小课

已于 2024-06-27 11:02:35 修改

阅读量1.6k

点赞数 12

分类专栏：大数据 Zookeeper Hadoop 文章标签：分布式 hadoop 大数据

于 2024-06-21 17:42:24 首次发布

本文链接：https://blog.csdn.net/qq_24330181/article/details/139866542

版权

大数据同时被 3 个专栏收录

144 篇文章 10 订阅

订阅专栏

Hadoop

13 篇文章 2 订阅

订阅专栏

Zookeeper

3 篇文章 0 订阅

订阅专栏

openEuler搭建hadoop 高可用完全分布式集群

spark01	spark02	spark03
192.168.10.101	192.168.10.102	192.168.10.103
zookeeper	zookeeper	zookeeper
namenode	namenode
recource manager	recource manager
journalnode	journalnode	journalnode
datanode	datanode	datanode
nodemanager	nodemanager	nodemanager
job history server
job log	job log	job log

一、准备

1、升级软件

yum -y update

2、安装常用软件

yum -y install gcc gcc-c++ autoconf automake cmake make \
 zlib zlib-devel openssl openssl-devel pcre-devel \
 rsync openssh-server vim man zip unzip net-tools tcpdump lrzsz tar wget

3、关闭防火墙

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0

systemctl stop firewalld
systemctl disable firewalld

4、修改主机名和IP地址

spark01

hostnamectl set-hostname spark01

spark02

hostnamectl set-hostname spark02

spark03

hostnamectl set-hostname spark03

vim /etc/sysconfig/network-scripts/ifcfg-ens32

参考如下：

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=eui64
NAME=ens32
UUID=55e7ac28-39d7-4f24-b6bf-0f9fb40b7595
DEVICE=ens32
ONBOOT=yes
IPADDR=192.168.10.101
PREFIX=24
GATEWAY=192.168.10.2
DNS1=192.168.10.2

5、修改hosts配置文件

vim /etc/hosts

修改内容如下：

192.168.10.101	spark01
192.168.10.102	spark02
192.168.10.103	spark03

重启系统

reboot

6、上传软件并配置环境变量

spark01

创建软件目录

mkdir -p /opt/soft

进入软件目录

cd /opt/soft

下载 JDK

下载 zookeeper

wget https://dlcdn.apache.org/zookeeper/zookeeper-3.9.2/apache-zookeeper-3.9.2-bin.tar.gz

下载 hadoop

wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz

解压 JDK 修改名称

tar -zxvf jdk-8u411-linux-x64.tar.gz

mv jdk1.8.0_411 jdk-8

解压 zookeeper 修改名称

tar -zxvf apache-zookeeper-3.9.2-bin.tar.gz

mv apache-zookeeper-3.9.2-bin zookeeper-3

解压 hadoop 修改名称

tar -zxvf hadoop-3.3.6.tar.gz

mv hadoop-3.3.6 hadoop-3

删除安装包（不推荐）

rm -f *.gz

配置环境变量

vim /etc/profile.d/my_env.sh

编写以下内容：

export JAVA_HOME=/opt/soft/jdk-8

export ZOOKEEPER_HOME=/opt/soft/zookeeper-3

export HDFS_NAMENODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_ZKFC_USER=root
export HDFS_JOURNALNODE_USER=root
export HADOOP_SHELL_EXECNAME=root

export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

export HADOOP_HOME=/opt/soft/hadoop-3
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native

export PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

生成新的环境变量

source /etc/profile

二、zookeeper

1. 编辑配置文件

cd $ZOOKEEPER_HOME/conf

vim zoo.cfg

注释版不推荐使用

# 心跳单位，2s
tickTime=2000
# zookeeper-3初始化的同步超时时间，10个心跳单位，也即20s
initLimit=10
# 普通同步：发送一个请求并得到响应的超时时间，5个心跳单位也即10s
syncLimit=5
# 内存快照数据的存储位置
dataDir=/home/zookeeper-3/data
# 事务日志的存储位置
dataLogDir=/home/zookeeper-3/datalog
# 当前zookeeper-3节点的端口 
clientPort=2181
# 单个客户端到集群中单个节点的并发连接数，通过ip判断是否同一个客户端，默认60
maxClientCnxns=1000
# 保留7个内存快照文件在dataDir中，默认保留3个
autopurge.snapRetainCount=7
# 清除快照的定时任务，默认1小时，如果设置为0，标识关闭清除任务
autopurge.purgeInterval=1
#允许客户端连接设置的最小超时时间，默认2个心跳单位
minSessionTimeout=4000
#允许客户端连接设置的最大超时时间，默认是20个心跳单位，也即40s，
maxSessionTimeout=300000
#zookeeper-3 3.5.5启动默认会把AdminService服务启动，这个服务默认是8080端口
admin.serverPort=9001
#集群地址配置
server.1=spark01:2888:3888
server.2=spark02:2888:3888
server.3=spark03:2888:3888

无注释版推荐

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/zookeeper-3/data
dataLogDir=/home/zookeeper-3/datalog 
clientPort=2181
maxClientCnxns=1000
autopurge.snapRetainCount=7
autopurge.purgeInterval=1
minSessionTimeout=4000
maxSessionTimeout=300000
admin.serverPort=9001
server.1=spark01:2888:3888
server.2=spark02:2888:3888
server.3=spark03:2888:3888

2. 保存后根据配置文件创建目录

在每台服务器上执行

mkdir -p /home/zookeeper-3/data
mkdir -p /home/zookeeper-3/datalog

3. 编写zookeeper-3开机启动脚本

在/etc/systemd/system/文件夹下创建一个启动脚本zookeeper-3.service

注意：在每台服务器上编写

cd /etc/systemd/system
vim zookeeper.service

内容如下：

[Unit]
Description=zookeeper
After=syslog.target network.target

[Service]
Type=forking
 指定zookeeper-3 日志文件路径，也可以在zkServer.sh 中定义
Environment=ZOO_LOG_DIR=/home/zookeeper-3/datalog
 指定JDK路径，也可以在zkServer.sh 中定义
Environment=JAVA_HOME=/opt/soft/jdk-8
ExecStart=/opt/soft/zookeeper-3/bin/zkServer.sh start
ExecStop=/opt/soft/zookeeper-3/bin/zkServer.sh stop
Restart=always
User=root
Group=root

[Install]
WantedBy=multi-user.target

[Unit]
Description=zookeeper
After=syslog.target network.target

[Service]
Type=forking
Environment=ZOO_LOG_DIR=/home/zookeeper-3/datalog
Environment=JAVA_HOME=/opt/soft/jdk-8
ExecStart=/opt/soft/zookeeper-3/bin/zkServer.sh start
ExecStop=/opt/soft/zookeeper-3/bin/zkServer.sh stop
Restart=always
User=root
Group=root

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
 等所有主机配置好后再执行以下命令
systemctl start zookeeper
systemctl enable zookeeper
systemctl status zookeeper

4、配置ssh免密钥登录

创建本地秘钥并将公共秘钥写入认证文件

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

ssh-copy-id root@spark01

ssh-copy-id root@spark02

ssh-copy-id root@spark03

5. 分发软件和配置文件

分发 ssh 免密钥

scp -r ~/.ssh root@spark02:~/
scp -r ~/.ssh root@spark03:~/

分发 hosts 文件

scp -r /etc/hosts root@spark02:/etc/
scp -r /etc/hosts root@spark03:/etc/

分发软件

scp -r /opt/soft/jdk-8 root@spark02:/opt/soft
scp -r /opt/soft/zookeeper-3 root@spark02:/opt/soft
scp -r /opt/soft/jdk-8 root@spark03:/opt/soft
scp -r /opt/soft/zookeeper-3 root@spark03:/opt/soft

分发环境变量

scp /etc/profile.d/my_env.sh root@spark02:/etc/profile.d/
scp /etc/profile.d/my_env.sh root@spark03:/etc/profile.d/

分发开启启动脚本

scp /etc/systemd/system/zookeeper.service root@spark02:/etc/systemd/system
scp /etc/systemd/system/zookeeper.service root@spark03:/etc/systemd/system

在所有主机节点使新的环境变量生效

source /etc/profile

6. myid

spark01

echo 1 > /home/zookeeper-3/data/myid
more /home/zookeeper-3/data/myid

spark02

echo 2 > /home/zookeeper-3/data/myid
more /home/zookeeper-3/data/myid

spark03

echo 3 > /home/zookeeper-3/data/myid
more /home/zookeeper-3/data/myid

7. 启动服务

在各节点执行以下命令

systemctl daemon-reload
systemctl start zookeeper
systemctl enable zookeeper
systemctl status zookeeper

8. 验证

jps

zkServer.sh status

9. 浏览器访问 AdminServer

浏览器访问：http://spark03:9001/commands

zookeeper

三、hadoop

1、修改配置文件

hadoop-env.sh

core-site.xml

hdfs-site.xml

workers

mapred-site.xml

yarn-site.xml

hadoop-env.sh

文档末尾追加以下内容：

export JAVA_HOME=/opt/soft/jdk-8

export HDFS_NAMENODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_ZKFC_USER=root
export HDFS_JOURNALNODE_USER=root
export HADOOP_SHELL_EXECNAME=root

export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://lihaozhe</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/data</value>
  </property>
  <property>
    <name>ha.zookeeper.quorum</name>
    <value>spark01:2181,spark02:2181,spark03:2181</value>
  </property>
  <property>
    <name>hadoop.http.staticuser.user</name>
    <value>root</value>
  </property>
  <property>
    <name>dfs.permissions.enabled</name>
    <value>false</value>
  </property>
  <property>
    <name>hadoop.proxyuser.root.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.root.groups</name>
    <value>*</value>
  </property>
</configuration>

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>dfs.nameservices</name>
    <value>lihaozhe</value>
  </property>
  <property>
    <name>dfs.ha.namenodes.lihaozhe</name>
    <value>nn1,nn2</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.lihaozhe.nn1</name>
    <value>spark01:8020</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.lihaozhe.nn2</name>
    <value>spark02:8020</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.lihaozhe.nn1</name>
    <value>spark01:9870</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.lihaozhe.nn2</name>
    <value>spark02:9870</value>
  </property>
  <property>
    <name>dfs.namenode.shared.edits.dir</name>
    <value>qjournal://spark01:8485;spark02:8485;spark03:8485/lihaozhe</value>
  </property>
  <property>
    <name>dfs.client.failover.proxy.provider.lihaozhe</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>
  <property>
    <name>dfs.ha.fencing.methods</name>
    <value>sshfence</value>
  </property>
  <property>
    <name>dfs.ha.fencing.ssh.private-key-files</name>
    <value>/root/.ssh/id_rsa</value>
  </property>
  <property>
    <name>dfs.journalnode.edits.dir</name>
    <value>/home/hadoop/journalnode/data</value>
  </property>
  <property>
    <name>dfs.ha.automatic-failover.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.safemode.threshold.pct</name>
    <value>1</value>
  </property>
</configuration>

workers

注意：

hadoop2.x中该文件名为slaves

hadoop3.x中该文件名为workers

spark01
spark02
spark03

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
  </property>
  <!-- yarn历史服务端口 -->
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>spark01:10020</value>
  </property>
  <!-- yarn历史服务web访问端口 -->
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>spark01:19888</value>
  </property>
</configuration>

yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

    <!-- Site specific YARN configuration properties -->
    <property>
		<name>yarn.resourcemanager.ha.enabled</name>
		<value>true</value>
    </property>
    <property>
		<name>yarn.resourcemanager.cluster-id</name>
		<value>cluster1</value>
    </property>
    <property>
		<name>yarn.resourcemanager.ha.rm-ids</name>
		<value>rm1,rm2</value>
    </property>
    <property>
		<name>yarn.resourcemanager.hostname.rm1</name>
		<value>spark01</value>
    </property>
    <property>
		<name>yarn.resourcemanager.hostname.rm2</name>
		<value>spark02</value>
    </property>
    <property>
		<name>yarn.resourcemanager.webapp.address.rm1</name>
		<value>spark01:8088</value>
    </property>
    <property>
		<name>yarn.resourcemanager.webapp.address.rm2</name>
		<value>spark02:8088</value>
    </property>
    <property>
		<name>yarn.resourcemanager.zk-address</name>
		<value>spark01:2181,spark02:2181,spark03:2181</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
		<value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
		<name>yarn.nodemanager.env-whitelist</name>
		<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
    <!-- 是否将对容器实施物理内存限制 -->
    <property>
		<name>yarn.nodemanager.pmem-check-enabled</name>
		<value>false</value>
    </property>

    <!-- 是否将对容器实施虚拟内存限制。 -->
    <property>
		<name>yarn.nodemanager.vmem-check-enabled</name>
		<value>false</value>
    </property>
    <!-- 开启日志聚集 -->
    <property>
		<name>yarn.log-aggregation-enable</name>
		<value>true</value>
    </property>

    <!-- 设置yarn历史服务器地址 -->
    <property>
		<name>yarn.log.server.url</name>
		<value>http://spark01:19888/jobhistory/logs</value>
    </property>

    <!-- 保存的时间7天 -->
    <property>
		<name>yarn.log-aggregation.retain-seconds</name>
		<value>604800</value>
    </property>
</configuration>

2、分发软件和环境变量

分发软件

scp -r /opt/soft/hadoop-3 root@spark02:/opt/soft
scp -r /opt/soft/hadoop-3 root@spark03:/opt/soft

分发环境变量

scp /etc/profile.d/my_env.sh root@spark02:/etc/profile.d/
scp /etc/profile.d/my_env.sh root@spark03:/etc/profile.d/

在所有节点是新的环境变量生效

source /etc/profile.d/my_env.sh

3、初始化集群

1.	zookeeper集群正常启动
2.	启动三个JournalNode：
	hadoop-daemon.sh start journalnode 或者 hdfs --daemon start journalnode
3.	在其中一个namenode上格式化：hdfs namenode -format
4.	把刚刚格式化之后的元数据拷贝到另外一个namenode上
    a)	启动刚刚格式化的namenode :
    	hadoop-daemon.sh start namenode 或者 hdfs --daemon start namenode
    b)	在没有格式化的namenode上执行：hdfs namenode -bootstrapStandby
    c)	启动第二个namenode: 
    	hadoop-daemon.sh start namenode 或者 hdfs --daemon start namenode
5.	在其中一个namenode上初始化 hdfs zkfc -formatZK
6.	停止上面节点：stop-dfs.sh
7.	全面启动：start-all.sh
8. 启动resourcemanager节点 
	yarn-daemon.sh start resourcemanager 或者	start-yarn.sh

http://dl.bintray.com/sequenceiq/sequenceiq-bin/hadoop-native-64-2.5.0.tar

不需要执行第 8 步

9. 启动历史服务
mapred --daemon start historyserver
10 11 12 不需要执行
10、安全模式

hdfs dfsadmin -safemode enter  
hdfs dfsadmin -safemode leave


11、查看哪些节点是namenodes并获取其状态
hdfs getconf -namenodes
hdfs haadmin -getServiceState nn1

12、强制切换状态
hdfs haadmin -transitionToActive --forcemanual nn1

重点提示：

 关机之前 依关闭服务
mapred --daemon stop historyserver
stop-yarn.sh
stop-dfs.sh
 开机后 依次开启服务
start-dfs.sh
start-yarn.sh
mapred --daemon start historyserver

或者

 关机之前关闭服务
mapred --daemon stop historyserver
stop-all.sh
 开机后开启服务
start-all.sh
mapred --daemon start historyserver

#jps 检查进程正常后开启胡哦关闭在再做其它操作

4、修改windows下hosts文件

C:\Windows\System32\drivers\etc\hosts

追加以下内容：

192.168.10.101	spark01
192.168.10.102	spark02
192.168.10.103	spark03

Windows11 注意修改权限

开始搜索 cmd

找到命令头提示符以管理身份运行

cmd

进入 C:\Windows\System32\drivers\etc 目录
```
cd drivers/etc
```
打开 hosts 配置文件
```
start hosts
```

追加以下内容后保存

192.168.10.101	spark01
192.168.10.102	spark02
192.168.10.103	spark03

5、测试

5.1 浏览器访问hadoop

浏览器访问: http://spark01:9870

namenode

浏览器访问: http://spark02:9870

namenode

浏览器访问:http://spark01:8088

resourcemanager

浏览器访问:http://spark01:19888

jobHistoryServer

5.2 测试 hdfs

本地文件系统创建测试文件 wcdata.txt

vim wcdata.txt

Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive

在 HDFS 上创建目录 /wordcount/input

hdfs dfs -mkdir -p /wordcount/input

查看 HDFS 目录结构

hdfs dfs -ls /

hdfs dfs -ls /wordcount

hdfs dfs -ls /wordcount/input

上传本地测试文件 wcdata.txt 到 HDFS 上 /wordcount/input

hdfs dfs -put wcdata.txt /wordcount/input

检查文件是否上传成功

hdfs dfs -ls /wordcount/input

hdfs dfs -cat /wordcount/input/wcdata.txt

5.3 测试 mapreduce

计算 PI 的值

hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar pi 10 10

单词统计

hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /wordcount/input/wcdata.txt /wordcount/result

hdfs dfs -ls /wordcount/result

hdfs dfs -cat /wordcount/result/part-r-00000

mHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive


> 在 HDFS 上创建目录 /wordcount/input

```bash
hdfs dfs -mkdir -p /wordcount/input

查看 HDFS 目录结构

hdfs dfs -ls /

hdfs dfs -ls /wordcount

hdfs dfs -ls /wordcount/input

上传本地测试文件 wcdata.txt 到 HDFS 上 /wordcount/input

hdfs dfs -put wcdata.txt /wordcount/input

检查文件是否上传成功

hdfs dfs -ls /wordcount/input

hdfs dfs -cat /wordcount/input/wcdata.txt

5.3 测试 mapreduce

计算 PI 的值

hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar pi 10 10

单词统计

hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /wordcount/input/wcdata.txt /wordcount/result

hdfs dfs -ls /wordcount/result

hdfs dfs -cat /wordcount/result/part-r-00000

李昊哲小课

关注

12
点赞
踩
26

收藏

觉得还不错? 一键收藏
打赏
0
评论
openEuler搭建hadoop 高可用完全分布式集群

openEuler搭建hadoop 高可用完全分布式集群
复制链接

扫一扫

专栏目录