Hadoop集群搭建(win10+vmware+centos7+hadoop 3.1.2)

最新推荐文章于 2020-08-07 20:06:50 发布

zmywhhit

最新推荐文章于 2020-08-07 20:06:50 发布

阅读量856

点赞数

分类专栏： hapoop

原文链接：https://blog.csdn.net/iflytop/article/details/82413694

版权

hapoop 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

大致步骤为

1、安装vmware

2、安装操作系统

3、配置网络

4、配置主机名与hosts文件

5、配置免密登录

6、安装JDK

7、安装hadoop

8、执行示例验证

1、VMWARE安装

注意下载wmware pro，我前期下载的vmware player，后面配置网络总是有问题，功能不够强大。

a、VMARE安装包

官网下载地址 15版本

官网地址：https://www.vmware.com/products/workstation-pro/workstation-pro-evaluation.html

b、破解秘钥工具

链接：https://pan.baidu.com/s/1Ck8PjexzGK_12PbYXerMCA
提取码：qp2m
复制这段内容后打开百度网盘手机App，操作更方便哦

安装后使用工具生成秘钥即可

2、安装操作系统

手里没有资源直接，参考如下博客安装即可

https://blog.csdn.net/babyxue/article/details/80970526

3、配置网络

后续会使用主机名和配置网络，所以需要固定IP，采用配置NAT模式。

1、配置VMWARENAT网络设置。（先点击更改设置）

2、设置主机对应的虚拟网关，后续配置静态IP的时候需要输入192.168.40.2作为网关。

3、设置虚拟机网络设置选择之前配置的VMNet8(NAT模式)

4、配置操作系统静态IP。

a、root用户登录操作系统，修改网络适配器

cd /etc/sysconfig/network-scripts

静态IP、开启启动、IP地址、网关、DNS可以百度找一个本省或者本市的DNS。

b、重启网络服务

service network restart

c、尝试ping下baidu.com(如果找不到DNS就本机ping下获取IP地址，centos上ping下IP地址)

c、设置本机的网络可以与虚拟机主机互通，方便后续本机ssh登录操作。

网络配置和虚拟操作系统设置及一样，设置IP、相同的网关等。

验证本机到虚拟机的网络，ping下网络。

4、配置主机名与hosts文件

a、配置主机名

修改成容易识别的centos+IP末尾，方便后续配置，wq保存退出。

b、配置hosts，把其他主机名称与IP映射添加到hosts文件中。

vi /etc/hosts

5、配置免密登录

a、创建hadoop用户、修改密码

useradd -d /home/hd -m hd

passwd hd

b、生成秘钥

ssh-keygen -t rsa

会在用户目录中生成.ssh目录其中生成秘钥信息。

b、接下来需要将公钥拷贝到要免密登陆的目标机器上，给本机也复制一份。

首先进入个人home/.ssh 目录下执行：

ssh-copy-id centos101

ssh-copy-id centos102

ssh-copy-id centos103

验证：在102上ssh centos100，直接登录无需输入密码。

关闭防火墙

systemctl stop firewalld.service 关闭防火墙

systemctl disable firewalld.service 禁止防火墙开机自启

6、安装JDK1.8

下载地址：https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

下载的1.8的压缩包版本。

b、本地通过sftp上传到虚拟机系统。

在/home/hd目录下mkdir apps，安装文件都上传安装到该目录下。

SFTP连接服务器，标题栏右键sftp连接会话。

sftp控制台输入cd /home/hp/apps

使用put 上传

c、解压安装回到ssh命令行之行解压。

cd /home/hd/apps/

执行：tar -zxvf jdk-8u221-linux-x64.tar.gz

d、配置环境变量

cd /home/hp

vi ./.bash_profile

新增java_home和java执行路径

java -verison验证安装成功

7、安装hadoop（首先只安装主节点，其他节点复制过去。）

a、官网现在hadoop，并同样方法上传到/home/hp/apps目录下。

镜像服务：http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.1.2/

上传完成后进行解压

cd /home/hd/apps

b、tar -zxvf hadoop-3.1.0.tar.gz

c、修改配置参数

/home/hd/apps/hadoop-3.1.2/etc/hadoop目录下如下5个配置文件修改参数如下

文件1：hadoop-env.sh 设置JAVA_HOME参数

文件2：core-site.xml指定hadoop所使用的文件系统，以及将centos100作为namenode节点，设置hadoop运行时文件存储目录.

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://centos100:9000</value>
</property>
<property>
        <name>hadoop.tmp.dir</name>
        <value>/home/hd/hadoopdata</value>
</property>
</configuration>

文件3：hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
        <name>dfs.replication</name>
        <value>2</value>
</property>
</configuration>

文件4：mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
</configuration>
~

文件5：yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
<property>
        <name>yarn.resourcemanger.hostname</name>
        <value>centos100</value>
</property>
<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
</property>
</configuration>

6、workers

在此文件添加我们作为DataNode节点的两台台机器的主机名，如果我们只将centos100 作为 namenode不做为datanode的话，就不用在此文件中添加centos100了，我这里三台机器都做为 DataNode 节点

centos100
centos101
centos102

d、将Hadoop发到另外两台机器

配置文件修改好后，我们需要将 hadoop 复制到centos72和centos73

执行命令：

scp -r /home/hd/apps/hadoop-3.1.2/ hp@centos101:/home/hd/apps/

scp -r /home/hd/apps/hadoop-3.1.2/ hp@centos102:/home/hd/apps/

e、修改hadoop环境变量

vi /home/hp/.bash_profile

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin
#Java Env
export JAVA_HOME=/home/hd/apps/jdk1.8.0_221
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
export HADOOP_HOME=/home/hd/apps/hadoop-3.1.2
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export PATH

使环境变量生效

[hd@centos100 ~]$ source /home/hd/.bash_profile