搭建hdfs分布式文件系统

最新推荐文章于 2024-03-19 20:00:00 发布

Master_slaves

最新推荐文章于 2024-03-19 20:00:00 发布

阅读量660

点赞数

分类专栏： hadoop 文章标签： Hadoop

本文链接：https://blog.csdn.net/master_chaoandqi/article/details/88375334

版权

hadoop 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

搭建采用windows108G内存 centos6.6 jdk1.8 4台虚拟机

1 上传JDK至虚拟机解压文件

 tar -zxvf jdk-8u141-linux-x64.tar.gz

2 配置环境变量增加JAVA环境变量

vi  /etc/profile

export JAVA_HOME=/app/jdk1.8
export PATH=.:$PATH:$JAVA_HOME/bin

3 配置完成后:wq退出输入在etc目录下输入此命令让配置文件生效

[root@centos01 etc]# source profile

4 输入java -version命令查看版本信息

[root@centos01 etc]# java -version
java version "1.8.0_141"
Java(TM) SE Runtime Environment (build 1.8.0_141-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode)

jdk配置就完成了

5 配置主机名与域名映射（后面使用主机名完成访问）修改主机名为centos01

[root@centos01 etc]# vi sysconfig/network

增加如下配置

NETWORKING=yes
HOSTNAME=centos01

6 修改hosts文件配置域名映射

[root@centos01 etc]# vi hosts

增加如下配置
192.168.131.3   centos01   linux机器Ip地址映射
192.168.131.2   windows    windows ip地址映射

检查是否配置成功

[root@centos01 etc]# ping centos01
PING centos01 (192.168.131.3) 56(84) bytes of data.
64 bytes from centos01 (192.168.131.3): icmp_seq=1 ttl=64 time=0.028 ms
64 bytes from centos01 (192.168.131.3): icmp_seq=2 ttl=64 time=0.040 ms
64 bytes from centos01 (192.168.131.3): icmp_seq=3 ttl=64 time=0.040 ms
64 bytes from centos01 (192.168.131.3): icmp_seq=4 ttl=64 time=0.039 ms
64 bytes from centos01 (192.168.131.3): icmp_seq=5 ttl=64 time=0.039 ms
64 bytes from centos01 (192.168.131.3): icmp_seq=6 ttl=64 time=0.042 ms

 ping windows
PING windows (192.168.131.2) 56(84) bytes of data.
64 bytes from windows (192.168.131.2): icmp_seq=1 ttl=128 time=16.0 ms
64 bytes from windows (192.168.131.2): icmp_seq=2 ttl=128 time=0.284 ms
64 bytes from windows (192.168.131.2): icmp_seq=3 ttl=128 time=0.330 ms
64 bytes from windows (192.168.131.2): icmp_seq=4 ttl=128 time=0.267 ms
^C

域名映射就配置完成了。

7 克隆出3台虚拟机（以centos01为例）

7.1修改Ip地址【centos01：192.168.131.3】【centos02：192.168.131.4】【centos03：192.168.131.5】【centos03：192.168.131.6】 centos02到04 分别需要修改网卡名称为ifcfg-eth1

[root@centos01 etc]# cat sysconfig/network-scripts/ifcfg-eth0 
DEVICE="eth0"
TYPE="Ethernet"
ONBOOT="yes"
NM_CONTROLLED="yes"
BOOTPROTO="static"
IPADDR=192.168.131.3
NETMASK=255.255.255.0
GETAWAY=192.168.131.2
DNS1=8.8.8.8

[root@centos01 etc]#

7.2 修改主机名 vi sysconfig/network centos01 centos02 centos03 centos04

[root@centos01 etc]# cat sysconfig/network
NETWORKING=yes
HOSTNAME=centos01
GETAWAY=192.168.131.2

7.3 配置域名映射

[root@centos01 etc]# cat hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.131.3   centos01
192.168.131.4   centos02
192.168.131.5   centos03
192.168.131.6   centos04
192.168.131.2   windows

7.4 安装jdk(已完成)克隆的母机已安装

7.5 关闭防火墙

[root@centos01 /]# service iptables status
iptables: Firewall is not running.

7.6 ssh免密登陆

本地免密码登录
    　ssh-keygen

 ssh-copy-id centos02

7. 7 时间同步（date -s '2019-03-11 20:23:10'）

8 配置本地yum源（https://www.cnblogs.com/gaoxu007/p/7173112.html）

mkdir /mnt/cdrom  创建文件

# mount -o loop /dev/cdrom /mnt/cdrom/   



# cd /etc/yum.repos.d/
# rm -rf *              //删除下面的所有文件或者命名为其他名字
# touch yum.repo       创建一个文件
# vi  yum.repo       编辑文件 加入以下内容




[rhel]
name =Red Hat enterprise Linux 6
baseurl=file:///mnt/cdrom
enable=1
gpgkey=1
gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-CentOS-6  此文件在/mnt/cdrom/目录下


修改 /etc/fstab  这个文件   再最后面加入

/dev/cdrom/             /mnt/cdrom              iso9660 ro              0 0
  


[root@centos01 /]# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Sun Mar 10 05:58:00 2019
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_centos01-lv_root /                       ext4    defaults        1 1
UUID=b29da312-2468-4116-b915-a57868afaf49 /boot                   ext4    defaults        1 2
/dev/mapper/vg_centos01-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/cdrom/             /mnt/cdrom              iso9660 ro              0 0

9 安装 yum install openssh-clients

用scp命令拷贝文件至其他机器

10 上传hadoop2.8.3的压缩包并解压

11 修改配置文件hdfs-site.xml文件

配置nameNode 在哪台环境上运行

配置nameNode的数据存储目录

配置dataNode的数据目录

配置secondarynamenode在哪台机器上运行

<configuration>
<property>
<name>dfs.nameode.rpc-address</name>
<value>centos01:9000</value>
</property>

<property>
<name>dfs.namenode.name.dir</name>
<value>/hadoop/hdpdata/name</value>

</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/hadoop/hdpdata/data</value>

</property>


</configuration>

12 修改vi hadoop-env.sh 修改本地jdk配置 export JAVA_HOME=/app/jdk1.8

13 修改etc/profile文件加上hadoop的环境变量

done
unset i
unset -f pathmunge
export JAVA_HOME=/app/jdk1.8
export HADOOP_HOME=/hadoop/hadoop-2.8.3
export PATH=.:$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin

14 通过scp命令拷贝文件至centos02 centos03 centos04 机器

scp profile centos04:/etc/

15 格式化存储目录 hadoop namenode -format

19/03/12 23:39:31 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
19/03/12 23:39:31 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
19/03/12 23:39:31 INFO util.GSet: Computing capacity for map NameNodeRetryCache
19/03/12 23:39:31 INFO util.GSet: VM type       = 64-bit
19/03/12 23:39:31 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
19/03/12 23:39:31 INFO util.GSet: capacity      = 2^15 = 32768 entries
19/03/12 23:39:31 INFO namenode.FSImage: Allocated new BlockPoolId: BP-393041277-192.168.131.3-1552405171820
19/03/12 23:39:31 INFO common.Storage: Storage directory /hadoop/hdpdata/name has been successfully formatted.
19/03/12 23:39:31 INFO namenode.FSImageFormatProtobuf: Saving image file /hadoop/hdpdata/name/current/fsimage.ckpt_0000000000000000000 using no compression
19/03/12 23:39:32 INFO namenode.FSImageFormatProtobuf: Image file /hadoop/hdpdata/name/current/fsimage.ckpt_0000000000000000000 of size 321 bytes saved in 0 seconds.
19/03/12 23:39:32 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
19/03/12 23:39:32 INFO util.ExitUtil: Exiting with status 0
19/03/12 23:39:32 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at centos01/192.168.131.3

16 启动hdfs的namenode 和datanode

hadoop-daemon.sh start namenode
-bash: hadoop-daemon.sh: command not found

错误原因未配置hadoop/sbin的环境目录

17 配置sbin的环境变量 vi etc/profile 文件 Path后增加：$HADOOP_HOME/sbin

18 启动namenode 报错没有权限

.defaultFS is file:///
2019-03-12 23:52:36,318 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-03-12 23:52:36,615 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
        at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:644)
        at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddressCheckLogical(DFSUtilClient.java:673)
        at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:635)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.getRpcServerAddress(NameNode.java:532)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:663)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:683)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:906)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:885)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1694)
2019-03-12 23:52:36,622 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2019-03-12 23:52:36,629 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at centos01/192.168.131.3
************************************************************/

18 未配置core-site.xml文件

<configuration>



<property> 

    <name>fs.default.name</name> 

    <value>hdfs://centos01:9000</value> 

  </property> 

</configuration>

19 启动namenode和3台datanode机器

hadoop-daemon.sh start datanode

hadoop-daemon.sh start namenode

20 修改本机hosts文件增加ip映射

C:\Windows\System32\drivers\etc 


192.168.131.3 centos01
192.168.131.4 centos02
192.168.134.5 centos03
192.168.131.6 centos04

21 浏览器访问http://centos01:50070/dfshealth.html#tab-datanode

22 环境搭建完毕在浏览器中可以看到节点信息

Master_slaves

关注

0
点赞
踩
5

收藏

觉得还不错? 一键收藏
打赏
0
评论
搭建hdfs分布式文件系统

搭建采用windows108G内存 centos6.6 jdk1.8 4台虚拟机1 上传JDK至虚拟机解压文件 tar -zxvf jdk-8u141-linux-x64.tar.gz 2 配置环境变量增加JAVA环境变量vi /etc/profileexport JAVA_HOME=/app/jdk1.8export PATH=.:$PATH:$JA...
复制链接

扫一扫