完全分布式的部署
实验背景:
172.25.2.3 ser3 namenode
172.25.2.4 ser4 dadanode
172.25.2.5 ser5 datanode
172.25.2.6 ser6 datanode
关闭几台主机的火墙,selinx
nfs的实现
如何让新加的结点,同步之前结点的数据?
1.首先关闭ser3上之前的伪分布式
[yxx@ser3 hadoop]$ sbin/stop-dfs.sh
Stopping namenodes on [ser3]
Stopping datanodes
Stopping secondary namenodes [ser3]
- 所有结点安装nfs-utils
[yxx@ser3 hadoop]$ logout
You have mail in /var/spool/mail/root
[root@ser3 ~]# yum install nfs-utils -y
[root@ser4 ~]# yum install nfs-utils -y
[root@ser5 ~]# yum install nfs-utils -y
在 ser3上
[root@ser3 ~]# id yxx
uid=1001(yxx) gid=1001(yxx) groups=1001(yxx)
[root@ser3 ~]# cat /etc/exports
/home/yxx *(rw,anonuid=1001,anongid=1001)
[root@ser3 ~]# exportfs -rv
exporting *:/home/yxx
[root@ser3 ~]# systemctl enable --now nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
[root@ser3 ~]# showmount -e
Export list for ser3:
/home/yxx *
在ser4和ser5上
[root@ser4 ~]# useradd yxx
[root@ser4 ~]# id yxx
uid=1001(yxx) gid=1001(yxx) groups=1001(yxx)
[root@ser5 ~]# useradd yxx
[root@ser5 ~]# id yxx
uid=1001(yxx) gid=1001(yxx) groups=1001(yxx)
注意保持全平台一致,ser3.ser4,ser5上的yxx用户的uid,gid应该一致
所有结点开启rpc-bind
[root@ser3 ~]# systemctl start rpcbind
在ser4和ser5上挂载ser3的用户主目录
[root@ser4 ~]# showmount -e 172.25.2.3
Export list for 172.25.2.3:
/home/yxx *
[root@ser4 ~]# mount 172.25.2.3:/home/yxx/ /home/yxx
[root@ser4 ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/rhel-root 17811456 3726040 14085416 21% /
devtmpfs 924072 0 924072 0% /dev
tmpfs 941100 0 941100 0% /dev/shm
tmpfs 941100 9372 931728 1% /run
tmpfs 941100 0 941100 0% /sys/fs/cgroup
/dev/sda1 1038336 181140 857196 18% /boot
tmpfs 188224 12 188212 1% /run/user/42
tmpfs 188224 0 188224 0% /run/user/0
172.25.2.3:/home/yxx 17811456 6137856 11673600 35% /home/yxx
设置几台主机的免密,因为它们挂载的是一样的公钥和私钥
[yxx@ser3 ~]$ ssh 172.25.2.4
[yxx@ser3 ~]$ ssh 172.25.2.5
对master(ser3)进行设置
[yxx@ser3 ~]$ cd hadoop
[yxx@ser3 hadoop]$ cd etc/hadoop/
[yxx@ser3 hadoop]$ vim hdfs-site.xml
修改副本数为2
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>
修改worker结点
[yxx@ser3 hadoop]$ vim workers
[yxx@ser3 hadoop]$ cat workers
172.25.2.4
172.25.2.5
[yxx@ser3 hadoop]$ rm -fr /tmp/*
重新进行初始化
hdfs namenode -format
重新启动,在ser3上启动nn,在ser4和ser5上启动datanode
[yxx@ser3 ~]$ cd hadoop
[yxx@ser3 hadoop]$ sbin/start-dfs.sh
Starting namenodes on [ser3]
Starting datanodes
Starting secondary namenodes [ser3]
ser3上此时少了一个datanode
[yxx@ser3 hadoop]$ jps
17329 Jps
17196 SecondaryNameNode
16957 NameNode
ser4和ser5已经成为了datanode
[root@ser4 ~]# su - yxx
Last login: Sun May 31 12:29:48 CST 2020 from 172.25.2.3 on pts/1
[yxx@ser4 ~]$ jps
14100 DataNode
14245 Jps
[root@ser5 ~]# su - yxx
Last login: Sun May 31 12:29:58 CST 2020 from ser3 on pts/2
[yxx@ser5 ~]$ jps
15303 DataNode
15439 Jps
测试
测试
因为之前定义的是存储两份,因此实际上会消耗400m的存储,
[yxx@ser3 hadoop]$ dd if=/dev/zero of=bigfile bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 0.344952 s, 608 MB/s
上传至分布式系统
因为之前进行了清理,所以现在需要重新创建用户主目录,上传时不指定文件路径,会直接上传至用户主目录
[yxx@ser3 hadoop]$ hdfs dfs -mkdir /user
[yxx@ser3 hadoop]$ hdfs dfs -mkdir /user/yxx
[yxx@ser3 hadoop]$ hdfs dfs -put bigfile
2020-05-31 12:51:08,326 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2020-05-31 12:51:39,862 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
可以看到每一份拷贝被分为两个block,因为这里一份最大存128M
元数据信息都存储在nn上,nn存储了block表和datanode结点的结点
dn出现故障,nn的两张表都要更新。
存储结点(dn)的热添加
在集群的状态为running的情况下,手动添加ser6加入集群,做datanode。
[root@ser6 ~]# useradd yxx
[root@ser6 ~]# id yxx
uid=1001(yxx) gid=1001(yxx) groups=1001(yxx)
[root@ser6 ~]# yum install nfs-utils -y
[root@ser6 ~]# systemctl start rpcbind
[root@ser6 ~]# mount 172.25.2.3:/home/yxx /home/yxx
[root@ser6 ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/rhel-root 17811456 3720276 14091180 21% /
devtmpfs 924072 0 924072 0% /dev
tmpfs 941100 0 941100 0% /dev/shm
tmpfs 941100 9412 931688 2% /run
tmpfs 941100 0 941100 0% /sys/fs/cgroup
/dev/sda1 1038336 181140 857196 18% /boot
tmpfs 188224 12 188212 1% /run/user/42
tmpfs 188224 0 188224 0% /run/user/0
172.25.2.3:/home/yxx 17811456 6341632 11469824 36% /home/yxx
在ser3上做免密
[yxx@ser3 hadoop]$ ssh 172.25.2.6
The authenticity of host '172.25.2.6 (172.25.2.6)' can't be established.
ECDSA key fingerprint is SHA256:Tsxe8LyfOAjPlH9iB0gv5RIN1GBLFjhC7nw3UrVZng4.
ECDSA key fingerprint is MD5:2e:97:50:30:c7:83:8b:93:c4:e4:e2:38:ec:d9:97:a2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.25.2.6' (ECDSA) to the list of known hosts.
Last login: Sun May 31 14:31:09 2020
添加worker
[root@ser6 ~]# su - yxx
Last login: Sun May 31 14:32:57 CST 2020 from ser3 on pts/2
[yxx@ser6 ~]$ cd hadoop
[yxx@ser6 hadoop]$ cd etc/hadoop/
[yxx@ser6 hadoop]$ vim workers
[yxx@ser6 hadoop]$ cat workers
172.25.2.4
172.25.2.5
172.25.2.6
启动ser6,让它只做datanode
[yxx@ser6 hadoop]$ jps
14405 Jps
[yxx@ser6 hadoop]$
[yxx@ser6 hadoop]$ pwd
/home/yxx/hadoop
[yxx@ser6 hadoop]$ cd s
sbin/ share/
[yxx@ser6 hadoop]$ cd sbin/
[yxx@ser6 sbin]$ hdfs --daemon start datanode
[yxx@ser6 sbin]$ jps
14482 DataNode
14498 Jps
[yxx@ser6 sbin]$
datanode会向nn汇报信息
设置数据保存三份
[yxx@ser6 hadoop]$ cd etc/hadoop/
[yxx@ser6 hadoop]$ vim hdfs-site.xml
[yxx@ser6 hadoop]$
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
目前为止,我们只做了数据存储层,HDFS。这一核心组件。
客户端需要进行文件切分和文件的副本数的控制
client -> nn (master)–> dn(slave)
之前的实验,是把nn充当为客户端,dn也可以当客户端,第一份数据存储在离自己最接近的机架上,当dn1是客户端时,第一份数据应该保存在同一机架上,第二份数据应该存储在第二个机架上,第三份数据存储在第二个机架上的不同dn上,为了减少数据 传输的损耗。hadoop具有机架 感应技术。同一个机架上,同一副本数只能存在两份。
客户端在外部时,第一个结点是随机的,随机在某个机架上。第二份和第一份不能在同一个机架上,第三份和第二份可以在同一个机架上。