Linux中配置文件分发步骤,hadoop详细文档(四) hadoop之HDFS安装详解(附带详细讲解视频)...

免费视频教程https://www.51doit.com/ 或者联系博主微信 17710299606

1 安装详细步骤

1.1 上传HDP的安装包到Linux01 机器的指定目录下

1.2  解压到指定的目录下

tar -zxvf hadoop-2.8.5.tar.gz

1.3 目录结构

1.4 修改配置文件

配置文件的目录在/opt/apps/hadoop-2.8.5/etc/hadoop/下

1.4.1  vi hadoop-env.sh

配置Hadoop使用的java环境变量 ,使用绝对路径

# The java implementation to use.

export JAVA_HOME=/opt/apps/jdk1.8.0_231

1.4.2  vi  hdfs-site.xml

dfs.namenode.rpc-address

linux01:9000

dfs.namenode.name.dir

/opt/hdpdata/name

dfs.datanode.data.dir

/opt/hdpdata/data

dfs.namenode.secondary.http-address

linux02:50090

1.4.3 vi core-site.xml

配置HDFS客户端默认操作的文件系统为HDFS分布式文件系统

fs.defaultFS

hdfs://linux01:9000

1.5  将配置好的安装包 分发到集群中的其他机器

apps>  scp -r hadoop-2.8.5  linux02:$PWD

apps>  scp -r hadoop-2.8.5  linux03:$PWD

1.6 初始化namenode

bin/hadoop  namenode  -format

初始化以后会在namenode节点上的指定的目录中生成文件夹

/opt/hdpdata/name

2 启动

2,1 单节点启动

启动 namenode

sbin/hadoop-daemon.sh   start namenode

linux01启动一个datanode

sbin/hadoop-daemon.sh   start datanode

linux02启动一个datanode

sbin/hadoop-daemon.sh   start datanode

linux03启动一个datanode

sbin/hadoop-daemon.sh   start datanode

查看当前机器的java进程  jps

[root@linux01 sbin]# jps

6053 DataNode

8600 Jps

5915 NameNode

http://linux01:50070访问HDFSwebUI页面

2.2  一键启动

查看 sbin下所有的命令

[root@linux01 sbin]# ll

total 124

-rwxr-xr-x. 1 reba reba 2752 Sep 10  2018 distribute-exclude.sh

-rwxr-xr-x. 1 reba reba 6467 Sep 10  2018 hadoop-daemon.sh

-rwxr-xr-x. 1 reba reba 1360 Sep 10  2018 hadoop-daemons.sh

-rwxr-xr-x. 1 reba reba 1640 Sep 10  2018 hdfs-config.cmd

-rwxr-xr-x. 1 reba reba 1427 Sep 10  2018 hdfs-config.sh

-rwxr-xr-x. 1 reba reba 2339 Sep 10  2018 httpfs.sh

-rwxr-xr-x. 1 reba reba 3763 Sep 10  2018 kms.sh

-rwxr-xr-x. 1 reba reba 4134 Sep 10  2018 mr-jobhistory-daemon.sh

-rwxr-xr-x. 1 reba reba 1648 Sep 10  2018 refresh-namenodes.sh

-rwxr-xr-x. 1 reba reba 2145 Sep 10  2018 slaves.sh

-rwxr-xr-x. 1 reba reba 1779 Sep 10  2018 start-all.cmd

-rwxr-xr-x. 1 reba reba 1471 Sep 10  2018 start-all.sh

-rwxr-xr-x. 1 reba reba 1128 Sep 10  2018 start-balancer.sh

-rwxr-xr-x. 1 reba reba 1401 Sep 10  2018 start-dfs.cmd

-rwxr-xr-x. 1 reba reba 3734 Sep 10  2018 start-dfs.sh

-rwxr-xr-x. 1 reba reba 1357 Sep 10  2018 start-secure-dns.sh

-rwxr-xr-x. 1 reba reba 1571 Sep 10  2018 start-yarn.cmd

-rwxr-xr-x. 1 reba reba 1347 Sep 10  2018 start-yarn.sh

-rwxr-xr-x. 1 reba reba 1770 Sep 10  2018 stop-all.cmd

-rwxr-xr-x. 1 reba reba 1462 Sep 10  2018 stop-all.sh

-rwxr-xr-x. 1 reba reba 1179 Sep 10  2018 stop-balancer.sh

-rwxr-xr-x. 1 reba reba 1455 Sep 10  2018 stop-dfs.cmd

-rwxr-xr-x. 1 reba reba 3206 Sep 10  2018 stop-dfs.sh

-rwxr-xr-x. 1 reba reba 1340 Sep 10  2018 stop-secure-dns.sh

-rwxr-xr-x. 1 reba reba 1642 Sep 10  2018 stop-yarn.cmd

-rwxr-xr-x. 1 reba reba 1340 Sep 10  2018 stop-yarn.sh

-rwxr-xr-x. 1 reba reba 4295 Sep 10  2018 yarn-daemon.sh

-rwxr-xr-x. 1 reba reba 1353 Sep 10  2018 yarn-daemons.sh

vi  etc/hadoop/slaves   配置从节点要在那些机器中启动

在这三个节点上分别启动/停止DN

linux01

linux02

linux03

系统环境变量的配置

export  JAVA_HOME=/opt/apps/jdk1.8.0_231

export  HADOOP_HOME=/opt/apps/hadoop-2.8.5

export  PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

.  或者   source /etc/profile

在任意位置输入start-dfs.sh 或者是stop-dfs.shs 一键启动或者是停止集群

webUI访问页面无法请求,数据无法存储副本等问题 ,学习阶段建议关闭所有的防火墙服务

-- 查看防火墙状态 systemctl status firewalld

● firewalld.service - firewalld - dynamic firewall daemon

Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)

Active: active (running) since Tue 2020-06-30 17:58:19 EDT; 6h ago

Docs: man:firewalld(1)

Main PID: 7105 (firewalld)

CGroup: /system.slice/firewalld.service

└─7105 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Jun 30 17:58:18 localhost.localdomain systemd[1]: Starting firewalld - dynami...

Jun 30 17:58:19 localhost.localdomain systemd[1]: Started firewalld - dynamic...

Hint: Some lines were ellipsized, use -l to show in full.

-- 停止防火墙 systemctl stop firewalld

[root@bogon ~]# systemctl status firewalld

firewalld.service - firewalld - dynamic firewall daemon

Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)

Active: inactive (dead) since Wed 2020-07-01 00:52:08 EDT; 6s ago

Docs: man:firewalld(1)

Process: 7105 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)

Main PID: 7105 (code=exited, status=0/SUCCESS)

--启动防火墙 systemctl start firewalld

--配置防火墙的开启自启和关闭

上述启动和停止仅仅对防火墙的临时操作 , 配置了防火墙的禁用/启用以后永久生效

[root@bogon ~]# systemctl disable firewalld 禁用

Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.

Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

[root@bogon ~]# systemctl enable firewalld 启动

Created symlink from /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service to /usr/lib/systemd/system/firewalld.service.

Created symlink from /etc/systemd/system/multi-user.target.wants/firewalld.service to /usr/lib/systemd/system/firewalld.service.