Hadoop
一、环境搭建VMware ,CentOS7,Ubuntu
1、VMware Workstation 15 Pro安装图解教程
详细见：https://www.shuzhiduo.com/A/Gkz1PGeQdR/
2、VMware Workstation 15 Pro安装带图形化界面的CentOS7
详细见：https://www.shuzhiduo.com/A/qVde08Gg5P/
3、VMware Workstation 15 Pro中安装ubuntu1804
详细见：https://www.shuzhiduo.com/A/MAzAPGWnd9/
注释：强烈安利复制粘贴、适应屏幕（安装VMware Tools）
二.创建普通用户(本人一开始用了，后面全部直接用的root)
1.在当前用户下面创建hadoop用户
[root@localhost ~]# useradd -m hadoop -s /bin/bash
[root@localhost ~]# passwd hadoop
2.如果不是root用户
[用户@localhost ~]# sudo useradd -m hadoop -s /bin/bash
[用户@localhost ~]# sudo passwd hadoop
3.为hadoop用户添加管理员权限
[root@localhost ~]# adduser hadoop sudo
三.安装jdk
[hadoop@localhost ~]$ su
密码：
[root@localhost hadoop]# ls
hadoop-3.3.0.tar.gz jdk-11.0.10_linux-x64_bin.rpm 模板 图片 下载 桌面
java 公共 视频 文档 音乐
[root@localhost hadoop]# pwd
/home/hadoop
[root@localhost hadoop]# cd ..
[root@localhost home]# cd ..
[root@localhost /]# ls
bin dev Hello.java lib media opt proc run software sys usr
boot etc home lib64 mnt passwd root sbin srv tmp var
[root@localhost /]# cd /home/hadoop
[root@localhost hadoop]# ls
hadoop-3.3.0.tar.gz jdk-11.0.10_linux-x64_bin.rpm 模板 图片 下载 桌面
java 公共 视频 文档 音乐
[root@localhost hadoop]# rpm -i jdk-11.0.10_linux-x64_bin.rpm
警告：jdk-11.0.10_linux-x64_bin.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID ec551f03: NOKEY
[root@localhost hadoop]#
[root@localhost hadoop]# ls
hadoop-3.3.0.tar.gz jdk-11.0.10_linux-x64_bin.rpm 模板 图片 下载 桌面
java 公共 视频 文档 音乐
[root@localhost hadoop]# java
用法：java [options] <主类> [args...]
（执行类）
或 java [options] -jar <jar 文件> [args...]
（执行 jar 文件）
或 java [options] -m <模块>[/<主类>] [args...]
java [options] --module <模块>[/<主类>] [args...]
（执行模块中的主类）
或 java [options] <源文件> [args]
（执行单个源文件程序）
将主类、源文件、-jar <jar 文件>、-m 或
--module <模块>/<主类> 后的参数作为参数
传递到主类。
其中，选项包括：
-cp <目录和 zip/jar 文件的类搜索路径>
-classpath <目录和 zip/jar 文件的类搜索路径>
--class-path <目录和 zip/jar 文件的类搜索路径>
使用 : 分隔的, 用于搜索类文件的目录, JAR 档案
和 ZIP 档案列表。
-p <模块路径>
--module-path <模块路径>...
用 : 分隔的目录列表, 每个目录
都是一个包含模块的目录。
--upgrade-module-path <模块路径>...
用 : 分隔的目录列表, 每个目录
都是一个包含模块的目录, 这些模块
用于替换运行时映像中的可升级模块
--add-modules <模块名称>[,<模块名称>...]
除了初始模块之外要解析的根模块。
<模块名称> 还可以为 ALL-DEFAULT, ALL-SYSTEM,
ALL-MODULE-PATH.
--list-modules
列出可观察模块并退出
-d <module name>
--describe-module <模块名称>
描述模块并退出
--dry-run 创建 VM 并加载主类, 但不执行 main 方法。
此 --dry-run 选项对于验证诸如
模块系统配置这样的命令行选项可能非常有用。
--validate-modules
验证所有模块并退出
--validate-modules 选项对于查找
模块路径中模块的冲突及其他错误可能非常有用。
-D<名称>=<值>
设置系统属性
-verbose:[class|module|gc|jni]
启用详细输出
-version 将产品版本输出到错误流并退出
--version 将产品版本输出到输出流并退出
-showversion 将产品版本输出到错误流并继续
--show-version
将产品版本输出到输出流并继续
--show-module-resolution
在启动过程中显示模块解析输出
-? -h -help
将此帮助消息输出到错误流
--help 将此帮助消息输出到输出流
-X 将额外选项的帮助输出到错误流
--help-extra 将额外选项的帮助输出到输出流
-ea[:<程序包名称>...|:<类名>]
-enableassertions[:<程序包名称>...|:<类名>]
按指定的粒度启用断言
-da[:<程序包名称>...|:<类名>]
-disableassertions[:<程序包名称>...|:<类名>]
按指定的粒度禁用断言
-esa | -enablesystemassertions
启用系统断言
-dsa | -disablesystemassertions
禁用系统断言
-agentlib:<库名>[=<选项>]
加载本机代理库 <库名>, 例如 -agentlib:jdwp
另请参阅 -agentlib:jdwp=help
-agentpath:<路径名>[=<选项>]
按完整路径名加载本机代理库
-javaagent:<jar 路径>[=<选项>]
加载 Java 编程语言代理, 请参阅 java.lang.instrument
-splash:<图像路径>
使用指定的图像显示启动屏幕
自动支持和使用 HiDPI 缩放图像
(如果可用)。应始终将未缩放的图像文件名 (例如, image.ext)
作为参数传递给 -splash 选项。
将自动选取提供的最合适的缩放
图像。
有关详细信息, 请参阅 SplashScreen API 文档
@argument 文件
一个或多个包含选项的参数文件
-disable-@files
阻止进一步扩展参数文件
--enable-preview
允许类依赖于此发行版的预览功能
要为长选项指定参数, 可以使用 --<名称>=<值> 或
--<名称> <值>。
[root@localhost hadoop]# ls
hadoop-3.3.0.tar.gz jdk-11.0.10_linux-x64_bin.rpm 模板 图片 下载 桌面
java 公共 视频 文档 音乐
[root@localhost hadoop]# cd /usr
[root@localhost usr]# ls
bin games java lib64 local share tmp
etc include lib libexec sbin src
[root@localhost usr]# cd java
[root@localhost java]# ls
default jdk-11.0.10 jdk-11.0.10_linux-x64_bin.rpm latest
[root@localhost java]# vim /etc/profile
[root@localhost java]# source /etc/profile
[root@localhost java]# javac -version
javac 11.0.10
[root@localhost java]# java -version
java version "11.0.10" 2021-01-19 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.10+8-LTS-162)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.10+8-LTS-162, mixed mode)
[root@localhost java]#
[root@localhost java]# vim /etc/profile
在最下面添加环境变量
#Java_HOME
export JAVA_HOME=/usr/java/jdk-11.0.10
export PATH=${PATH}:${JAVA_HOME}/bin
四、hadoop 不在 sudoers 文件中。此事将被报告。
详细见：https://my.oschina.net/u/4410397/blog/3428595
五、虚拟机网络配置
[root@hadoop001 ~]# cd /etc
[root@hadoop001 etc]# ls
abrt hosts purple
adjtime hosts.allow python
aliases hosts.deny qemu-ga
aliases.db hp qemu-kvm
alsa idmapd.conf radvd.conf
alternatives init.d ras
anacrontab inittab rc0.d
asound.conf inputrc rc1.d
at.deny iproute2 rc2.d
audisp ipsec.conf rc3.d
audit ipsec.d rc4.d
avahi ipsec.secrets rc5.d
bash_completion.d iscsi rc6.d
bashrc issue rc.d
binfmt.d issue.net rc.local
bluetooth kdump.conf rdma
brltty kernel redhat-release
brltty.conf krb5.conf request-key.conf
centos-release krb5.conf.d request-key.d
centos-release-upstream ksmtuned.conf resolv.conf
chkconfig.d ld.so.cache resolv.conf.YW7P00
chrony.conf ld.so.conf rpc
chrony.keys ld.so.conf.d rpm
cifs-utils libaudit.conf rsyncd.conf
cron.d libibverbs.d rsyslog.conf
cron.daily libnl rsyslog.d
cron.deny libreport rwtab
cron.hourly libuser.conf rwtab.d
cron.monthly libvirt samba
crontab locale.conf sane.d
cron.weekly localtime sasl2
crypttab login.defs scl
csh.cshrc logrotate.conf securetty
csh.login logrotate.d security
cups lsm selinux
cupshelpers lvm services
dbus-1 machine-id sestatus.conf
dconf magic setroubleshoot
default mail.rc setuptool.d
depmod.d makedumpfile.conf.sample sgml
dhcp man_db.conf shadow
DIR_COLORS mcelog shadow-
DIR_COLORS.256color mke2fs.conf shells
DIR_COLORS.lightbgcolor modprobe.d skel
dleyna-server-service.conf modules-load.d smartmontools
dnsmasq.conf motd sos.conf
dnsmasq.d mtab speech-dispatcher
dracut.conf mtools.conf ssh
dracut.conf.d multipath ssl
drirc my.cnf statetab
e2fsck.conf my.cnf.d statetab.d
enscript.cfg nanorc subgid
environment netconfig subuid
ethertypes NetworkManager sudo.conf
exports networks sudoers
exports.d nfs.conf sudoers.d
favicon.png nfsmount.conf sudo-ldap.conf
fcoe nsswitch.conf sysconfig
festival nsswitch.conf.bak sysctl.conf
filesystems nsswitch.conf.rpmnew sysctl.d
firefox ntp systemd
firewalld numad.conf system-release
flatpak oddjob system-release-cpe
fonts oddjobd.conf tcsd.conf
fprintd.conf oddjobd.conf.d terminfo
fstab openldap tmpfiles.d
fuse.conf opt trusted-key.key
gconf os-release tuned
gcrypt PackageKit udev
gdbinit pam.d udisks2
gdbinit.d passwd updatedb.conf
gdm passwd- UPower
geoclue pbm2ppa.conf usb_modeswitch.conf
GeoIP.conf pinforc vconsole.conf
GeoIP.conf.default pkcs11 vimrc
ghostscript pki virc
gnupg plymouth vmware-tools
GREP_COLORS pm wgetrc
groff pnm2ppa.conf wpa_supplicant
group polkit-1 wvdial.conf
group- popt.d X11
grub2.cfg postfix xdg
grub.d ppp xinetd.d
gshadow prelink.conf.d xml
gshadow- printcap yum
gss profile yum.conf
gssproxy profile.d yum.repos.d
host.conf protocols
hostname pulse
[root@hadoop001 etc]# cd sysconfig/
[root@hadoop001 sysconfig]# ls
anaconda ip6tables network run-parts
atd ip6tables-config network-scripts samba
authconfig iptables nfs saslauthd
cbq iptables-config ntpdate selinux
chronyd iptables.save qemu-ga smartmontools
console irqbalance radvd sshd
cpupower kdump raid-check sysstat
crond kernel rdisc sysstat.ioconf
ebtables-config ksm readonly-root virtlockd
fcoe libvirtd rpcbind virtlogd
firewalld man-db rpc-rquotad wpa_supplicant
grub modules rsyncd
init netconsole rsyslog
[root@hadoop001 sysconfig]# cd network-scripts/
[root@hadoop001 network-scripts]# ls
ifcfg-ens33 ifdown-post ifup-eth ifup-sit
ifcfg-ens33~ ifdown-ppp ifup-ib ifup-Team
ifcfg-lo ifdown-routes ifup-ippp ifup-TeamPort
ifdown ifdown-sit ifup-ipv6 ifup-tunnel
ifdown-bnep ifdown-Team ifup-isdn ifup-wireless
ifdown-eth ifdown-TeamPort ifup-plip init.ipv6-global
ifdown-ib ifdown-tunnel ifup-plusb network-functions
ifdown-ippp ifup ifup-post network-functions-ipv6
ifdown-ipv6 ifup-aliases ifup-ppp
ifdown-isdn ifup-bnep ifup-routes
#修改ifcfg-ens33配置文件
[root@hadoop001 network-scripts]# vi ifcfg-ens33
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static #修改
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=1167f30e-5f62-4002-9627-43659abebb33
DEVICE=ens33
ONBOOT=yes #修改
IPADDR=192.168.1.100 #配置ip
MACADDR=00:0C:29:80:73:BE
NETMASK=255.255.255.0 #配置mac,虚拟机-->设置-->网络适配器NAT-->高级-->MAC
GATEWAY=192.168.1.2 #容易错误配置，编辑-->虚拟网咯编辑器-->VMnet8-->NAT设置-->网关
DNS1=192.168.1.2 #与网关一样就可以
DNS2=8.8.8.8
IPV6_PRIVACY=no
#重启网络服务
[root@hadoop001 network-scripts]# service network restart
Restarting network (via systemctl): [ 确定 ]
[root@hadoop001 network-scripts]#
#查看ip地址
[root@hadoop001 network-scripts]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:80:73:be brd ff:ff:ff:ff:ff:ff
inet 192.168.1.100/24 brd 192.168.1.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::467a:1602:431:275/64 scope link
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:80:d1:02 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:80:d1:02 brd ff:ff:ff:ff:ff:ff
[root@hadoop001 network-scripts]#
六、主机名设置
1.编辑网络配置文件
[root@hadoop001 network-scripts]# vim /etc/sysconfig/network
# Created by anaconda
NETWORKING=yes
HOSTNAME=hadoop001
2.永久设置主机名
[root@hadoop001 network-scripts]# hostnamectl set-hostname hadoop001
3.重启
[root@hadoop001 network-scripts]# hostnamectl set-hostname hadoop001
4.查看主机名是否有效
[root@hadoop001 ~]# hostname
hadoop001
另外几台分别同样操作设置为hadoop002,hadoop003
七、hosts文件配置
配置IP和hostname的映射关系
[root@hadoop001 ~]# vim /etc/hosts
#增加
192.168.1.100 hadoop001
192.168.1.101 hadoop002
192.168.1.102 hadoop003
其他台配置一样
八、安装本机ssh免密登陆
1.关闭防火墙(永久)和SELinux（root权限）
[root@hadoop001 ~]# vim /etc/selinux/config
修改 /etc/selinux/config 文件中的 SELINUX=enforcing 修改为 SELINUX=disabled ，然后重启。
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
#SELINUX=enforcing
SELINUX=disabled #修改为disabled
# SELINUXTYPE= can take one of three two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
~
~
~
~
"/etc/selinux/config" 15L, 565C
2.ssh免密登录
[root@hadoop001 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:F0VeLRWK+eOPDIccQgNhUfKWbo1tYrp28j9NHzw3HYk root@hadoop001
The key's randomart image is:
+---[RSA 2048]----+
| *+. .o .+o|
| . + .oo.o .|
| *.o.....|
| + =..E o |
| SB.= o. o|
| +.= +..=o|
| . +oo. =|
| o.. .+.o. |
| ..+....o . |
+----[SHA256]-----+
[root@hadoop001 ~]#
3.向本机复制公钥
[root@hadoop001 ~]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
4.登录hadoop001验证(第一次输入yes回车进入就表示免密码登录配置成功)
[root@hadoop001 ~]# ssh hadoop001
The authenticity of host 'hadoop001 (192.168.1.100)' can't be established.
ECDSA key fingerprint is SHA256:RcMX0Yg6dBCZlIAM3cS0Rh/fSwNjJSCr2TNb3GPup4I.
ECDSA key fingerprint is MD5:d5:1a:ac:1f:33:c0:30:3f:95:68:41:a5:33:17:20:4f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoop001,192.168.1.100' (ECDSA) to the list of known hosts.
Last login: Sun Mar 28 14:40:59 2021
[root@hadoop001 ~]#
5.把公钥复制到另外两台虚拟机(途中输入密码)
[root@hadoop001 ~]# ssh-copy-id -i hadoop002
[root@hadoop001 ~]# ssh-copy-id -i hadoop003
6.验证在hadoop001上面使用ssh命令确认集群免密码登录是否成功，可以免密码登录说明是没有问题
[root@hadoop001 ~]# ssh hadoop002
Last login: Sun Mar 28 14:45:37 2021
[root@hadoop002 ~]# exit
登出
Connection to hadoop002 closed.
[root@hadoop001 ~]# ssh hadoop003
Last login: Sun Mar 28 14:48:32 2021
[root@hadoop003 ~]# exit
登出
Connection to hadoop003 closed.
[root@hadoop001 ~]#
[root@hadoop001 ~]# ssh-copy-id -i hadoop002
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop002 (192.168.1.101)' can't be established.
ECDSA key fingerprint is SHA256:RcMX0Yg6dBCZlIAM3cS0Rh/fSwNjJSCr2TNb3GPup4I.
ECDSA key fingerprint is MD5:d5:1a:ac:1f:33:c0:30:3f:95:68:41:a5:33:17:20:4f.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop002's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'hadoop002'"
and check to make sure that only the key(s) you wanted were added.
[root@hadoop001 ~]# ssh-copy-id -i hadoop002
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
(if you think this is a mistake, you may want to use -f option)
[root@hadoop001 ~]# ssh-copy-id -i hadoop003
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop003 (192.168.1.102)' can't be established.
ECDSA key fingerprint is SHA256:RcMX0Yg6dBCZlIAM3cS0Rh/fSwNjJSCr2TNb3GPup4I.
ECDSA key fingerprint is MD5:d5:1a:ac:1f:33:c0:30:3f:95:68:41:a5:33:17:20:4f.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop003's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'hadoop003'"
and check to make sure that only the key(s) you wanted were added.
[root@hadoop001 ~]# ssh-copy-id -i hadoop003
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
(if you think this is a mistake, you may want to use -f option)
[root@hadoop001 ~]# ssh hadoop002
Last login: Sun Mar 28 14:45:37 2021
[root@hadoop002 ~]# exit
登出
Connection to hadoop002 closed.
[root@hadoop001 ~]# ssh hadoop003
Last login: Sun Mar 28 14:48:32 2021
[root@hadoop003 ~]# exit
登出
Connection to hadoop003 closed.
[root@hadoop001 ~]#
或者(差不多)
2.配置sshd（root权限）
[hadoop@localhost root]$ sudo vi /etc/ssh/sshd_config
vim进去修改配置
43 PubkeyAuthentication yes
47 AuthorizedKeysFile .ssh/authorized_keys
添加部分
140 RSAAuthentication yes
重启 sshd 服务，使用命令：systemctl restart sshd.service 。
[hadoop@localhost root]$ systemctl restart sshd.service
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: hadoop
Password:
==== AUTHENTICATION COMPLETE ===
[hadoop@localhost root]$
3.生成秘钥（普通用户权限）
从 root 用户切换到hadoop 用户，可使用命令：su hadoop。然后， 执行命令 ssh-keygen -t rsa 来生成秘钥。无需指定口令密码，遇到询问直接回车，命令执行完毕后会在 hadoop 用户的家目录中（/home/hadoop/.ssh）生成两个文件：
id_rsa 私钥
id_rsa.pub 公钥
[hadoop@localhost root]$ su hadoop
密码：
[hadoop@localhost root]$ ssh-keygen -trsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:3E8juUADf2abvIQypfJSeaecamsXbe7ddtzFXAkFBZw hadoop@localhost.localdomain
The key's randomart image is:
+---[RSA 2048]----+
| . .o*o |
| o E |
| = + . .|
| * O + ..|
| . * S.X o o.|
| + =.*o* . +|
| . . ++o . .o|
| .o.. .. .. +|
| ooo .. .... |
+----[SHA256]-----+
[hadoop@localhost root]$
[hadoop@localhost root]$ vi /home/hadoop/.ssh
[hadoop@localhost root]$ cd /home/hadoop/.ssh
[hadoop@localhost .ssh]$ ls
id_rsa id_rsa.pub
[hadoop@localhost .ssh]$ cd id_rsa
bash: cd: id_rsa: 不是目录
[hadoop@localhost .ssh]$ cat id_rsa
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA0YJvPvHbZPVjORoqfi2yZy4kBDuK8NWZDA4XXKaAq6K6aQD9
nWXn54xYNpLIp402FOheOcWcYXoZAaNwZyif7JkNlOE3F+z6SAX5AUnO0FbmqU9i
A63elh9l074CRB/kFR5FyS7wVbMMf9pBzPH0YwvkCVHnIIJv43LojLZqZkkm18LK
vw0Jexo5mcPilLrO5GFjWwk6r9JncBVBOIvinffhnUEh8yw84iMaHTyCF7eQtyyV
YQ+sbt7dzx2kDIPDHtaqCs3KzTTshPONrXOwnV8uy9qrIXlSdJZ+uDKF3iwBjua/
krv7uFOIXL/oSybUI3i+dLGjsU9cC2KyTTZM+wIDAQABAoIBAE8IFxL1hT92WbGm
rzTd5BiMDUYTd/wVdyBXCHUI0izsu8j0eLcxQ+PSy0v63vpliGsmpLTCWJVky54U
m0guyPUlXzw0IOZDnL4ikqXrw7pPrb9clKqyoe6bdXwEhzJPnWhh/Q1BSaPqYOKx
4HSBKSFb0O+7F6bpzW9NX3AFN+nRK1gnGPA0iGOGNUyP/0UVDoMfctW5VXhD1iu4
RptvpLUh1nPwOzhZRBrJnbPyE94bJH18KJPlgw01phG1TwnlqqM6u2tFbc6ThtgH
wEgdh5f7HDGmej9Aza48jRCUmiY3r77MUOaTrivohi3cjOBQz2BCJ6+uGlEZ9jN+
hyhloVECgYEA98aZH0k2egS8edxK9cqYjNJD3267QQIQvLo+rjsf+7QhVFxwO2ge
KOYg0V9YUIbpttRaE2f+X6xmTcH2ybL18wK60y9AEQiGHHN1XfaQiNFvsMnn/eFY
U3RDTCGlfc0e7tpQboHxfJrKwiB6iCpPj4YqdwthfZ0USWR+34xp4zcCgYEA2Hau
Gc/fC6QGtz2tKHcqPFpVH3dKJvnvqO8OXUU4tGSaZISyga2S8115d4KZniXYfXMQ
FCqVwMq/cNj9LRMxmuqeHv3XUaMJGE95AW2jD2JeKo7VSjd2CysMRaaK7qWGX6jt
OF6ijzDZiyppQkFME8oLfSK2v2KIk/j/q/6cTl0CgYBpSuIDI4+c5qpZdr38GW4e
WbQyHNJCW+hU6yh7zfBXfEK1oNqoxCQc6T6E+umCvvJOmYr1uDmm2pJW9Ng0+nH+
JOjTmb61/lNPf9keZwsguS+nhwWpI7vvKvb2QU4cWbCNfAS2EU5Xz0femwK3HpPU
wAUHtbRmNvxJ/ATWZssQnQKBgQDR5pOAmB/TK+UPPxFwEc205GtyrbwL+4S4Lcei
DgOkeYF1Q2/Na8D5mIS2rL/FqTE6xJ6sz3aTkob9KIyobtpFPIjDyKZIlW22Uyol
hmj9/AcQAZ018H3Y6o9l4s8KBxw8GpSderbrXxU0a5XSF3tsHRny5/yJrUR/KI7T
+3saSQKBgQCybQycXkJNhBQQxpXGA/moo0dxNdKxdp0YFnkRbDC7F3gYqH9/v3J8
A9BfmBImJjHA1ihZm2JAO03rp2sB3TYQwIlkRejCUNp7V4FA/QcR+RU8D2guB1rO
YSQQRb227YmP0W5jkZ//M0ysEjI6RZa+CXP2gAW5IlxUHUyKwqom5g==
-----END RSA PRIVATE KEY-----
[hadoop@localhost .ssh]$ cd ..
[hadoop@localhost ~]$ cd ..
[hadoop@localhost home]$
4.将公钥导入到认证文件（普通用户权限）
[hadoop@localhost home]$ cat /home/hadoop/.ssh/id_rsa.pub >> /home/hadoop/.ssh/authorized_keys
[hadoop@localhost home]$
5.设置认证文件访问权限（普通用户权限）
[hadoop@localhost home]$ chmod 700 /home/hadoop/.ssh
[hadoop@localhost home]$ chmod 600 /home/hadoop/.ssh/authorized_keys
[hadoop@localhost home]$
九、安装hadoop3.3.0
1、解压hadoop
[root@localhost hadoop330]# cd /home
[root@localhost home]# ls
hadoop sf zkycentos7
[root@localhost home]# cd hadoop/
[root@localhost hadoop]# ls
hadoop-3.3.0.tar.gz java jdk-11.0.10_linux-x64_bin.rpm 公共 模板 视频 图片 文档 下载 音乐 桌面
[root@localhost hadoop]# sudo cp hadoop-3.3.0.tar.gz /usr/hadoop330/
[root@localhost hadoop]#
[root@localhost hadoop]# cd /usr/hadoop330/
[root@localhost hadoop]# tar -zxvf hadoop-2.9.2.tar.gz
[root@localhost hadoop330]# ls
hadoop-3.3.0 hadoop-3.3.0.tar.gz
[root@localhost hadoop330]#
配置hadoop-env.sh文件，增加环境变量信息
[root@hadoop001 hadoop]# cd /usr/hadoop330/hadoop-3.3.0/etc/hadoop/
[root@hadoop001 hadoop]# ls
capacity-scheduler.xml kms-site.xml
configuration.xsl log4j.properties
container-executor.cfg mapred-env.cmd
core-site.xml mapred-env.sh
hadoop-env.cmd mapred-queues.xml.template
hadoop-env.sh mapred-site.xml
hadoop-metrics2.properties shellprofile.d
hadoop-policy.xml ssl-client.xml.example
hadoop-user-functions.sh.example ssl-server.xml.example
hdfs-rbf-site.xml user_ec_policies.xml.template
hdfs-site.xml vim
httpfs-env.sh workers
httpfs-log4j.properties yarn-env.cmd
httpfs-site.xml yarn-env.sh
kms-acls.xml yarnservice-log4j.properties
kms-env.sh yarn-site.xml
kms-log4j.properties
[root@hadoop001 hadoop]# vim hadoop-env.sh
#添加如下内容
export JAVA_HOME=/usr/java/jdk-11.0.10
export HADOOP_LOG_DIR=/usr/hadoop330/hadoop-3.3.0/tmp/logs/hadoop
2、配置hadoop环境变量
（1）vi /etc/profile 添加如下配置
#hadoop3.3.0
export HADOOP_HOME=/usr/hadoop330/hadoop-3.3.0
export PATH=$HADOOP_HOME/bin:$PATH
（2）然后刷新配置：source /etc/profile
（3）完成后命令：hadoop version，出现版本号则成功。
[root@localhost hadoop330]# hadoop version
Hadoop 3.3.0
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r aa96f1871bfd858f9bac59cf2a81ec470da649af
Compiled by brahma on 2020-07-06T18:44Z
Compiled with protoc 3.7.1
From source with checksum 5dc29b802d6ccd77b262ef9d04d19c4
This command was run using /usr/hadoop330/hadoop-3.3.0/share/hadoop/common/hadoop-common-3.3.0.jar
[root@localhost hadoop330]#
十、修改hadoop相关配置文件
1.然后切换到配置文件目录
[root@hadoop001 hadoop]# cd /usr/hadoop330/hadoop-3.3.0/etc/hadoop
[root@hadoop001 hadoop]# ls
capacity-scheduler.xml kms-log4j.properties
configuration.xsl kms-site.xml
container-executor.cfg log4j.properties
core-site.xml mapred-env.cmd
hadoop-env.cmd mapred-env.sh
hadoop-env.sh mapred-queues.xml.template
hadoop-metrics2.properties mapred-site.xml
hadoop-policy.xml shellprofile.d
hadoop-user-functions.sh.example ssl-client.xml.example
hdfs-rbf-site.xml ssl-server.xml.example
hdfs-site.xml user_ec_policies.xml.template
httpfs-env.sh workers
httpfs-log4j.properties yarn-env.cmd
httpfs-site.xml yarn-env.sh
kms-acls.xml yarnservice-log4j.properties
kms-env.sh yarn-site.xml
[root@hadoop001 hadoop]#
2.配置core-site.xml文件 ，添加如下配置
[root@hadoop001 hadoop]# vim core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<!-- 这里填的是你自己的ip，端口默认-->
<!-- 这样貌似也可以<value>hdfs://192.168.187.128:9000</value> -->
<value>hdfs://hadoop001:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<!-- 这里填的是你自定义的hadoop工作的目录，端口默认-->
<value>/usr/hadoop330/hadoop-3.3.0/tmp</value>
</property>
</configuration>
3.配置hdfs-site.xml文件,把hdfs中文件副本的数量设置为2
[root@hadoop001 hadoop]# vim hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop001:50090</value>
</property>
</configuration>
4.配置mapred-site.xml文件,设置mapreduce使用资源调度的框架
[root@hadoop001 hadoop]# vim mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
5.配置yarn-site.xml文件,设置yarn上支持运行的服务和环境变量白名单
[root@hadoop001 hadoop]# vim yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CL
ASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop001</value>
</property>
</configuration>
6.修改workers文件，增加所有从节点的主机名，一个一行
[root@hadoop001 hadoop]# vim workers
#删除localhost，添加下面内容
hadoop002
hadoop003
7.修改启动脚本
[root@hadoop001 hadoop]# cd /usr/hadoop330/hadoop-3.3.0/sbin
[root@hadoop001 sbin]# ls
distribute-exclude.sh start-all.sh stop-balancer.sh
FederationStateStore start-balancer.sh stop-dfs.cmd
hadoop-daemon.sh start-dfs.cmd stop-dfs.sh
hadoop-daemons.sh start-dfs.sh stop-secure-dns.sh
httpfs.sh start-secure-dns.sh stop-yarn.cmd
kms.sh start-yarn.cmd stop-yarn.sh
mr-jobhistory-daemon.sh start-yarn.sh workers.sh
refresh-namenodes.sh stop-all.cmd yarn-daemon.sh
start-all.cmd stop-all.sh yarn-daemons.sh
[root@hadoop001 sbin]#
8.在start-dfs.sh和stop-dfs.sh增加下面内容
[root@hadoop001 sbin]# vim start-dfs.sh
[root@hadoop001 sbin]# vim stop-dfs.sh
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
9.在start-yarn.sh和stop-yarn.sh增加下面内容
[root@hadoop001 sbin]# vim start-yarn.sh
[root@hadoop001 sbin]# vim stop-yarn.sh
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
10.格式化namenode
[root@hadoop001 sbin]# cd /usr/hadoop330/hadoop-3.3.0/
[root@hadoop001 hadoop-3.3.0]# ls
bin include LICENSE-binary NOTICE-binary sbin
core-site.xml lib licenses-binary NOTICE.txt share
etc libexec LICENSE.txt README.txt
[root@hadoop001 hadoop-3.3.0]# bin/hdfs namenode -format
#或者直接
[root@hadoop001 ~]# cd $HADOOP_HOME
[root@hadoop001 hadoop-3.3.0]# bin/hdfs namenode -format
WARNING: /usr/hadoop330/hadoop-3.3.0/logs does not exist. Creating.
2021-03-28 15:58:37,512 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = hadoop001/192.168.1.100
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 3.3.0
STARTUP_MSG: classpath = /usr/hadoop330/hadoop-3.3.0/etc/hadoop:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/zookeeper-jute-3.5.6.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/avro-1.7.7.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/javax.activation-api-1.2.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/hadoop-auth-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jetty-webapp-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jersey-servlet-1.19.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jetty-server-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/accessors-smart-1.2.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerb-server-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jackson-annotations-2.10.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/gson-2.2.4.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jettison-1.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/commons-beanutils-1.9.4.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/httpclient-4.5.6.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/hadoop-annotations-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/asm-5.0.4.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jetty-util-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerb-core-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerby-util-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jetty-security-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jsch-0.1.55.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/commons-compress-1.19.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/curator-framework-4.2.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jersey-core-1.19.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/commons-net-3.6.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/metrics-core-3.2.4.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jersey-json-1.19.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/dnsjava-2.1.7.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/curator-recipes-4.2.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jetty-io-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/guava-27.0-jre.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/paranamer-2.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/curator-client-4.2.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/commons-codec-1.11.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerb-client-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jetty-xml-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/stax2-api-3.1.4.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/token-provider-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.0.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/httpcore-4.4.10.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerb-util-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/commons-lang3-3.7.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/commons-daemon-1.0.13.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jackson-databind-2.10.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/checker-qual-2.5.2.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerb-common-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jetty-http-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jersey-server-1.19.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/json-smart-2.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jackson-core-2.10.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/commons-text-1.4.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerby-config-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/netty-3.10.6.Final.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jsr305-3.0.2.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/commons-io-2.5.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/re2j-1.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/zookeeper-3.5.6.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/failureaccess-1.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/nimbus-jose-jwt-7.9.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/snappy-java-1.0.5.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/jetty-servlet-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/hadoop-registry-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/hadoop-nfs-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/hadoop-common-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/hadoop-kms-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/common/hadoop-common-3.3.0-tests.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/zookeeper-jute-3.5.6.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/avro-1.7.7.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/javax.activation-api-1.2.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/hadoop-auth-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-webapp-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-server-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/accessors-smart-1.2.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-annotations-2.10.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/gson-2.2.4.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jettison-1.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-beanutils-1.9.4.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/httpclient-4.5.6.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/hadoop-annotations-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/asm-5.0.4.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-util-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-util-ajax-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-security-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jsch-0.1.55.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-compress-1.19.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/curator-framework-4.2.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-net-3.6.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/dnsjava-2.1.7.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/curator-recipes-4.2.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-io-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/paranamer-2.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/curator-client-4.2.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/netty-all-4.1.50.Final.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-xml-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/stax2-api-3.1.4.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.0.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/httpcore-4.4.10.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/woodstox-core-5.0.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-lang3-3.7.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/okio-1.6.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-databind-2.10.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-http-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/json-smart-2.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jackson-core-2.10.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-text-1.4.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/netty-3.10.6.Final.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/commons-io-2.5.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/re2j-1.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/zookeeper-3.5.6.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/nimbus-jose-jwt-7.9.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/jetty-servlet-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-nfs-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-client-3.3.0-tests.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.0-tests.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-3.3.0-tests.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.0-tests.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-client-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/hdfs/hadoop-hdfs-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.0-tests.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/bcprov-jdk15on-1.60.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.10.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/asm-tree-7.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/asm-analysis-7.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/jetty-jndi-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/jetty-plus-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-server-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/guice-4.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/jna-5.2.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/asm-commons-7.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/jline-3.9.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/jackson-jaxrs-base-2.10.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/snakeyaml-1.16.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/jersey-client-1.19.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/javax.websocket-api-1.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-servlet-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-common-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/jakarta.xml.bind-api-2.3.2.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/jetty-client-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/json-io-2.5.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/java-util-1.9.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/jakarta.activation-api-1.2.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/javax-websocket-server-impl-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-api-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/bcpkix-jdk15on-1.60.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.10.3.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/javax-websocket-client-impl-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/websocket-client-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/objenesis-2.6.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/fst-2.50.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/jetty-annotations-9.4.20.v20190813.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/lib/javax.websocket-client-api-1.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-tests-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-services-api-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-api-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-client-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-services-core-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-applications-mawo-core-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-common-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-common-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-registry-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-router-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.3.0.jar:/usr/hadoop330/hadoop-3.3.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.3.0.jar
STARTUP_MSG: build = https://gitbox.apache.org/repos/asf/hadoop.git -r aa96f1871bfd858f9bac59cf2a81ec470da649af; compiled by 'brahma' on 2020-07-06T18:44Z
STARTUP_MSG: java = 11.0.10
************************************************************/
2021-03-28 15:58:37,525 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2021-03-28 15:58:37,654 INFO namenode.NameNode: createNameNode [-format]
2021-03-28 15:58:38,366 INFO namenode.NameNode: Formatting using clusterid: CID-40be2aef-0936-4b00-8bcc-083c49a6f93d
2021-03-28 15:58:38,456 INFO namenode.FSEditLog: Edit logging is async:true
2021-03-28 15:58:38,493 INFO namenode.FSNamesystem: KeyProvider: null
2021-03-28 15:58:38,495 INFO namenode.FSNamesystem: fsLock is fair: true
2021-03-28 15:58:38,495 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2021-03-28 15:58:38,552 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE)
2021-03-28 15:58:38,553 INFO namenode.FSNamesystem: supergroup = supergroup
2021-03-28 15:58:38,553 INFO namenode.FSNamesystem: isPermissionEnabled = true
2021-03-28 15:58:38,553 INFO namenode.FSNamesystem: isStoragePolicyEnabled = true
2021-03-28 15:58:38,553 INFO namenode.FSNamesystem: HA Enabled: false
2021-03-28 15:58:38,630 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2021-03-28 15:58:38,639 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2021-03-28 15:58:38,639 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2021-03-28 15:58:38,641 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2021-03-28 15:58:38,642 INFO blockmanagement.BlockManager: The block deletion will start around 2021 3月 28 15:58:38
2021-03-28 15:58:38,643 INFO util.GSet: Computing capacity for map BlocksMap
2021-03-28 15:58:38,643 INFO util.GSet: VM type = 64-bit
2021-03-28 15:58:38,645 INFO util.GSet: 2.0% max memory 498 MB = 10.0 MB
2021-03-28 15:58:38,645 INFO util.GSet: capacity = 2^20 = 1048576 entries
2021-03-28 15:58:38,651 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled
2021-03-28 15:58:38,651 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2021-03-28 15:58:38,656 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.999
2021-03-28 15:58:38,657 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2021-03-28 15:58:38,657 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2021-03-28 15:58:38,657 INFO blockmanagement.BlockManager: defaultReplication = 2
2021-03-28 15:58:38,657 INFO blockmanagement.BlockManager: maxReplication = 512
2021-03-28 15:58:38,657 INFO blockmanagement.BlockManager: minReplication = 1
2021-03-28 15:58:38,657 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
2021-03-28 15:58:38,657 INFO blockmanagement.BlockManager: redundancyRecheckInterval = 3000ms
2021-03-28 15:58:38,657 INFO blockmanagement.BlockManager: encryptDataTransfer = false
2021-03-28 15:58:38,657 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
2021-03-28 15:58:38,677 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911
2021-03-28 15:58:38,677 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215
2021-03-28 15:58:38,677 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215
2021-03-28 15:58:38,677 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215
2021-03-28 15:58:38,694 INFO util.GSet: Computing capacity for map INodeMap
2021-03-28 15:58:38,694 INFO util.GSet: VM type = 64-bit
2021-03-28 15:58:38,695 INFO util.GSet: 1.0% max memory 498 MB = 5.0 MB
2021-03-28 15:58:38,695 INFO util.GSet: capacity = 2^19 = 524288 entries
2021-03-28 15:58:38,695 INFO namenode.FSDirectory: ACLs enabled? true
2021-03-28 15:58:38,695 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2021-03-28 15:58:38,695 INFO namenode.FSDirectory: XAttrs enabled? true
2021-03-28 15:58:38,695 INFO namenode.NameNode: Caching file names occurring more than 10 times
2021-03-28 15:58:38,701 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2021-03-28 15:58:38,703 INFO snapshot.SnapshotManager: SkipList is disabled
2021-03-28 15:58:38,707 INFO util.GSet: Computing capacity for map cachedBlocks
2021-03-28 15:58:38,707 INFO util.GSet: VM type = 64-bit
2021-03-28 15:58:38,707 INFO util.GSet: 0.25% max memory 498 MB = 1.2 MB
2021-03-28 15:58:38,707 INFO util.GSet: capacity = 2^17 = 131072 entries
2021-03-28 15:58:38,716 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2021-03-28 15:58:38,716 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2021-03-28 15:58:38,716 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2021-03-28 15:58:38,718 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2021-03-28 15:58:38,718 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2021-03-28 15:58:38,720 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2021-03-28 15:58:38,720 INFO util.GSet: VM type = 64-bit
2021-03-28 15:58:38,720 INFO util.GSet: 0.029999999329447746% max memory 498 MB = 153.0 KB
2021-03-28 15:58:38,720 INFO util.GSet: capacity = 2^14 = 16384 entries
2021-03-28 15:58:38,747 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1630466842-192.168.1.100-1616918318738
#successful成功啦
2021-03-28 15:58:38,769 INFO common.Storage: Storage directory /usr/hadoop330/hadoop-3.3.0/tmp/dfs/name has been successfully formatted.
2021-03-28 15:58:38,817 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/hadoop330/hadoop-3.3.0/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2021-03-28 15:58:38,946 INFO namenode.FSImageFormatProtobuf: Image file /usr/hadoop330/hadoop-3.3.0/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 396 bytes saved in 0 seconds .
2021-03-28 15:58:38,967 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2021-03-28 15:58:38,973 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2021-03-28 15:58:38,975 INFO namenode.NameNode: SHUTDOWN_MSG:
#该部分为成功
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop001/192.168.1.100
************************************************************/
[root@hadoop001 hadoop-3.3.0]#
注意：如果多次格式化，需要先删除虚拟机的/usr/hadoop330/hadoop-3.3.0/tmp目录（core-site.xml文件中设置的目录），再格式化才能成功
11.启动集群(出问题了java环境前面没有加到hadoop目录cd /usr/hadoop330/hadoop-3.3.0/etc/hadoop/的 hadoop-env.sh，具体见–九、安装hadoop3.3.3的–>2.配置hadoop环境变量)
#重新复制java环境
[root@hadoop001 hadoop-3.3.0]# scp -rq /usr/java/jdk-11.0.10 hadoop002:/usr/java/
[root@hadoop001 hadoop-3.3.0]# scp -rq /etc/profile hadoop002:/etc/
[root@hadoop001 hadoop-3.3.0]# scp -rq /usr/java/jdk-11.0.10 hadoop003:/usr/java/
[root@hadoop001 hadoop-3.3.0]# scp -rq /etc/profile hadoop003:/etc/
后面复制按照前面步骤格式化namenode
scp -rq /usr/hadoop330/hadoop-3.3.0 hadoop002:/usr/hadoop330/
scp -rq /usr/hadoop330/hadoop-3.3.0 hadoop003:/usr/hadoop330/
#查看了下cp过去的文件。有点无聊
[root@hadoop002 ~]# cd /usr/hadoop330/hadoop-3.3.0/etc/hadoop
[root@hadoop002 hadoop]# ls
capacity-scheduler.xml kms-log4j.properties
configuration.xsl kms-site.xml
container-executor.cfg log4j.properties
core-site.xml mapred-env.cmd
hadoop-env.cmd mapred-env.sh
hadoop-env.sh mapred-queues.xml.template
hadoop-metrics2.properties mapred-site.xml
hadoop-policy.xml shellprofile.d
hadoop-user-functions.sh.example ssl-client.xml.example
hdfs-rbf-site.xml ssl-server.xml.example
hdfs-site.xml user_ec_policies.xml.template
httpfs-env.sh workers
httpfs-log4j.properties yarn-env.cmd
httpfs-site.xml yarn-env.sh
kms-acls.xml yarnservice-log4j.properties
kms-env.sh yarn-site.xml
[root@hadoop002 hadoop]# vim core-site.xml
[root@hadoop002 hadoop]# vim hdfs-site.xml
[root@hadoop002 hadoop]# vim mapred-site.xml
[root@hadoop002 hadoop]# vim yarn-site.xml
十一、验证
1、启动，jps
[root@hadoop001 ~]# $HADOOP_HOME
bash: /usr/hadoop330/hadoop-3.3.0: 是一个目录
[root@hadoop001 ~]# cd $HADOOP_HOME
[root@hadoop001 hadoop-3.3.0]# sbin/start-all.sh
Starting namenodes on [hadoop001]
上一次登录：日 3月 28 19:22:59 CST 2021pts/1 上
hadoop001: namenode is running as process 27966. Stop it first.
Starting datanodes
上一次登录：日 3月 28 19:32:59 CST 2021pts/0 上
hadoop002: datanode is running as process 18915. Stop it first.
hadoop003: datanode is running as process 18581. Stop it first.
Starting secondary namenodes [hadoop001]
上一次登录：日 3月 28 19:33:00 CST 2021pts/0 上
hadoop001: secondarynamenode is running as process 28264. Stop it first.
Starting resourcemanager
上一次登录：日 3月 28 19:33:03 CST 2021pts/0 上
resourcemanager is running as process 28546. Stop it first.
Starting nodemanagers
上一次登录：日 3月 28 19:33:12 CST 2021pts/0 上
hadoop002: nodemanager is running as process 19051. Stop it first.
hadoop003: nodemanager is running as process 18716. Stop it first.
#jps
[root@hadoop001 hadoop-3.3.0]# jps
28546 ResourceManager
28264 SecondaryNameNode
27966 NameNode
30607 Jps
2、WEB UI查看
#linux里面的火狐验证网页是否能打开
http://192.168.1.100:8088/cluster
http://192.168.1.100:9870/dfshealth.html#tab-overview
3、运行PI实例检查集群是否成功
[root@hadoop001 mapreduce]# hadoop jar /usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar pi 10 10
[root@hadoop001 hadoop-3.3.0]# ls
bin hadoop-env.sh libexec LICENSE.txt NOTICE.txt share
core-site.xml include LICENSE-binary logs README.txt tmp
etc lib licenses-binary NOTICE-binary sbin
[root@hadoop001 hadoop-3.3.0]# cd share/
[root@hadoop001 share]# ls
doc hadoop
[root@hadoop001 share]# cd hadoop/
[root@hadoop001 hadoop]# ls
client common hdfs mapreduce tools yarn
[root@hadoop001 hadoop]# cd mapreduce/
[root@hadoop001 mapreduce]# ls
hadoop-mapreduce-client-app-3.3.0.jar
hadoop-mapreduce-client-common-3.3.0.jar
hadoop-mapreduce-client-core-3.3.0.jar
hadoop-mapreduce-client-hs-3.3.0.jar
hadoop-mapreduce-client-hs-plugins-3.3.0.jar
hadoop-mapreduce-client-jobclient-3.3.0.jar
hadoop-mapreduce-client-jobclient-3.3.0-tests.jar
hadoop-mapreduce-client-nativetask-3.3.0.jar
hadoop-mapreduce-client-shuffle-3.3.0.jar
hadoop-mapreduce-client-uploader-3.3.0.jar
hadoop-mapreduce-examples-3.3.0.jar
jdiff
lib-examples
sources
[root@hadoop001 mapreduce]# pwd
/usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce
[root@hadoop001 mapreduce]# hadoop jar /usr/hadoop330/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar pi 10 10
Number of Maps = 10
Samples per Map = 10
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
2021-03-28 19:42:10,007 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at hadoop001/192.168.1.100:8032
2021-03-28 19:42:10,414 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1616930585147_0001
2021-03-28 19:42:10,582 INFO input.FileInputFormat: Total input files to process : 10
2021-03-28 19:42:10,663 INFO mapreduce.JobSubmitter: number of splits:10
2021-03-28 19:42:10,887 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1616930585147_0001
2021-03-28 19:42:10,887 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-03-28 19:42:11,041 INFO conf.Configuration: resource-types.xml not found
2021-03-28 19:42:11,042 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-03-28 19:42:11,442 INFO impl.YarnClientImpl: Submitted application application_1616930585147_0001
2021-03-28 19:42:11,472 INFO mapreduce.Job: The url to track the job: http://hadoop001:8088/proxy/application_1616930585147_0001/
2021-03-28 19:42:11,472 INFO mapreduce.Job: Running job: job_1616930585147_0001
2021-03-28 19:42:21,650 INFO mapreduce.Job: Job job_1616930585147_0001 running in uber mode : false
2021-03-28 19:42:21,653 INFO mapreduce.Job: map 0% reduce 0%
2021-03-28 19:42:39,321 INFO mapreduce.Job: map 20% reduce 0%
2021-03-28 19:43:01,602 INFO mapreduce.Job: map 40% reduce 7%
2021-03-28 19:43:04,633 INFO mapreduce.Job: map 70% reduce 7%
2021-03-28 19:43:05,682 INFO mapreduce.Job: map 80% reduce 7%
2021-03-28 19:43:06,690 INFO mapreduce.Job: map 100% reduce 7%
2021-03-28 19:43:07,700 INFO mapreduce.Job: map 100% reduce 100%
2021-03-28 19:43:08,721 INFO mapreduce.Job: Job job_1616930585147_0001 completed successfully
2021-03-28 19:43:08,858 INFO mapreduce.Job: Counters: 54
File System Counters
FILE: Number of bytes read=226
FILE: Number of bytes written=2904869
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2630
HDFS: Number of bytes written=215
HDFS: Number of read operations=45
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=10
Launched reduce tasks=1
Data-local map tasks=10
Total time spent by all maps in occupied slots (ms)=347743
Total time spent by all reduces in occupied slots (ms)=26244
Total time spent by all map tasks (ms)=347743
Total time spent by all reduce tasks (ms)=26244
Total vcore-milliseconds taken by all map tasks=347743
Total vcore-milliseconds taken by all reduce tasks=26244
Total megabyte-milliseconds taken by all map tasks=356088832
Total megabyte-milliseconds taken by all reduce tasks=26873856
Map-Reduce Framework
Map input records=10
Map output records=20
Map output bytes=180
Map output materialized bytes=280
Input split bytes=1450
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=280
Reduce input records=20
Reduce output records=0
Spilled Records=40
Shuffled Maps =10
Failed Shuffles=0
Merged Map outputs=10
GC time elapsed (ms)=3557
CPU time spent (ms)=24070
Physical memory (bytes) snapshot=2350845952
Virtual memory (bytes) snapshot=31071526912
Total committed heap usage (bytes)=1931476992
Peak Map Physical memory (bytes)=257482752
Peak Map Virtual memory (bytes)=2827128832
Peak Reduce Physical memory (bytes)=166195200
Peak Reduce Virtual memory (bytes)=2830069760
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1180
File Output Format Counters
Bytes Written=97
Job Finished in 59.022 seconds
Estimated value of Pi is 3.20000000000000000000
[root@hadoop001 mapreduce]#
第一次接触，如果有错误欢迎批评指正~~~~
扫一扫
1196
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
