Hadoop-3.3.0安装 Centos 8.2安装Hadoop-3.3.0 Hadoop-3.3.0安装指南

Centos 8.2 安装Hadoop 3.3.0

[root@iZuf61u64hvs1nquuaipy8Z /]# clear

[root@iZuf61u64hvs1nquuaipy8Z i-data/]# git clone https://github.com/grondo/pdsh.git
[root@iZuf61u64hvs1nquuaipy8Z i-data]# cd pdsh
[root@iZuf61u64hvs1nquuaipy8Z pdsh]# ./bootstrap
[root@iZuf61u64hvs1nquuaipy8Z pdsh]# ./configure
[root@iZuf61u64hvs1nquuaipy8Z pdsh]# make
[root@iZuf61u64hvs1nquuaipy8Z pdsh]# make install
[root@iZuf61u64hvs1nquuaipy8Z pdsh]$ pdsh -V
pdsh-2.32-28-g00f66b2
rcmd modules: rsh,exec (default: rsh)
misc modules: (none)
[root@iZuf61u64hvs1nquuaipy8Z /]# su hadoop
su: 用户 hadoop 不存在
[root@iZuf61u64hvs1nquuaipy8Z /]# sudo useradd -m hadoop -s /bin/bash
[root@iZuf61u64hvs1nquuaipy8Z /]# sudo passwd hadoop
更改用户 hadoop 的密码 。
新的 密码：
重新输入新的 密码：
passwd：所有的身份验证令牌已经成功更新。

[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ sudo yum update

我们信任您已经从系统管理员那里了解了日常注意事项。
总结起来无外乎这三点：

#1) 尊重别人的隐私。
#2) 输入前要先考虑(后果和风险)。
#3) 权力越大，责任越大。

[sudo] hadoop 的密码：
hadoop 不在 sudoers 文件中。此事将被报告。
[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ su root
密码：

[root@iZuf61u64hvs1nquuaipy8Z hadoop]# vi /etc/sudoers
hadoop ALL=(ALL) ALL

[root@iZuf61u64hvs1nquuaipy8Z hadoop]# su hadoop
[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ yum update
错误：运行此命令需要管理员特权（多数系统下是root用户）。
[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ sudo yum update
[sudo] hadoop 的密码：
[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ sudo dnf update
[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ sudo yum install openssh-server
[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ ssh localhost
The authenticity of host ‘localhost (127.0.0.1)’ can’t be established.
ECDSA key fingerprint is SHA256:q99DX17U2+t45and1n3bSzbTPWZ4j7WETDcl26E60fA.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added ‘localhost’ (ECDSA) to the list of known hosts.
hadoop@localhost’s password:

Welcome to Alibaba Cloud Elastic Compute Service !

Activate the web console with: systemctl enable --now cockpit.socket

[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ cd .ssh
[hadoop@iZuf61u64hvs1nquuaipy8Z .ssh]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:NpqKb+17FOTXBRdDE/PTHtmQDnewADHSHeJzXmwU+Rw hadoop@iZuf61u64hvs1nquuaipy8Z
The key’s randomart image is:
±–[RSA 3072]----+
| …=+++&B |
| .o.oo==E=|
| o o…==B=|
| o .+.o…=|
| So . .|
| +… |
| .o. |
| … . |
| .oo.oo |
±—[SHA256]-----+
[hadoop@iZuf61u64hvs1nquuaipy8Z .ssh]$ cat ./id_rsa.pub >> ./authorized_keys
[hadoop@iZuf61u64hvs1nquuaipy8Z .ssh]$ chmod 600 ~/.ssh/authorized_keys
[hadoop@iZuf61u64hvs1nquuaipy8Z .ssh]$ chmod 700 ~/.ssh
[hadoop@iZuf61u64hvs1nquuaipy8Z .ssh]$ ssh localhost
[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ exit;
注销
Connection to localhost closed.
[hadoop@iZuf61u64hvs1nquuaipy8Z .ssh]$

访问官网http://hadoop.apache.org/获取安装包和源代码

[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ sudo wget https://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz
[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ sudo wget https://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.3.0/hadoop-3.3.0-src.tar.gz

[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ ls
hadoop-3.3.0-src.tar.gz hadoop-3.3.0.tar.gz

[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ sudo tar -zxvf ./hadoop-3.3.0.tar.gz -C /usr/local
[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ cd /usr/local
[hadoop@iZuf61u64hvs1nquuaipy8Z local]$ sudo mkdir apache
[hadoop@iZuf61u64hvs1nquuaipy8Z local]$ sudo mv hadoop-3.3.0 ./apache/
[hadoop@iZuf61u64hvs1nquuaipy8Z apache]$ sudo chown -R hadoop ./hadoop-3.3.0
[hadoop@iZuf61u64hvs1nquuaipy8Z apache]$ ./hadoop-3.3.0/bin/hadoop version
ERROR: JAVA_HOME is not set and could not be found.

[hadoop@iZuf61u64hvs1nquuaipy8Z apache]$ which java
/usr/bin/java
[hadoop@iZuf61u64hvs1nquuaipy8Z apache]$ ls -lrt /usr/bin/java
lrwxrwxrwx 1 root root 22 1月 30 21:08 /usr/bin/java -> /etc/alternatives/java
[hadoop@iZuf61u64hvs1nquuaipy8Z apache]$ ls -lrt /etc/alternatives/java
lrwxrwxrwx 1 root root 73 1月 30 21:08 /etc/alternatives/java -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/bin/java
[hadoop@iZuf61u64hvs1nquuaipy8Z apache]$ sudo vim ~/.bashrc

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre
export CLASSPATH=.:$JAVA_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH

[hadoop@iZuf61u64hvs1nquuaipy8Z apache]$ source ~/.bashrc

[hadoop@iZuf61u64hvs1nquuaipy8Z apache]$ hadoop-3.3.0/bin/hadoop version
Hadoop 3.3.0
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r aa96f1871bfd858f9bac59cf2a81ec470da649af
Compiled by brahma on 2020-07-06T18:44Z
Compiled with protoc 3.7.1
From source with checksum 5dc29b802d6ccd77b262ef9d04d19c4
This command was run using /usr/local/apache/hadoop-3.3.0/share/hadoop/common/hadoop-common-3.3.0.jar

[hadoop@iZuf61u64hvs1nquuaipy8Z apache]$ hadoop-3.3.0/sbin/start-dfs.sh
Starting namenodes on [iZuf61u64hvs1nquuaipy8Z]
iZuf61u64hvs1nquuaipy8Z: Warning: Permanently added ‘izuf61u64hvs1nquuaipy8z,172.21.89.148’ (ECDSA) to the list of known hosts.
iZuf61u64hvs1nquuaipy8Z: WARNING: /usr/local/apache/hadoop-3.3.0/logs does not exist. Creating.
Starting datanodes
Starting secondary namenodes [iZuf61u64hvs1nquuaipy8Z]

官方示例测试一下：

[hadoop@iZuf61u64hvs1nquuaipy8Z apache]$ cd hadoop-3.3.0
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ mkdir input
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ cp ./etc/hadoop/*.xml ./input
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar grep ./input ./output 'dfs[a-z]+'

[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ cat ./output/*
1 dfsadmin

单机模式完成！

伪分布式配置：

[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop]$ cd ~
[hadoop@iZuf61u64hvs1nquuaipy8Z ~]$ cd /usr/local/apache/hadoop-3.3.0/etc/hadoop
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop]$ vim hdfs-site.xml

<configuration>
	<property>
		<name>dfs.replication</name>
		<value>1</value>
	</property>
</configuration>

[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop]$ vim core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://localhost:9000</value>
        </property>
</configuration>

[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop]$ cd ../../

[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ ls

bin  include  lib      LICENSE-binary   LICENSE.txt  NOTICE-binary  output      sbin
etc  input    libexec  licenses-binary  logs         NOTICE.txt     README.txt  share

[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ ./bin/hdfs namenode -format

…
common.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.
…

[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ sbin/start-all.sh
WARNING: Attempting to start all Apache Hadoop daemons as hadoop in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [iZuf61u64hvs1nquuaipy8Z]
Starting resourcemanager
Starting nodemanagers

[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ jps
bash: jps: 未找到命令
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ rpm -qa | grep openjdk
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ sudo yum install -y java-1.8.0-openjdk-devel
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ jps
59554 SecondaryNameNode
59220 NameNode
59894 NodeManager
59772 ResourceManager
61164 Jps
59341 DataNode

[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ ./bin/hdfs dfs -mkdir -p /user/hadoop
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ ./bin/hdfs dfs -ls /
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2021-01-31 02:35 /user
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ ./bin/hdfs dfs -ls .
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ ./bin/hdfs dfs -mkdir input
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ ./bin/hdfs dfs -ls .
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2021-01-31 02:37 input
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ ./bin/hdfs dfs -ls /user/hadoop
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2021-01-31 02:37 /user/hadoop/input
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$ ./bin/hdfs dfs -put ./etc/hadoop/*.xml input
[hadoop@iZuf61u64hvs1nquuaipy8Z hadoop-3.3.0]$

9870

8088