安装步骤
- root用户安装JDK
- 创建目录:mkdir /usr/java,上传jdk-8u45-linux-x64.gz包到该目录
- 解压 tar -zxvf jdk-8u45-linux-x64.gz
- 修改用户和用户组:chown -R root:root jdk1.8.0_45
- 建立软连接:ln -s jdk1.8.0_45 jdk
- 修改环境变量:vi /etc/profile
export JAVA_HOME=/usr/java/jdk
export PATH=$JAVA_HOME/bin:$PATH
- 创建用户:useradd hadoop,切换目录su - hadoop
- 创建目录:mkdir app software sourcecode log tmp data lib
drwxrwxr-x 2 hadoop hadoop 6 Nov 27 21:32 app 解压的文件夹,软连接
drwxrwxr-x 2 hadoop hadoop 6 Nov 27 21:32 data 数据
drwxrwxr-x 2 hadoop hadoop 6 Nov 27 21:32 lib 第三方的jar
drwxrwxr-x 2 hadoop hadoop 6 Nov 27 21:32 log 日志文件夹
drwxrwxr-x 2 hadoop hadoop 6 Nov 27 21:32 software 压缩包
drwxrwxr-x 2 hadoop hadoop 6 Nov 27 21:32 sourcecode 源代码编译
drwxrwxr-x 2 hadoop hadoop 6 Nov 27 21:32 tmp 临时文件夹 ???/tmp
- 上传文件到software目录,
- 解压文件到app目录:tar -zxvf hadoop-2.6.0-cdh5.16.2.tar.gz -C /home/hadoop/app/
- 进入app目录做软连接:ln -s hadoop-2.6.0-cdh5.16.2 hadoop
- 配置hadoop的JAVA_HOME显性配置:vi /home/hadoop/app/hadoop/etc/hadoop/hadoop-env.sh
# The java implementation to use.
export JAVA_HOME=/usr/java/jdk
- 配置文件
vi /home/hadoop/app/hadoop/etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop01:9000</value>
</property>
</configuration>
vi /home/hadoop/app/hadoop/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<!--配置snn-->
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop01:50090</value>
</property>
<property>
<name>dfs.namenode.secondary.https-address</name>
<value>hadoop01:50091</value>
</property>
</configuration>
vi /home/hadoop/app/hadoop/etc/hadoop/slaves
hadoop01
- ssh无密码信任关系
[hadoop@hadoop01 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory \'/home/hadoop/.ssh\'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:pBLxyxiDd/ou7f5LQjTsc81VTaqPbVuoboCqCZMcq+k ruoze@ruozedata001
The key\'s randomart image is:
+---[RSA 2048]----+
| . .o.|
| . + . ..|
| . = * . . . |
| . @ = o . . |
| . + B S.o . |
| . + + o. . + . |
| * .o.. .. = .|
| o o.ooo .o o |
|+E o=+.o. oo . |
+----[SHA256]-----+
[hadoop@hadoop01 ~]$ ls -a
. .. app .bash_history .bash_logout .bash_profile .bashrc data lib log software sourcecode .ssh tmp
[hadoop@hadoop01 ~]$ cd .ssh
[hadoop@hadoop01 .ssh]$ ls
id_rsa id_rsa.pub
[hadoop@hadoop01 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@hadoop01 .ssh]$ ll
total 12
-rw-rw-r-- 1 hadoop hadoop 397 Oct 31 22:46 authorized_keys
-rw------- 1 hadoop hadoop 1675 Oct 31 22:43 id_rsa
-rw-r--r-- 1 hadoop hadoop 397 Oct 31 22:43 id_rsa.pub
[hadoop@hadoop01 .ssh]$ chmod 600 authorized_keys
[hadoop@hadoop01 .ssh]$ ll
total 12
-rw------- 1 hadoop hadoop 397 Oct 31 22:46 authorized_keys
-rw------- 1 hadoop hadoop 1675 Oct 31 22:43 id_rsa
-rw-r--r-- 1 hadoop hadoop 397 Oct 31 22:43 id_rsa.pub
[hadoop@hadoop01 .ssh]$ ssh hadoop01 date
The authenticity of host \'hadoop01 (172.17.0.5)' can't be established.
ECDSA key fingerprint is SHA256:RHKs1HfxMV8qdj04EqUgtl+xUH1ExUJqnnD6GrNg4+4.
ECDSA key fingerprint is MD5:85:0c:a2:eb:a4:79:fa:a9:ae:40:8c:20:17:82:b7:97.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added \'hadoop01,172.17.0.5' (ECDSA) to the list of known hosts.
Sun Oct 31 22:47:21 CST 2021
[hadoop@hadoop01 .ssh]$ ssh hadoop01 date
Sun Oct 31 22:48:34 CST 2021
[hadoop@hadoop01 .ssh]$ ls
authorized_keys id_rsa id_rsa.pub known_hosts
[hadoop@hadoop01 .ssh]$ cat known_hosts
hadoop01,192.168.0.3 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG9N5IGRTqwqGGHZcyNJ2i7lG54isK19GMq+Zw3VDIr64dS2sqoZ79n+8Ibz8ZJsU1aNiaJJTzYUvuxZv5W4iHQ=
当ssh登录到某台机器的时候,会把相应的ip和主机名追加到known_hosts中
15.配置环境变量vi .bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
export HADOOP_HOME=/home/hadoop/app/hadoop
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH
export HADOOP_CONF_DIR=/home/hadoop/app/hadoop/etc/hadoop
- source .bashrc
- which hadoop
- 格式化:hdfs namenode -format
- 第一次启动:start-dfs.sh
start-dfs.sh 21/11/01 21:19:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Starting namenodes on [hadoop01] hadoop01: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-hadoop-namenode-hadoop01.out hadoop01: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-hadoop-datanode-hadoop01.out Starting secondary namenodes [hadoop01] hadoop01: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-hadoop-secondarynamenode-hadoop01.out 21/11/01 21:19:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
- namenode:hadoop01,fs.defaultFS控制的;datanode:slaves控制的,secondarynamenode:dfs.namenode.secondary.http-address,dfs.namenode.secondary.https-address控制的
- namenode:主节点,名称节点(老大),读写请求先经过它;datanode:从节点,数据节点(小弟),存储数据和检索数据;secondarynamenode:第二名称节点(老二)