Hadoop伪分布式部署

最新推荐文章于 2024-04-08 23:12:27 发布

钻石大门

最新推荐文章于 2024-04-08 23:12:27 发布

阅读量105

点赞数

分类专栏： Hadoop

本文链接：https://blog.csdn.net/weixin_41158277/article/details/94365787

版权

Hadoop 专栏收录该内容

13 篇文章 0 订阅

订阅专栏

前置条件与软件说明
Hadoop版本:hadoop-2.6.0-cdh5.7.0
JDK版本：jdk-8u45-linux-x64
Linux安装ssh服务

1.创建用户和上传hadoop软件
wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
rz命令上传

[root@10-9-15-140 ~]# useradd hadoop
[root@10-9-15-140 ~]# su - hadoop
[hadoop@10-9-15-140 ~]$
[hadoop@10-9-15-140 ~]$ mkdir app
[hadoop@10-9-15-140 ~]$ cd app/
[hadoop@10-9-15-140 app]$ wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
或
[hadoop@10-9-15-140 app]$ rz

2.部署jdk
CDH环境:
mkdir /usr/java jdk部署
mkdir /usr/share/java 部署CDH需要mysql jdbc jar包
rz上传 jdk-8u45-linux-x64.gz

解压
[root@10-9-15-140 java]# tar -xzvf jdk-8u45-linux-x64.gz
[root@10-9-15-140 java]# ll
total 169216
drwxr-xr-x 8 uucp      143      4096 Apr 11  2015 jdk1.8.0_45
-rw-r--r-- 1 hadoop hadoop 173271626 Jun 30 23:59 jdk-8u45-linux-x64.gz

权限修正
[root@10-9-15-140 java]# chown -R root:root jdk1.8.0_45
[root@10-9-15-140 java]# ll
total 169216
drwxr-xr-x 8 root   root        4096 Apr 11  2015 jdk1.8.0_45
-rw-r--r-- 1 hadoop hadoop 173271626 Jun 30 23:59 jdk-8u45-linux-x64.gz
[root@10-9-15-140 java]# vi /etc/profile

#env
export JAVA_HOME=/usr/share/java/jdk1.8.0_45
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JER_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JER_HOME/bin:$PATH


[root@10-9-15-140 java]# source /etc/profile
[root@10-9-15-140 java]# which java
/usr/share/java/jdk1.8.0_45/bin/java

3.解压hadoop

[hadoop@10-9-15-140 app]$ tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@10-9-15-140 app]$ cd hadoop-2.6.0-cdh5.7.0
[hadoop@10-9-15-140 app]$ ll
total 304288
drwxr-xr-x 14 hadoop hadoop      4096 Mar 24  2016 hadoop-2.6.0-cdh5.7.0
-rw-r--r--  1 hadoop hadoop 311585484 Jun 30 23:59 hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@10-9-15-140 app]$ cd hadoop-2.6.0-cdh5.7.0
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ ll
total 76
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin		*可执行脚本
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin-mapreduce1
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 cloudera
drwxr-xr-x  6 hadoop hadoop  4096 Mar 24  2016 etc		*配置目录(conf)
drwxr-xr-x  5 hadoop hadoop  4096 Mar 24  2016 examples
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 examples-mapreduce1
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 include
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 lib		*jar包目录
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 libexec
-rw-r--r--  1 hadoop hadoop 17087 Mar 24  2016 LICENSE.txt
-rw-r--r--  1 hadoop hadoop   101 Mar 24  2016 NOTICE.txt
-rw-r--r--  1 hadoop hadoop  1366 Mar 24  2016 README.txt
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 sbin		*hadoop组件的启动、停止脚本
drwxr-xr-x  4 hadoop hadoop  4096 Mar 24  2016 share
drwxr-xr-x 17 hadoop hadoop  4096 Mar 24  2016 src

4.Configuration

Use the following:
/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>


/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/hdfs-site.xml:
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

5.配置ssh localhost无密码信任关系

[hadoop@10-9-15-140 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
53:21:86:1c:af:cd:f3:c4:45:a4:ca:72:ea:83:58:37 hadoop@10-9-15-140
The key's randomart image is:
+--[ RSA 2048]----+
|     ..oo ..o    |
|      oo . +     |
|        . o .    |
|       = + .     |
|      o S o      |
|    . E+ =       |
|   o o..  .      |
|  . ...          |
|      ..         |
+-----------------+
[hadoop@10-9-15-140 ~]$ cd .ssh
[hadoop@10-9-15-140 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1675 Jul  2 01:18 id_rsa
-rw-r--r-- 1 hadoop hadoop  400 Jul  2 01:18 id_rsa.pub
[hadoop@10-9-15-140 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@10-9-15-140 .ssh]$ ll
total 12
-rw-rw-r-- 1 hadoop hadoop  400 Jul  2 01:18 authorized_keys
-rw------- 1 hadoop hadoop 1675 Jul  2 01:18 id_rsa		私钥
-rw-r--r-- 1 hadoop hadoop  400 Jul  2 01:18 id_rsa.pub	公钥

ssh localhost date 是需要输入密码，但是这个用户是没有配置密码。
我们应该在没有配置密码情况下去完成无密码信任呢？

改权限
[hadoop@10-9-15-140 .ssh]$ chmod 600 authorized_keys
[hadoop@10-9-15-140 .ssh]$ ssh localhost date
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is 68:d4:3b:cb:d1:f9:dc:23:65:81:a9:a0:fd:e9:ec:3b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Tue Jul  2 01:19:38 CST 2019
#ssh测试通过
[hadoop@10-9-15-140 .ssh]$ ssh localhost date
Tue Jul  2 01:19:53 CST 2019

6.格式化

[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs namenode -format

7.启动

[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
19/07/02 09:48:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-10-9-15-140.out
localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-10-9-15-140.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-10-9-15-140.out
19/07/02 09:48:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ jps
29539 Jps
29222 DataNode
29405 SecondaryNameNode
29134 NameNode

Two issues:

Issue One:
start-dfs.sh启动出现Error:cannot find configuration directory:/etc/hadoop
将hadoop-env.sh中的

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

改为绝对路径

export HADOOP_CONF_DIR=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop

Issue Two:
Hadoop安装完后，启动时报Error: JAVA_HOME is not set and could not be found.

解决办法：

修改/etc/hadoop/hadoop-env.sh中设JAVA_HOME。

应当使用绝对路径。

export JAVA_HOME=$JAVA_HOME                  //错误，原来就这样的 不该就报错

export JAVA_HOME=/usr/share/java/jdk1.8.0_45/bin/java        //正确，应该这么改

8.环境变量

[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ cat ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_PREFIX/bin:$PATH

9.正式使用

通过浏览器查看NameNode
http://NameNodeIP机器IP地址:50070/
http://106.75.17.244:50070

注意：云主机需要在防火墙添加端口
在这里插入图片描述

10.停止NameNode和DataNode
$ sbin/stop-dfs.sh

[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ sbin/stop-dfs.sh
19/07/03 00:10:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
19/07/03 00:10:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@10-9-15-140 hadoop-2.6.0-cdh5.7.0]$ jps
23642 Jps

参考资料：
apache hadoop： hadoop.apache.org
cdh hadoop: http://archive.cloudera.com/cdh5/cdh/5/

钻石大门

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Hadoop伪分布式部署

参考资料：apache hadoop： hadoop.apache.orgcdh hadoop: http://archive.cloudera.com/cdh5/cdh/5/1.创建用户和上传hadoop软件wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gzrz命令上传useradd ha...
复制链接

扫一扫

专栏目录