hadoop伪分布式搭建

Hadoop
伪分布模式（Pseudo-Distributed Mode）

伪分布模式在“单节点集群”上运行Hadoop，其中所有的守护进程都运行在同一台机器上。
该模式在单机模式之上增加了代码调试功能，允许你检查内存使用情况，HDFS输入输出，以及其他的守护进程交互。
比如namenode，datanode，secondarynamenode，jobtracer，tasktracer这5个进程，都能在集群上看到。
1.下载解压

2.软连接

3.设置环境变量

4.环境要求java和ssh

5.配置hadoop用户的ssh信任关系

6.配置文件

7.格式化 和 启动

8.检查服务是否OK

下面是具体步骤：

[root@hadoop001 ~]# useradd hadoop
[root@hadoop001 ~]# vi /etc/sudoers
hadoop ALL=(ALL) ALL

 

1.下载解压
[root@hadoop001 software]# rz
rz waiting to receive.
Starting zmodem transfer. Press Ctrl+C to cancel.
Transferring hadoop-2.8.1.tar.gz...
100% 414604 KB 11845 KB/sec 00:00:35 0 Errors
[root@hadoop001 software]# tar -xzvf hadoop-2.8.1.tar.gz

2.软连接
[root@hadoop002 software]# ln -s /opt/software/hadoop-2.8.1 /opt/software/hadoop
[root@hadoop002 software]# chown -R hadoop:hadoop hadoop/*
[root@hadoop002 software]# chown -R hadoop:hadoop hadoop-2.8.1
[root@hadoop002 software]# chown -R hadoop:hadoop hadoop-2.8.1/*
[root@hadoop001 software]# cd hadoop
[root@hadoop001 hadoop]# rm -f *.txt
[root@hadoop001 hadoop]# ll
total 32
drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14:24 bin
drwxrwxr-x. 3 hadoop hadoop 4096 Jun 2 14:24 etc
drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14:24 include
drwxrwxr-x. 3 hadoop hadoop 4096 Jun 2 14:24 lib
drwxrwxr-x. 2 hadoop hadoop 4096 Aug 20 13:59 libexec
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 13:59 logs
drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14:24 sbin
drwxrwxr-x. 4 hadoop hadoop 4096 Jun 2 14:24 share
[root@hadoop001 hadoop]#

3.设置环境变量
[root@hadoop001 hadoop]# vi /etc/profile
export HADOOP_HOME=/opt/software/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop001 hadoop]# source /etc/profile

4.环境要求java和ssh
[root@hadoop001 hadoop-2.8.1]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

[root@hadoop001 hadoop-2.8.1]# service sshd status
openssh-daemon (pid 1447) is running...

5.切换用户hadoop
[root@hadoop002 software]# su - hadoop
[hadoop@hadoop002 ~]$ cd /opt/software/hadoop
[hadoop@hadoop002 hadoop]$ ll
total 152
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 15:12 bin
drwxr-xr-x. 3 hadoop hadoop 4096 Aug 20 15:12 etc
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 15:12 include
drwxr-xr-x. 3 hadoop hadoop 4096 Aug 20 15:12 lib
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 15:12 libexec
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 15:12 sbin
drwxr-xr-x. 3 hadoop hadoop 4096 Aug 20 15:12 share

bin: 可执行文件
etc：配置文件
sbin: shell脚本 启动关闭hdfs,yarn

6.配置hadoop用户的ssh信任关系
[hadoop@hadoop001 ~]$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Created directory '/home/hadoop/.ssh'.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
b2:d3:a5:38:ff:b6:60:ba:0c:7d:1f:6f:72:69:21:58 hadoop@hadoop001
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| |
| E |
| . So. |
| . =.o. . |
| . * * .. o |
| o B ooo= |
| +..o+*. |
+-----------------+
[hadoop@hadoop001 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@hadoop001 ~]$ chmod 0600 ~/.ssh/authorized_keys
[hadoop@hadoop001 ~]$

[hadoop@hadoop001 ~]$ ssh hadoop001 date
The authenticity of host 'hadoop001 (192.168.137.130)' can't be established.
RSA key fingerprint is 09:f6:4a:f1:a0:bd:79:fd:34:e7:75:94:0b:3c:83:5a.
Are you sure you want to continue connecting (yes/no)? yes #第一次回车输入yes
Warning: Permanently added 'hadoop001,192.168.137.130' (RSA) to the list of known hosts.
Sun Aug 20 14:22:28 CST 2017
[hadoop@hadoop001 ~]$ ssh hadoop001 date #不需要回车输入yes,即OK
Sun Aug 20 14:22:29 CST 2017

[hadoop@hadoop001 ~]$ ssh localhost date
[hadoop@hadoop001 ~]$ ssh localhost date #不需要回车输入yes,即OK
Sun Aug 20 14:22:29 CST 2017

[hadoop@hadoop001 ~]$ cd /opt/software/hadoop

 

7.配置文件
[hadoop@hadoop001 hadoop]$ vi etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop001:9000</value>
</property>
</configuration>

[hadoop@hadoop001 hadoop]$ vi etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

8. 格式化 和 启动
#添加环境变量
[hadoop@hadoop001 hadoop]$ vi /opt/software/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_45

[hadoop@hadoop001 hadoop]$ which hdfs
/opt/software/hadoop/bin/hdfs
[hadoop@hadoop001 hadoop]$ hdfs namenode -format
........
17/08/20 16:43:19 INFO common.Storage: Storage directory
/tmp/hadoop-hadoop/dfs/name has been successfully formatted.

[hadoop@hadoop001 hadoop]$ which start-dfs.sh
/opt/software/hadoop/sbin/start-dfs.sh
[hadoop@hadoop001 hadoop]$
[hadoop@hadoop001 hadoop]$ start-dfs.sh
# 0.0.0.0 也要输入yes

9.检查服务是否OK
[hadoop@hadoop002 hadoop]$ jps
19536 DataNode
19440 NameNode
19876 Jps
19740 SecondaryNameNode
[hadoop@hadoop002 logs]$

http://192.168.137.131:50070

YARN伪分布式部署
[hadoop@hadoop000 hadoop]$ cp mapred-site.xml.template mapred-site.xml
[hadoop@hadoop000 hadoop]$ vi mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

[hadoop@hadoop000 hadoop]$ vi yarn-site.xml:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

[hadoop@hadoop000 sbin]$ ./start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/software/hadoop-2.8.1/logs/yarn-hadoop-resourcemanager-hadoop000.out
192.168.137.251: starting nodemanager, logging to /opt/software/hadoop-2.8.1/logs/yarn-hadoop-nodemanager-hadoop000.out
[hadoop@hadoop000 sbin]$
[hadoop@hadoop000 sbin]$ jps
18576 SecondaryNameNode
17793 ResourceManager
18755 Jps
17893 NodeManager
18422 DataNode
18317 NameNode

http://192.168.137.131:50070
http://192.168.137.131:8088

 

-------------------------------------------------------------
ERROR：
1.
Starting namenodes on [hadoop002]
hadoop002: Error: JAVA_HOME is not set and could not be found.
localhost: Error: JAVA_HOME is not set and could not be found.

转载于:https://www.cnblogs.com/rookieLearn/p/7420507.html