Linux安装Hadoop

最新推荐文章于 2024-04-28 18:32:26 发布

胖毁青春，瘦解百病

最新推荐文章于 2024-04-28 18:32:26 发布

阅读量161

点赞数

分类专栏： # 软件安装文章标签： hadoop 大数据 linux hdfs centos

本文链接：https://blog.csdn.net/llb_3601478/article/details/117012981

版权

软件安装专栏收录该内容

11 篇文章 0 订阅

订阅专栏

1、下载http://archive.apache.org/dist/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz

2、添加环境变量

vim /etc/profile 或者 vim ~/.bashrc

#Hadoop Environment 
export HADOOP_HOME=$USER_HOME/software/hadoop-2.7.3
export PATH=$PATH:$HADOOP_HOME/bin
source /etc/profile 或者 source ~/.bashrc

3、修改配置文件

vim $HADOOP_HOME/etc/hadoop/core-site.xml

<configuration>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/home/hadoop/software/hadoop-2.7.3/tmp</value>
	</property>
	<property>
		<name>fs.default.name</name>
		<value>hdfs://hadoop:9000</value>
	</property>
</configuration>

vim $HADOOP_HOME/etc/hadoop/hdfs-site.xml

<configuration>
	<property>
		<name>dfs.datanode.data.dir</name>
		<value>/home/hadoop/software/hadoop-2.7.3/data</value>
	</property>
	<property>
		<name>dfs.namenode.name.dir</name>
		<value>/home/hadoop/software/hadoop-2.7.3/name</value>
	</property>
	<property>
		<name>dfs.http.address</name>
		<value>0.0.0.0:8100</value><!-- 默认端口50070 -->
	</property>
	<property>
		<name>dfs.replication</name>
		<value>1</value>
	</property>
</configuration>

4、namenode格式化

hdfs namenode -format

成功标志：INFO util.ExitUtil: Exiting with status 0

5、启动Namenode和Datanode

sbin/start-dfs.sh

执行报错：

hadoop@hadoop:~/software/hadoop-2.7.3/bin$ ../sbin/start-dfs.sh
Starting namenodes on [localhost]
localhost: ssh: connect to host localhost port 22: Connection refused
localhost: ssh: connect to host localhost port 22: Connection refused
Starting secondary namenodes [0.0.0.0]
0.0.0.0: ssh: connect to host 0.0.0.0 port 22: Connection refused

执行：sudo apt-get install openssh-server 安装服务端
测试是否安装成功

su hadoop
ssh localhost

再执行sbin/start-dfs.sh

hadoop@hadoop:~/software/hadoop-2.7.3/sbin$ ./start-dfs.sh
Starting namenodes on [localhost]
hadoop@localhost's password: 
localhost: Error: JAVA_HOME is not set and could not be found.
vim $HADOOP_HOME/etc/hadoop/hadoop-env.sh

增加 export JAVA_HOME=/home/hadoop/software/jdk1.8.0_201，再执行hadoop/sbin/start-dfs.sh
然后执行jps命令查看：

hadoop@hadoop:~/software/hadoop-2.7.3/sbin$ jps
76051 Jps
73282 NameNode
73685 SecondaryNameNode
73466 DataNode

6、HDFS验证:
可以在浏览器里面输入下面网址：http://hadoop:8100 默认端口为50070

可以在shell里面操作hdfs文件系统：

bin/hadoop fs -ls /        //查看hdfs根目录的文件树
bin/hadoop fs -mkdir /test        //创建test文件夹
bin/hadoop fs -cp 文件 文件        //拷贝文件

7、安装yarn：

cp $HADOOP_HOME/etc/hadoop/mapred-site.xml.template $HADOOP_HOME/etc/hadoop/mapred-site.xml

vim $HADOOP_HOME/etc/hadoop/mapred-site.xml

<configuration>
	<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
	</property>
</configuration>

vim $HADOOP_HOME/etc/hadoop/yarn-site.xml

<configuration>
	<!-- Site specific YARN configuration properties -->
	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
	</property>
	<property>
		<name>yarn.resourcemanager.scheduler.address</name>
		<value>hadoop:8030</value>
	</property>
	<property>
		<name>yarn.resourcemanager.resource-tracker.address</name>
		<value>hadoop:8031</value>
	</property>
	<property>
		<name>yarn.resourcemanager.address</name>
		<value>hadoop:8032</value>
	</property>
</configuration>

启动yarn

hadoop/sbin/start-yarn.sh
hadoop@hadoop:~/software/hadoop-2.7.3/sbin$ jps
10578 NodeManager
10243 ResourceManager
10614 Jps

8、YARN验证:
可以在浏览器里面输入下面网址：http://localhost:8088/cluster
统计单词出现的个数：
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /bbb.txt /out

执行报错：
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /bbb.txt /out
19/04/12 00:52:27 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
解决方案：没有启动dfs的进程，执行sbin/stop-all.sh之后执行sbin/start-all.sh
然后再执行报错：
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /bbb.txt /out

19/04/12 01:30:43 INFO client.RMProxy: Connecting to ResourceManager at hadoop/192.168.126. :8032
19/04/12 01:30:44 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/hadoop/.staging/job_1555057828784_0001
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot delete /tmp/hadoop-yarn/staging/hadoop/.staging/job_1555057828784_0001. Name node is in safe mode.
The reported blocks 6 has reached the threshold 0.9990 of total blocks 6. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 0 seconds.
解决方案：bin/hadoop dfsadmin -safemode leave 关闭安全模式
hadoop@hadoop:~/software/hadoop-2.7.3$ bin/hadoop dfsadmin -safemode leave
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Safe mode is OFF

再执行
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /bbb.txt /output
报错：hdfs.DFSClient: Caught exception java.lang.InterruptedException
这个报错不影响结果，可以忽略。建议将文件放在/usr/data/input下面，不提示错误。
执行成功：19/04/14 19:03:36 INFO mapreduce.Job: map 100% reduce 100%
19/04/14 19:03:36 INFO mapreduce.Job: Job job_1555293092368_0002 completed successfully
执行 bin/hdfs dfs -ls -R /
-rw-r--r-- 1 hadoop supergroup 0 2019-04-14 19:03 /output/_SUCCESS
-rw-r--r-- 1 hadoop supergroup 157 2019-04-14 19:03 /output/part-r-00000
然后执行bin/hdfs dfs -cat /output/part-r-00000，结果如下：
0000000 1
11111 1
2222 1
222222 1

9、启停脚本
$HADOOP_HOME/sbin/start-all.sh
$HADOOP_HOME/sbin/stop-all.sh

10、使用到的命令

rm -rf /home/hadoop/software/hadoop-2.7.3/tmp 删除tmp目录
rm -rf /home/hadoop/software/hadoop-2.7.3/data 删除data目录
rm -rf /home/hadoop/software/hadoop-2.7.3/name 删除name目录
hdfs namenode -format 格式化
hadoop version 版本