centos7安装hadoop伪分布式系统(4)之04安装hadoop伪分布式集群

1上传解压hadoop安装文件

配置env

/HADOOP：表示hadoop的安装文件

cd /home/hadoop/bigdater/hadoop/etc/hadoop

1.1修改hadoop文件夹etc下的hadoop-env.sh

export JAVA_HOME=${JAVA_HOME}改成export JAVA_HOME=/opt/jdk

添加

export HADOOP_PID_DIR=/home/hadoop/bigdater/hadoop/hdfs/tmp

1.2修改mapred-env.sh

添加

export HADOOP_MAPRED_PID_DIR=/home/hadoop/bigdater/hadoop/hdfs/tmp

1.3修改yarn-env.sh

添加

export YARN_PID_DIR=/home/hadoop/bigdater/hadoop/hdfs/tmp

2配置集群xml

2.1修改core-site.xml　

vi core-site.xml

<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hh:8020</value> </property>

<property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/bigdater/hadoop/hdfs/tmp</value> </property> </configuration>

2.2配置hdfs-site.xml　　

<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/home/hadoop/bigdater/hadoop/hdfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/home/hadoop/bigdater/hadoop/hdfs/data</value> </property>

<property> <name>dfs.permissions.enabled</name> <value>false</value> </property> </configuration>　

2.3配置mapred-site.xml　

cp mapred-site.xml.template mapred-site.xml

添加

<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>

2.4配置yarn-site.xml

　<configuration> <!-- 指定reducer获取数据的方式--> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>

2.5配置slaves

vi slaves 添加hh(hh表示主机名)

3配置环境变量

export HADOOP_HOME=/home/hadoop/bigdater/hadoop export HADOOP_PREFIX=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_PREFIX

export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop

export HADOOP_HDFS_HOME=$HADOOP_PREFIX

export HADOOP_MAPRED_HOME=$HADOOP_PREFIX

export HADOOP_YARN_HOME=$HADOOP_PREFIX

PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

 

4启动hadoop

4.1格式化

　hdfs namenode -format

4.2启动hadoop集群

start-all.sh

4.3查看hadoop文件夹下的logs日志文件

4.4查看jps中的java进程

查看服务器的50070端口的web界面

http://192.168.1.106:50070/dfshealth.html#tab-overview

查看服务器的8088端口的web界面

http://192.168.1.106:8088/cluster

使用hadoop的例子程序测试hadoop安装

1.创建一个测试文件test.txt

vi test.txt 输入一些字符串

2.上传到hdfs上

hadoop dfs -put test.txt /

3.开启测试程序

cd /home/hadoop/bigdater/hadoop/share/hadoop/mapreduce

hadoop jar hadoop-mapreduce-examples-2.5.0-cdh5.3.6.jar wordcount /test.txt output/1234

4.下载

hadoop dfs -ls output/1234/part-*

查看计算结果:

hadoop dfs -text output/1234/part-*