Hadoop的安装与配置即错误排查

1.上传hadoop

2.解压：tar -zxvf hadoop-3.1.3.tar.gz -C /usr/local/src

（解压目录根据需要）

3.切换到所解压的目录之下：cd /usr/local/src

 

4.重命名：mv hadoop-3.1.3/ hadoop

 

5.修改环境变量：vi /etc/profile

export HADOOP_HOME=/usr/

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

保存退出：wq

6.生效环境变量：source /etc/profile

7.修改配置文件：cd hadoop/etc/hadoop

 

(1)vi hadoop-env.sh

 export JAVA_HOME=/usr/local/src/jdk

(2)vim yarn-env.sh

(3)vim mapred-env.sh

   export JAVA_HOME=/usr/local/src/jdk

(4)vim core-site.xml

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>

<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/src/hadoop</value>
</property>

</configuration>

(5)vim hdfs-site.xml

<configuration>

<property>
<name>dfs.replication</name>
<value>2</value>
</property>

<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/src/hadoop/hdfs/name</value>
<final>true</final>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/src/hadoop/hdfs/data</value>
<final>true</final>
</property>

</configuration>

(6)vim yarn-site.xml

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

<property>
<name>yarn.resourcemanager.address</name>
<value>master:18040</value>
</property>

<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:18030</value>
</property>

<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:18025</value>
</property>

<property>
<name>yarn.resourcemanager.resourcemanager.admin.address</name>
<value>master:18141</value>
</property>

<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:18088</value>
</property>

<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>8192</value>
</property>

<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
</property>

<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>4096</value>
</property>

<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value></value>
</property>

<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>5</value>
</property>

<property>
<name>yarn.nodemanager.pmem-check-enable</name>
<value>false</value>
</property>

<property>
<name>yarn.nodemanager.vmem-check-enable</name>
<value>false</value>
</property>

<!-- Site specific YARN configuration properties -->
</configuration>

 (7)vim mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

 (8)vim workers

master

slave1

slave2

8.（1）将master上的hadoop分发到slave1和slave2上：

scp -r /usr/local/src/hadoop root@slave1/2:/usr/local/src

 （2）分发profile文件

scp /etc/profile root@slave1:/etc/profile

 分发完之后在slave1和slave2上 生效配置文件：source /etc/profile

9.在master上进行格式化：hdfs namenode -format

10.启动：start-all.sh

*******************************Hadoop排错*****************************

一·

1.错误：节点不全
2.原因：hadoop中没有默认的root用户
3.解决：在环境变量中（/etc/profile）添加
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
然后source /etc/profile
分发：scp /etc/profile root@slave1/2:/etc/profile
source /etc/profile

二：启动Hadoop时从节点没有DataNode
原因：因为格式化的问题，导致了主节点和从节点的clusterId不一致，所以才导致datanode没有启动成功
解决：
进入到你的集群的current目录下去找VERSION这个文件
修改的clusterId，从主节点中的namenode，打开并和从节点对比，并修改为namenode的clusterId