记一次Hadoop配置操作记录

最近一周都是再看大数据方面的知识,计划着这段时间来学习大数据的知识,然后在搭建Hadoop等环境的时候,也是用了一些时间吧，感觉还是很有必要来记录一下这方面的配置知识。

---

先安装一太,安装好JDK的环境,然后其他的几台直接克隆就可以了,就不用一个一个的安装,要是一个一个的安装还是比较要时间的。

export JAVA_HOME=/opt/jdk1.8.0_181

export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib

 这里的/opt/jdk1.8.0_181 就根据你安装JDK的路径了。 然后克隆出几台服务出来。

 

配置免密登陆

ssh-keygen -t rsa

在~/.ssh 目录下生成二个文件,id_rsa为私钥,id_rsa.pub为公钥。

修改/etc/hosts中IP和Hostname的映射

192.168.119.141 master.yang.com master
192.168.119.142 slave1.yang.com slave1
192.168.119.143 slave2.yang.com slave2

导入公钥到认证文件中 : ssh-copy-id -i /root/.ssh/id_rsa.pub master

 

分别执行 :

ssh-copy-id -i /root/.ssh/id_rsa.pub slave1

ssh-copy-id -i /root/.ssh/id_rsa.pub slave2

这里是根据你克隆了几台服务器来操作,比如我就只有二个,所以我这里自然也就只有二个.

 

测试下看下免密登陆的配置是否成功

ssh master \ ssh slave1 \ ssh slave2

如果你没有要密码就可以登陆成功的话,那么就说明你是配置成功的。

 

 Hadoop配置

上传Hadoop hadoop-2.7.3.tar.gz 我这里上传的是2.7.3;对文件进行解压

 tar -zxvf hadoop-2.7.3.tar.gz -C /usr/local

 接下来就是具体修改配置文件信息了

 

 都是在/usr/local/hadoop-2.7.3/etc/hadoop 目录下

hadoop-env.sh

 这个文件主要是修改JDK的路径

 export JAVA_HOME=/opt/jdk1.8.0_181

 

yarn-env.sh 也是修改JDK的路径

这个文件也是修改JDK的指向

export JAVA_HOME=/opt/jdk1.8.0_181

 

salves如果存在就修改,不存在就自己创建一个 touch salves

master
slave1
slave2

 

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://master:8020</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/log/hadoop/tmp</value>
  </property>
</configuration>

 

 hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
       <name>dfs.namenode.name.dir</name>
       <value>file:///data/hadoop/hdfs/name</value>
    </property>
        <property>
           <name>dfs.datanode.data.dir</name>
           <value>file:///data/hadoop/hdfs/data</value>
        </property>
    <property>
           <name>dfs.namenode.secondary.http-address</name>
           <value>master:50090</value>
        </property>
    <property>
       <name>dfs.replication</name>
       <value>3</value>
    </property>
</configuration>

 

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
    </property>
       <property>
     <name>mapreduce.jobhistory.address</name>
         <value>master:10020</value>
       </property>
       <property>
         <name>mapreduce.jobhistory.webapp.address</name>
         <value>master:19888</value>    
       </property>
</configuration>

 

yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>master</value>
</property>
<property>
  <name>yarn.resourcemanager.address</name>
  <value>${yarn.resourcemanager.hostname}:8032</value>
</property>

<property>
  <name>yarn.resourcemanager.scheduler.address</name>
  <value>${yarn.resourcemanager.hostname}:8030</value>
</property>

<property>
  <name>yarn.resourcemanager.webapp.address</name>
  <value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.https.address</name>
  <value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
  <name>yarn.resourcemanager.resource-tracker.address</name>
  <value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
  <name>yarn.resourcemanager.admin.address</name>
  <value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
  <name>yarn.nodemanager.local-dirs</name>
  <value>/data/hadoop/yarn/local</value>
</property>

<property>
  <name>yarn.log-aggregation-enable</name>
  <value>true</value>
</property>

<property>
  <name>yarn.nodemanager.remote-app-log-dir</name>
  <value>/data/tmp/logs</value>
</property>

<property>
  <name>yarn.log.server.url</name>
  <value>http://master:19888/jobhistory/logs/</value>
  <desc>URL for jobhistory server</desc>
</property>

<property>
  <name>yarn.nodemanager.vmem-check-enabled</name>
  <value>false</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>

<property> 
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> 
  <value>org.apache.hadoop.mapred.ShuffleHandler</value> 
</property> 
</configuration>

这个上面就基本是一台上面的Hadoop配置完成，但是其他的几台不可能也是这么配置吧;使用scp 直接copy过去就可以了

scp -r /usr/local/hadoop-2.7.3/ slave1:/usr/local/

scp -r /usr/local/hadoop-2.7.3/ slave2:/usr/local/

 这里还是看你的服务器有多少台来进行决定

在Master那台上面格式化NameNode

在bin目录下

./hdfs namenode -format

xxx has been successully formatted的话,就说明是成功了。

 

启动在master节点上面

sbin 目录下

./start-dfs.sh

./start-yarn.sh

./mr-jobhistory-daemon.sh start historyserver 

来进行启动

然后你可以使用 Jps 来查询启动的进程

 

关闭的话就是

./stop-dfs.sh

./stop-yarn.sh

./mr-jobhistory-daemon.sh stop historyserver  

 

• http://192.168.119.141:50070/dfshealth.html#tab-overview   NameNode
• http://192.168.119.141:8088/cluster  ResourceManager
• http://192.168.119.141:19888/jobhistory  MapReduce JobHistroy Server

如果都可以访问成功的话,并且上面的节点都有的话,那么就说明你是配置成功了，没有问题。