最近一周都是再看大数据方面的知识,计划着这段时间来学习大数据的知识,然后在搭建Hadoop等环境的时候,也是用了一些时间吧,感觉还是很有必要来记录一下这方面的配置知识。
先安装一太,安装好JDK的环境,然后其他的几台直接克隆就可以了,就不用一个一个的安装,要是一个一个的安装还是比较要时间的。
export JAVA_HOME=/opt/jdk1.8.0_181
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib这里的/opt/jdk1.8.0_181 就根据你安装JDK的路径了。 然后克隆出几台服务出来。
配置免密登陆
ssh-keygen -t rsa
在~/.ssh 目录下生成二个文件,id_rsa为私钥,id_rsa.pub为公钥。
修改/etc/hosts中IP和Hostname的映射
192.168.119.141 master.yang.com master
192.168.119.142 slave1.yang.com slave1
192.168.119.143 slave2.yang.com slave2导入公钥到认证文件中 : ssh-copy-id -i /root/.ssh/id_rsa.pub master
分别执行 :
ssh-copy-id -i /root/.ssh/id_rsa.pub slave1
ssh-copy-id -i /root/.ssh/id_rsa.pub slave2
这里是根据你克隆了几台服务器来操作,比如我就只有二个,所以我这里自然也就只有二个.
测试下看下免密登陆的配置是否成功
ssh master \ ssh slave1 \ ssh slave2
如果你没有要密码就可以登陆成功的话,那么就说明你是配置成功的。
Hadoop配置
上传Hadoop hadoop-2.7.3.tar.gz 我这里上传的是2.7.3;对文件进行解压
tar -zxvf hadoop-2.7.3.tar.gz -C /usr/local
接下来就是具体修改配置文件信息了
都是在/usr/local/hadoop-2.7.3/etc/hadoop 目录下
hadoop-env.sh
这个文件主要是修改JDK的路径
export JAVA_HOME=/opt/jdk1.8.0_181
yarn-env.sh 也是修改JDK的路径
这个文件也是修改JDK的指向
export JAVA_HOME=/opt/jdk1.8.0_181
salves如果存在就修改,不存在就自己创建一个 touch salves
master
slave1
slave2
core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
--><!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/log/hadoop/tmp</value>
</property>
</configuration>
hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
--><!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///data/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
--><!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration><!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property><property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property><property>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/data/hadoop/yarn/local</value>
</property><property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property><property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/data/tmp/logs</value>
</property><property>
<name>yarn.log.server.url</name>
<value>http://master:19888/jobhistory/logs/</value>
<desc>URL for jobhistory server</desc>
</property><property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property><property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property><property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
这个上面就基本是一台上面的Hadoop配置完成,但是其他的几台不可能也是这么配置吧;使用scp 直接copy过去就可以了
scp -r /usr/local/hadoop-2.7.3/ slave1:/usr/local/
scp -r /usr/local/hadoop-2.7.3/ slave2:/usr/local/
这里还是看你的服务器有多少台来进行决定
在Master那台上面格式化NameNode
在bin目录下
./hdfs namenode -format
xxx has been successully formatted的话,就说明是成功了。
启动在master节点上面
sbin 目录下
./start-dfs.sh
./start-yarn.sh
./mr-jobhistory-daemon.sh start historyserver
来进行启动
然后你可以使用 Jps 来查询启动的进程
关闭的话就是
./stop-dfs.sh
./stop-yarn.sh
./mr-jobhistory-daemon.sh stop historyserver
- http://192.168.119.141:50070/dfshealth.html#tab-overview NameNode
- http://192.168.119.141:8088/cluster ResourceManager
- http://192.168.119.141:19888/jobhistory MapReduce JobHistroy Server
如果都可以访问成功的话,并且上面的节点都有的话,那么就说明你是配置成功了,没有问题。