Hadoop 2.8.0 + CentOS7.3搭建

最新推荐文章于 2022-08-16 15:06:44 发布

xiaoxiao树

最新推荐文章于 2022-08-16 15:06:44 发布

阅读量507

点赞数

分类专栏： other数据库

本文链接：https://blog.csdn.net/u010719917/article/details/74995516

版权

other数据库专栏收录该内容

10 篇文章 0 订阅

订阅专栏

Hadoop 2.8.0 + CentOS7.3
一、安装jdk1.8
tar zxvf jdk-8u65-linux-x64.tar.gz
mv jdk-8u65-linux-x64 /usr/src/jdk

在/etc/profile中添加如下
JAVA_HOME=/usr/src/jdk/
PATH=$JAVA_HOME/bin:/usr/local/xtrabackup/bin:$PATH
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME
export PATH

二、安装hadoop
tar zxvf hadoop-2.8.0.tar.gz
mv hadoop-2.8.0 /usr/src/hadoop

在/etc/profile中添加环境变量
export CLASSPATH
HADOOP_LOG_DIR=/usr/src/hadoop
HADOOP_PREFIX=/usr/src/hadoop
export HADOOP_PREFIX
export HADOOP_HOME=/usr/src/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export HADOOP_MAPARED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export LD_LIBRARY_PATH=${HADOOP_HOME}/lib/native/:$LD_LIBRARY_PATH
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC"

三、服务器相关设置
1, /etc/hosts设置
[root@centos128 hadoop]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.44.128 centos128 localhost
192.168.44.129 centos129
192.168.44.130 centos130

centos128 是master
centos129,centos130 是slave

2. firewalld关闭
systemctl disable firewalld

3. ssh trusts设置
ssh-keygen -t rsa
ssh-keygen -t dsa
cd ~/.ssh
ssh-copy-id -i id_rsa.pub centos128
ssh-copy-id -i id_dsa.pub centos128
ssh-copy-id -i id_rsa.pub centos129
ssh-copy-id -i id_dsa.pub centos129
ssh-copy-id -i id_rsa.pub centos130
ssh-copy-id -i id_dsa.pub centos130

在其它服务器做同样的设置

4. 存放路径的创建
mkdir /data/hadoop/name -p
mkdir /data/hadoop/tmp -p
mkdir /Data1 -p
mkdir /Data2 -p

四、设置配置文件
主要几个配置文件
a,在hadoop-env.sh中将
export JAVA_HOME=${JAVA_HOME}
改成
export JAVA_HOME=/usr/src/jdk

b,
etc/hadoop/core-site.xml 配置NameNode URI
etc/hadoop/hdfs-site.xml 配置NameNode ，配置DataNode，
etc/hadoop/yarn-site.xml 配置ResourceManager ，配置NodeManager ，配置History Server
etc/hadoop/mapred-site.xml 配置MapReduce Applications，配置 MapReduce JobHistory Server
etc/hadoop/slaves         添加slave的IP

1,etc/hadoop/core-site.xml 配置如下：

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://centos128:9000</value>
    </property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/data/hadoop/tmp</value>
</property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
    </property>
<property>
    <name>fs.trash.interval</name>
    <value>10080</value>
</property>
</configuration>

其中hdfs://centos128:9000是 DataName uri地址

2, etc/hadoop/hdfs-site.xml 配置如下：
<configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/data/hadoop/name</value>
    </property>
    <property>
        <name>dfs.blocksize</name>
        <value>268435456</value>
    </property>
    <property>
        <name>dfs.namenode.handler.count</name>
        <value>100</value>
    </property>
   
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/Data1,/Data2</value>
    </property>
</configuration>

dfs.namenode.name.dir namenode物理路径
dfs.replication        默认为3个副本
dfs.datanode.data.dir datanode放存物理路径

3, etc/hadoop/yarn-site.xml 配置如下：
含义参考：http://blog.csdn.net/u010719917/article/details/73917217


<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>centos128</value>
</property>
<property>
    <name>yarn.resourcemanager.address</name>
    <value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>

<property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>0</value>
</property>
<property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>512</value>
</property>


<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>1024</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>2.1</value>
</property>
<property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>${hadoop.tmp.dir}/nm-local-dir</value>
</property>
<property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>${yarn.log.dir}/userlogs</value>
</property>
   <property>
      <name>yarn.nodemanager.log.retain-seconds</name>
    <value>10800</value>
</property>
   <property>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/logs</value>
</property>
<property>
    <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
    <value>logs</value>
</property>
   <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

    <property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>-1</value>
</property>
<property>
    <name>yarn.log-aggregation.retain-check-interval-seconds</name>
    <value>-1</value>
</property>


</configuration>

4. etc/hadoop/mapred-site.xml 配置如下：
含义参考：http://blog.csdn.net/u010719917/article/details/73917217
<configuration>
    
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.map.memory.mb</name>
        <value>1536</value>
    </property>
    <property>
        <name>mapreduce.map.java.opts</name>
        <value>-Xmx1024M</value>
    </property>
    <property>
        <name>mapreduce.reduce.memory.mb</name>
        <value>3072</value>
    </property>
    <property>
        <name>mapreduce.reduce.java.opts</name>
        <value>Xmx2560M</value>
    </property>
    <property>
        <name>mapreduce.task.io.sort.mb</name>
        <value>512</value>
    </property>
    <property>
        <name>mapreduce.task.io.sort.factor</name>
        <value>100</value>
    </property>
    <property>
        <name>mapreduce.reduce.shuffle.parallelcopies</name>
        <value>50</value>
    </property>
    
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>centos128:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>centos128:19888</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.intermediate-done-dir</name>
        <value>/mr-history/tmp</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.done-dir</name>
        <value>/mr-history/done</value>
    </property>
</configuration>

5. etc/hadoop/slaves 如下
[root@centos128 hadoop]# cat slaves
centos129
centos130

6，日志路径：
[root@centos128 logs]# pwd
/usr/src/hadoop/logs
[root@centos128 logs]# ll
total 456
-rw-r--r-- 1 root root 135437 Jul 1 14:06 hadoop-root-namenode-centos128.log
-rw-r--r-- 1 root root   5069 Jul 1 13:19 hadoop-root-namenode-centos128.out
-rw-r--r-- 1 root root 22419 Jul 1 12:54 hadoop-root-secondarynamenode-centos128.log
-rw-r--r-- 1 root root    716 Jul 1 12:54 hadoop-root-secondarynamenode-centos128.out
-rw-r--r-- 1 root root 34891 Jul 1 14:07 mapred-root-historyserver-centos128.log
-rw-r--r-- 1 root root   1477 Jul 1 13:18 mapred-root-historyserver-centos128.out
-rw-r--r-- 1 root root      0 Jul 1 12:53 SecurityAuth-root.audit
-rw-r--r-- 1 root root 20165 Jul 1 13:06 yarn-root-proxyserver-centos128.log
-rw-r--r-- 1 root root    702 Jul 1 13:06 yarn-root-proxyserver-centos128.out
-rw-r--r-- 1 root root 87905 Jul 1 14:06 yarn-root-resourcemanager-centos128.log
-rw-r--r-- 1 root root   1524 Jul 1 13:04 yarn-root-resourcemanager-centos128.out
-rw-r--r-- 1 root root    702 Jul 1 13:00 yarn-root-resourcemanager-centos128.out.1
-rw-r--r-- 1 root root    702 Jul 1 12:58 yarn-root-resourcemanager-centos128.out.2

五 , 其它服务安装hadoop
将配置好的hadoop包，jdk,profile,host 复制到centos129,centos130
cd /
tar cvf hd.tar.gz /usr/src/hadoop/ /usr/src/jdk/ /etc/profile /etc/hosts

六. hadoop脚本及页面
Hadoop Startup
要启动hdfs的HDFS，首先HDFS 格式化：
[hdfs]$ $HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name>

开启HDFS的 NameNode：
[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode

开启HDFS的 DataNode：
[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs start datanode

If etc/hadoop/slaves and ssh trusted access is configured (see Single Node Setup), all of the HDFS processes can be started with a utility script. As hdfs:
如果etc/hadoop/slaves和ssh trusted都被配置，以上所有进程，可以用如下脚本开启
[hdfs]$ $HADOOP_PREFIX/sbin/start-dfs.sh

开启YARN的ResourceManager：
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager

开启YARN的NodeManager：
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemons.sh --config $HADOOP_CONF_DIR start nodemanager

Start a standalone WebAppProxy server. Run on the WebAppProxy server as yarn. If multiple servers are used with load balancing it should be run on each of them:
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start proxyserver

If etc/hadoop/slaves and ssh trusted access is configured (see Single Node Setup), all of the YARN processes can be started with a utility script. As yarn:
直接开启yarn全部功能：
[yarn]$ $HADOOP_PREFIX/sbin/start-yarn.sh

开启mapred的MapReduce JobHistory Server服务：
[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver

Hadoop Shutdown
关闭hdfs的namenode：
[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode

关闭hdfs的DataNode:
[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode

If etc/hadoop/slaves and ssh trusted access is configured (see Single Node Setup), all of the HDFS processes may be stopped with a utility script. As hdfs:
[hdfs]$ $HADOOP_PREFIX/sbin/stop-dfs.sh

关闭yarn的ResourceManager：
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager

关闭yarn的NodeManager：
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemons.sh --config $HADOOP_CONF_DIR stop nodemanager

If etc/hadoop/slaves and ssh trusted access is configured (see Single Node Setup), all of the YARN processes can be stopped with a utility script. As yarn:
[yarn]$ $HADOOP_PREFIX/sbin/stop-yarn.sh

Stop the WebAppProxy server. Run on the WebAppProxy server as yarn. If multiple servers are used with load balancing it should be run on each of them:
[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop proxyserver

关闭mapred的MapReduce JobHistory Serve：
[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver

Web Interfaces
Once the Hadoop cluster is up and running check the web-ui of the components as described below:
Daemon                           Web Interface                Notes
NameNode                         http://nn_host:port/          Default HTTP port is 50070.
ResourceManager                 http://rm_host:port/         Default HTTP port is 8088.
MapReduce JobHistory Server    http://jhs_host:port/        Default HTTP port is 19888.