Ubuntu 16.04+Hadoop-2.5.0 伪分布式搭建

最新推荐文章于 2022-05-01 01:23:46 发布

理想国的柠檬

最新推荐文章于 2022-05-01 01:23:46 发布

阅读量269

点赞数

分类专栏： UBUNTU hadoop

UBUNTU 同时被 2 个专栏收录

3 篇文章 0 订阅

订阅专栏

hadoop

3 篇文章 0 订阅

订阅专栏

1. https://archive.apache.org/dist/hadoop/core/hadoop-2.5.0/

下载hadoop-2.5.0版本

解压到/usr路径下

2. 确保jkd安装，并配置环境变量

JAVA_HOME=/usr/java/jdk1.8.0_171

export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin

export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib

3.配置hadoop环境变量

HADOOP_HOME=/usr/hadoop-2.5.0

export PATH=$PATH:$HADOOP_HOME/bin

export PATH=$PATH:$HADOOP_HOME/sbin

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

export HADOOP_HDFS_HOME=${HADOOP_HOME}

export HADOOP_YARN_HOME=${HADOOP_HOME}

4.创建相关文件（这里容易遇到权限问题，后续启动hadoop时候要将/home/hadoop/hadoop-2.5.0 权限改为当前用户）

# mkdir -p /home/hadoop

# mkdir -p /home/hadoop/hadoop-2.5.0

# mkdir -p /home/hadoop/hadoop-2.5.0/tmp

# mkdir -p /home/hadoop/hadoop-2.5.0/dfs

# mkdir -p /home/hadoop/hadoop-2.5.0/dfs/name

# mkdir -p /home/hadoop/hadoop-2.5.0/dfs/data

5.配置文件修改

①

vim /usr/hadoop-2.5.0/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_171

②

vim /usr/hadoop-2.5.0/etc/hadoop/core-site.xml

添加如下记录

在<configuration></configuration>之间加入如下记录：
<property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/hadoop-2.5.0/tmp</value>
</property>
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
</property>

vim /usr/hadoop-2.5.0/etc/hadoop/hdfs-site.xml

添加如下记录

在<configuration></configuration>之间加入如下记录：
<property>
    <name>dfs.name.dir</name>
    <value>/home/hadoop/hadoop-2.5.0/dfs/name</value>
</property>
<property>
    <name>dfs.data.dir</name>
    <value>/home/hadoop/hadoop-2.5.0/dfs/data</value>
</property>
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
<property>
    <name>dfs.permissions</name>
    <value>false</value>
</property>

④

vim /usr/hadoop-2.5.0/etc/hadoop/mapred-site.xml.template

添加如下记录

在<configuration></configuration>之间加入如下记录：
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

复制

cp /usr/hadoop-2.5.0/etc/hadoop/mapred-site.xml.template /usr/hadoop-2.5.0/etc/hadoop/mapred-site.xml

⑤

vim /usr/hadoop-2.5.0/etc/hadoop/yarn-site.xml

添加如下记录

在<configuration></configuration>之间加入如下记录：
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>localhost</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
  <name>yarn.resourcemanager.scheduler.class</name>
  <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>

vim /usr/hadoop-2.5.0/etc/hadoop/slaves

删除文件中的记录并添加如下记录

localhost

6.格式化HDFS文件系统

cd /usr/hadoop-2.5.0/bin/hdfs namenode -format

7.启动hadoop集群和停止hadoop集群

/usr/hadoop-2.5.0/sbin/start-all.sh     //启动
/usr/hadoop-2.5.0/sbin/stop-all.sh     //停止

8.在hadoop集群启动着的情况下执行jps指令查看输出，如果看见如下输出说明搭建成功。

Jps
NameNode
SecondaryNameNode
DataNode
NodeManager
ResourceManager

最终结果：

xin@xin-OptiPlex-790:/usr/hadoop-2.5.0/sbin$ ./start-dfs.sh
Starting namenodes on [localhost]
xin@localhost's password:
localhost: starting namenode, logging to /usr/hadoop-2.5.0/logs/hadoop-xin-namenode-xin-OptiPlex-790.out
xin@localhost's password:
localhost: starting datanode, logging to /usr/hadoop-2.5.0/logs/hadoop-xin-datanode-xin-OptiPlex-790.out
Starting secondary namenodes [0.0.0.0]
xin@0.0.0.0's password:
0.0.0.0: starting secondarynamenode, logging to /usr/hadoop-2.5.0/logs/hadoop-xin-secondarynamenode-xin-OptiPlex-790.out

xin@xin-OptiPlex-790:/usr/hadoop-2.5.0/sbin$ jps

26032 org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar
27713 SecondaryNameNode
27384 NameNode
27832 Jps

27530 DataNode

xin@xin-OptiPlex-790:/usr/hadoop-2.5.0/sbin$ ./start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/hadoop-2.5.0/logs/yarn-xin-resourcemanager-xin-OptiPlex-790.out
xin@localhost's password:
localhost: starting nodemanager, logging to /usr/hadoop-2.5.0/logs/yarn-xin-nodemanager-xin-OptiPlex-790.out

xin@xin-OptiPlex-790:/usr/hadoop-2.5.0/sbin$ jps

28304 Jps
26032 org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar
27713 SecondaryNameNode
27891 ResourceManager
27384 NameNode
27530 DataNode

28206 NodeManager

26032 org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar为什么会有这个不太明白……、

参考博客：https://blog.csdn.net/looc_246437/article/details/53522036