CentOS 7 安装配置Hadoop2.6

2019独角兽企业重金招聘Python工程师标准>>>

CentOS 7 安装配置Hadoop2.6

1. 安装JDK并配置环境变量

2. 配置SSH无密码登陆

1. ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
2. cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
3. 验证ssh, ssh localhost 
不需要输入密码即可登录

3.下载Hadoop2.6

1. 创建hadoop存放目录

mkdir /root/hadoop

2. 进入hadoop目录

cd /root/hadoop

3. 下载hadoop2.6

wget http://apache.fayea.com/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz

Hadoop所有版本下载:http://apache.fayea.com/hadoop/common/

4. 解压

tar -zxvf hadoop-2.6.0.tar.gz
解压后目录为: /root/hadoop/hadoop-2.6.0 

5. 创建hdfs和tmp目录

mkdir /root/hadoop/tmp 
mkdir /root/hadoop/hdfs 
mkdir /root/hadoop/hdfs/data 
mkdir /root/hadoop/hdfs/name

4. 设置Hadoop环境变量

1. 编辑环境变量文件

vi ~/.bash_profile

2. 添加hadoop环境变量

# set hadoop path
export HADOOP_HOME=/root/hadoop/hadoop-2.6.0
export PATH=$PATH:$HADOOP_HOME/bin

3. 使环境变量生效

source ~/.bash_profile

5. 配置Hadoop

进入配置文件目录

cd /root/hadoop/hadoop-2.6.0/etc/hadoop

需要配置的的有4个文件

hadoop-2.6.0/etc/hadoop/core-site.xml 
hadoop-2.6.0/etc/hadoop/hdfs-site.xml 
hadoop-2.6.0/etc/hadoop/mapred-site.xml 
hadoop-2.6.0/etc/hadoop/yarn-site.xml

1. 配置core-site.xml

<configuration>
<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
    <description>HDFS的URI，文件系统://namenode标识:端口号</description>
</property>

<property>
    <name>hadoop.tmp.dir</name>
    <value>/root/hadoop/tmp</value>
    <description>namenode上本地的hadoop临时文件夹</description>
</property>
</configuration>

2.配置hdfs-site.xml

<configuration>
<property>
    <name>dfs.name.dir</name>
    <value>/root/hadoop/hdfs/name</value>
    <description>namenode上存储hdfs名字空间元数据 </description> 
</property>

<property>
    <name>dfs.data.dir</name>
    <value>/root/hadoop/hdfs/data</value>
    <description>datanode上数据块的物理存储位置</description>
</property>

<property>
    <name>dfs.replication</name>
    <value>1</value>
    <description>副本个数，配置默认是3,应小于datanode机器数量</description>
</property>
</configuration>

3.配置mapred-site.xml

<configuration>
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>
</configuration>

4.配置yarn-site.xml

<configuration>
<!-- Site specific YARN configuration properties -->
<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
</property>
<property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>${yarn.resourcemanager.hostname}:8099</value>
</property>
</configuration>

6. 启动Hadoop

1. 格式化Hadoop

cd /root/hadoop/hadoop-2.6.0/bin
[root@localhost bin]# hdfs namenode –format
17/02/02 23:31:31 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = localhost.localdomain/127.0.0.1
STARTUP_MSG:   args = [–format]
STARTUP_MSG:   version = 2.6.0

2. 启动NameNode 和 DataNode 守护进程

./root/hadoop/hadoop-2.6.0/sbin/start-dfs.sh

3. 启动ResourceManager 和 NodeManager 守护进程

./root/hadoop/hadoop-2.6.0/sbin/start-yarn.sh

7. 验证Hadoop是否启动

1. 输入jps,有如下进程,表示Hadoop正常启动

[root@localhost hadoop]# jps
19942 Jps
16327 DataNode
16975 ResourceManager
16681 SecondaryNameNode
17077 NodeManager

2. 查看YARN的ResourceManager的界面,默认端口是8088,上面配置文件改成了8099

http://ip:8099

参考:http://www.cnblogs.com/lovezhaolei/p/5594115.html

转载于:https://my.oschina.net/himrliu/blog/832411