CentOS搭建Hive(一) 搭建Hadoop(单机版)

1. 安装Java

将Java包解压到/usr/lib下,并重命名为jkd1.8
修改/etc/profile文件,添加如下环境变量

export JAVA_HOME=/usr/lib/jdk1.8
export PATH=$JAVA_HOME/bin:$PATH

执行如下命令使得环境变量文件立刻生效

source /etc/profile

2. 解压Hadoop

这里用到的Hadoop版本是2.7.7,解压后重命名为hadoop-2.7.7,并移到/usr/local/soft下面
添加环境变量

export HADOOP_PREFIX=/usr/local/soft/hadoop-2.7.7
export HADOOP_HOME=$HADOOP_PREFIX
export HADOOP_MAPRED_HOME=$HADOOP_PREFIX
export HADOOP_COMMON_HOME=$HADOOP_PREFIX
export HADOOP_HDFS_HOME=$HADOOP_PREFIX
export YARN_HOME=$HADOOP_PREFIX
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOO_PREFIX/lib/native
export HADOOP_INSTALL=$HADOOP_PREFIX
export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin

执行如下命令使得环境变量文件立刻生效

source /etc/profile

3. 修改Hadoop配置

创建如下目录:
/usr/local/data/hadoop/log
/usr/local/data/hadoop/namenode
/usr/local/data/hadoop/datanode

3.1 hadoop-env.sh

export HADOOP_NAMENODE_OPTS=" -Xms1024m -Xmx1024m -XX:+UseParallelGC"
export HADOOP_DATANODE_OPTS=" -Xms1024m -Xmx1024m"
export HADOOP_LOG_DIR=/usr/local/data/hadoop/log

export JAVA_HOME=/usr/lib/jdk1.8

3.2 core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://pigg01:9000</value>
    </property>
</configuration>

3.3 hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/data/hadoop/namenode</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/data/hadoop/datanode</value>
    </property>
</configuration>

4. 启动HDFS

# 格式化namenode 目录(只执行一次)
hadoop namenode -format

# 启动HDFS
start-dfs.sh

# 查看已经启动的hdfs进程,如果有NameNode和DataNode说明启动成功
jps

5. 配置yarn模式

5.1 mapred-site.xml

之前只有mapred-site.xml.template这个文件,复制一个这个,并重命名为mapred-site.xml,添加如下配置

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

5.2 yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

5.3 启动yarn

start-yarn.sh

判断如下本机地址是否可以访问
http://localhost:8088/
http://localhost:50070/