搭建Hadoop单机版
1. 安装Java
将Java包解压到/usr/lib下,并重命名为jkd1.8
修改/etc/profile文件,添加如下环境变量
export JAVA_HOME=/usr/lib/jdk1.8
export PATH=$JAVA_HOME/bin:$PATH
执行如下命令使得环境变量文件立刻生效
source /etc/profile
2. 解压Hadoop
这里用到的Hadoop版本是2.7.7,解压后重命名为hadoop-2.7.7,并移到/usr/local/soft下面
添加环境变量
export HADOOP_PREFIX=/usr/local/soft/hadoop-2.7.7
export HADOOP_HOME=$HADOOP_PREFIX
export HADOOP_MAPRED_HOME=$HADOOP_PREFIX
export HADOOP_COMMON_HOME=$HADOOP_PREFIX
export HADOOP_HDFS_HOME=$HADOOP_PREFIX
export YARN_HOME=$HADOOP_PREFIX
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOO_PREFIX/lib/native
export HADOOP_INSTALL=$HADOOP_PREFIX
export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin
执行如下命令使得环境变量文件立刻生效
source /etc/profile
3. 修改Hadoop配置
创建如下目录:
/usr/local/data/hadoop/log
/usr/local/data/hadoop/namenode
/usr/local/data/hadoop/datanode
3.1 hadoop-env.sh
export HADOOP_NAMENODE_OPTS=" -Xms1024m -Xmx1024m -XX:+UseParallelGC"
export HADOOP_DATANODE_OPTS=" -Xms1024m -Xmx1024m"
export HADOOP_LOG_DIR=/usr/local/data/hadoop/log
export JAVA_HOME=/usr/lib/jdk1.8
3.2 core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://pigg01:9000</value>
</property>
</configuration>
3.3 hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/data/hadoop/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/data/hadoop/datanode</value>
</property>
</configuration>
4. 启动HDFS
# 格式化namenode 目录(只执行一次)
hadoop namenode -format
# 启动HDFS
start-dfs.sh
# 查看已经启动的hdfs进程,如果有NameNode和DataNode说明启动成功
jps
5. 配置yarn模式
5.1 mapred-site.xml
之前只有mapred-site.xml.template这个文件,复制一个这个,并重命名为mapred-site.xml,添加如下配置
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
5.2 yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
5.3 启动yarn
start-yarn.sh
判断如下本机地址是否可以访问
http://localhost:8088/
http://localhost:50070/