hadoop单机模式部署
1、下载hadoop安装包
wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz
2、解压安装包
tar zxvf hadoop-3.2.0.tar.gz
3、配置hadoop的环境变量
用vi /etc/profile命令
#Hadoop 3.2.0
export HADOOP_HOME=/home/hadoop/hadoop-3.2.0
export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin
export HADOOP_COMMON_HOME=$HADOOP_PREFIX
export HADOOP_HDFS_HOME=$HADOOP_PREFIX
export HADOOP_MAPRED_HOME=$HADOOP_PREFIX
export HADOOP_YARN_HOME=$HADOOP_PREFIX
export HADOOP_INSTALL=$HADOOP_PREFIX
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_PREFIX/lib/native
export HADOOP_CONF_DIR=$HADOOP_PREFIX
export HADOOP_PREFIX=$HADOOP_PREFIX
export HADOOP_LIBEXEC_DIR=$HADOOP_PREFIX/libexec
export JAVA_LIBRARY_PATH=$HADOOP_PREFIX/lib/native:$JAVA_LIBRARY_PATH
export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
4,修改hosts
使用vi /etc/hosts命令在hosts文件中追加:
192.168.1.4 node101
5,编辑hdfs-site.xml
vi /home/hadoop/hadoop-3.0.0/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://node101:9000</value>
<description>HDFS的URI,文件系统://namenode标识:端口</description>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop</value>
<description>namenode上传到hadoop的临时文件夹</description>
</property>
<property>
<name>fs.checkpoint.period</name>
<value>3600</value>
<description>用来设置检查点备份日志的最长时间</description>
</property>
</configuration>
6,编辑/core-site.xml
vi /home/hadoop/hadoop-3.0.0/etc/hadoop/core-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>副本个数,默认配置是3,应小于datanode机器数量</description>
</property>
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/hadoop-3.0.0/hdfs/name</value>
<description>datanode上存储hdfs名字空间元数据</description>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/hadoop-3.0.0/hdfs/data</value>
<description>datanode上数据块的物理存储位置</description>
</property>
</configuration>
7,设置java_home(注意hadoop官网有介绍,hadoop3.x版本的jdk要1.8以上)
vi /home/hadoop/hadoop-3.0.0/etc/hadoop/hadoop-env.sh
追加export JAVA_HOME=/usr/java/jdk1.8.0_11
8、初始化namenode节点
/home/hadoop/hadoop-3.0.0/bin/hadoop namenode -format
9、启动hdfs
/home/hadoop/hadoop-3.0.0/sbin/start-dfs.sh
10、检查hdfs集群启动情况
jps命令下看到如下图所示则说明没问题
hadoop启动的时候可能会遇到的问题:
1,Unable to load native_hadoop library 的解决办法
找到hadoop下的etc目录下的log4j.properties文件,然后使用vim编辑并追加
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
2,(Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password))
这是因为主机没有设置无密码登 录,按照以下步骤即可:
首先使用 ssh localhost命令查看能否无密码登录本地服务器,如果不能则按步骤执行以下命令
1)ssh-keygen -t rsa -P ‘’ -f ~/.ssh/id_rsa
2) cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
3) chmod 0600 ~/.ssh/authorized_keys
3,WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX
1)把hadoop_prefix变量全部改成hadoop_prefix,
使用vim /etc/profile 命令将HADOOP_PREFIX改成HADOOP_HOME(注意大写)
2)取消hadoop_prefix变量
unset HADOOP_PREFIX
3)source一下使修改后的变量生效
source /etc/profile