docker-compose版本
使用docker-compose请参见基于docker-compose的hadoop集群搭建
非docker版本
下载
wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz && tar -zxvf hadoop-3.2.0.tar.gz && cd hadoop-3.2.0/etc/hadoop && vi core-site.xml
修改配置文件
修改core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://qbitcluster</value>
</property>
<!--默认的路径-->
<property>
<name>hadoop.tmp.dir</name>
<value>/home/qbit/data/tmp</value>
</property>
<!--hadoop的临时目录,如果需要配置多个目录,需要用逗号隔开-->
<property>
<name>ha.zookeeper.quorum</name>
<value>192.168.1.146:2181,192.168.1.144:2181,192.168.1.141:2181</value>
</property>
<!--配置zookeeper 管理HDFS-->
</configuration>
修改hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
设置系统参数
export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64/"
export HADOOP_HOME="/home/qbit/hadoop-3.2.0"
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
在任意地方,输入hadoop会自动补全。说明配置正确。
上面是临时生效,希望永久生效则修改~/.bashrc把上面几句话添加到文件后面。
master节点配置
修改hadoop环境变量
vi /home/qbit/hadoop-3.2.0/etc/hadoop/hadoop-env.sh
需要设置JAVA_HOME和HADOOP_HOME
设置环境变量
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
初始化
hadoop namenode -format
结果里(倒数8行的样子)出现下列说明成功
2019-07-26 08:26:07,056 INFO common.Storage: Storage directory /home/qbit/data/tmp/dfs/name has been successfully formatted.
2019-07-26 08:26:07,085 INFO namenode.FSImageFormatProtobuf: Saving image file /home/qbit/data/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2019-07-26 08:26:07,255 INFO namenode.FSImageFormatProtobuf: Image file /home/qbit/data/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 399 bytes saved in 0 seconds .
2019-07-26 08:26:07,292 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2019-07-26 08:26:07,305 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubuntu/192.168.1.146
************************************************************/
启动namenode
hdfs --daemon start namenode
然后ps看下启动没
修改slave
vi /home/qbit/hadoop-3.2.0/etc/hadoop/slaves
按情况添加ip,前提是做了免密登录
192.168.1.146
192.168.1.144
192.168.1.141
启动集群
/home/qbit/hadoop-3.2.0/sbin/start-dfs.sh