Centos搭建hadoop集群

搭建hadoop之前保证已经配置好集群机器间免密登录并搭建好zookeeper

1.解压hadoop-3.0.0.tar.gz 到 /opt/software目录下

以安装 hadoop-3.0.0.tar.gz 为例, 下载地址 Index of /dist/hadoop/core/hadoop-3.0.0

tar -xvf hadoop-3.0.0.tar.gz
mv hadoop-3.0.0 /opt/software

如果解压缩后权限不一致, 可以采用如下解压命令

tar --no-same-owner -zxf hadoop-3.0.0.tar.gz

2.创建相关的文件夹

mkdir /hdata
mkdir /hdata/hadoop_data
mkdir /hdata/hadoop_data/datanode
mkdir /hdata/hadoop_data/namenode
mkdir /hdata/hadoop_data/temDir

3.编辑/etc/profile,增加hadoop环境变量

vim /etc/profile

加入:

#hadoop
export HADOOP_PREFIX=/opt/software/hadoop-3.0.0
export HADOOP_HOME=/opt/software/hadoop-3.0.0
export HADOOP_HDFS_HOME=/opt/software/hadoop-3.0.0
export HADOOP_CONF_DIR=/opt/software/hadoop-3.0.0/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=/opt/software/hadoop-3.0.0/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

执行:

source /etc/profile

4.修改 hadoop-env.sh 配置

cd  /opt/software/hadoop-3.0.0/etc/hadoop
vim hadoop-env.sh

增加以下内容:

export JAVA_HOME=/usr/java/jdk1.8.0_151
export HDFS_NAMENODE_USER="root"
export HDFS_DATANODE_USER="root"
export HDFS_SECONDARYNAMENODE_USER="root"
export YARN_RESOURCEMANAGER_USER="root"
export YARN_NODEMANAGER_USER="root"

5.编辑 core-site.xml

vim core-site.xml

增加以下内容:

<configuration>
 <property>
     <name>fs.defaultFS</name>
     <value>hdfs://pure.evil1:9000</value>
   </property>
   <property>
     <name>hadoop.proxyuser.root.hosts</name>
     <value>*</value>
   </property>
   <property>
     <name>hadoop.proxyuser.root.groups</name>
     <value>*</value>
   </property>
  <property>
      <name>hadoop.tmp.dir</name>
      <value>/hdata/hadoop_data/temDir</value>
  </property>
  <property>
     <name>fs.hdfs.impl</name>
     <value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
     <description>The FileSystem for hdfs: uris.</description>
   </property>
</configuration>

6.编辑 hdfs-site.xml

vim hdfs-site.xml

增加以下内容:

<configuration>
<property>
   <name>dfs.replication</name>
   <value>2</value>
</property>
<property>
   <name>dfs.namenode.name.dir</name>
   <value>/hdata/hadoop_data/namenode</value>
</property>
<property>
   <name>dfs.datanode.data.dir</name>
   <value>/hdata/hadoop_data/datanode</value>
</property>
<property>
   <name>dfs.namenode.http-address</name>
   <value>pure.evil1:50070</value>
</property>
</configuration>

7.编辑 mapred-site.xml

vim mapred-site.xml

增加以下内容:

<configuration>
<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

8.编辑 yarn-site.xml

vim yarn-site.xml

增加以下内容:

<configuration>
<!-- Site specific YARN configuration properties -->
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>pure.evil1</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>
<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>49152</value>
</property>
<property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>49152</value>
</property>
</configuration>

9.编辑workers文件

vim workers

加入:

pure.evil1
pure.evil2
pure.evil3

10.将hadoop-3.0.0复制到其他节点 /opt/software目录下

scp -r hadoop-3.0.0 root@192.168.1.1:/opt/software
scp -r hadoop-3.0.0 root@192.168.1.2:/opt/software
scp -r hadoop-3.0.0 root@192.168.1.3:/opt/software

11.依次重复2, 3步

建立相关的文件夹

修改其他节点的/etc/profile文件，并使用source命令使修改生

12.在主节点上格式化namenode

最好只格式化一次,格式化多次容易出现问题

hadoop namenode -format

13.启动hadoop集群(在主节点执行)

cd /opt/software/hadoop-3.0.0/sbin
./start-all.sh

14 相关测试命令

hadoop fs -mkdir /test

hadoop fs -put realtime.csv /test

hadoop fs -cat /test/realtime.csv

hadoop fs -ls /test