Hadoop搭建

改权限chown hadoop:hadoop -R spark-2.2.0-bin-hadoop2.6

1.jdk

tar -zvxf jdk-8u211-linux-x64.tar.gz -C /opt/java/
vi /etc/profile

export JAVA_HOME=/opt/java/jdk1.8.0_211
export JRE_HOME=/opt/java/jdk1.8.0_211/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

source /etc/profile
java -version
scp -r /opt/java root@hadoop2:/opt
scp -r /etc/profile root@hadoop2:/etc

2.Hadoop

1.tar -zvxf hadoop-2.6.5.tar.gz -C /opt/hadoop

2.在目录/opt/hadoop/hadoop-2.6.5/etc/hadoop/修改hadoop-env.sh
jdk目录
# export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/opt/java/jdk1.8.0_211
存放pid
#export HADOOP_PID_DIR=${HADOOP_PID_DIR}
#export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_PID_DIR=/opt/hadoop/pidDir
export HADOOP_SECURE_DN_PID_DIR=/opt/hadoop/pidDir

pid修改集合
A.修改hadoop-env.sh，如果没有相关配置，可用直接添加

export HADOOP_PID_DIR=/opt/hadoop/pidDir export
export HADOOP_SECURE_DN_PID_DIR=/opt/hadoop/pidDir
上述配置，影响NameNode DataNode SecondaryNameNode进程pid存储

B.修改mapred-env.sh

export HADOOP_MAPRED_PID_DIR=/opt/hadoop/pidDir
上述配置，影响JobHistoryServer进程pid存储

C.修改yarn-env.sh

export YARN_PID_DIR=/opt/hadoop/pidDir
上述配置，影响 NodeManager ResourceManager 进程pid存储


3.修改core-site.xml(在configuration标签下配置)

 <configuration>
	<property>
   		<name>fs.defaultFS</name>
    	<value>hdfs://hadoop1:9000</value>
	</property>
</configuration>

4.配置hdfs-site.xml

<configuration>
<property> ---副本数量，应小于datanode
        <name>dfs.replication</name>
        <value>2</value>
</property>
<property>
	<name>dfs.namenode.name.dir</name>
	<value>/opt/hadoop/dfs/name</value>
</property>
<property>
	<name>dfs.datanode.data.dir</name>
	<value>/opt/hadoop/dfs/data</value>
</property>
<property>---把hdfs的权限检测关了
       <name>dfs.permissions.enabled</name>
       <value>false</value>
   </property>
</configuration>

5.配置yarn-site.xml

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.resourcemanager.address</name>
    <value>hadoop1:8032</value>
</property>
<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>hadoop1:8030</value>
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>hadoop1:8031</value>
</property>
<property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>hadoop1:8033</value>
</property>
<property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>hadoop1:8088</value>
</property>
<property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

6.配置slaves
hadoop1
hadoop2

7.环境变量
export JAVA_HOME=/opt/java/jdk1.8.0_211
export JRE_HOME=/opt/java/jdk1.8.0_211/jre
export HADOOP_HOME=/opt/hadoop/hadoop-2.6.5

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

source /etc/profile

8.分发
scp -r /opt/hadoop hadoop2:/opt
scp -r /etc/profile root@hadoop2:/etc
source /etc/profile

9.接着格式化hdfs

在主节点hadoop1上执行以下命令

hdfs namenode -format（想再次格式化的话，一定将hdfs-site.xml配置的路径删干净）

注：命令执行完之后从结果上很难看出是否格式化成功了，这时可以紧接着通过echo $? 命令查看hdfs namenode -format命令是否执行成功。如下，如果输出0，则说明上一条执行成功。

10.启动hdfs，在hadoop1上执行以下命令 50070

cd /sbin
./start-dfs.sh

11.启动yarn，在hadoop1上执行以下命令 8088

cd  /sbin

./start-yarn.sh