Hadoop环境搭建

需要准备的软件和压缩包

• hadoop,hbase,hive压缩包
  链接：https://pan.baidu.com/s/14JgPUQr8Uu7DKXugqT_-MA
  提取码：8p0x
  
• 安装并配置jdk
  具体教程:https://blog.csdn.net/qq_40375298/article/details/106817387

具体步骤

• 首先修改IP地址,主机名和主机列表

  //修改IP地址
  vi /etc/sysconfig/network-scripts/ifcfg-enp0s3
  //修改主机名
  vi /etc/hostname
  //修改主机列表
  vi /etc/hosts

• 生成密钥
  ssh-keygen -t rsa -P “”
  cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

• 配置解压后的Hadoop文件下的etc/hadoop目录下的文件
  1.hadoop-env.sh
  修改JAVA_HOME的jdk安装目录

 	The java implementation to use.
	export JAVA_HOME=/opt/jdk1.8.0_221

2. core-site.xml

<configuration>
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://192.168.56.10:9000</value> //这里需要填写当前虚拟机的IP地址
</property>
<property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/tmp</value>				//这里是临时文件的地址,需要有这个文件夹	
</property>
<property>
        <name>hadoop.proxyuser.root.hosts</name>
        <value>*</value>
</property>
<property>
        <name>hadoop.proxyuser.root.groups</name>
        <value>*</value>
</property>
</configuration>

3. hdfs-site.xml

<configuration>
<property>
        <name>dfsl.replication</name>   //
        <value>3</value>
</property>
<property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>zcy00:50090<>
</property>

4. mapred-site.xml

cp mapred-site.xml.template mapred-site.xml
rm -rf mapred-site.xml.template
vi mapred-site.xml

<configuration>
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>
<property>
        <name>mapreduce.jobhistory.address</name>
        <value>192.168.56.10:10020</value>
</property>
<property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>192.168.56.10:19888</value>
</property>

5. yarn-site.xml

<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
</property>
<property>
        <name>yarn.nodemanager.aux-services.mapreduce.shffle.class</name>
        <value>org.apche.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
        <name>yarn.resourcemanager.hostname</name>
        <value>zcy00</value>
</property>
<property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
</property>
<property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
</property>

6.配置Hadoop的环境变量
	vi /etc/profile

export HADOOP_HOME=/opt/hadoop					//hadoop的安装目录
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

配置完成后需要 source /etc/profile

6.格式化HDFS
	hadoop namenode -format     //如果前面配置文件有错这里会提示报错的
7.启动hadoop
	start-all.sh
 	可以用jps命令来查看模块是否安装全

8.访问Hadoop
http://192.168.56.137:50070 HDFS页面
http://192.168.56.137:8088 YARN的管理界面
http://192.168.56.126:19888/