hadoop安装步骤

1、安装centos7

1.1 配置VirtualBox

1.2 centos环境配置

关闭防火墙：

停止防火墙服务
systemctl stop firewalld
禁用防火墙，下次开机启动后防火墙服务不再启动
systemctl disable firewalld

生成秘钥：

ssh-keygen -t rsa -P “”
cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

开启远程免密登录配置：

[root@cjh1 ~]# ssh-copy-id -i .ssh/id_rsa.pub -p22 root@192.168.56.101

远程登录：

[root@cjh1 ~]# ssh -p 22 root@192.168.56.101

往远程服务器拷贝文件：

[root@cjh1 ~]# scp /etc/hosts root@192.168.56.102:/etc/

配置网卡：

[root@localhost .ssh]# cd /etc/sysconfig/network-scripts/

修改：

ifcfg-enp0s3    
	BOOTPROTO=static
	ONBOOT=yes
	IPADDR=192.168.56.101
	
ifcfg-enp0s8
	BOOTPROTO=dhcp
	ONBOOT=yes

配置完成后重启网卡,重新启动网络：

systemctl restart network.service  或者  service network restart

1.3 配置主机名

vi /etc/hostname

cjh1
[root@centos7 ~]$ hostnamectl set-hostname cjh1     # 使用这个命令会立即生效且重启也生效
[root@centos7 ~]$ hostname                          # 查看下cjh1
[root@centos7 ~]$ vi /etc/hosts                     # 编辑下hosts文件， 给127.0.0.1添加hostname
[root@centos7 ~]$ cat /etc/hosts                    # 检查127.0.0.1   localhost

vi /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.56.101  cjh1 

[root@localhost network-scripts]# vi /etc/hostname

1.4 配置JDK环境变量

export JAVA_HOME=/opt/bigdata/jdk180
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

2、配置Hadoop

2.1配置Hadoop ./etc/hadoop目录下的文件

hadoop-env.sh

# The java implementation to use.
export JAVA_HOME=/opt/bigdata/jdk180

core-site.xml

<configuration>
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://cjh1:9000</value>
</property>
<property>
   <name>hadoop.tmp.dir</name>
   <value>/opt/bigdata/hadoop260/hadoop2</value>
</property>
<property>
   <name>hadoop.proxyuser.root.hosts</name>
   <value>*</value>
 </property>
<property>
   <name>hadoop.proxyuser.root.groups</name>
   <value>*</value>
 </property>
</configuration>

hdfs-site.xml

<configuration>
<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>
<property>
  <name>dfs.namenode.secondary.http-address</name>
  <value>cjh1:50090</value>
</property>
</configuration>

mapred-site.xml

<configuration>
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>
<property>
        <name>mapreduce.jobhistory.address</name>
        <value>cjh1:10020</value>
</property>
<property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>cjh1:19888</value>
</property>
</configuration>

yarn-site.xml

<configuration>
<!-- reducer获取数据方式 -->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<!-- 指定YARN的ResourceManager的地址 -->
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>cjh1</value>
</property>
<!-- 日志聚集功能使用 -->
<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>
<!-- 日志保留时间设置7天 -->
<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
</property>
</configuration>

vi ./slaves  
cjh1

2.2 Hadoop环境变量配置

进入配置
vi /etc/profile

export HADOOP_HOME=/opt/bigdata/hadoop260
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

然后source一下使配置生效

source /etc/profile

2.3 格式化HDFS

hadoop namenode -format

2.4 启动hadoop

start-all.sh

启动历史服务

[root@cjh1 hadoop260]# mr-jobhistory-daemon.sh start historyserver

2.5 访问Hadoop

http://192.168.56.101:50070   HDFS页面
http://192.168.56.101:8088   YARN的管理界面
http://192.168.56.101:19888/