Hadoop 初级安装

1、配置hosts，假设你linux的地址是192.168.1.123，且在windows中能够正常访问.用root用户修改/etc/hosts文件，添加如下解析

 

192.168.1.123 single.hadoop.com

因Hadoop使用端口比较多，建议关闭防火墙避免出现不必要的问题，生产环境中可以对相应端口做安全控制

 

[root@single-hadoop-dajiangtai-com ~]# service iptables stop

[root@single-hadoop-dajiangtai-com ~]# chkconfig iptables off

 

注意：有的同学可能装的是CentOS7，这时要注意，CentOS 7.0默认使用的是firewall作为防火墙，关闭firewall使用如下命令

 

[root@single-hadoop-dajiangtai-com ~]# systemctl stop firewalld.service #停止firewall

[root@single-hadoop-dajiangtai-com ~]# systemctl disable firewalld.service #禁止firewall开机启动

 

 

2、准备Hadoop专用用户和组

 

[root@single-hadoop-dajiangtai-com ~]# groupadd hadoop //创建用户组

[root@single-hadoop-dajiangtai-com ~]# useradd -g hadoop hadoop //新建hadoop用户并增加到hadoop工作组

[root@single-hadoop-dajiangtai-com ~]# passwd hadoop //设置密码

//为hadoop设置密码：dajiangtai

3、下载并解压Hadoop2.2.0，小讲是把文件放到/usr/java/目录下，注意：下载前确认一下你的Linux系统是64位系统还是32位系统，分别下载对应的版本，如果下载错了，后面会有很多问题

 

32位系统：

 

[root@single-hadoop-dajiangtai-com ~]$ cd /usr/java/ //root用户下，将 hadoop 下载到/usr/java目录下

[root@single-hadoop-dajiangtai-com java]$ wget http://hadoop.f.dajiangtai.com/hadoop2.2/hadoop-2.2.0.tar.gz //在线下载 hadoop-2.2.0.tar.gz

[root@single-hadoop-dajiangtai-com java]$ tar zxvf hadoop-2.2.0.tar.gz //解压

[root@single-hadoop-dajiangtai-com java]$ chown -R hadoop:hadoop hadoop-2.2.0 //将hadoop-2.2.0操作权限赋给hadoop用户

[root@single-hadoop-dajiangtai-com java]$ rm hadoop-2.2.0.tar.gz //删除hadoop安装包

64位系统：

 

[root@single-hadoop-dajiangtai-com ~]$ cd /usr/java/

[root@single-hadoop-dajiangtai-com java]$ wget http://hadoop.f.dajiangtai.com/hadoop2.2/hadoop-2.2.0-x64.tar.gz //在线下载 hadoop-2.2.0-x64.tar.gz

[root@single-hadoop-dajiangtai-com java]$ tar zxvf hadoop-2.2.0-x64.tar.gz //解压

[root@single-hadoop-dajiangtai-com java]$ chown -R hadoop:hadoop hadoop-2.2.0-x64 //将hadoop-2.2.0-x64操作权限赋给hadoop用户

[root@single-hadoop-dajiangtai-com java]$ rm hadoop-2.2.0-x64.tar.gz //删除hadoop安装包

4、创建Hadoop数据目录。

 

[root@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ mkdir -p /data/dfs/name

[root@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ mkdir -p /data/dfs/data

[root@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ mkdir -p /data/tmp

[root@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ chown -R hadoop:hadoop /data //将/data文件权限赋给hadoop

5、修改etc/hadoop/core-site.xml配置文件，添加如下信息。

 

[hadoop@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ vi etc/hadoop/core-site.xml

 

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://single.hadoop.com:9000</value>

</property>

<property>

<name>io.file.buffer.size</name>

<value>131072</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>file:/data/tmp</value>

<description>Abase for other temporary directories.</description>

</property>

<property>

<name>hadoop.proxyuser.hadoop.hosts</name>

<value>*</value>

</property>

<property>

<name>hadoop.proxyuser.hadoop.groups</name>

<value>*</value>

</property>

</configuration>

6、修改etc/hadoop/hdfs-site.xml配置文件，添加如下信息。

 

[hadoop@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ vi etc/hadoop/hdfs-site.xml

 

<?xml version="1.0" encoding="UTF-8"?>

<configuration>

<property>

<name>dfs.namenode.name.dir</name>

<value>/data/dfs/name</value>

<description>Determineswhere on the local filesystem the DFS name node should store the name table. Ifthis is a comma-delimited list of directories then the name table is replicatedin all of the directories, for redundancy. </description>

<final>true</final>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>/data/dfs/data</value>

<description>Determineswhere on the local filesystem an DFS data node should store its blocks. If thisis a comma-delimited list of directories, then data will be stored in all nameddirectories, typically on different devices.Directories that do not exist areignored.

</description>

<final>true</final>

</property>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.permissions</name>

<value>false</value>

</property>

</configuration>

7、修改etc/hadoop/mapred-site.xml配置文件，添加如下信息。

 

[hadoop@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ vi etc/hadoop/mapred-site.xml

 

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

8、修改etc/hadoop/yarn-site.xml配置文件，添加如下信息。

 

[hadoop@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ vi etc/hadoop/yarn-site.xml

 

<?xml version="1.0"?>

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

9、设置etc/hadoop/slaves，添加如下信息。

 

[hadoop@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ vi etc/hadoop/slaves

 

single.hadoop.dajiangtai.com

10、设置Hadoop环境变量，使用root账户创建/etc/profile.d/hadoop.sh 文件，并输入如下内容

 

[root@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]# vi /etc/profile.d/hadoop.sh

32位系统

 

HADOOP_HOME=/usr/java/hadoop-2.2.0

PATH=$HADOOP_HOME/bin:$PATH

export HADOOP_HOME PATH

64位系统

 

HADOOP_HOME=/usr/java/hadoop-2.2.0-x64

PATH=$HADOOP_HOME/bin:$PATH

export HADOOP_HOME PATH

SSH无密码验证配置

由于hadoop需要无密码登录作为datanode的节点，而由于部署单节点的时候，当前节点既是namenode又是datanode，所以此时需要生成无密码登录的ssh。方法如下：

 

[root@single-hadoop-dajiangtai-com ~]# su hadoop //切换到 hadoop用户下

[hadoop@single-hadoop-dajiangtai-com root]$ cd //直接输入cd会自动切换到/home/hadoop根目录下

[hadoop@single-hadoop-dajiangtai-com ~]$ mkdir .ssh //创建‘.ssh’目录

[hadoop@single-hadoop-dajiangtai-com ~]$ ssh-keygen -t rsa //输入此命令后，一直按 Enter 键

[hadoop@single-hadoop-dajiangtai-com ~]$ cd .ssh //切换到 .ssh 目录下

[hadoop@single-hadoop-dajiangtai-com .ssh]$ cp id_rsa.pub authorized_keys //把生成的 id_rsa.pub 复制一份，命名为authorized_keys

[hadoop@single-hadoop-dajiangtai-com .ssh]$ cd .. //后退到根目录下

[hadoop@single-hadoop-dajiangtai-com ~]$ chmod 700 .ssh //.ssh 文件夹权限必须是 700

[hadoop@single-hadoop-dajiangtai-com ~]$ chmod 600 .ssh/* //“.ssh” 里面的文件

[hadoop@single-hadoop-dajiangtai-com ~]$ ssh single.hadoop.dajiangtai.com //第一次登陆需要密码，第二次以后登录就不需要密码，此时表明设置成功

测试运行

1、格式化文件系统，并启动Hadoop

 

[hadoop@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ bin/hadoop namenode -format

[hadoop@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ sbin/start-all.sh

2、检查一个各个守护进程是否已经启动，在linux命令行中执行jps命令，如果成功会看到以下几个进程

 

[hadoop@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ jps

2336 Jps

1819 SecondaryNameNode

1561 NameNode

2052 NodeManager

1951 ResourceManager

1653 DataNode

3、在Windows上可以通过 http://192.168.1.123:50070 访问WebUI，查看NameNode，集群、文件系统的状态。

 

4、创建一个文件到HDFS中，执行:bin/hadoop fs -mkdir /haiyan 命令

 

[hadoop@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ bin/hadoop fs -mkdir /haiyan

查看HDFS文件系统，执行:bin/hadoop fs -ls / 命令，出现如下结果:

 

[hadoop@single-hadoop-dajiangtai-com hadoop-2.2.0-x64]$ bin/hadoop fs -ls /

Found 1 items

drwxr-xr-x - hadoop supergroup 0 2015-01-16 23:39 /haiyan

5、执行自带的wordcount程序，首先在/home/hadoop用户目录下建一个djt.txt文件，内容如下。

 

Hi this is Haiyan

Haiyan is an IT study platform

Hello hadoop

Hello Haiyan

把创建好的djt.txt文件上传到HDFS系统的/haiyan 目录，命令如下：

 

[hadoop@single-hadoop-com hadoop-2.2.0-x64]$ bin/hadoop fs -put /home/hadoop/djt.txt /haiyan

查看文件是否上传成功，结果如下：

 

textpop-up

[hadoop@single-hadoop-com hadoop-2.2.0-x64]$ bin/hadoop fs -ls /haiyan

Found 1 items

-rw-r--r-- 1 hadoop supergroup 87 2015-01-16 23:57 /haiyan/djt.txt

转载于:https://my.oschina.net/wind2015/blog/1517655