hadoop安装笔记

Hadoop集群搭建

Hadoop版本—2.6.0稳定版

所需软件

Linux和Windows所需软件包括:

1.   Java^TM1.5.x，必须安装，建议选择Sun公司发行的Java版本。

2.   ssh 必须安装并且保证 sshd一直运行，以便用Hadoop 脚本管理远端Hadoop守护进程。

Windows下的附加软件需求

1.   Cygwin - 提供上述软件之外的shell支持。

安装软件

安装ssh,rsync及其他依赖

yum -yinstall  openssh-clients gcc-c++ vim   make gcc unzip  wget subversion  openssh zlib  zlib-devel  rsync

rsync –version

解压

tar -zxvf hadoop-2.6.0.tar.gz

单机版本安装测试

编辑etc/hadoop/hadoop-env.sh 

#set to the root of your Java installation

  export JAVA_HOME=/usr/java/latest

 

  # Assuming your installation directory is/usr/local/hadoop

  export HADOOP_PREFIX=/usr/local/hadoop

执行bin/hadoop

显示帮助文档

单机操作

$ mkdir input

  $ cp etc/hadoop/*.xml input

  $ bin/hadoop jarshare/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output'dfs[a-z.]+'

  $ cat output/*

 

$ bin/hadoop jarshare/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output1'con[a-z.]+'

上面的操作是打包conf里的配置作为input然后根据正则表达式去匹配,将结果集写入output.

单机伪分布式模式测试

配置

etc/hadoop/core-site.xml:

<configuration>

    <property>

       <name>fs.defaultFS</name>

       <value>hdfs://localhost:9000</value>

    </property>

</configuration>

 

etc/hadoop/hdfs-site.xml:

<configuration>

    <property>

       <name>dfs.replication</name>

        <value>1</value>

    </property>

</configuration>

设置ssh免密码登录

测试

ssh localhost

 

必须要输入密码的话执行:

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

  $ cat ~/.ssh/id_dsa.pub >>~/.ssh/authorized_keys

启动

格式化namenode

bin/hdfsnamenode -format

开启namenode和datanode守护进程

sbin/start-dfs.sh

查看namenode状态通过50070端口

创建hdfs所需的hdfs目录用来执行mapreduce

bin/hdfsdfs -mkdir /user

bin/hdfsdfs -mkdir /user/Hadoop

bin/hdfsdfs -ls /

把input放入hdfs

bin/hdfsdfs -mkdir /input

bin/hdfsdfs -put etc/hadoop /input 

跑下例子

bin/hadoopjar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep /inputoutput 'dfs[a-z.]+'

 

bin/hadoopjar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep /input/*output3 'con[a-z.]+'

把结果集从hdfs拿下来

bin/hdfsdfs -get output3 output5

catoutput5/*

停止命令

sbin/stop-dfs.sh

单节点yarn

配置

etc/hadoop/mapred-site.xml:

<configuration>

    <property>

       <name>mapreduce.framework.name</name>

       <value>yarn</value>

    </property>

</configuration>

etc/hadoop/yarn-site.xml:

<configuration>

    <property>

        <name>yarn.nodemanager.aux-services</name>

       <value>mapreduce_shuffle</value>

    </property>

</configuration>

启动ResourceManager和NodeManager守护进程

sbin/start-yarn.sh

 

查看ResourceManager的web页面在端口8088

停止yarn

sbin/stop-yarn.sh

 

完全分布式集群搭建

3台机器

修改hostname

vim/etc/sysconfig/network

修改hostname=yourname

Hostname yourname

修改host

vim/etc/hosts

yourmaster master

yourslaveslave1

yourslaveslave2

scp/etc/hosts root@slave1:/etc/

scp/etc/hosts root@slave2:/etc/

 

添加用户

useradd -u 600 hadoop

passwd Hadoop

su – hadoop

在Slave1执行命令

ssh-keygen-t dsa -P '' -f ~/.ssh/id_dsa

cat~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

 

scp /home/hadoop/.ssh/id_dsa.pub.130 hadoop@master:/home/hadoop/.ssh/

# vi /etc/ssh/sshd_config   #开启RSA认证

RSAAuthentication yes

PubkeyAuthentication yes

AuthorizedKeysFile     .ssh/authorized_keys

#service sshd restart

 

同理slave2

在master执行

catid_dsa.pub.130 >> authorized_keys

cat id_dsa.pub.131>> authorized_keys

权限很重要

chmod 700 .ssh/

chmod 600 .ssh/authorized_keys

scp/home/hadoop/.ssh/authorized_keys  hadoop@slave1:/home/hadoop/.ssh/

scp /home/hadoop/.ssh/authorized_keys  hadoop@slave2:/home/hadoop/.ssh/

在master测试

Ssh slave1 ssh slave2

 

解压hadoop安装包

 

chown hadoop.hadoop -R/export/servers/hadoop-2.6.0

编辑环境变量

vi /etc/profile

HADOOP_HOME=/home/hadoop/hadoop-2.3.0/

PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export HADOOP_HOME PATH

source/etc/profile

编辑slaves

[hadoop@master hadoop]$ vi slaves

slave1

slave2

core-site.xml

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://master:9000</value>

</property>

<property>

<name>io.file.buffer.size</name>

<value>131072</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>file:/export/servers/hadoop-2.6.0/tmp</value>

</property>

</configuration>

 

hdfs-site.xml

<configuration>

<property>

<name>dfs.namenode.name.dir</name>

<value>file:/export/servers/hadoop-2.6.0/dfs/name</value>

</property>

<property>

<name>dfs.namenode.data.dir</name>

<value>file:/export/servers/hadoop-2.6.0/dfs/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>2</value>  //数据副本数量 默认为3

</property>

</configuration>

yarn-site.xml

<configuration>

<property>

<name>yarn.resourcemanager.address</name>

<value>master:8032</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>master:8030</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>master:8031</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address</name>

<value>master:8033</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value>master:8088</value>

</property>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

   <value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

</configuration>

 

mapred-site.xml

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>master:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>master:19888</value>

</property>

</configuration>

格式化文件系统并启动

hdfs namenode –format

启动hdfs文件系统

start-dfs.sh

jps

查看守护进程

352NameNode

531SecondaryNameNode

645 Jps

启动mapreduce架构

start-yarn.sh

查看进程

1870ResourceManager

查看slave

sshslave1

jps

26043 NodeManager

查看

http://192.168.159.129:8088/cluster/nodes

可以用浏览器打开NameNode, ResourceManager和各个NodeManager的web界面，

·        NameNode web UI, http://master:50070/

·        ResourceManager web UI, http://master:8088/

·        NodeManager web UI, http://slave01:8042

验证

hdfs dfs-mkdir /user

hdfs dfs-mkdir /user/hadoop

hdfs dfs-put ./etc/hadoop  input

运行著名的WordCount

hadoopjar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount inputoutput

 

 

查看结果集

hdfs dfs -cat output/*

停止集群

$ stop-yarn.sh

$ stop-hdfs.sh

 

后记:

因为测试环境机器每周都会重启所以经常面临重启集群的情况:

重启时

hdfs namenode -format

hdfs datanode -format

sbin/start-dfs.sh;tail -f logs/*

 sbin/start-yarn.sh;tail -f logs/*

注意查看日志报错