Hadoop cdh版本配置

               Hadoop cdh版本配置

1.  准备3台虚拟机 192.168.48.121  iflysse01 192.168.48.122  iflysse02 192.168.48.123  iflysse03 在iflysse01创建图中文件夹并把tar包放到cdh中

2.  vim /etc/sysconfig/network

3.  vim /etc/hosts

4.  vim/etc/sysconfig/network-scripts/ifcfg-eth0

Init 0 关机重启

查看ifconfig

打开C:\Windows\System32\drivers\etc\hosts

修改权限（允许）

在控制台ping 名称和ip

安装jdk环境和hadoop环境 进入/opt/softwares/cdh/文件夹 解压tar -zxvf jdk-7u79-linux-i586.tar.gz -C /opt/app/ tar -zxvf hadoop-2.5.0-cdh5.3.6.tar.gz -C /opt/app/

vim/etc/profile或者打开profile

测试hadoop环境 在opt下新建data文件夹touch hello.txt->vim hello.txt hadood jar /opt/app/hadoop2.5.0cdh5.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.6.jar wordcount file:///opt/data/hello.txt file:///opt/data/out

配置：

    *hdfs

        * hadoop-env.sh

        * core-site.xml

<configuration> <property>         <name>fs.defaultFS</name>         <value>hdfs://iflysse01:8020</value>     </property>     <property>         <name>hadoop.tmp.dir</name>         <value>/opt/app/hadoop-2.5.0-cdh5.3.6/data/tmp</value>     </property>     <property>         <name>fs.trash.interval</name>         <value>420</value>     </property> </configuration>

        * hdfs-site.xml

<configuration>   <property>         <name>dfs.namenode.secondary.http-address</name>         <value>iflysse03:50090</value>     </property> </configuration>

        * slaves

iflysse01 iflysse02 iflysse03

    *yarn

        * yarn-env.sh

      

 * yarn-site.xml

<configuration>   <property>         <name>yarn.resourcemanager.hostname</name>         <value>iflysse02</value>     </property>     <property>         <name>yarn.nodemanager.aux-services</name>         <value>mapreduce_shuffle</value>     </property>     <property>         <name>yarn.log-aggregation-enable</name>         <value>true</value>     </property>     <property>         <name>yarn.log-aggregation.retain-seconds</name>         <value>420</value>     </property> </configuration>

        * slaves

    *mapredue

        * mapred-env.sh

export JAVA_HOME=/opt/app/jdk1.7.0_79

        * mapred-site.xml

<configuration>    <property>         <name>mapreduce.framework.name</name>         <value>yarn</value>     </property>     <property>         <name>mapreduce.jobhistory.address</name>         <value>iflysse01:10020</value>     </property>     <property>         <name>mapreduce.jobhistory.webapp.address</name>         <value>iflysse01:19888</value>     </property> </configuration>

在iflysse01上配置完hadoop之后，需要把hadoop的安装目录远程拷贝到其他节点 切换到 cd /opt/app下 scp -r hadoop-2.5.0-cdh5.3.6/ root@iflysse02:/opt/app/  scp -r hadoop-2.5.0-cdh5.3.6/ root@iflysse03:/opt/app/

设置免密码登录SSH

Cd /root/.ssh/     Ssh-keygen –t rsa 然后四个回车 Ssh-copy-id iflysse01第一次要输入密码 Ssh-copy-id iflysse02 Ssh-copy-id iflysse03 在iflysse02也要进行相同的操作

时间同步

关闭防火墙，service iptables stop

# rpm -qa|grep ntp # vi /etc/sysconfig/ntpd     # Drop root to id 'ntp:ntp' by default.     SYNC_HWCLOCK=yes     OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"     [root@iflysse01 hadoop-2.5.0-cdh5.3.6]# service ntpd status ntpd is stopped [root@iflysse01 hadoop-2.5.0-cdh5.3.6]# service ntpd start Starting ntpd: [  OK  ] [root@iflysse01 hadoop-2.5.0-cdh5.3.6]# chkconfig ntpd on [root@iflysse01 hadoop-2.5.0-cdh5.3.6]# crontab -e 0-59/10 * * * * /usr/sbin/ntpdate iflysse01

格式化

Hdfs namenode –format

启动进程

Start-dfs.sh

Start-yarn.sh

mr-jobhistory-daemon.shstart historyserver单独启动