基础准备（Hadoop/Spark/Mahout安装准备）

1、基础准备

1.1 操作系统

CentOS6，或者ubuntu 12。

1.2 JDK安装

1.2.1检查本机已安装JDK版本

卸载系统自带的jdk版本及旧版本。

[root @master ~]$rpm -qa |grep java

tzdata-java-2013b-1.el6.noarch

java-1.6.0-openjdk-1.6.0.0-1.61.1.11.11.el6_4.x86_64

java-1.7.0-openjdk-1.7.0.19-2.3.9.1.el6_4.x86_64

#卸载Openjdk

[root @master ~]$rpm -e --nodeps java-1.7.0-openjdk-1.7.0.19-2.3.9.1.el6_4.x86_64

[root @master ~]$rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.61.1.11.11.el6_4.x86_64

[root @master ~]$ rpm-e --nodeps tzdata-java-2013b-1.el6.noarch

1.2.2下载JDK

JDK1.7官方下载页面：

http://www.oracle.com/technetwork/cn/java/javase/downloads/jdk7-downloads-1880260-zhs.html

选择相应版本下载，64位的JDK：

http://www.oracle.com/technetwork/cn/java/javase/downloads/jdk7-downloads-1880260-zhs.html

1.2.3解压及环境变量配置

#1 rpm 安装

[root @master ~]$ chmod755 ./jdk-7u15-linux-x64-demos.rpm

[root @master ~]$rpm -ivh jdk-7u15-linux-x64-demos.rpm

#2 或者tar 安装

[root @master ~]$tar zxvf ./jdk-7u15-linux-x64-demos.tar.gz  -C /usr/local /

 

环境变量配置

[root@master ~]$ vi /etc/profile

添加：

#jdk

exportJAVA_HOME=/usr/local/jdk1.7.0_03

exportCLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

exportPATH=$JAVA_HOME/bin:$PATH

保存退出，输入以下命令使之立即生效

[root@master ~]$ source /etc/profile

测试下JDK是否安装成功

[root@master ~]$java -version 

java version"1.7.0_03"

Java(TM) SE RuntimeEnvironment (build 1.7.0_03-b04)

Java HotSpot(TM)64-Bit Server VM (build 22.1-b02, mixed mode)

 

1.3 Maven安装

1.3.1下载Maven

Manen官方下载页面：http://maven.apache.org/download.cgi

选择相应版本下载，Maven 3.2.3 (Binary tar.gz)：

http://mirror.bit.edu.cn/apache/maven/maven-3/3.2.3/binaries/apache-maven-3.2.3-bin.tar.gz

1.3.2 Maven解压

[root@master ~]$tar zxvf ./ apache-maven-3.2.3-bin.tar.gz

拷贝至目录

[root@master ~]$ cpapache-maven-3.2.3 /usr/local/ maven-3.2.3

1.3.3 环境变量配置

[root@master ~]$ vi /etc/profile

添加：

#maven

export MAVEN_HOME=/usr/local/ maven-3.2.3

export PATH=$MAVEN_HOME/bin:$PATH

保存退出，输入以下命令使之立即生效

[root@master ~]$ source /etc/profile

 

测试

[root@master ~]$ mvn -v 

Apache Maven 3.2.3 (45f7c06d68e745d05611f7fd14efb6594181933e;2014-06-17T21:51:42+08:00)

Maven home: /home/jb-huangmeiling/lib/maven-3.2.2

Java version: 1.7.0_03, vendor: Oracle Corporation

Java home: /usr/local/jdk1.7.0_03/jre

Default locale: en_US, platform encoding: ANSI_X3.4-1968

OS name: "linux", version: "2.6.32-71.el6.x86_64",arch: "amd64", family: "unix"

 

给maven添加本地仓库

[root@master ~]$ vi settings.xml

 找到第53行，把注释去掉，修改成：

<localRepository> /usr/data/maven/maven-dependcies</localRepository>

 

由于maven国外服务器可能连不上，先给maven配置一下国内镜像，在maven目录下，conf/settings.xml,在<mirrors></mirros>里添加，原本的不要动:

<mirror>  

     <id>nexus-osc</id>  

      <mirrorOf>*</mirrorOf>  

  <name>Nexusosc</name>  

  <url>http://maven.oschina.net/content/groups/public/</url>  

</mirror>  

同样，在<profiles></profiles>内新添加:

<profile>  

       <id>jdk-1.7</id>  

       <activation>  

         <jdk>1.4</jdk>  

       </activation>  

       <repositories>  

         <repository>  

           <id>nexus</id>  

           <name>local private nexus</name>  

           <url>http://maven.oschina.net/content/groups/public/</url>  

           <releases>  

             <enabled>true</enabled>  

           </releases>  

           <snapshots>  

             <enabled>false</enabled>  

           </snapshots>  

         </repository>  

       </repositories>  

       <pluginRepositories>  

         <pluginRepository>  

           <id>nexus</id>  

          <name>local private nexus</name>  

           <url>http://maven.oschina.net/content/groups/public/</url>  

           <releases>  

             <enabled>true</enabled>  

           </releases>  

           <snapshots>  

             <enabled>false</enabled>  

           </snapshots>  

         </pluginRepository>  

       </pluginRepositories>  

     </profile>  

 

1.4 系统配置

1.4.1关闭防火墙

[root@master ~]$service iptables stop

#永久关闭

[root@master ~]$chkconfig iptables on/off

 

1.4.2关闭selinux

[root@master ~]$setenforce 0

[root@master ~]$ vi/etc/selinux/config

SELINUX=enforcing

#改为

SELINUX= disabled

 1.4.3时间同步

[root@master ~]$ cp/usr/share/zoneinfo/Asia/Shanghai /etc/localtime

[root@master ~]$ntpdate us.pool.ntp.org

#如示安装ntpdate，可以使用yum install ntpdate

[root@master ~]$ yuminstall ntpdate

 1.4.5配置域名(hosts)

[root@master ~]$ vi/etc/sysconfig/network

HOSTNAME=localhost

#改为

HOSTNAME=master

 1.4.5 配置主机名

[root@master ~]$ vi /etc/hosts

#增加如下信息，此处根据自己实际IP进行配置，其它slave节点也需要配置该信息

192.168.2.140 master.hadoop

192.168.2.141 slave1.hadoop

192.168.2.142 slave2.hadoop

1.4.2.143        slave3.hadoop

 1.4.6 SSH配置

[root@master ~]$ ssh-keygen -t rsa

#下面直接回车既可

Generating public/private rsa key pair.

Enter file in which to save the key(/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in/root/.ssh/id_rsa.

Your public key has been saved in/root/.ssh/id_rsa.pub.

The key fingerprint is:

5b:59:bf:2e:20:bf:d6:81:a9:ae:8d:3e:66:78:38:03root@master

The key's randomart image is:

+--[ RSA 2048]----+

|                |

|                |

|           .    |

|          o .   |

|        S oo .  |

|  E     .o+ . . |

|   . o  .+ o .. |

|    = =o. oo.   |

|    B++o... ..  |

+-----------------+

[root@master ~]$ chmod 700 .ssh

[root@master ~]$ cat ~/.ssh/id_rsa.pub>>~/.ssh/authorized_keys

[root@master ~]$ chmod 600 ~/.ssh/authorized_keys

#修改无密码登陆配置文件

[root@master ~]$ vi /etc/ssh/sshd_config

#默认前面都有个 "#"， 去掉既可

RSAAuthentication yes

PubkeyAuthentication yes

AuthorizedKeysFile .ssh/authorized_keys

#测试ssh是否可用

[root@master ~]# ssh localhost

#如此处不需要输入密码， 说明配置成功。 重复以上步骤完成其它节点上SSH无密码登陆

#将生成的公钥复制到其它节点上

[root@master ~]# scp ~/.ssh/id_rsa.pubslave1.hadoop:/root/

#将公钥追加到authorized_keys中

[root@master ~]#cat ~/id_rsa.pub>>~/.ssh/authorized_keys

 1.4.7启动httpd

[root@master ~]#service httpd start

 

转载请注明出处：

http://blog.csdn.net/sunbow0/article/details/41961403