配置Hadoop

1. 安装jdk

   1. 从甲骨文官网下载对应系统的jdk安装文件

   2. 将jdk移动到/usr/local/java目录下

      mkdir /etc/local/java
      mv ./jdk-8u77-linux-x64.tar.gz /usr/local/java/
      cd /usr/local/java

   3. 解压

        tar xvf ./jdk-8u77-linux-x64.tar.gz
      

   4. 修改环境变量
      vi /etc/profile在最底部添加

      export JAVA_HOME=/usr/local/java/jdk1.8.0_75
      export JRE_HOME=/usr/local/java/jdk1.8.0_75/jre
      export PATH=$PATH:/usr/local/java/jdk1.8.0_75/bin
      export CLASSPATH=./:/usr/local/java/jdk1.8.0_75/lib:/usr/local/java/jdk1.7.0_75/jre/lib

   5. 更新profile文件

      source /etc/profile

2. 安装Hadoop

   1. 下载Hadoop：从[Apache官网下载]binary格式(http://hadoop.apache.org/releases.html)Hadoop安装包;
   2. 使用tar -xvf hadoop.xxx 指令解压hadoop，将解压出来的文件夹移动到某个位置：cd hadoop.xxx ~/hadoop

   3. 更新/etc/profile文件：

      export HADOOP_INSTALL=/Users/weixin/hadoop
      export PATH=/opt/local/bin:$PATH:$HADOOP_INSTALL/bin
      输入source /etc/profile立即更新profile文件设置。

   4. 检验是否安装成功：
      

3. hadoop伪分布式配置

   1. 配置$HADOOP_INSTALL/etc/hadoop/core-site.xml:

      <configuration>
        <property>
          <name>hadoop.tmp.dir</name>
          <value>hdfs://localhost:9000</value>
          <description>A base for other temporary directories.</description>
        </property>
        <property>
          <name>fs.default.name</name>
           <value>hdfs://localhost:8020</value>
        </property>
      </configuration>
      

   2. 配置$HADOOP_INSTALL/etc/hadoop/hdfs-site.xml:

      <configuration>
        <property>
          <name>dfs.replication</name>
          <value>1</value>
        </property>
      
      </configuration>
      

   3. 配置$HADOOP_INSTALL/etc/hadoop/mapred-site.xml:

      <configuration>
        <property>
          <name>mapred.job.tracker</name>
          <value>hdfs://localhost:9001<value>
        </property>
        <property>
          <name>mapred.tasktracker.map.tasks.maximum</name>
          <value>2</value>
        </property>
        <property>
          <name>mapred.tasktracker.reduce.tasks.maximum</name>
          <value>2</value>
        </property>
      
      </configuration>
      

   4, 配置ssh免密码登录

       ssh-keygen -t rsa -P '' ~/.ssh/id_rsa
       cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
   

   如果文件~/.ssh/authorized_keys不存在，则通过touch ~/.ssh/authorized_keys新建
   通过ssh localhost来测试是否还需要密码登录。

   1. 格式化文件系统： hadoop namenode -format