Hadoop全分布

• 连接xshell跟xftp
• 创建文件夹
  • mkdir h3cu
• 使用xftp上传hadoop包跟jdk包至h3cu
• 解压两个包至/usr/local/src/ 并修改文件夹名称
  • tar -zxvf jdk-8u91-linux-x64.tar.gz -C /usr/local/src/
  • tar -zxvf hadoop-2.6.0-cdh5.15.1.tar.gz -C /usr/local/src/
  • cd /usr/local/src/
  • mv hadoop~/ hadoop
  • mv jdk~/ jdk
• 修改.bash_profile
  • vi ~/.bash_profile
  • export JAVA_HOME=/usr/local/src/jdk
  • export HADOOP_HOME=/usr/local/src/hadoop
  • export PATH=$JAVA_HOME/bin:$PATH
  • export PATH=$HADOOP_HOME/bin:$PATH
  • export PATH=$HADOOP_HOME/sbin:$PATH
• 验证变量
  • source ~/.bash_profile
  • java -version 查看java
  • hadoop 查看hadoop
• 配置hadoop文件
  • cd /usr/local/src/hadoop/etc/hadoop
  • vi hadoop-env.sh
    • 修改jdk路径 = /usr/local/src/jdk
  • vi core-site.xml
    • <property>
      • <name>fs.defaultFS</name>
      • <value>hdfs://centos001:9000</value>
    • </property>
  • vi hdfs-site.xml
    • <property>
      • <name>dfs.namenode.name.dir</name>
      • <value>/usr/local/src/hadoop/namenode</value>
    • </property>
    • <property>
      • <name>dfs.namenode.data.dir</name>
      • <value>/usr/local/src/hadoop/disk1</value>
    • </property>
    • <property>
      • <name>dfs.namenode.data.dir</name>
      • <value>/usr/local/src/hadoop/disk2</value>
    • </property>
  • vi yarn-site.xml
    • <property>
      • <name>yarn.resourcemanager.hostname</name>
      • <value>centos001</value>
    • </property>
    • <property>
      • <name>yarn.nodemanager.aux-services</name>
      • <value>mapreduce_shuffle</value>
    • </property>
    • <property>
      • <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
      • <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    • </property>
  • cp mapred-site.xml.template mapred-site.xml
    • <property>
      • <name>mapreduce.framework.name</name>
      • <value>yarn</value>
    • </property>
  • vi slaves
    • centos001
    • centos002
    • centos003
• 分发配置好的文件
  •  scp -r /usr/local/src/hadoop/ /usr/local/src/jdk/ centos002:/usr/local/src
  •  scp -r /usr/local/src/hadoop/ /usr/local/src/jdk/ centos003:/usr/local/src
  • scp -r ~/.bash_profile centos002:~
  • scp -r ~/.bash_profile centos003:~
• 格式化namenode
  • hdfs namenode -format
• 启动Hadoop
  • start-dfs.sh
  • start-yarn.sh
  • jps查看节点
    • ResourceManager
    • NodeManager
    • Jps
    • NameNode
    • DataNode
• 进入网站查看nnamenode页面
  • ip:50070

  •