hadoop64位编译及开发环境搭建（二）

最新推荐文章于 2022-12-20 12:37:14 发布
最新推荐文章于 2022-12-20 12:37:14 发布
阅读量3.4k 收藏 1
点赞数
分类专栏： java hadoop 文章标签： hadoop 64位 namenode mapreduce yarn
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qyl445/article/details/41623609
版权
java 同时被 2 个专栏收录
12 篇文章 0 订阅
订阅专栏
hadoop
4 篇文章 0 订阅
订阅专栏

上次已经编译好了64位的hadoop可执行文件，现在正式开始安装hadoop。

        一.先准备三台服务器，一台主节点，2台数据节点，分别装上系统，我这里用的系统是centos7.0-minimal 版本。

在这里需要注意的是，在格式化硬盘时，创建swap区。

        二.安装jdk，此处使用jdk1.7；

            1.换目录到/usr 目录下，  cd /usr

    2.创建java  目录 ，  mkdir java

            3.将jkd复制到/usr/java目录     cp jdk-7u55-linux-x64.tar.gz  /usr/java

            4.解压jdk，  tar zxvf   jdk-7u55-linux-x64.tar.gz

            5.配置环境变量   vi /etc/profile

增加记录  ：

export JAVA_HOME=/usr/java/jdk1.7.0_55
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$PATH:$JAVA_HOME/bin

            6.使环境变量生效   source /etc/profile

            7.验证是否生效   java -version

       三.修改主机名称

            1. vi /etc/sysconfig/network    增加如下：

NETWORKING=yes
HOSTNAME=master

           2.vi /etc/hosts   修改并增加后为  

 127.0.0.1         localhost
::1                       localhost
192.168.22.30  master 

192.168.22.31 slave1

192.168.22.32 slave2

后面连个记录是另外两个主机的ip地址

          3.重启reboot    

          4.重启成功后，查看修改是否成功   hostname    看看显示的是不是修改后的机器名称，我的机器就不是，后面发现，还需要修改一个地方，

          5.vi /etc/hostname,  改成master   ，如果4步验证通过，就不用修改了。

把其他2台机器也经行同样的操作。

     四.  制作ssh无密码登录

        1.用hadoop用户登录master主机，

        2.制作无密码密钥    ssh-keygen -t rsa  然后一路回车

    3.将密钥复制到其他机器   ssh-copy-id -i  ~/.ssh/id_rsa.pub hadoop@slave1  ，输入yes，输入slave1的hadoop用户密码

    4.验证：ssh slave1

    5.将密钥拷贝到slave2机器。

    6.对slave1，slave2也执行3，4步的操作，这样就实现了ssh无密码登录

    7.cd ~/.ssh  ,并执行  cat id_rsa.pub >> auth.....自动补全   回车  ，三个机器执行同样的操作，这样就实现了本机无密码ssh登录

  五.hadoop安装  此操作在master机上进行，用hadoop用户

   1.复制  hadoop-2.5.2-x64.tar.gz  到/home/hadoop/ 目录下，解压  tar zxvf hadoop-2.5.2-x64.tar.gz

   2.修改配置文件，cd hadoop-2.5.2/etc/hadoop/  

  3.修改core-site.xml文件     

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/hadoop/tmp</value>
    </property>
   <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>4096</value>
    </property>
</configuration>

    4.修改hdfs-site.xml

<configuration>
    <property>
        <name>dfs.nameservices</name>
        <value>hadoop-cluster1</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>master:50090</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///home/hadoop/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///home/hadoop/dfs/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>
    5.修改mapred-site.xml 

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jbohistory.address</name>
        <value>master:10020</value>
    </property>
    <property>
        <name>mapreduce.jbohistory.webap.address</name>
        <value>master:19888</value>
    </property>
</configuration>
  6.修改yarn-site.xml 

<configuration>

<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>master:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>master:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
       <value><span style="font-family: Arial, Helvetica, sans-serif;">master</span>:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value><span style="font-family: Arial, Helvetica, sans-serif;">master</span>:8088</value>
    </property>
</configuration>
   7.配置运行环境  vi  hadoop-env.sh ， 
# The java implementation to use.
export JAVA_HOME=/usr/java/jdk1.7.0_55
   8.配置slaves        

slave1
slave2
   至此，nameNode配置完成。

下面安装dataNode

   9.安装dataNode  

    cd /home/hadoop

   scp -r hadoop-2.5.2 hadoop@slave1:~/

   scp -r ahdoop-2.5.2 hadoop@slave2:~/

   scp -r tmp hadoop@slave1:~/

  scp -r tmp hadoop@slave2:~/

 10.格式化文件系统

   cd hadoop-2.5.2

   bin/hdfs namenode -format 


14/11/30 19:45:09 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master/127.0.0.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.5.2
STARTUP_MSG:   classpath = /home/hadoop/hadoop-2.5.2/etc/hadoop:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/hadoop-auth-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/junit-4.11.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/hadoop-annotations-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/xz-1.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/avro-1.7.4.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jetty-6.1.26.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/hadoop-common-2.5.2-tests.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/hadoop-nfs-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/common/hadoop-common-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/hadoop-hdfs-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/hdfs/hadoop-hdfs-2.5.2-tests.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jline-0.9.94.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-common-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-api-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-client-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-common-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.2.jar:/home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.2-tests.jar:/contrib/capacity-scheduler/*.jar
STARTUP_MSG:   build = Unknown -r Unknown; compiled by 'hadoop' on 2014-11-28T19:36Z
STARTUP_MSG:   java = 1.7.0_55
************************************************************/
14/11/30 19:45:09 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
14/11/30 19:45:09 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-0c5e9190-2c94-4fa4-85ea-1685ed7fc406
14/11/30 19:45:12 INFO namenode.FSNamesystem: fsLock is fair:true
14/11/30 19:45:12 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
14/11/30 19:45:12 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
14/11/30 19:45:12 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
14/11/30 19:45:12 INFO blockmanagement.BlockManager: The block deletion will start around 2014 Nov 30 19:45:12
14/11/30 19:45:12 INFO util.GSet: Computing capacity for map BlocksMap
14/11/30 19:45:12 INFO util.GSet: VM type       = 64-bit
14/11/30 19:45:12 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
14/11/30 19:45:12 INFO util.GSet: capacity      = 2^21 = 2097152 entries
14/11/30 19:45:13 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
14/11/30 19:45:13 INFO blockmanagement.BlockManager: defaultReplication         = 2
14/11/30 19:45:13 INFO blockmanagement.BlockManager: maxReplication             = 512
14/11/30 19:45:13 INFO blockmanagement.BlockManager: minReplication             = 1
14/11/30 19:45:13 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
14/11/30 19:45:13 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
14/11/30 19:45:13 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
14/11/30 19:45:13 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
14/11/30 19:45:13 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
14/11/30 19:45:13 INFO namenode.FSNamesystem: fsOwner             = hadoop (auth:SIMPLE)
14/11/30 19:45:13 INFO namenode.FSNamesystem: su

pergroup          = supergroup
14/11/30 19:45:13 INFO namenode.FSNamesystem: isPermissionEnabled = true
14/11/30 19:45:13 INFO namenode.FSNamesystem: Determined nameservice ID: hadoop-cluster1
14/11/30 19:45:13 INFO namenode.FSNamesystem: HA Enabled: false
14/11/30 19:45:13 INFO namenode.FSNamesystem: Append Enabled: true
14/11/30 19:45:14 INFO util.GSet: Computing capacity for map INodeMap
14/11/30 19:45:14 INFO util.GSet: VM type       = 64-bit
14/11/30 19:45:14 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
14/11/30 19:45:14 INFO util.GSet: capacity      = 2^20 = 1048576 entries
14/11/30 19:45:14 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/11/30 19:45:14 INFO util.GSet: Computing capacity for map cachedBlocks
14/11/30 19:45:14 INFO util.GSet: VM type       = 64-bit
14/11/30 19:45:14 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
14/11/30 19:45:14 INFO util.GSet: capacity      = 2^18 = 262144 entries
14/11/30 19:45:14 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
14/11/30 19:45:14 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
14/11/30 19:45:14 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
14/11/30 19:45:14 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
14/11/30 19:45:14 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
14/11/30 19:45:14 INFO util.GSet: Computing capacity for map NameNodeRetryCache
14/11/30 19:45:14 INFO util.GSet: VM type       = 64-bit
14/11/30 19:45:14 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
14/11/30 19:45:14 INFO util.GSet: capacity      = 2^15 = 32768 entries
14/11/30 19:45:14 INFO namenode.NNConf: ACLs enabled? false
14/11/30 19:45:14 INFO namenode.NNConf: XAttrs enabled? true
14/11/30 19:45:14 INFO namenode.NNConf: Maximum size of an xattr: 16384
14/11/30 19:45:14 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1378118471-127.0.0.1-1417347914574
14/11/30 19:45:15 INFO common.Storage: Storage directory /home/hadoop/dfs/name has been successfully formatted.
14/11/30 19:45:16 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
14/11/30 19:45:16 INFO util.ExitUtil: Exiting with status 0
14/11/30 19:45:16 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/127.0.0.1
************************************************************/
    11.启动hadoop   

./start-dfs.sh

Starting namenodes on [master]
master: starting namenode, logging to /home/hadoop/hadoop-2.5.2/logs/hadoop-hadoop-namenode-master.out
slave1: starting datanode, logging to /home/hadoop/hadoop-2.5.2/logs/hadoop-hadoop-datanode-slave1.out
slave2: starting datanode, logging to /home/hadoop/hadoop-2.5.2/logs/hadoop-hadoop-datanode-slave2.out
./start-yarn.sh 

starting yarn daemons
starting resourcemanager, logging to /home/hadoop/hadoop-2.5.2/logs/yarn-hadoop-resourcemanager-master.out
slave1: starting nodemanager, logging to /home/hadoop/hadoop-2.5.2/logs/yarn-hadoop-nodemanager-slave1.out
slave2: starting nodemanager, logging to /home/hadoop/hadoop-2.5.2/logs/yarn-hadoop-nodemanager-slave2.out
12 .验证是否启动成功

   jps 

3526 ResourceManager
3796 Jps
3295 NameNode
ssh slave1 , jps  

4330 Jps
4074 DataNode

ssh slave2 ,jps  

3928 Jps
3667 DataNode

停止hadoop   

stop-yarn.sh

stop-dfs.sh



发现一个问题，yarn框架启动后，slave上的nodemanage经常一会就死了，后来发现是因为防火墙的原因，使datanode无法与namenode通讯，

关闭centos7的防火墙，systemctl stop firewalld.service

                                           systemctl disable firewalld.service  


查看集群运行情况 http://192.168.22.30:50070/    

http://192.168.22.30:8088/




  






     

第8个狼
关注
专栏目录
CentOS 64位上编译 Hadoop 2.6.0
(Andrew)的专栏
03-19 8346
1.操作系统编译环境 yum install cmake lzo-devel zlib-devel gcc gcc-c++ autoconf automake libtool ncurses-devel openssl-devel libXtst 2.安装JDK 下载JDK1.7，注意只能用1.7，否则编译会出错 http://www.oracle.com/tech
编译好的64位hadoop-2.7.0.tar.gz分卷3
07-29
上传大小限制了，所以拆分 每个1分，评论返分，支持作者下载更好的资源来学习和分享 hadoop-2.7.0.tar.gz分卷1 分卷1：hadoop-2.7.0.z01 分卷2：hadoop-2.7.0.z02 分卷3：hadoop-2.7.0.zip 下载完毕后选中分卷1，解压即可
Windows环境下编译Hadoop源代码
光晕
07-28 248
1.先从Git取代码 git clone git://git.apache.org/hadoop-common.git 2.下载Protocol Buffers 2.4.1 目前2.5.0还有问题，不能用。 Google Code上有编译好的Windows binary，下载下来放到PATH的路径里。 再把2.4.1的source下载下来，在java目录里面用maven编译。编译之前
hadoop源码编译环境安装配置
owen-li
07-19 2549
hadoop运行报错： 在运行hadoop的时候，出现警告： WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable 原因： hadoop官方编译的tar包是32位的，我们的系统是64位的，故
Ubuntu 64位安装Hadoop
.
05-15 225
从官网下载hadoop-3.1.0.tar，并解压到/usr/hadoop目录中； 更改Hadoop文件的相关配置： core-site.xml 配置如下：　　　　 　　　　&amp;amp;lt;configuration&amp;amp;gt; 　　　　　　&amp;amp;lt;property&amp;amp;gt; 　　　　　　　　&amp;amp;lt;name&amp;amp;gt;hadoop.tmp.dir&amp;amp;lt
Hadoop 2.7.4 Windows 64位 编译bin
01-18
总之，Hadoop 2.7.4 Windows 64位编译bin包为Windows用户提供了在本地开发和测试大数据处理应用的便利。通过理解和掌握上述知识点，用户可以在Windows环境下高效地利用Hadoop进行大数据处理任务。
Hadoop 2.7.4 Windows 7 64位 编译bin（包含winutils.exe, hadoop.dll等）
08-17
标题中的“Hadoop 2.7.4 Windows 7 64位 编译bin（包含winutils.exe, hadoop.dll等）”指的是一个专为Windows 7 64位操作系统编译的Hadoop二进制包，包含了关键组件winutils.exe和hadoop.dll。这些文件对于在Windows...
Hadoop 2.7.5 Windows 7 64位 编译bin（包含winutils.exe, hadoop.dll等）
12-22
本资源针对的是Windows 7 64位操作系统用户，帮助他们在本地环境中搭建Hadoop开发和测试平台。 在Windows系统上编译Hadoop通常是一项挑战，因为Hadoop最初设计是运行在Linux环境下的。然而，通过一些额外的步骤，...
使用Maven搭建Hadoop开发环境
09-01
至此，我们已经成功地使用Maven搭建了一个Hadoop开发环境，可以编写、编译和测试Hadoop应用程序。当然，实际开发过程中，你可能还需要根据自己的需求调整`pom.xml`文件，例如更新Hadoop的版本，或者添加其他依赖库。...
hadoop 2.6 64位
12-05
hadoop-2.6.0 大数据框架的64位编译。我自己已经编译出来了，并且通过了测试。因为，考虑到其他的人也会遇到这种情况。所以希望对大家有帮组。
hadoop 64位本地库
01-13
我们从官网上下载的hadoop的本地库是在32位系统中编译。所以，如果你的linux系统是64位的那么运行的时候可能会出现一些错误导致namenode上hdfs不能启动，这时需要去下载64位本地库来进行替换。
hadoop 2.6.0 64位安装包
12-23
最新版hadoop 64位安装包 centos 6.4 64位机器上编译 由于超过100MB，所以给的网盘链接地址
Hadoop2.4.1-x64& Hadoop2.6.0-x64
09-08
文件为百度云下载链接，包含2.4.1 64位和32位，2.6.0 64位，编译环境均为CentOS 64 --编译环境：CentOS6.5 64 hadoop-2.4.1-x64.tar.gz ----2.4.1 64位 hadoop-2.4.1-x86.tar.gz ----2.4.1 32位 hadoop-2.6.0-x64.tar.gz ----2.6.0 64位
hadoop3.1.0
06-25
hadoop3.1.0在win10系统使用vs2017编译的本地库，包括hadoop.dll,hdfs.dll,winutils.exe及其.lib文件
hadoop64位编译
菜鸟的专栏
04-04 1095
官方提供的hadoop-2.x版本貌似都是32位的，在64位机子下使用可能会报错，最好使用官方提供的源码进行本地编译，编译成适合本地硬件环境的64位软件包。关于native 　 Hadoop是使用Java语言开发的，但是有一些需求和操作并不适合使用java，所以就引入了本地库（Native Libraries）的概念，通过本地库，Hadoop可以更加高效地执行某一些操作。　 native位于h
hadoop64位系统编译
GodfatherTye的博客
12-06 307
hadoop源码编译64位系统版本 写在前面： 因为Hadoop不提供64位编译好的版本，只能用源码自行编译64位版本，所以本文以hadoop2.6.0为例进行源码编译。值得注意的是如果想编译2.6.0版本的hadoop需要用JDK7，如果用JDK8编译会报这样的错误: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hado
Hadoop2.x源码64位编译
最新发布
weixin_42465125的博客
12-20 512
Hadoop2.x 64Centos7 上源码编译
在 win7 64 上编译 hadoop 2.7.3的经历
buxinchun的专栏
05-07 3313
主要讲述了自己在 Win7 64 系统上编译 Hadoop 的亲身经历。
Ubuntu 12.04上编译Hadoop 2.2.0 64位步骤详解
在Ubuntu 12.04环境下，本文档详尽地介绍了如何将32位的Hadoop 2.2.0版本升级为64位。首先，你需要确保系统中已安装必要的开发工具...通过遵循文档中的指南，开发者可以避免常见问题并顺利搭建Hadoop 2.2.0的64位环境。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值