hadoop 集群配置(1.0.4)

最新推荐文章于 2015-09-22 10:09:00 发布

置顶 East271536394

最新推荐文章于 2015-09-22 10:09:00 发布

阅读量993

点赞数

分类专栏： hadoop

本文链接：https://blog.csdn.net/east271536394/article/details/8962761

版权

hadoop 专栏收录该内容

3 篇文章 0 订阅

订阅专栏

在/home/mongodb/workspace/hadoop-1.0.4环境下，进行Hadoop集群配置，创建数据目录/home/mongodb/data/hdfs/namesecondary，并通过hadoop jar命令运行WordCount示例，处理/user/mongodb/input/wordcount.txt文件，输出结果到/user/mongodb/output/wordcount1。

摘要由CSDN通过智能技术生成

1. 服务器
202.173.8.57 test01.east.cn (master--namenode)
202.173.8.58 test02.east.cn (slave1--datanode)
202.173.8.59 test03.east.cn (slave2--datanode)

2. 配置host vim /etc/hosts 新增加

202.173.8.57 test01.east.cn
202.173.8.58 test02.east.cn
202.173.8.59 test03.east.cn

3. 配置ssh 登录(mongodb 用户)
a)在master 执行 ssh-keygen -q -t rsa -N "" -f /home/mongodb/.ssh/id_rsa
b)查看 /home/mongodb/.ssh/id_rsa
[mongodb@test01 ~]$ cat /home/mongodb/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEApmN/5CF6diQIyglLEUDk/Co6cUZZCidx4cXUEuDhSp/6Y5wWGJ0JO49+cIPuAbw/Gse9CeduL5nJoz2kcq+TbFcGwxqW0MoCOMGCV/G500+6lz6ajr5qS3qLGRyOvGz80nigXEQe+9g9xz2en5bGu/VMQyfNlNdXekzPmir8QIap/GqWJZjgVZMOeqBUfEkGZDZX/brG9kqU5RZBRxFxW2e2yWP43ThmZxig0lNmyu4scVLTs6N22BLL81gYGUoMKcGIWxkwoEiWtsMv0Qro/eF6AEGBEr+1fDZU7SxV5McVP5OXQLW/BG+HShf8XmS5IgEXiWbZal7LSTg8j6cZGw== mongodb@test01.east.cn
c)cat id_rsa.pub > authorized_keys
d) chmod go-rwx authorized_keys
e) 将authorized_keys copy 到slave1 slave2
scp authorized_keys 202.173.8.58:/home/mongodb/.ssh
scp authorized_keys 202.173.8.59:/home/mongodb/.ssh
4. 配置目录
mkdir -p /home/mongodb/app
mkdir -p /home/mongodb/software
mkdir -p /home/mongodb/workspace
mkdir -p /home/mongodb/data
5. 安装JDK到/home/mongodb/workspace/jdk1.7.0_04

6. 配置hadoop-env.sh
export JAVA_HOME=/home/mongodb/workspace/jdk1.7.0_04
7. 配置环境hadoop_home
vim .bash_profile 新增加
export HADOOP_HOME=/home/mongodb/workspace/hadoop-1.0.4
export PATH=$PATH:$HADOOP_HOME/bin
[mongodb@test01 ~]$ echo $JAVA_HOME
/home/mongodb/workspace/jdk1.7.0_04
[mongodb@test01 ~]$ echo $HADOOP_HOME

/home/mongodb/workspace/hadoop-1.0.4

8. 配置master 及slaves
[mongodb@test01 conf]$ cat masters
test01.east.cn
[mongodb@test01 conf]$ cat slaves
test02.east.cn
test03.east.cn
[mongodb@test01 conf]$

9. 创建文件

mkdir -p /home/mongodb/data/hdfs/namesecondary

mkdir -p /home/mongodb/data/hdfs/name
mkdir -p /home/mongodb/data/hdfs/data
mkdir -p /home/mongodb/data/mapred/mrlocal
mkdir -p /home/mongodb/data/mapred/mrsystem
mkdir -p /home/mongodb/data/tmp
10.设置配置文件,配置完成后，分发到其它服务器上
a)core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<!-- Put site-specific property overrides in this file. -->


<configuration>


  <property>
    <name>fs.default.name</name>
    <value>hdfs://test01.east.cn:9000</value>
  </property>


  <property>
    <name>fs.checkpoint.dir</name>
    <value>/home/mongodb/data/hdfs/namesecondary</value>
  </property>
  
  <property>
    <name>fs.checkpoint.period</name>
    <value>1800</value>
  </property>


  <property>
    <name>fs.checkpoint.size</name>
    <value>33554432</value>
  </property>
   
   <property>
    <name>io.compression.codes</name>
    <value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec</value>
  </property>
   
   <property>
    <name>fs.trash.interval</name>
    <value>1440</value>
    <description>Hadoop</description>
  </property>
  
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/mongodb/data/tmp</value>
  </property>


</configuration>

b)hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<!-- Put site-specific property overrides in this file. -->


<configuration>


   <property>
    <name>dfs.name.dir</name>
    <value>/home/mongodb/data/hdfs/name</value>
    <description>desc</description>
   </property>


   <property>
    <name>dfs.data.dir</name>
    <value>/home/mongodb/data/hdfs/data</value>
    <description>desc</description>
  </property>


   <property>
    <name>dfs.http.address</name>
    <value>202.173.8.57:50070</value>
  </property>


   <property>
    <name>dfs.secondary.http.address</name>
    <value>202.173.8.57:50090</value>
  </property>
   
   <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>


   <property>
    <name>dfs.datanode.du.reserved</name>
    <value>1073741824</value>
  </property>
   


    <property>
     <name>dfs.block.size</name>
     <value>134217728</value>
    </property>


    <property>
      <name>dfs.permissions</name>
      <value>false</value>
    </property>
</configuration>

c)mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<!-- Put site-specific property overrides in this file. -->


<configuration>


  <property>
    <name>mapred.job.tracker</name>
    <value>202.173.8.57:9001</value>
  </property>


  <property>
    <name>mapred.local.dir</name>
    <value>/home/mongodb/data/mapred/mrlocal</value>
    <final>true</final>
  </property>
   
  <property>
    <name>mapred.system.dir</name>
    <value>/home/mongodb/data/mapred/mrsystem</value>
    <final>true</final>
  </property>


   <property>
    <name>mapred.tasktracker.map.tasks.maximum</name>
    <value>2</value>
    <final>true</final>
  </property>


   <property>
    <name>mapred.tasktracker.reduce.tasks.maximum</name>
    <value>1</value>
    <final>true</final>
  </property>


   <property>
    <name>io.sort.mb</name>
    <value>32</value>
    <final>true</final>
  </property>


  <property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx64M</value>
  </property>


  <property>
    <name>mapred.compress.map.output</name>
    <value>true</value>
  </property>


</configuration>

11 初始化hdfs
hadoop namenode -format
[mongodb@test01 data]$ hadoop namenode -format
Warning: $HADOOP_HOME is deprecated.

13/05/16 15:40:51 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = test01.east.cn/202.173.8.57
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
13/05/16 15:40:51 INFO util.GSet: VM type = 64-bit
13/05/16 15:40:51 INFO util.GSet: 2% max memory = 17.77875 MB
13/05/16 15:40:51 INFO util.GSet: capacity = 2^21 = 2097152 entries
13/05/16 15:40:51 INFO util.GSet: recommended=2097152, actual=2097152
13/05/16 15:40:51 INFO namenode.FSNamesystem: fsOwner=mongodb
13/05/16 15:40:51 INFO namenode.FSNamesystem: supergroup=supergroup
13/05/16 15:40:51 INFO namenode.FSNamesystem: isPermissionEnabled=false
13/05/16 15:40:51 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/05/16 15:40:51 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/05/16 15:40:51 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/05/16 15:40:51 INFO common.Storage: Image file of size 113 saved in 0 seconds.
13/05/16 15:40:51 INFO common.Storage: Storage directory /home/mongodb/data/hdfs/name has been successfully formatted.
13/05/16 15:40:51 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at test01.east.cn/202.173.8.57
************************************************************/

12 启动hadoop
hadoop_home/bin/start-all.sh
[mongodb@test01 bin]$ ./start-all.sh
Warning: $HADOOP_HOME is deprecated.

starting namenode, logging to /home/mongodb/workspace/hadoop-1.0.4/libexec/../logs/hadoop-mongodb-namenode-test01.east.cn.out
test03.east.cn: starting datanode, logging to /home/mongodb/workspace/hadoop-1.0.4/libexec/../logs/hadoop-mongodb-datanode-test03.east.cn.out
test02.east.cn: starting datanode, logging to /home/mongodb/workspace/hadoop-1.0.4/libexec/../logs/hadoop-mongodb-datanode-ss.east.cn.out
test01.east.cn: starting secondarynamenode, logging to /home/mongodb/workspace/hadoop-1.0.4/libexec/../logs/hadoop-mongodb-secondarynamenode-test01.east.cn.out
starting jobtracker, logging to /home/mongodb/workspace/hadoop-1.0.4/libexec/../logs/hadoop-mongodb-jobtracker-test01.east.cn.out
test03.east.cn: starting tasktracker, logging to /home/mongodb/workspace/hadoop-1.0.4/libexec/../logs/hadoop-mongodb-tasktracker-test03.east.cn.out
test02.east.cn: starting tasktracker, logging to /home/mongodb/workspace/hadoop-1.0.4/libexec/../logs/hadoop-mongodb-tasktracker-ss.east.cn.out
[mongodb@test01 bin]$ jps
29456 SecondaryNameNode
29740 Jps
29566 JobTracker
29242 NameNode
13测试

hadoop jar hadoop-examples-1.0.4.jar wordcount /user/mongodb/input/wordcount.txt /user/mongodb/output/wordcount1