hadoop集群配置实例

最新推荐文章于 2022-10-24 15:42:40 发布

weixin_30508309

最新推荐文章于 2022-10-24 15:42:40 发布

阅读量72

点赞数

文章标签：大数据运维 java

原文链接：http://www.cnblogs.com/hanhuilee/p/5221304.html

版权

1）ssh配置
http://allthingshadoop.com/2010/04/20/hadoop-cluster-setup-ssh-key-authentication/
2) 修改打开文件数限制 - 主从配置一致
/etc/security/limits.conf中增加如下配置提高打开文件限制数
hadoop soft nofile 10240
hadoop hard nofile 65536

3）配置core-site.xml - 主从配置一致(配置文件在hadoop/conf目录，下同)
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
   <property>
       <name>fs.default.name</name>
       <value>hdfs://server:6060/</value>
       <description>who is the namenode</description>
   </property>
</configuration>
4）配置hdfs-site.xml - 主从配置一致
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
   <property>
       <name>dfs.replication</name>
       <value>1</value>
       <final>true</final>
       <description>default 3 in production environment</description>
   </property>
   <property>
       <name>dfs.data.dir</name>
       <value>/home/hadoop/hdfs/data</value>
       <description>where data file is saved</description>
   </property>
   <property>
       <name>dfs.name.dir</name>
       <value>/home/hadoop/hdfs/name</value>
       <description>where data meta file is saved (on namenode) </description>
   </property>
   <property>
       <name>hadoop.tmp.dir</name>
       <value>/home/hadoop/hdfs/tmp</value>
       <description>A base for other temporary directories.</description>
   </property>
</configuration>

目录权限
A）data目录权限
chmod 755 /home/hadoop/hdfs/data
B）namenode的数据目录不需要创建，hadoop namenode -format将自动创建
rm -rf /home/hadoop/hdfs/name

C）如果发现datanode也启动不了了
rm -rf /home/hadoop/hdfs/data

D）namenode几点显示只有一个datanode - 本机
从节点日志显示中显示
2013-11-04 15:37:29,138 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: server/192.168.16.133:6060. Already tried 0 time(s).
从节点无法连接到namenode，
修改方式：
在namenode上，hostname对应地址为127.0.0.1，需要修改/etc/hosts中的127.0.0.1（注释掉）
E) 如果主从仍旧无法连接，关闭防火墙

5）修改masters - 主从配置一致
server

6)修改slaves - 主动配置一致
server
slave1
slave2
7）mapred-site.xml配置 - 主动配置一致
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
   <property>
       <name>mapred.job.tracker</name>
       <value>server:8021</value>
       <final>true</final>
   </property>
   <property>
       <name>mapred.local.dir</name>
       <value>/home/hadoop/mapred/local</value>
       <final>true</final>
   </property>
   <property>
       <name>mapred.system.dir</name>
       <value>/home/hadoop/mapred/system</value>
       <final>true</final>
   </property>
   <property>
       <name>mapred.tasktracker.map.tasks.maximum</name>
       <value>7</value>
       <final>true</final>
   </property>
   <property>
       <name>mapred.tasktracker.reduce.tasks.maximum</name>
       <value>7</value>
       <final>true</final>
   </property>
   <property>
       <name>mapred.child.java.opts</name>
       <value>-Xmx400m</value>
   </property>
</configuration>
8）hadoop-evn.sh中配置

export JAVA_HOME=/path/to/jdk

9）初始化、启动

$ bin/hadoop namenode -format
$ bin/hadoop start-all.sh

$ hadoop dfsadmin -report
Warning: $HADOOP_HOME is deprecated.

Configured Capacity: 288730824704 (268.9 GB)
Present Capacity: 188958699520 (175.98 GB)
DFS Remaining: 188958601216 (175.98 GB)
DFS Used: 98304 (96 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 3 (3 total, 0 dead)

Name: 192.168.16.133:50010
Decommission Status : Normal
Configured Capacity: 119873626112 (111.64 GB)
DFS Used: 40960 (40 KB)
Non DFS Used: 7385841664 (6.88 GB)
DFS Remaining: 112487743488(104.76 GB)
DFS Used%: 0%
DFS Remaining%: 93.84%
Last contact: Mon Nov 04 16:03:57 CST 2013

Name: 192.168.16.113:50010
Decommission Status : Normal
Configured Capacity: 97833930752 (91.11 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 86688358400 (80.73 GB)
DFS Remaining: 11145543680(10.38 GB)
DFS Used%: 0%
DFS Remaining%: 11.39%
Last contact: Mon Nov 04 16:03:55 CST 2013

Name: 192.168.16.156:50010
Decommission Status : Normal
Configured Capacity: 71023267840 (66.15 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 5697925120 (5.31 GB)
DFS Remaining: 65325314048(60.84 GB)
DFS Used%: 0%
DFS Remaining%: 91.98%
Last contact: Mon Nov 04 16:03:56 CST 2013

Hadoop版本1.0.3

转载于:https://www.cnblogs.com/hanhuilee/p/5221304.html