四节点服务器安装什么系统,Hadoop 三节点集群安装配置详细实例

排错

[www.linuxidc.com@Hadoop47 logs]$ vi hadoop-zhouhh-datanode-Hadoop47.log

2012-05-23 17:17:14,230 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting DataNode

STARTUP_MSG: host = Hadoop47/192.168.10.47

STARTUP_MSG: args = []

STARTUP_MSG: version = 1.0.3

STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1335192; compiled by ‘hortonfo’ on Tue May 8 20:31:25 UTC 2012

************************************************************/

2012-05-23 17:17:14,762 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties

2012-05-23 17:17:14,772 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.

2012-05-23 17:17:14,772 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).

2012-05-23 17:17:14,772 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started

2012-05-23 17:17:14,907 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.

2012-05-23 17:17:15,064 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library

2012-05-23 17:17:15,187 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: file:///

at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)

at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:198)

at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:228)

at org.apache.hadoop.hdfs.server.namenode.NameNode.getServiceAddress(NameNode.java:222)

at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:337)

at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:299)

at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)

at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)

at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)

at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)

at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)

2012-05-23 17:17:15,187 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down DataNode at Hadoop47/192.168.10.47

************************************************************/

同样，需要配置相关的端口

[www.linuxidc.com@Hadoop48 bin]$ start-mapred.sh

[www.linuxidc.com@Hadoop48 ~]$ ssh Hadoop46

Last login: Wed May 23 17:33:05 2012 from hadoop47

[www.linuxidc.com@Hadoop46 ~]$ cd hadoop-1.0.3/logs

[www.linuxidc.com@Hadoop46 logs]$ vi hadoop-zhouhh-datanode-Hadoop46.log

2012-05-23 17:38:46,062 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: Hadoop48/192.168.10.48:54310. Already tried 0 time(s).

2012-05-23 17:38:47,065 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: Hadoop48/192.168.10.48:54310. Already tried 1 time(s).

[www.linuxidc.com@Hadoop46 logs]$ vi hadoop-zhouhh-tasktracker-Hadoop46.log

2012-05-23 17:58:13,356 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 54550: starting

2012-05-23 17:58:14,428 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: Hadoop48/192.168.10.48:54311. Already tried 0 time(s).

2012-05-23 17:58:15,430 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: Hadoop48/192.168.10.48:54311. Already tried 1 time(s).

[www.linuxidc.com@Hadoop48 conf]$ netstat -antp | grep 54310

(Not all processes could be identified, non-owned process info

will not be shown, you would have to be root to see it all.)

tcp 0 0 192.168.10.48:54310 192.168.20.188:30300 ESTABLISHED 20469/python

[www.linuxidc.com@Hadoop48 conf]$ netstat -antp | grep 54311

(Not all processes could be identified, non-owned process info

will not be shown, you would have to be root to see it all.)

tcp 0 0 192.168.10.48:54311 192.168.20.188:30300 TIME_WAIT -

原来端口被占用了，将相关占用端口python程序停掉。

[www.linuxidc.com@Hadoop48 hadoop-1.0.3]$ stop-mapred.sh

[www.linuxidc.com@Hadoop48 hadoop-1.0.3]$ stop-dfs.sh

[www.linuxidc.com@Hadoop48 hadoop-1.0.3]$ start-dfs.sh

starting namenode, logging to /home/zhouhh/hadoop-1.0.3/libexec/../logs/hadoop-zhouhh-namenode-Hadoop48.out

Hadoop47: starting datanode, logging to /home/zhouhh/hadoop-1.0.3/libexec/../logs/hadoop-zhouhh-datanode-Hadoop47.out

Hadoop46: starting datanode, logging to /home/zhouhh/hadoop-1.0.3/libexec/../logs/hadoop-zhouhh-datanode-Hadoop46.out

Hadoop48: starting secondarynamenode, logging to /home/zhouhh/hadoop-1.0.3/libexec/../logs/hadoop-zhouhh-secondarynamenode-Hadoop48.out

[www.linuxidc.com@Hadoop48 hadoop-1.0.3]$ netstat -antp | grep 54310

(Not all processes could be identified, non-owned process info

will not be shown, you would have to be root to see it all.)

tcp 0 0 192.168.10.48:54310 0.0.0.0:* LISTEN 24716/java

tcp 0 0 192.168.10.48:51040 192.168.10.48:54310 TIME_WAIT -

tcp 0 0 192.168.10.48:51038 192.168.10.48:54310 TIME_WAIT -

tcp 0 0 192.168.10.48:54310 192.168.10.46:38202 ESTABLISHED 24716/java

[www.linuxidc.com@Hadoop48 hadoop-1.0.3]$ start-mapred.sh

starting jobtracker, logging to /home/zhouhh/hadoop-1.0.3/libexec/../logs/hadoop-zhouhh-jobtracker-Hadoop48.out

Hadoop46: starting tasktracker, logging to /home/zhouhh/hadoop-1.0.3/libexec/../logs/hadoop-zhouhh-tasktracker-Hadoop46.out

Hadoop47: starting tasktracker, logging to /home/zhouhh/hadoop-1.0.3/libexec/../logs/hadoop-zhouhh-tasktracker-Hadoop47.out

[www.linuxidc.com@Hadoop48 hadoop-1.0.3]$ netstat -antp | grep 54311

(Not all processes could be identified, non-owned process info

will not be shown, you would have to be root to see it all.)

tcp 0 0 192.168.10.48:54311 0.0.0.0:* LISTEN 25238/java

tcp 0 0 192.168.10.48:54311 192.168.10.46:33561 ESTABLISHED 25238/java

tcp 0 0 192.168.10.48:54311 192.168.10.47:55277 ESTABLISHED 25238/java

查看DataNode的log，已经正常。

[www.linuxidc.com@Hadoop48 hadoop-1.0.3]$ jps

24716 NameNode

25625 Jps

25238 JobTracker

24909 SecondaryNameNode

[www.linuxidc.com@Hadoop46 ~]$ jps

10649 TaskTracker

10352 DataNode

10912 Jps