Hadoop 安装笔记

Hadoop 安装笔记

在验证集群是否搭建成功的时候，首先通过web ui是可以正常访问页面的，以为大工告成。但没先到在执行一个map-reduce例子的时候，遇到吊销问题。

1. 访问web ui界面

• hdfs集群访问地址

http://192.168.51.110:9870/

 

• yarn集群访问地址

http://192.168.51.110:8088

 

• jobhistory访问地址：

http://192.168.51.110:19888

 

2. 执行一个map-reduce例子

2.1 程序卡死

执行一个map-reduce例子 。 如下图所示，程序似乎卡死在。

INFO mapreduce.Job: Running job: job_1617543497587_0001

注意到提示信息中有:INFO conf.Configuration: resource-types.xml not found。但教程里并没有 resource-types.xml的说明，百度网上也没有找到比较有价值的答案。而注意到该问题的日志级别仅仅是INFO，所以也有可能跟这个问题无关。

 

[hadoop@node01 bin]$ hadoop jar /kkb/install/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.4.jar pi 5 5
Number of Maps  = 5
Samples per Map = 5
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Starting Job
2021-04-05 07:37:34,038 INFO client.RMProxy: Connecting to ResourceManager at node01/192.168.51.110:8032
2021-04-05 07:37:45,541 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1617543497587_0001
2021-04-05 07:37:45,765 INFO input.FileInputFormat: Total input files to process : 5
2021-04-05 07:37:45,880 INFO mapreduce.JobSubmitter: number of splits:5
2021-04-05 07:37:46,196 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1617543497587_0001
2021-04-05 07:37:46,198 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-04-05 07:37:46,430 INFO conf.Configuration: resource-types.xml not found
2021-04-05 07:37:46,430 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-04-05 07:37:47,005 INFO impl.YarnClientImpl: Submitted application application_1617543497587_0001
2021-04-05 07:37:47,106 INFO mapreduce.Job: The url to track the job: http://node01:8088/proxy/application_1617543497587_0001/
2021-04-05 07:37:47,107 INFO mapreduce.Job: Running job: job_1617543497587_0001
​
​
​
^[[A^[[A
​
​
^C

2.2 重启Hadoop

无奈之下想起了重启大法。可是重启之后更让我吓一跳

[hadoop@node01 bin]$ xcall jps
============= node01 jps =============
15043 JobHistoryServer
14372 NameNode
15115 Jps
14621 SecondaryNameNode
14941 NodeManager
14479 DataNode
14831 ResourceManager
============= node02 jps =============
18211 Jps
17974 DataNode
18057 NodeManager
============= node03 jps =============
18096 DataNode
18179 NodeManager
18334 Jps
[hadoop@node01 bin]$ hadoop jar /kkb/install/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.4.jar pi 5 5
Number of Maps  = 5
Samples per Map = 5
java.net.ConnectException: Call From node01.kaikeba.com/192.168.51.110 to node01:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
    at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566)
    at org.apache.hadoop.ipc.Client.call(Client.java:1508)
    at org.apache.hadoop.ipc.Client.call(Client.java:1405)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:904)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1666)
    at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1582)
    at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1594)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1683)
    at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:279)
    at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:360)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:368)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:699)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812)
    at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636)
    at org.apache.hadoop.ipc.Client.call(Client.java:1452)
    ... 38 more

​
​

但再次访问Web UI界面也是正常的。所以考虑到可能是程序刚启动，端口初始化工作还没完成导致。于是重新执行。

2.3 重新执行map-reduce例子

[hadoop@node01 bin]$ hadoop jar /kkb/install/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.4.jar pi 5 5
Number of Maps  = 5
Samples per Map = 5
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Starting Job
2021-04-05 07:52:04,523 INFO client.RMProxy: Connecting to ResourceManager at node01/192.168.51.110:8032
2021-04-05 07:52:15,658 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1617580080143_0001
2021-04-05 07:52:15,888 INFO input.FileInputFormat: Total input files to process : 5
2021-04-05 07:52:16,012 INFO mapreduce.JobSubmitter: number of splits:5
2021-04-05 07:52:16,235 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1617580080143_0001
2021-04-05 07:52:16,236 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-04-05 07:52:16,453 INFO conf.Configuration: resource-types.xml not found
2021-04-05 07:52:16,453 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-04-05 07:52:16,915 INFO impl.YarnClientImpl: Submitted application application_1617580080143_0001
2021-04-05 07:52:16,963 INFO mapreduce.Job: The url to track the job: http://node01:8088/proxy/application_1617580080143_0001/
2021-04-05 07:52:16,964 INFO mapreduce.Job: Running job: job_1617580080143_0001
2021-04-05 07:52:54,516 INFO mapreduce.Job: Job job_1617580080143_0001 running in uber mode : true
2021-04-05 07:52:54,518 INFO mapreduce.Job:  map 0% reduce 0%
2021-04-05 07:52:57,641 INFO mapreduce.Job:  map 40% reduce 0%
2021-04-05 07:52:58,812 INFO mapreduce.Job:  map 80% reduce 0%
2021-04-05 07:52:59,817 INFO mapreduce.Job:  map 100% reduce 0%
2021-04-05 07:53:00,825 INFO mapreduce.Job:  map 100% reduce 100%
2021-04-05 07:53:00,840 INFO mapreduce.Job: Job job_1617580080143_0001 completed successfully
2021-04-05 07:53:01,009 INFO mapreduce.Job: Counters: 56
    File System Counters
        FILE: Number of bytes read=416
        FILE: Number of bytes written=1316
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=6922
        HDFS: Number of bytes written=1423170
        HDFS: Number of read operations=169
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=33
    Job Counters 
        Launched map tasks=5
        Launched reduce tasks=1
        Other local map tasks=5
        Total time spent by all maps in occupied slots (ms)=0
        Total time spent by all reduces in occupied slots (ms)=0
        TOTAL_LAUNCHED_UBERTASKS=6
        NUM_UBER_SUBMAPS=5
        NUM_UBER_SUBREDUCES=1
        Total time spent by all map tasks (ms)=3592
        Total time spent by all reduce tasks (ms)=1293
        Total vcore-milliseconds taken by all map tasks=0
        Total vcore-milliseconds taken by all reduce tasks=0
        Total megabyte-milliseconds taken by all map tasks=0
        Total megabyte-milliseconds taken by all reduce tasks=0
    Map-Reduce Framework
        Map input records=5
        Map output records=10
        Map output bytes=90
        Map output materialized bytes=140
        Input split bytes=725
        Combine input records=0
        Combine output records=0
        Reduce input groups=2
        Reduce shuffle bytes=140
        Reduce input records=10
        Reduce output records=0
        Spilled Records=20
        Shuffled Maps =5
        Failed Shuffles=0
        Merged Map outputs=5
        GC time elapsed (ms)=82
        CPU time spent (ms)=4370
        Physical memory (bytes) snapshot=4254257152
        Virtual memory (bytes) snapshot=18626322432
        Total committed heap usage (bytes)=3810525184
        Peak Map Physical memory (bytes)=902742016
        Peak Map Virtual memory (bytes)=3103236096
        Peak Reduce Physical memory (bytes)=904392704
        Peak Reduce Virtual memory (bytes)=3111657472
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=590
    File Output Format Counters 
        Bytes Written=97
Job Finished in 56.587 seconds
Estimated value of Pi is 3.68000000000000000000

这下大功告成了。

3. 刷新Web UI界面

执行成功后，可以看到Web UI界面有如下图。