一:启动,通过UI界面查看Hadoop运行状态
释:
Hadoop集群正常启动后,它默认开放了两个端口9870
和8088
,分别用于监控HDFS集群和YARN集群。通过UI界面可以方便地进行集群的管理和查看,只需要在本地操作系统的浏览器输入集群服务的IP和对应的端口号即可访问。
Hadoop2和Hadoop端口的区别表:
查看HDFS集群状态:
浏览器中访问:http://master:9870 【或者http://master的IP地址 :9870】
注:
若上面不能通过主机名master加端口9870
的方式,原因在于没有在hosts
文件里IP与主机名的映射,现在只能通过IP地址加端口号的方式访问:http://192.168.1.101:9870
则:修改主机的C:\Windows\System32\drivers\etc\hosts
文件,增加hadoop集群主机名与IP地址的映射,则可成功 【或者你防火墙未关】
点击Overview可查看HDFS集群状态显示正常
也可以 点击Datandoes,查看数据节点信息
点击Utilities下拉菜单,选择Browse the file system ,可以看到内部为空
在HDFS上创建一个目录BigData,执行: hdfs dfs -mkdir /BigData
在Hadoop WebUI界面查看刚才创建的目录
查看YARN集群状态
输入:http://master:8088/cluster,查看YARN集群状态显示正常
如下例子:
二.词频统计
启动Hadoop集群,执行:start-all.sh
2.创建文件,先执行:vim test.txt 再执行:cat test.txt
/
3.文件上传到HDFS指定目录【上传到HDFS的/BigData目录】,执行:hdfs dfs -put test.txt /BigData
4.查看是否上传成功,执行HDFS命令:hdfs dfs -ls /BigData
5.查看文件是否上传成功
6.运行词频统计程序的jar包,查看Hadoop自带的jar包,先执行:cd $HADOOP_HOME/share/hadoop/mapreduce
7.执行:hadoop jar ./hadoop-mapreduce-examples-3.3.4.jar wordcount /BigData/test.txt /wc_result
注【以下为正确运行】:
[root@master mapreduce]# hadoop jar ./hadoop-mapreduce-examples-3.3.4.jar wordcount /BigData/test.txt /wc_result
2022-10-04 12:40:30,559 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at master/192.168.1.101:8032
2022-10-04 12:40:31,501 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1664901602347_0001
2022-10-04 12:40:32,416 INFO input.FileInputFormat: Total input files to process : 1
2022-10-04 12:40:32,911 INFO mapreduce.JobSubmitter: number of splits:1
2022-10-04 12:40:33,204 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1664901602347_0001
2022-10-04 12:40:33,205 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-10-04 12:40:33,418 INFO conf.Configuration: resource-types.xml not found
2022-10-04 12:40:33,418 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2022-10-04 12:40:33,939 INFO impl.YarnClientImpl: Submitted application application_1664901602347_0001
2022-10-04 12:40:33,974 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1664901602347_0001/
2022-10-04 12:40:33,974 INFO mapreduce.Job: Running job: job_1664901602347_0001
2022-10-04 12:40:45,146 INFO mapreduce.Job: Job job_1664901602347_0001 running in uber mode : false
2022-10-04 12:40:45,147 INFO mapreduce.Job: map 0% reduce 0%
2022-10-04 12:40:52,319 INFO mapreduce.Job: map 100% reduce 0%
2022-10-04 12:40:59,512 INFO mapreduce.Job: map 100% reduce 100%
2022-10-04 12:41:00,531 INFO mapreduce.Job: Job job_1664901602347_0001 completed successfully
2022-10-04 12:41:00,694 INFO mapreduce.Job: Counters: 54
File System Counters
FILE: Number of bytes read=98
FILE: Number of bytes written=552153
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=191
HDFS: Number of bytes written=60
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=4860
Total time spent by all reduces in occupied slots (ms)=5021
Total time spent by all map tasks (ms)=4860
Total time spent by all reduce tasks (ms)=5021
Total vcore-milliseconds taken by all map tasks=4860
Total vcore-milliseconds taken by all reduce tasks=5021
Total megabyte-milliseconds taken by all map tasks=4976640
Total megabyte-milliseconds taken by all reduce tasks=5141504
Map-Reduce Framework
Map input records=6
Map output records=16
Map output bytes=154
Map output materialized bytes=98
Input split bytes=100
Combine input records=16
Combine output records=8
Reduce input groups=8
Reduce shuffle bytes=98
Reduce input records=8
Reduce output records=8
Spilled Records=16
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=235
CPU time spent (ms)=1180
Physical memory (bytes) snapshot=318394368
Virtual memory (bytes) snapshot=5478535168
Total committed heap usage (bytes)=141692928
Peak Map Physical memory (bytes)=211103744
Peak Map Virtual memory (bytes)=2734690304
Peak Reduce Physical memory (bytes)=107290624
Peak Reduce Virtual memory (bytes)=2743844864
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=91
File Output Format Counters
Bytes Written=60
8.查看输出目录/wc_result
,执行:hdfs dfs -ls /wc_result
如下所示:
9.查看词频统计结果,查看Hadoop自带示例的jar包,执行:hdfs dfs -cat /wc_result/*
在HDFS集群UI界面查看结果文件,在UI界面,查看/wc_result
点击结果文件下载,利用Notepad++打开结果文件
查看程序运行状态,访问8088,看到FINISHED和SUCCEEDED,单击应用标识,查看应用的运行详情