Hadoop性能基准测试

最新推荐文章于 2023-05-07 23:45:00 发布

滑冰选手库里

最新推荐文章于 2023-05-07 23:45:00 发布

阅读量599

点赞数

分类专栏： Hadoop

本文链接：https://blog.csdn.net/weixin_40096730/article/details/105560252

版权

本文详细介绍了对Hadoop HDFS的性能基准测试，包括写入和读取10个128M文件的测试，记录了吞吐量和平均IO率。同时，还通过Sort程序评估了MapReduce的性能，利用RandomWriter生成随机数据并执行Sort，最后验证了排序的正确性。

摘要由CSDN通过智能技术生成

1）测试HDFS写性能

测试内容：向HDFS集群写10个128M的文件

hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 128MB

输出日志：

20/04/16 04:07:54 INFO fs.TestDFSIO: TestDFSIO.1.8
20/04/16 04:07:54 INFO fs.TestDFSIO: nrFiles = 10
20/04/16 04:07:54 INFO fs.TestDFSIO: nrBytes (MB) = 128.0
20/04/16 04:07:54 INFO fs.TestDFSIO: bufferSize = 1000000
20/04/16 04:07:54 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
20/04/16 04:07:56 INFO fs.TestDFSIO: creating control file: 134217728 bytes, 10 files
20/04/16 04:07:57 INFO fs.TestDFSIO: created control files for: 10 files
20/04/16 04:07:57 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.187.203:8032
20/04/16 04:07:58 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.187.203:8032
20/04/16 04:07:58 INFO mapred.FileInputFormat: Total input paths to process : 10
20/04/16 04:07:59 INFO mapreduce.JobSubmitter: number of splits:10
20/04/16 04:08:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1587019021463_0001
20/04/16 04:08:01 INFO impl.YarnClientImpl: Submitted application application_1587019021463_0001
20/04/16 04:08:02 INFO mapreduce.Job: The url to track the job: http://hadoop103:8088/proxy/application_1587019021463_0001/
20/04/16 04:08:02 INFO mapreduce.Job: Running job: job_1587019021463_0001
20/04/16 04:08:16 INFO mapreduce.Job: Job job_1587019021463_0001 running in uber mode : false
20/04/16 04:08:16 INFO mapreduce.Job:  map 0% reduce 0%
20/04/16 04:09:13 INFO mapreduce.Job:  map 7% reduce 0%
20/04/16 04:09:47 INFO mapreduce.Job:  map 17% reduce 0%
20/04/16 04:10:04 INFO mapreduce.Job:  map 17% reduce 3%
20/04/16 04:10:31 INFO mapreduce.Job:  map 23% reduce 3%
20/04/16 04:10:42 INFO mapreduce.Job:  map 27% reduce 3%
20/04/16 04:10:57 INFO mapreduce.Job:  map 33% reduce 7%
20/04/16 04:10:59 INFO mapreduce.Job:  map 37% reduce 7%
20/04/16 04:11:08 INFO mapreduce.Job:  map 40% reduce 10%
20/04/16 04:11:12 INFO mapreduce.Job:  map 40% reduce 13%
20/04/16 04:11:24 INFO mapreduce.Job:  map 47% reduce 13%
20/04/16 04:11:27 INFO mapreduce.Job:  map 53% reduce 13%
20/04/16 04:11:52 INFO mapreduce.Job:  map 57% reduce 13%
20/04/16 04:11:57 INFO mapreduce.Job:  map 57% reduce 17%
20/04/16 04:12:00 INFO mapreduce.Job:  map 67% reduce 17%
20/04/16 04:12:03 INFO mapreduce.Job:  map 67% reduce 20%
20/04/16 04:12:13 INFO mapreduce.Job:  map 70% reduce 20%
20/04/16 04:12:15 INFO mapreduce.Job:  map 70% reduce 23%
20/04/16 04:12:17 INFO mapreduce.Job:  map 77% reduce 23%
20/04/16 04:12:20 INFO mapreduce.Job: Task Id : attempt_1587019021463_0001_m_000003_1, Status : FAILED
Error: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /benchmarks/TestDFSIO/io_data/test_io_3 (inode 16853): File does not exist. Holder DFSClient_attempt_1587019021463_0001_m_000003_1_-167239712_1 does not have any open files.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3428)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.comp

最低0.47元/天解锁文章