Hadoop2.7.0学习——基准测试

最新推荐文章于 2024-07-25 11:42:13 发布
最新推荐文章于 2024-07-25 11:42:13 发布
阅读量5.2k 收藏 2
点赞数
分类专栏： hadoop 文章标签： hadoop 基准测试
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/flygoa/article/details/52127382
版权

Hadoop2.7.0学习——基准测试

Hadoop 带有一些基准测试程序，可以最少的准备成本轻松运行。基准测试被打包在测试程序JAR文件中，通过无参调用JAR文件可以得到其列表
——《Hadoop权威指南》

查看信息

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.0-tests.ja

执行结果

[root@hadoop-master hadoop-2.7.0]# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.0-tests.jar 
An example program must be given as the first argument.
Valid program names are:
  DFSCIOTest: Distributed i/o benchmark of libhdfs.
  DistributedFSCheck: Distributed checkup of the file system consistency.
  JHLogAnalyzer: Job History Log analyzer.
  MRReliabilityTest: A program that tests the reliability of the MR framework by injecting faults/failures
  NNdataGenerator: Generate the data to be used by NNloadGenerator
  NNloadGenerator: Generate load on Namenode using NN loadgenerator run WITHOUT MR
  NNloadGeneratorMR: Generate load on Namenode using NN loadgenerator run as MR job
  NNstructureGenerator: Generate the structure to be used by NNdataGenerator
  SliveTest: HDFS Stress Test and Live Data Verification.
  TestDFSIO: Distributed i/o benchmark.
  fail: a job that always fails
  filebench: Benchmark SequenceFile(Input|Output)Format (block,record compressed and uncompressed), Text(Input|Output)Format (compressed and uncompressed)
  largesorter: Large-Sort tester
  loadgen: Generic map/reduce load generator
  mapredtest: A map/reduce test check.
  minicluster: Single process HDFS and MR cluster.
  mrbench: A map/reduce benchmark that can create many small jobs
  nnbench: A benchmark that stresses the namenode.
  sleep: A job that sleeps at each map and reduce task.
  testbigmapoutput: A map/reduce program that works on a very big non-splittable file and does identity map/reduce
  testfilesystem: A test for FileSystem read/write.
  testmapredsort: A map/reduce program that validates the map-reduce framework's sort.
  testsequencefile: A test for flat files of binary key value pairs.
  testsequencefileinputformat: A test for sequence file input format.
  testtextinputformat: A test for text input format.
  threadedmapbench: A map/reduce benchmark that compares the performance of maps with multiple spills over maps with 1 spill

写入速度测试

测试写入10个10M文件

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.0-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 10MB

[root@hadoop-master hadoop-2.7.0]# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.0-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 10MB
16/08/05 10:11:42 INFO fs.TestDFSIO: TestDFSIO.1.8
16/08/05 10:11:42 INFO fs.TestDFSIO: nrFiles = 10
16/08/05 10:11:42 INFO fs.TestDFSIO: nrBytes (MB) = 10.0
16/08/05 10:11:42 INFO fs.TestDFSIO: bufferSize = 1000000
16/08/05 10:11:42 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/08/05 10:11:43 INFO fs.TestDFSIO: creating control file: 10485760 bytes, 10 files
16/08/05 10:11:44 INFO fs.TestDFSIO: created control files for: 10 files
16/08/05 10:11:45 INFO client.RMProxy: Connecting to ResourceManager at hadoop-master/192.168.20.141:8032
16/08/05 10:11:45 INFO client.RMProxy: Connecting to ResourceManager at hadoop-master/192.168.20.141:8032
16/08/05 10:11:46 INFO mapred.FileInputFormat: Total input paths to process : 10
16/08/05 10:11:46 INFO mapreduce.JobSubmitter: number of splits:10
16/08/05 10:11:46 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470403413050_0001
16/08/05 10:11:46 INFO impl.YarnClientImpl: Submitted application application_1470403413050_0001
16/08/05 10:11:47 INFO mapreduce.Job: The url to track the job: http://hadoop-master:8088/proxy/application_1470403413050_0001/
16/08/05 10:11:47 INFO mapreduce.Job: Running job: job_1470403413050_0001
16/08/05 10:11:57 INFO mapreduce.Job: Job job_1470403413050_0001 running in uber mode : false
16/08/05 10:11:57 INFO mapreduce.Job:  map 0% reduce 0%
16/08/05 10:12:27 INFO mapreduce.Job:  map 40% reduce 0%
16/08/05 10:12:53 INFO mapreduce.Job:  map 47% reduce 13%
16/08/05 10:12:54 INFO mapreduce.Job:  map 53% reduce 13%
16/08/05 10:13:03 INFO mapreduce.Job:  map 60% reduce 13%
16/08/05 10:13:04 INFO mapreduce.Job:  map 67% reduce 13%
16/08/05 10:13:06 INFO mapreduce.Job:  map 80% reduce 13%
16/08/05 10:13:08 INFO mapreduce.Job:  map 83% reduce 13%
16/08/05 10:13:10 INFO mapreduce.Job:  map 90% reduce 13%
16/08/05 10:13:11 INFO mapreduce.Job:  map 100% reduce 13%
16/08/05 10:13:12 INFO mapreduce.Job:  map 100% reduce 17%
16/08/05 10:13:18 INFO mapreduce.Job:  map 100% reduce 20%
16/08/05 10:13:30 INFO mapreduce.Job:  map 100% reduce 67%
16/08/05 10:13:36 INFO mapreduce.Job:  map 100% reduce 100%
16/08/05 10:13:37 INFO mapreduce.Job: Job job_1470403413050_0001 completed successfully
16/08/05 10:13:37 INFO mapreduce.Job: Counters: 51
    File System Counters
        FILE: Number of bytes read=840
        FILE: Number of bytes written=1269539
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=2390
        HDFS: Number of bytes written=104857676
        HDFS: Number of read operations=43
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=12
    Job Counters 
        Killed map tasks=2
        Launched map tasks=13
        Launched reduce tasks=1
        Data-local map tasks=11
        Rack-local map tasks=2
        Total time spent by all maps in occupied slots (ms)=646035
        Total time spent by all reduces in occupied slots (ms)=66302
        Total time spent by all map tasks (ms)=646035
        Total time spent by all reduce tasks (ms)=66302
        Total vcore-seconds taken by all map tasks=646035
        Total vcore-seconds taken by all reduce tasks=66302
        Total megabyte-seconds taken by all map tasks=661539840
        Total megabyte-seconds taken by all reduce tasks=67893248
    Map-Reduce Framework
        Map input records=10
        Map output records=50
        Map output bytes=734
        Map output materialized bytes=894
        Input split bytes=1270
        Combine input records=0
        Combine output records=0
        Reduce input groups=5
        Reduce shuffle bytes=894
        Reduce input records=50
        Reduce output records=5
        Spilled Records=100
        Shuffled Maps =10
        Failed Shuffles=0
        Merged Map outputs=10
        GC time elapsed (ms)=8218
        CPU time spent (ms)=15840
        Physical memory (bytes) snapshot=2184921088
        Virtual memory (bytes) snapshot=9295441920
        Total committed heap usage (bytes)=1374396416
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=1120
    File Output Format Counters 
        Bytes Written=76
16/08/05 10:13:37 WARN hdfs.DFSClient: DFSInputStream has been closed already
16/08/05 10:13:37 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/08/05 10:13:37 INFO fs.TestDFSIO:            Date & time: Fri Aug 05 10:13:37 EDT 2016
16/08/05 10:13:37 INFO fs.TestDFSIO:        Number of files: 10
16/08/05 10:13:37 INFO fs.TestDFSIO: Total MBytes processed: 100.0
16/08/05 10:13:37 INFO fs.TestDFSIO:      Throughput mb/sec: 1.4243796826482067
16/08/05 10:13:37 INFO fs.TestDFSIO: Average IO rate mb/sec: 6.660604000091553
16/08/05 10:13:37 INFO fs.TestDFSIO:  IO rate std deviation: 8.936692949846902
16/08/05 10:13:37 INFO fs.TestDFSIO:     Test exec time sec: 112.884
16/08/05 10:13:37 INFO fs.TestDFSIO:

测试读文件速度

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.0-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 10MB

执行结果

[root@hadoop-master hadoop-2.7.0]# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.0-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 10MB
16/08/05 10:25:55 INFO fs.TestDFSIO: TestDFSIO.1.8
16/08/05 10:25:55 INFO fs.TestDFSIO: nrFiles = 10
16/08/05 10:25:55 INFO fs.TestDFSIO: nrBytes (MB) = 10.0
16/08/05 10:25:55 INFO fs.TestDFSIO: bufferSize = 1000000
16/08/05 10:25:55 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/08/05 10:25:56 INFO fs.TestDFSIO: creating control file: 10485760 bytes, 10 files
16/08/05 10:25:57 INFO fs.TestDFSIO: created control files for: 10 files
16/08/05 10:25:57 INFO client.RMProxy: Connecting to ResourceManager at hadoop-master/192.168.20.141:8032
16/08/05 10:25:57 INFO client.RMProxy: Connecting to ResourceManager at hadoop-master/192.168.20.141:8032
16/08/05 10:25:58 INFO mapred.FileInputFormat: Total input paths to process : 10
16/08/05 10:25:58 INFO mapreduce.JobSubmitter: number of splits:10
16/08/05 10:25:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470403413050_0002
16/08/05 10:25:59 INFO impl.YarnClientImpl: Submitted application application_1470403413050_0002
16/08/05 10:25:59 INFO mapreduce.Job: The url to track the job: http://hadoop-master:8088/proxy/application_1470403413050_0002/
16/08/05 10:25:59 INFO mapreduce.Job: Running job: job_1470403413050_0002
16/08/05 10:26:07 INFO mapreduce.Job: Job job_1470403413050_0002 running in uber mode : false
16/08/05 10:26:07 INFO mapreduce.Job:  map 0% reduce 0%
16/08/05 10:26:36 INFO mapreduce.Job:  map 20% reduce 0%
16/08/05 10:26:37 INFO mapreduce.Job:  map 30% reduce 0%
16/08/05 10:26:38 INFO mapreduce.Job:  map 50% reduce 0%
16/08/05 10:26:39 INFO mapreduce.Job:  map 100% reduce 0%
16/08/05 10:26:44 INFO mapreduce.Job:  map 100% reduce 100%
16/08/05 10:26:46 INFO mapreduce.Job: Job job_1470403413050_0002 completed successfully
16/08/05 10:26:46 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=832
        FILE: Number of bytes written=1269501
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=104859990
        HDFS: Number of bytes written=77
        HDFS: Number of read operations=53
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=10
        Launched reduce tasks=1
        Data-local map tasks=10
        Total time spent by all maps in occupied slots (ms)=291173
        Total time spent by all reduces in occupied slots (ms)=5582
        Total time spent by all map tasks (ms)=291173
        Total time spent by all reduce tasks (ms)=5582
        Total vcore-seconds taken by all map tasks=291173
        Total vcore-seconds taken by all reduce tasks=5582
        Total megabyte-seconds taken by all map tasks=298161152
        Total megabyte-seconds taken by all reduce tasks=5715968
    Map-Reduce Framework
        Map input records=10
        Map output records=50
        Map output bytes=726
        Map output materialized bytes=886
        Input split bytes=1270
        Combine input records=0
        Combine output records=0
        Reduce input groups=5
        Reduce shuffle bytes=886
        Reduce input records=50
        Reduce output records=5
        Spilled Records=100
        Shuffled Maps =10
        Failed Shuffles=0
        Merged Map outputs=10
        GC time elapsed (ms)=3177
        CPU time spent (ms)=7570
        Physical memory (bytes) snapshot=2147774464
        Virtual memory (bytes) snapshot=9255862272
        Total committed heap usage (bytes)=1374396416
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=1120
    File Output Format Counters 
        Bytes Written=77
16/08/05 10:26:46 WARN hdfs.DFSClient: DFSInputStream has been closed already
16/08/05 10:26:46 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
16/08/05 10:26:46 INFO fs.TestDFSIO:            Date & time: Fri Aug 05 10:26:46 EDT 2016
16/08/05 10:26:46 INFO fs.TestDFSIO:        Number of files: 10
16/08/05 10:26:46 INFO fs.TestDFSIO: Total MBytes processed: 100.0
16/08/05 10:26:46 INFO fs.TestDFSIO:      Throughput mb/sec: 29.197080291970803
16/08/05 10:26:46 INFO fs.TestDFSIO: Average IO rate mb/sec: 47.35454177856445
16/08/05 10:26:46 INFO fs.TestDFSIO:  IO rate std deviation: 29.781282953365924
16/08/05 10:26:46 INFO fs.TestDFSIO:     Test exec time sec: 48.942
16/08/05 10:26:46 INFO fs.TestDFSIO:

命令查看

安装目录执行

cat TestDFSIO_results.log

执行结果

[root@hadoop-master hadoop-2.7.0]# cat TestDFSIO_results.log 
----- TestDFSIO ----- : write
           Date & time: Fri Aug 05 10:13:37 EDT 2016
       Number of files: 10
Total MBytes processed: 100.0
     Throughput mb/sec: 1.4243796826482067
Average IO rate mb/sec: 6.660604000091553
 IO rate std deviation: 8.936692949846902
    Test exec time sec: 112.884

----- TestDFSIO ----- : read
           Date & time: Fri Aug 05 10:26:46 EDT 2016
       Number of files: 10
Total MBytes processed: 100.0
     Throughput mb/sec: 29.197080291970803
Average IO rate mb/sec: 47.35454177856445
 IO rate std deviation: 29.781282953365924
    Test exec time sec: 48.942

web查看

http://hadoop-master:50070/

清除测试数据

将测试数据删除

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.0-tests.jar TestDFSIO -clean

单词统计测试

新建一个words.txt

vim words.txt

内容

hello hadoop hbase mytest
hadoop-node1
hadoop-master
hadoop-node2
this is my test

上传文件

bin/hadoop fs -put words.txt /tmp/

使用mapreduce统计单词个数
统计指定文件单词个数，并将结果输入到指定文件

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0.jar wordcount /tmp/words.txt /tmp/words_result.txt

结果 

hadoop  1
hadoop-master   1
hadoop-node1     1
hadoop-node2     1
hbase   1
hello   1
is  1
my  1
mytest  1
test    1
this    1

代码查看
查看根目录

bin/hadoop fs -ls /

查看指定文件

bin/hadoop fs -cat hdfs:///tmp/words_result.txt/part-r-00000

执行结果

[root@hadoop-master hadoop-2.7.0]# bin/hadoop fs -cat hdfs:///tmp/words_result.txt/part-r-00000
hadoop  1
hadoop-master   1
hadoop-node1    1
hadoop-node2    1
hbase   1
hello   1
is  1
my  1
mytest  1
test    1
this    1

更多Hadoop2.7.0常用操作HDFS的Shell命令

http://www.cuiweiyou.com/1405.html

参考

http://blog.itpub.net/25854343/viewspace-1425183/

-贫寒豌豆
关注
专栏目录
Hadoop——测试代码
Devin
04-23 947
1、本地测试代码class WCMap extends Mapper<LongWritable, Text, Text, IntWritable>{ /** * key: 偏移量 * value:每行文本内容 * context：输出内容上下文 */ @Override protected void map(LongWritable key, Text value,Contex...
hadoop测试代码
11-21
centos7安装hadoop2.6.0后，使用eclipse安装mapreduce插件，并测试是否正常使用
Hadoop基准测试
weixin_35994859的博客
12-07 149
基准测试 为了更加全面的了解系统，我们需要通过测试，发现瓶颈所在。通过测试，可以用来分析系统的性能。基准测试被打包在测试程序JAR文件中，通过无参调用JAR文件可以得到其列表 IO写操作 命令： hadoop jar /opt/module/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2-tes...
压测实操--TestDFSIO压测hdfs读写方案
最新发布
07-25 313
在集群硬件资源能给到最大条件下（比如namenode最大能给到8G，再大就会影响其他组件的内存使用），结合并行文件和文件大小对hdfs的i/o进行测试，需要重点关注io速率、rpc、namenode同步和集群是否告警情况。也可以将执行命令得到的io rate和throughput进行整理，形成曲线图查看趋势。更多技术信息请查看云掣官网t=yrgw。
【大数据Hadoop】Hadoop源代码调试（持续更新...）
如切如磋，如琢如磨，臻于至善。
01-29 257
hadoop 源代码 各个组件的调试，分析源代码，跟踪源代码，解析源代码，理解源代码，熟读源代码
hadoop安装与配置（本地模式）
lihuan1124的博客
06-22 1212
hadoop版本:hadoop-2.7.2虚拟机安装前期配置：IP地址、IP与主机映射、关闭防火墙一、安装java jdk   1、下载java jdk1.8版本，放在/home/software目录下，   2、解压：tar -zxvf java-jdk*****.jar   3、vim ~/.bash_profile12345#添加export JAVA_HOME=root/app/jdk1....
Hadoop2.7.0学习——Windows下hadoop-eclipse-plugin-2.7.0插件安装-附件资源
03-02
Hadoop2.7.0学习——Windows下hadoop-eclipse-plugin-2.7.0插件安装-附件资源
hadoop2.7.0 windows
09-21
Hadoop 2.7.0 是一个开源框架，主要用于分布式存储和计算，是大数据处理领域的重要工具。在Windows环境下安装和使用Hadoop可能与在Linux上有所不同，但同样可以实现强大的功能。本教程将详细解释如何在Windows系统上...
Hadoop高级编程——构建与实现大数据解决方案.rar
10-25
本压缩包“Hadoop高级编程——构建与实现大数据解决方案”将深入探讨如何利用Hadoop进行高效的数据操作，构建实际的大数据解决方案。 一、Hadoop概述 Hadoop是由Apache基金会开发的开源项目，主要由Hadoop ...
Hadoop运行WordCount示例出现的一些问题及解决办法
aaronhanfei的专栏
03-30 1035
博客标题起的真别扭。。 :evil: 先说两点： 1.用cygwin伪分布式环境运行的。 2.hadoop开发者第一期里有个DFSOperator的示例，那个更简单，怎么打jar包，怎么运行之类的问题可以先参见一个那篇文章。 运行WordCount示例遇到了两个问题： a. ***/work/tmp does not exist的问题 解决办法：配置conf/mapred-sit...
hadoop-mapreduce-client-app-2.7.1.jar
06-03
Java开发大数据jar包，为我们大数据分析提供我们专门的jar包，
linux运行hadoop实例wordcount详解
DaiDaiHan1的专栏
02-24 1717
在服务器上配置了hadoop准备自己从wordcount.java开始到跑起wordcount程序，但是就这简单的几步遇到了很多问题，现在总结一下我碰到的问题： 整个过程步骤是： 1.首先创建文件目录： mkdir  Test 2.编译wordcount.java文件生成.class文件，存放在第一步创建的文件夹Test中： javac  -classpath /hadoop*core*
Hadoop基础篇 01 Hadoop集群的部署与使用
qilie_32的博客
04-13 1166
集群节点类型 集群硬件配置 集群规模 集群网络拓扑 集群的建立与安装 集群基准测试 在云计算环境中使用Hadoop
hadoop--基压测试
没有合适的昵称
02-09 1595
在安装完集群后,我们都需要先对集群做一些测试,下面讲解测试读写的性能 写性能 包名: Apache: hadoop-mapreduce-client-jobclient-2.7.5-tests.jar CDH: hadoop-mapreduce-client-jobclient-3.0.0-cdh6.2.0-tests.jar 包路径: /home/hadoop-jrq/bigdata/hadoo...
Hadoop开发相关问题总结
06-14 4043
总结自己在Hadoop开发中遇到的问题，主要在mapreduce代码执行方面。大部分来自日常代码执行错误的解决方法，还有一些是对JAVA、Hadoop剖析。对于问题，通过查询stackoverflow、csdn找到了解决方法。汇总出来以后查询方便。内容将不定期更新。
hadoop 和spark的基准测试(1)
r8t7y6的博客
08-22 1506
Hadoop 2.8.0 基准测试 1.查看jar包命令 2.建立乱序100M数据 3.排序 4.删除文件     1. 执行: hadoop jar../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.0-tests.jar 结果:  Anexample program must be given a
hadoop实例WordCount程序一步一步运行
china1000的专栏
12-14 590
虽说现在用Eclipse下开发hadoop程序很方便了，但是命令行方式对于小程序开发验证很方便。这是初学hadoop时的笔记，记录下来以备查。 　　1. 经典的WordCound程序（WordCount.java），可参见 hadoop0.18文档 import java.io.IOException; import java.util.ArrayList; import java
【大数据实验2】note2：Hadoop测试工具
nosteglic
10-24 2106
note2：Hadoop测试工具
Hadoop2.7.0双节点集群搭建教程与通信配置
- Hadoop2.7.0采用主备NameNode架构，一个作为active节点提供服务，另一个standby节点用于备份并保持同步。 - 使用QJM（Quorum Journal Manager）方式实现HDFS元数据一致性，确保数据在多数JournalNode中写入成功...
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

-贫寒豌豆

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值