2.mr对wordcount性能测试50g数据

1.对4.1中生成50g单词进行统计

基于mr磁盘测试

开始:19/12/27 16:23:45

结束:19/12/27 16:26:22

 

 

 

19/12/27 16:26:15 INFO mapreduce.Job: map 100% reduce 75%

19/12/27 16:26:16 INFO mapreduce.Job: map 100% reduce 82%

19/12/27 16:26:17 INFO mapreduce.Job: map 100% reduce 87%

19/12/27 16:26:18 INFO mapreduce.Job: map 100% reduce 92%

19/12/27 16:26:19 INFO mapreduce.Job: map 100% reduce 95%

19/12/27 16:26:20 INFO mapreduce.Job: map 100% reduce 99%

19/12/27 16:26:21 INFO mapreduce.Job: map 100% reduce 100%

19/12/27 16:26:22 INFO mapreduce.Job: Job job_1577432091531_0001 completed successfully

19/12/27 16:26:22 INFO mapreduce.Job: Counters: 55

File System Counters

FILE: Number of bytes read=5603673641

FILE: Number of bytes written=3363374125

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=54698304302

HDFS: Number of bytes written=535602831

HDFS: Number of read operations=1850

HDFS: Number of large read operations=0

HDFS: Number of write operations=200

HDFS: Number of bytes read erasure-coded=0

Job Counters

Launched map tasks=450

Launched reduce tasks=100

Data-local map tasks=438

Rack-local map tasks=12

Total time spent by all maps in occupied slots (ms)=19917612

Total time spent by all reduces in occupied slots (ms)=3050464

Total time spent by all map tasks (ms)=19917612

Total time spent by all reduce tasks (ms)=3050464

Total vcore-milliseconds taken by all map tasks=19917612

Total vcore-milliseconds taken by all reduce tasks=3050464

Total megabyte-milliseconds taken by all map tasks=20395634688

Total megabyte-milliseconds taken by all reduce tasks=3123675136

Map-Reduce Framework

Map input records=1373955

Map output records=5042161744

Map output bytes=75172225085

Map output materialized bytes=1352230193

Input split bytes=63450

Combine input records=5042161744

Combine output records=117362307

Reduce input groups=23909852

Reduce shuffle bytes=1352230193

Reduce input records=117362307

Reduce output records=23909852

Spilled Records=349748802

Shuffled Maps =45000

Failed Shuffles=0

Merged Map outputs=45000

GC time elapsed (ms)=349516

CPU time spent (ms)=16941720

Physical memory (bytes) snapshot=358828400640

Virtual memory (bytes) snapshot=1499865440256

Total committed heap usage (bytes)=413318774784

Peak Map Physical memory (bytes)=720048128

Peak Map Virtual memory (bytes)=2736115712

Peak Reduce Physical memory (bytes)=562401280

Peak Reduce Virtual memory (bytes)=2844385280

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=54698240852

File Output Format Counters

Bytes Written=535602831

 

 

 

 

 

 

 

9/12/27 16:26:22 INFO mapreduce。工作:计数器:55

文件系统计数器

FILE:读取的字节数= 5603673641

FILE:写入的字节数= 3363374125

FILE:读取操作数= 0

FILE:大读取操作数= 0

FILE:写操作数= 0

HDFS:读取的字节数= 54698304302

HDFS:写入的字节数= 535602831

HDFS:读取操作数= 1850

HDFS:大型读取操作数= 0

HDFS:写入操作数= 200

HDFS:读取的字节数擦除编码= 0

工作柜台

启动的map任务= 450

启动减少任务= 100

数据局部map任务= 438

机架本地map任务= 12

所有map在占用的插槽中花费的总时间(ms)= 19917612

所有人花费的总时间减少了占用的时隙(ms)= 3050464

所有map任务花费的总时间(ms)= 19917612

所有缩减任务花费的总时间(毫秒)= 3050464

所有map任务花费的总vcore-毫秒数= 19917612

所有reduce任务花费的总vcore-毫秒数= 3050464

所有map任务花费的总兆字节毫秒数= 20395634688

所有缩减任务花费的总兆字节-毫秒= 3123675136

Map-Reduce框架

map输入记录= 1373955

map输出记录= 5042161744

映射输出字节= 75172225085

映射输出实例化字节= 1352230193

输入拆分字节= 63450

合并输入记录= 5042161744

合并输出记录= 117362307

减少输入组= 23909852

减少随机播放字节= 1352230193

减少输入记录= 117362307

减少输出记录= 23909852

溢出记录= 349748802

随机map= 45000

随机洗牌失败= 0

合并map输出= 45000

已用GC时间(ms)= 349516

花的CPU时间(毫秒)= 16941720

物理内存(字节)快照= 358828400640

虚拟内存(字节)快照= 1499865440256

已提交的总堆使用量(字节)= 413318774784

峰图物理内存(字节)= 720048128

峰图虚拟内存(字节)= 2736115712

峰值减少物理内存(字节)= 562401280

峰值减少虚拟内存(字节)= 2844385280

随机错误

BAD_ID = 0

连接= 0

IO_ERROR = 0

WRONG_LENGTH = 0

WRONG_MAP = 0

WRONG_REDUCE = 0

文件输入格式计数器

读取的字节数= 54698240852

文件输出格式计数器

写入的字节数= 535602831

 

 

 
发布了38 篇原创文章 · 获赞 19 · 访问量 4252
展开阅读全文

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: 大白 设计师: CSDN官方博客

分享到微信朋友圈

×

扫一扫,手机浏览