生产硬件规格
浪潮:20核,128G,10.98TB
浪潮:20核,128G,10.98TB
浪潮:20核,128G,10.98TB
浪潮:20核,128G,10.98TB
浪潮:20核,128G,10.98TB
引用1.测试集群
http://note.youdao.com/noteshare?id=3332b5f14230918c41f084a1524a7a03&sub=6F5522350B9A44E0B6C895103361CF6A
2.生产集群
a.randomwriter
操作:
MapReduce执行randomwriter向hdfs写入数据,一个MapReduce向每个节点写入10g数据
19/12/27 14:23:36 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=10984570
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=7140
HDFS: Number of bytes written=54665407316
HDFS: Number of read operations=300
HDFS: Number of large read operations=0
HDFS: Number of write operations=100
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=50
Other local map tasks=50
Total time spent by all maps in occupied slots (ms)=13675032
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=13675032
Total vcore-milliseconds taken by all map tasks=13675032
Total megabyte-milliseconds taken by all map tasks=14003232768
Map-Reduce Framework
Map input records=50
Map output records=81929283
Input split bytes=7140
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=30500
CPU time spent (ms)=2340830
Physical memory (bytes) snapshot=19369316352
Virtual memory (bytes) snapshot=135502348288
Total committed heap usage (bytes)=22564831232
Peak Map Physical memory (bytes)=457793536
Peak Map Virtual memory (bytes)=2738778112
org.apache.hadoop.examples.RandomTextWriter$Counters
BYTES_WRITTEN=53687109184
RECORDS_WRITTEN=81929283
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=54665407316
Job ended: Fri Dec 27 14:23:36 CST 2019
The job took 338 seconds.
19/12/27 14:23:36 INFO mapreduce。工作:计数器:35
文件系统计数器
FILE:读取的字节数= 0
FILE:写入的字节数= 10984570
FILE:读取操作数= 0
FILE:大读取操作数= 0
FILE:写操作数= 0
HDFS:读取的字节数= 7140
HDFS:写入的字节数= 54665407316
HDFS:读取操作数= 300
HDFS:大型读取操作数= 0
HDFS:写入操作数= 100
HDFS:读取的字节数擦除编码= 0
工作柜台
启动的地图任务= 50
其他本地地图任务= 50
所有地图在占用的插槽中花费的总时间(毫秒)= 13675032
所有设备花费的总时间减少,占用的时隙(ms)= 0
所有地图任务花费的总时间(毫秒)= 13675032
所有地图任务花费的总vcore-毫秒= 13675032
所有地图任务花费的总兆字节毫秒数= 14003232768
Map-Reduce框架
地图输入记录= 50
地图输出记录= 81929283
输入拆分字节= 7140
溢出记录= 0
随机洗牌失败= 0
合并地图输出= 0
已用GC时间(ms)= 30500
花的CPU时间(毫秒)= 2340830
物理内存(字节)快照= 19369316352
虚拟内存(字节)快照= 135502348288
已提交的总堆使用量(字节)= 22564831232
峰图物理内存(字节)= 457793536
峰图虚拟内存(字节)= 2738778112
org.apache.hadoop.examples.RandomTextWriter $ Counters
BYTES_WRITTEN = 53687109184
RECORDS_WRITTEN = 81929283
文件输入格式计数器
读取的字节数= 0
文件输出格式计数器
写入的字节数= 54665407316
工作已结束:2019年12月27日星期五12:23:36
这项工作花了338秒。