Microsoft Azure 云服务器Ambari测试

最新推荐文章于 2023-01-12 14:50:20 发布
最新推荐文章于 2023-01-12 14:50:20 发布
阅读量398 收藏
点赞数
分类专栏： 大数据平台测试
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/rainbowzhouj/article/details/90487396
版权

Microsoft Azure 云服务器Ambari测试

相关链接：

https://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/

https://www.slideshare.net/BartVandewoestyne/big-data-benchmarking

测试方法

1、测试云服务器的相关信息及IO

2、测试HDFS的IO

测试步骤

测试云服务器的相关信息及IO

使用UnixBench测试

yum -y install perl-Time-HiRes
wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/byte-unixbench/UnixBench5.1.3.tgz
tar -zxvf UnixBench5.1.3.tgz
cd UnixBench
make
./Run

通过测试得到以下信息：

========================================================================
   BYTE UNIX Benchmarks (Version 5.1.2)

   System: host1.cloudlocal.test: GNU/Linux
   OS: GNU/Linux -- 4.9.75-29.el7.x86_64 -- #1 SMP Fri Jan 5 19:42:28 UTC 2018
   Machine: x86_64 (x86_64)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 1: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 2: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 3: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 4: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 5: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 6: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 7: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 8: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 9: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 10: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 11: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 12: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 13: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 14: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   CPU 15: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (4199.8 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   06:53:38 up 15:10,  2 users,  load average: 0.43, 1.03, 0.91; runlevel 5

------------------------------------------------------------------------
Benchmark Run: 三 4月 25 2018 06:53:38 - 07:22:16
16 CPUs in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       33841196.1 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     4518.3 MWIPS (10.7 s, 7 samples)
Execl Throughput                               2049.1 lps   (29.8 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        764894.1 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          204490.0 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       2521457.9 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1162324.8 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 139672.3 lps   (10.0 s, 7 samples)
Process Creation                               6805.2 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   5691.3 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                    976.6 lpm   (60.0 s, 2 samples)
System Call Overhead                         995472.1 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   33841196.1   2899.8
Double-Precision Whetstone                       55.0       4518.3    821.5
Execl Throughput                                 43.0       2049.1    476.5
File Copy 1024 bufsize 2000 maxblocks          3960.0     764894.1   1931.6
File Copy 256 bufsize 500 maxblocks            1655.0     204490.0   1235.6
File Copy 4096 bufsize 8000 maxblocks          5800.0    2521457.9   4347.3
Pipe Throughput                               12440.0    1162324.8    934.3
Pipe-based Context Switching                   4000.0     139672.3    349.2
Process Creation                                126.0       6805.2    540.1
Shell Scripts (1 concurrent)                     42.4       5691.3   1342.3
Shell Scripts (8 concurrent)                      6.0        976.6   1627.6
System Call Overhead                          15000.0     995472.1    663.6
                                                                   ========
System Benchmarks Index Score                                        1096.1

------------------------------------------------------------------------
Benchmark Run: 三 4月 25 2018 07:22:16 - 07:50:46
16 CPUs in system; running 16 parallel copies of tests

Dhrystone 2 using register variables      251068998.6 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                    55154.7 MWIPS (9.9 s, 7 samples)
Execl Throughput                              30795.9 lps   (29.8 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        838158.5 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          214567.8 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       2752367.1 KBps  (30.0 s, 2 samples)
Pipe Throughput                             9622691.1 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                1170824.0 lps   (10.0 s, 7 samples)
Process Creation                              61039.3 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                  28312.5 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   5327.4 lpm   (60.1 s, 2 samples)
System Call Overhead                        6848982.3 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0  251068998.6  21514.1
Double-Precision Whetstone                       55.0      55154.7  10028.1
Execl Throughput                                 43.0      30795.9   7161.8
File Copy 1024 bufsize 2000 maxblocks          3960.0     838158.5   2116.6
File Copy 256 bufsize 500 maxblocks            1655.0     214567.8   1296.5
File Copy 4096 bufsize 8000 maxblocks          5800.0    2752367.1   4745.5
Pipe Throughput                               12440.0    9622691.1   7735.3
Pipe-based Context Switching                   4000.0    1170824.0   2927.1
Process Creation                                126.0      61039.3   4844.4
Shell Scripts (1 concurrent)                     42.4      28312.5   6677.5
Shell Scripts (8 concurrent)                      6.0       5327.4   8879.0
System Call Overhead                          15000.0    6848982.3   4566.0
                                                                   ========
System Benchmarks Index Score                                        5387.1

测试详解：

测试项目项目说明基准线
Dhrystone 2 using register variables测试处理字符串116700.0lps
Double-Precision Whetstone测试浮点数操作的速度和效率55.0MWIPS
Execl Throughput此测试考察每秒钟可以执行的 execl 系统调用的次数43.0lps
File Copy 1024 bufsize 2000 maxblocks测试从一个文件向另外一个文件传输数据的速率3960.0KBps
File Copy 256 bufsize 500 maxblocks测试从一个文件向另外一个文件传输数据的速率。1655.0KBps
File Read 4096 bufsize 8000 maxblocks测试从一个文件向另外一个文件传输数据的速率。5800.0KBps
Pipe-based Context Switching测试两个进程（每秒钟）通过一个管道交换一个不断增长的整数的次数12440.0lps
Pipe Throughput一秒钟内一个进程可以向一个管道写 512 字节数据然后再读回的次数4000.0lps
Process Creation测试每秒钟一个进程可以创建子进程然后收回子进程的次数（子进程一定立即退出）。126.0lps
Shell Scripts (8 concurrent)测试一秒钟内一个进程可以并发地开始一个shell 脚本的 n 个拷贝的次数，n 一般取值1，2，4，8.42.4lpm
System Call Overhead测试进入和离开操作系统内核的代价，即一次系统调用的代价。6.0lpm

使用superbench测试

wget -qO- --no-check-certificate https://raw.githubusercontent.com/oooldking/script/master/superbench.sh | bash

通过测试得到以下信息：


 CPU Model            : Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
 CPU Cores            : 4 Cores @ 2294.688 MHz x86_64
 CPU Cache            : 51200 KB
 OS                   : Ubuntu 16.04.4 LTS (64 Bit) Hyper-V
 Kernel               : 4.15.0-1013-azure
 Total Space          : 1167.0 GB (25.3 GB Used)
 Total RAM            : 28114 MB (7162 MB Used 12236 MB Buff)
 Total SWAP           : 7167 MB (0 MB Used)
 Uptime               : 0 days 4 hour 32 min
 Load average         : 2.24, 1.75, 1.62
 ASN & ISP            : AS8075, Microsoft Corporation
 Organization         : Microsoft Azure
 Location             : Virginia, United States / US
 Region               : Virginia
----------------------------------------------------------------------
 I/O Speed( 1.0GB )   : 52.7 MB/s
 I/O Speed( 1.0GB )   : 48.0 MB/s
 I/O Speed( 1.0GB )   : 45.8 MB/s
 Average I/O Speed    : 48.8 MB/s

通过测试得到磁盘挂载信息：

sshuser@hn0-sinocb:~$ df -TH
Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs   15G     0   15G   0% /dev
tmpfs          tmpfs     3.0G  302M  2.7G  11% /run
/dev/sda1      ext4      1.1T   19G  1.1T   2% /
tmpfs          tmpfs      15G   25k   15G   1% /dev/shm
tmpfs          tmpfs     5.3M     0  5.3M   0% /run/lock
tmpfs          tmpfs      15G     0   15G   0% /sys/fs/cgroup
/dev/sdb1      ext4      212G  7.8G  193G   4% /mnt
tmpfs          tmpfs     3.0G     0  3.0G   0% /run/user/2008
tmpfs          tmpfs     3.0G     0  3.0G   0% /run/user/202
tmpfs          tmpfs     3.0G     0  3.0G   0% /run/user/2007
tmpfs          tmpfs     3.0G     0  3.0G   0% /run/user/2002
tmpfs          tmpfs     3.0G     0  3.0G   0% /run/user/203
tmpfs          tmpfs     3.0G     0  3.0G   0% /run/user/2001
tmpfs          tmpfs     3.0G     0  3.0G   0% /run/user/2019

利用hdparm测试read的速度：

sshuser@hn0-sinocb:~$ sudo hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads: 124 MB in  3.07 seconds =  40.35 MB/sec

sshuser@hn0-sinocb:~$ sudo hdparm -t /dev/sdb1

/dev/sdb1:
 Timing buffered disk reads: 7854 MB in  3.00 seconds = 2617.24 MB/sec

利用dd测试服务器自身IO：

# write
time dd if=/dev/zero of=/var/test bs=8k count=1000000
# read
 time dd if=/var/test of=/dev/null bs=8k

测试HDFS的IO

HDFS 使用GET、PUT命令测试：

test_io_0的大小为2GB，耗时为44.1s，速度为46.4MB/s
sshuser@hn0-sinocb:~$ time hadoop fs -get /benchmarks/TestDFSIO/io_data/test_io_0 /tmp
real    0m44.108s
user    0m18.905s
sys     0m4.411s


test_io_0的大小为2GB，耗时为10.9s，速度为187.8MB/s
sshuser@hn0-sinocb:~$ time hadoop fs -put /tmp/test_io_0 /tmp

real    0m10.910s
user    0m20.676s
sys     0m3.023s

HDFS 使用TestDFSIO命令测试：

通过实验得出：有4个值对IO速度有影响分别是nrFiles、fileSize、replication、Blocksize。

# 写测试 速度为57MB/s
sshuser@hn0-sinocb:~$ hadoop jar /usr/hdp/2.6.5.8-7/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 128M
18/08/03 08:01:44 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
18/08/03 08:01:44 INFO fs.TestDFSIO:            Date & time: Fri Aug 03 08:01:44 UTC 2018
18/08/03 08:01:44 INFO fs.TestDFSIO:        Number of files: 10
18/08/03 08:01:44 INFO fs.TestDFSIO: Total MBytes processed: 1280.0
18/08/03 08:01:44 INFO fs.TestDFSIO:      Throughput mb/sec: 56.02486103208299
18/08/03 08:01:44 INFO fs.TestDFSIO: Average IO rate mb/sec: 57.28535079956055
18/08/03 08:01:44 INFO fs.TestDFSIO:  IO rate std deviation: 9.138790402525261
18/08/03 08:01:44 INFO fs.TestDFSIO:     Test exec time sec: 27.783

18/08/03 04:23:22 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write 
18/08/03 04:23:22 INFO fs.TestDFSIO: Date & time: Fri Aug 03 04:23:22 UTC 2018 
18/08/03 04:23:22 INFO fs.TestDFSIO: Number of files: 10 
18/08/03 04:23:22 INFO fs.TestDFSIO: Total MBytes processed: 20480.0 
18/08/03 04:23:22 INFO fs.TestDFSIO: Throughput mb/sec: 44.43027072526776 
18/08/03 04:23:22 INFO fs.TestDFSIO: Average IO rate mb/sec: 44.44496536254883 
18/08/03 04:23:22 INFO fs.TestDFSIO: IO rate std deviation: 0.818583127053945 
18/08/03 04:23:22 INFO fs.TestDFSIO: Test exec time sec: 112.405

18/08/03 04:25:14 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write 
18/08/03 04:25:14 INFO fs.TestDFSIO: Date & time: Fri Aug 03 04:25:14 UTC 2018 
18/08/03 04:25:14 INFO fs.TestDFSIO: Number of files: 10 
18/08/03 04:25:14 INFO fs.TestDFSIO: Total MBytes processed: 20480.0 
18/08/03 04:25:14 INFO fs.TestDFSIO: Throughput mb/sec: 45.32076390271969 
18/08/03 04:25:14 INFO fs.TestDFSIO: Average IO rate mb/sec: 45.32685089111328 
18/08/03 04:25:14 INFO fs.TestDFSIO: IO rate std deviation: 0.5280477142580408 
18/08/03 04:25:14 INFO fs.TestDFSIO: Test exec time sec: 83.44

# 读测试 速度为46MB/s
sshuser@hn0-sinocb:~$ hadoop jar /usr/hdp/2.6.5.8-7/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -read -nrFiles 1 -fileSize 1280M
18/08/03 08:09:02 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
18/08/03 08:09:02 INFO fs.TestDFSIO:            Date & time: Fri Aug 03 08:09:02 UTC 2018
18/08/03 08:09:02 INFO fs.TestDFSIO:        Number of files: 10
18/08/03 08:09:02 INFO fs.TestDFSIO: Total MBytes processed: 1280.0
18/08/03 08:09:02 INFO fs.TestDFSIO:      Throughput mb/sec: 46.53868528214078
18/08/03 08:09:02 INFO fs.TestDFSIO: Average IO rate mb/sec: 46.76382827758789
18/08/03 08:09:02 INFO fs.TestDFSIO:  IO rate std deviation: 3.126925903426747
18/08/03 08:09:02 INFO fs.TestDFSIO:     Test exec time sec: 28.243

18/08/03 04:27:20 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read 
18/08/03 04:27:20 INFO fs.TestDFSIO: Date & time: Fri Aug 03 04:27:20 UTC 2018 
18/08/03 04:27:20 INFO fs.TestDFSIO: Number of files: 10 
18/08/03 04:27:20 INFO fs.TestDFSIO: Total MBytes processed: 20480.0 
18/08/03 04:27:20 INFO fs.TestDFSIO: Throughput mb/sec: 46.67423299740648 
18/08/03 04:27:20 INFO fs.TestDFSIO: Average IO rate mb/sec: 46.69725799560547 
18/08/03 04:27:20 INFO fs.TestDFSIO: IO rate std deviation: 1.038977669688391 
18/08/03 04:27:20 INFO fs.TestDFSIO: Test exec time sec: 68.322

18/08/03 04:29:00 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read 
18/08/03 04:29:00 INFO fs.TestDFSIO: Date & time: Fri Aug 03 04:29:00 UTC 2018 
18/08/03 04:29:00 INFO fs.TestDFSIO: Number of files: 10 
18/08/03 04:29:00 INFO fs.TestDFSIO: Total MBytes processed: 20480.0 
18/08/03 04:29:00 INFO fs.TestDFSIO: Throughput mb/sec: 46.54725296204626 
18/08/03 04:29:00 INFO fs.TestDFSIO: Average IO rate mb/sec: 46.61547088623047 
18/08/03 04:29:00 INFO fs.TestDFSIO: IO rate std deviation: 1.8338779757659875 
18/08/03 04:29:00 INFO fs.TestDFSIO: Test exec time sec: 68.055

HDFS 使用TeraSort

# 生成1280M数据，耗时22s
sshuser@hn0-sinocb:~$ hadoop jar /usr/hdp/2.6.5.8-7/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen 12800000 /tera
File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=320258
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                WASB: Number of bytes read=252
                WASB: Number of bytes written=1280000000
                WASB: Number of read operations=0
                WASB: Number of large read operations=0
                WASB: Number of write operations=0
        Job Counters
                Launched map tasks=2
                Other local map tasks=2
                Total time spent by all maps in occupied slots (ms)=21315
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=21315
                Total vcore-milliseconds taken by all map tasks=21315
                Total megabyte-milliseconds taken by all map tasks=32739840
        Map-Reduce Framework
                Map input records=12800000
                Map output records=12800000
                Input split bytes=167
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=882
                CPU time spent (ms)=28800
                Physical memory (bytes) snapshot=2392748032
                Virtual memory (bytes) snapshot=5661786112
                Total committed heap usage (bytes)=2059927552
        org.apache.hadoop.examples.terasort.TeraGen$Counters
                CHECKSUM=27488073979810235
        File Input Format Counters
                Bytes Read=0
        File Output Format Counters
                Bytes Written=1280000000


# 整理1280M数据，耗时66s
sshuser@hn0-sinocb:~$ hadoop jar /usr/hdp/2.6.5.8-7/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3.2.6.5.8-7.jar terasort /tera /tmp/tera_rp
  File System Counters
                FILE: Number of bytes read=2447891722
                FILE: Number of bytes written=3779899611
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                WASB: Number of bytes read=1280263628
                WASB: Number of bytes written=1280000000
                WASB: Number of read operations=0
                WASB: Number of large read operations=0
                WASB: Number of write operations=0
        Job Counters
                Launched map tasks=4
                Launched reduce tasks=1
                Rack-local map tasks=4
                Total time spent by all maps in occupied slots (ms)=84380
                Total time spent by all reduces in occupied slots (ms)=41598
                Total time spent by all map tasks (ms)=84380
                Total time spent by all reduce tasks (ms)=41598
                Total vcore-milliseconds taken by all map tasks=84380
                Total vcore-milliseconds taken by all reduce tasks=41598
                Total megabyte-milliseconds taken by all map tasks=129607680
                Total megabyte-milliseconds taken by all reduce tasks=63894528
        Map-Reduce Framework
                Map input records=12800000
                Map output records=12800000
                Map output bytes=1305600000
                Map output materialized bytes=1331200024
                Input split bytes=664
                Combine input records=0
                Combine output records=0
                Reduce input groups=12800000
                Reduce shuffle bytes=1331200024
                Reduce input records=12800000
                Reduce output records=12800000
                Spilled Records=36337420
                Shuffled Maps =4
                Failed Shuffles=0
                Merged Map outputs=4
                GC time elapsed (ms)=1500
                CPU time spent (ms)=97490
                Physical memory (bytes) snapshot=4674580480
                Virtual memory (bytes) snapshot=14154989568
                Total committed heap usage (bytes)=5284298752
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=1280261968
        File Output Format Counters
                Bytes Written=1280000000


# Tera的Teravalidate，耗时53s
sshuser@hn0-sinocb:~$ hadoop jar /usr/hdp/2.6.5.8-7/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3.2.6.5.8-7.jar teravalidate /tera /tmp/report
 File System Counters
                FILE: Number of bytes read=672045548
                FILE: Number of bytes written=1344572754
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                WASB: Number of bytes read=1280000498
                WASB: Number of bytes written=659244666
                WASB: Number of read operations=0
                WASB: Number of large read operations=0
                WASB: Number of write operations=0
        Job Counters
                Launched map tasks=2
                Launched reduce tasks=1
                Rack-local map tasks=2
                Total time spent by all maps in occupied slots (ms)=53714
                Total time spent by all reduces in occupied slots (ms)=13983
                Total time spent by all map tasks (ms)=53714
                Total time spent by all reduce tasks (ms)=13983
                Total vcore-milliseconds taken by all map tasks=53714
                Total vcore-milliseconds taken by all reduce tasks=13983
                Total megabyte-milliseconds taken by all map tasks=82504704
                Total megabyte-milliseconds taken by all reduce tasks=21477888
        Map-Reduce Framework
                Map input records=12800000
                Map output records=6400438
                Map output bytes=659244660
                Map output materialized bytes=672045548
                Input split bytes=332
                Combine input records=0
                Combine output records=0
                Reduce input groups=6
                Reduce shuffle bytes=672045548
                Reduce input records=6400438
                Reduce output records=6400434
                Spilled Records=12800876
                Shuffled Maps =2
                Failed Shuffles=0
                Merged Map outputs=2
                GC time elapsed (ms)=1680
                CPU time spent (ms)=52150
                Physical memory (bytes) snapshot=2986717184
                Virtual memory (bytes) snapshot=8509726720
                Total committed heap usage (bytes)=3184525312
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=1280000000
        File Output Format Counters
                Bytes Written=659244666

测试结论

Ambari的HDFS测试在当前配置下，IO速度为50MB/s。

rainbowzhouj
关注
专栏目录
Hadoop 集群规划与部署最佳实践
程序员光剑
08-04 927
2009年2月2日，Apache Hadoop项目诞生。它是一个开源的分布式系统基础架构，用于存储、处理和分析海量的数据。Hadoop具有高容错性、可靠性、可扩展性、适应性等特征，因而广泛应用于数据仓库、日志分析、网络流量监测、推荐引擎、搜索引擎等领域。由于Hadoop采用“分而治之”的架构设计理念，因此可以轻松应对数据量、计算能力和存储成本的增长。2013年底，随着云计算的火爆，基于云平台的Hadoop服务逐渐成为当下热门话题。
大数据相关开源项目汇总
Null的博客
12-06 2472
调度与管理服务 **Azkaban **是一款基于Java编写的任务调度系统任务调度，来自LinkedIn公司，用于管理他们的Hadoop批处理工作流。Azkaban根据工作的依赖性进行排序，提供友好的Web用户界面来维护和跟踪用户的工作流程。 **YARN **是一种新的Hadoop资源管理器，它是一个通用资源管理系统，可为上层应用提供统一的资源管理和调度，解决了旧MapReduce框架的性...
ambari 部署与测试
08-02
ambari 安装，以及使用ambari 部署大数据集群，ambari metric 监控等
azure 测试服务器性能,测试 Azure VM 网络吞吐量 | Microsoft Docs
weixin_39714275的博客
08-11 443
您现在访问的是微软AZURE全球版技术文档网站，若需要访问由世纪互联运营的MICROSOFT AZURE中国区技术文档网站，请访问 https://docs.azure.cn.带宽/吞吐量测试 (NTTTCP)10/06/2020本文内容在 Azure 中测试网络吞吐量性能时，最好使用以要测试的网络为目标并能最大程度减少其他资源(这些资源可能会对性能产生影响)的使用的工具。 建议使用 NTTTCP...
ambari 搭建hadoop大数据平台系列6-验证测试
weixin_33755649的博客
04-04 147
ambari 搭建hadoop大数据平台系列6-验证测试web 检查组件服务是否正常启动shell 端检查服务是否健康hdfs fsck / -files -blocks -locationsjava api/ python 检查服务是否正常 转载于:https://blog.51cto.com/fuyuan2016/1912881...
centos7安装ambari (已测试成功)
shileimohan的专栏
08-04 202
参考1： https://blog.csdn.net/Happy_Sunshine_Boy/article/details/86595945?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.channel_param&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLear
阿里云服务器搭建及安装Ambari环境
L_x_4的博客
12-28 823
实验报告 第四组 服务器搭建及安装Ambari环境 Ambari简介 Ambari 跟 Hadoop 等开源软件一样，也是 Apache Software Foundation 中的一个项目。 Ambari的作用就是创建、管理、监视 Hadoop 的集群，但是这里的 Hadoop 是广义，指的是 Hadoop 整个生态圈（例如 Hive，Hbase，Spark，Kafka，Zookeeper ...
C#码农的大数据之路 - 使用C#编写MR作业
dotNET跨平台
06-08 1620
写在前面 从Hadoop出现至今，大数据几乎就是Java平台专属一般。虽然Hadoop或Spark也提供了接口可以与其他语言一起使用，但作为基于JVM运行的框架，Java系语言有着天生优势。而且能找到的与大数据框架如Hadoop等使用介绍的文章也都以Java语言作为示例居多。许多C#er为了转投大数据怀抱也开始学习Java。微软为了拥抱大数据在这方面也做了许多，提供了一些工具及库使C#可以更好的
polybase配置 sql_Hadoop 的 PolyBase 配置和安全
weixin_39876739的博客
12-20 259
Hadoop 的 PolyBase 配置和安全PolyBase configuration and security for Hadoop04/23/2019本文内容适用于：Applies to: SQL ServerSQL Server(所有支持的版本)SQL ServerSQL Server (all supported versions) - 仅限 Windows Azure SQL 托管实...
云计算复习笔记三
爱学习的小蜗牛的博客
01-02 1336
云计算复习笔记三
Azure云服务器搭建
qq_38211852的博客
05-27 1万+
1. 创建学生账号，获取免费使用权登陆教育专业人员专属网站并验证，网址如下：https://azure.microsoft.com/zh-cn/education/2.仪表板中创建虚拟机2.1 选择所需的虚拟机类型：这里我们选择ubuntu系统2.2 配置虚拟机参数：这里我们需要关注用户名及密码2.3 配置完成的主页面：这里我们主要关注生成的IP地址...
Azure微软云（部署一台虚拟机云服务器）
热门推荐
qq_45631844的博客
05-08 1万+
文章目录一、前言1.1 概述1.2 虚拟机1.3 虚拟网络虚拟网络在Azure中的功能二、虚拟网络的创建2.1 登录注册Azure2.2 登录到Azure上2.3 虚拟网络的部署三、虚拟机的部署3.1 在搜索栏输入虚拟机，点击进入3.2 点击创建--设置如下参数3.3 点击下一步到网络，配置如下选项3.4 然后直接点查看和创建，最终创建效果如下四、公共IP地址的创建4.1 搜索栏输入公共--点击公共IP地址进入4.2 点击创建4.3 将公共IP地址绑定到Windows虚拟机上4.4 绑定效果五、连接测试5.
微软云 Azure 云服务器 Web应用服务云计算解决方案
02-23 2033
一、 Microsoft Azure 微软云简介 Microsoft Azure 是 Microsoft 为客户提供的公有云平台。客户可以多种不同方式使用此平台。例如，客户可以使用 Microsoft Azure 构建一个在 Microsoft 数据中心运行并存储其数据的 Web 应用程序。同时，亦可使用 Microsoft Azure 创建虚拟机以用于开发和测试，Microsoft Azure平台提供了种类广泛的服务，可以帮助用户按需使用、降低拥有成本、提升交付速度，提升IT支...
azure连接微软服务器,微软云服务产品 增应用与Azure连接功能
weixin_33208391的博客
08-05 190
8月31日消息，据国外媒体报道，微软本周推出了云服务产品Windows Azure Mobile Services，此举增加了Windows应用与Windows Azure云端平台的连接功能。微软服务器与工具项目管理副总裁Scott Guthrie表示，该技术的推出使得用户将客户端和基于Win 8的移动应用与可扩展的云端服务相连接成为可能。它允许用户能够轻松地在云服务中存储结构化的数据，只需经过用...
云服务器哪家强？AWS、Azure、阿里云、腾讯云、华为云深度评测
zhanglongfei1016的专栏
12-08 3671
上云在今天几乎已经成了很多企业的标配，云服务器因为不需要运维人员去机房维护，也不需要企业去建设机房等，大大降低了企业的IT资源门槛，可以帮助用户快速实现业务部署。云服务器的随需随买、灵活弹性也让企业可以从容应对流量高峰。 随着云计算行业发展越发成熟，厂商们服务器迭代越来越快、服务器的种类也是五花八门，不同的厂商的云服务器性能也让企业们非常关注。 在国际市场，亚马逊AWS、微软Azure和阿里云分列前三。在国内市场，阿里云占据了近一半份额，紧随其后的是腾讯云和华为云等。 因此，笔者选取了上述几家云厂商的
Azure 测试项目分析
qq_37756099的博客
12-16 192
Lab类 属性： Lab name Subscription resource group location 方法： CreateLab() BrowseLab() DeleteLab() VM类 属性： 方法：
Azure云服务简单实现
最新发布
weixin_48358407的博客
01-12 569
然后我们在虚拟机的基本信息中添加我们刚刚创建的资源组，并在其网络配置中选择刚刚的虚拟网络最后我们进行付费就可以创建成功了，之后我们再为该虚拟机创建网络安全组然后通过VM实例的公共网络地址就能访问到该虚拟机了。然后我们根据官网的指引就可以一步一步的完成堡垒机的创建了，之后我们就可以在选择连接方式的时候选择使用Bastion然后输入虚拟机的名称就可以访问虚拟机的桌面了，堡垒机唯一的不足就是不能实现文件的复制粘贴。按照要求一步一步的创建资源组，最后右上角会显示资源组是否成功创建。然后我们点击使用堡垒机。
ambari diskusage: data unavaliable
weixin_33721344的博客
08-29 459
今天在查看ambari集群时，发现有一台服务器的磁盘统计没有数据，显示：ambari diskusage:data unavaliable. 不知道是什么原因，去对应的机器上，在/var/log/ambari-metrics-monitor目录下，查看日志。tail -f ambari-metrics-monitor.out，发现报如下错误：OSError: [Errno 1...
微软云服务Azure产品简介
b2473195597的博客
12-05 1066
微软云服务Azure所有产品简介
微软Windows Azure云开发详解
"微软Windows Azure云应用开发实践探讨了如何利用微软的云计算平台进行应用程序的构建，涵盖了Windows Azure、SQL Azure以及AppFabric等关键组件。Windows Azure作为云操作系统，负责资源管理和任务调度；SQL Azure...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值