本文介绍了CDH5的搭建步骤及遇到的问题和解决方案。
参考自:CDH5.7Hadoop集群搭建(离线版) - 数据民工张三省 - 博客园及Cloudera Manager 5.9 和 CDH 5.9 离线安装指南及个人采坑填坑记 - 破击手 - 博客园
准备工作
本案例使用公司现成的三台机器,如需自己搭建和配置虚拟机,可参考别的教程。
- 三节点的集群,查看机器配置如下:
IP地址 | 主机别名 | 说明 | 可用内存 | 可用硬盘 |
10.1.4.18 | b3 | 主节点master(namenode),datanode | 4.3GB | 44GB |
10.1.4.19 | b4 | datanode | 4.9GB | 44GB |
10.1.4.20 | b5 | datanode | 4.8GB | 44GB |
- 机器系统信息:
[root@bi-zhaopeng03 songzehao]# cat /proc/version
Linux version 3.10.0-229.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP Fri Mar 6 11:36:42 UTC 2015
[root@bi-zhaopeng03 songzehao]# uname -a
Linux bi-zhaopeng03 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
搭建工作
JDK安装
因为hadoop是由Java写的,很多启动程序都需要java的环境。JDK安装没什么特别的介绍,安装完查看信息。
[root@bi-zhaopeng03 songzehao]# java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
[root@bi-zhaopeng03 songzehao]# echo $JAVA_HOME
/usr/local/java
相关安装介质获取
- Mysql:
https://dev.mysql.com/get/Downloads/MySQL-5.5/mysql-5.5.62-linux-glibc2.12-x86_64.tar.gz
- JDBC驱动:
https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz
- cloudera manager包:
http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz
- CDH包:
http://archive.cloudera.com/cdh5/parcels/5.7.2/CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel
http://archive.cloudera.com/cdh5/parcels/5.7.2/CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha1
http://archive.cloudera.com/cdh5/parcels/5.7.2/manifest.json
配置hostname(每台机器)
[root@bi-zhaopeng03 songzehao]# vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=bi-zhaopeng03
[root@bi-zhaopeng03 songzehao]# vi /etc/hosts
10.1.4.18 bi-zhaopeng03 b3 (说明:b3也是hostname的别名)
10.1.4.19 bi-zhaopeng04 b4
10.1.4.20 bi-zhaopeng05 b5
关闭selinux(每台机器)
配置selinux需重启后生效
[root@bi-zhaopeng03 songzehao]# vi /etc/sysconfig/selinux
SELINUX=disabled
[root@bi-zhaopeng03 songzehao]# sestatus -v
SELinux status: disabled
关闭防火墙(每台机器)
[root@bi-zhaopeng03 songzehao]# systemctl stop firewalld
[root@bi-zhaopeng03 songzehao]# systemctl disable firewalld
rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'
rm '/etc/systemd/system/basic.target.wants/firewalld.service'
[root@bi-zhaopeng03 songzehao]# systemctl status firewalld
firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled)
Active: inactive (dead)
NTP网络时间同步 (每台机器)
首先安装ntp服务
[root@bi-zhaopeng03 songzehao]# yum -y install ntp
或
[root@bi-zhaopeng03 songzehao]# yum install -y ntpdate
然后同步时间,如下是CentOS系统使用NTP来从一个时间服务器同步,把当前时区调整为上海就是+8区,想改其他时区也可以去看看/usr/share/zoneinfo目录
[root@bi-zhaopeng03 songzehao]# cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
[root@bi-zhaopeng03 songzehao]# ntpdate us.pool.ntp.org
启动和查看ntp服务
[root@bi-zhaopeng03 songzehao]# systemctl start ntpd //启动ntp服务
[root@bi-zhaopeng03 songzehao]# systemctl status ntpd //查看ntp服务状态
SSH无密码登录配置 (每台机器)
检验机器之间是否已配置好SSH免密登录,可ssh hostname测试,我这里的三台机器已被配置好。下面贴出其他小伙伴的操作示范
[root@cdh1 /]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
/root/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
1d:e9:b4:ed:1d:e5:c6:a7:f3:23:ac:02:2b:8c:fc:ca root@cdh1
The key's randomart image is:
+--[ RSA 2048]----+
| |
| . |
| + .|
| + + + |
| S + . . =|
| . . . +.|
| . o o o + |
| .o o . . o + |
| Eo.. ... . o|
+-----------------+
[root@cdh1 /]# ssh-copy-id 192.168.42.129
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.42.129's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '192.168.42.129'"
and check to make sure that only the key(s) you wanted were added.
MySQL安装
可参考文首的文章使用rpm方式安装,这里参考百度经验:https://jingyan.baidu.com/article/a378c9609eb652b3282830fd.html,解压安装(如下仅为解压操作,省略的部分配置步骤安装百度经验即可)
[root@bi-zhaopeng03 songzehao]# tar mysql-5.5.62-linux-glibc2.12-x86_64.tar.gz
[root@bi-zhaopeng03 songzehao]# cp mysql-5.5.62-linux-glibc2.12-x86_64 /usr/local/mysql -r
...
安装完查看信息
[root@bi-zhaopeng03 songzehao]# mysql --help |grep Distrib
mysql Ver 14.14 Distrib 5.5.62, for linux-glibc2.12 (x86_64) using readline 5.1
修改密码解压JDBC驱动(需在各节点创建目录/usr/share/java/,并将驱动包重命名后放进去)
[root@bi-zhaopeng03 songzehao]# mysql -u root -p //登录进行去更改密码
mysql> SET PASSWORD=PASSWORD('123456');
mysql> update user set host='%' where user='root' and host='localhost'; //允许mysql远程访问
Query OK, 1 row affected (0.05 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
[root@bi-zhaopeng03 songzehao]# chkconfig mysql on //配置开机启动
[root@bi-zhaopeng03 songzehao]# tar -zcvf mysql-connector-java-5.1.44.tar.gz // 解压mysql-connector-java-5.1.44.tar.gz得到mysql-connector-java-5.1.44-bin.jar
[root@bi-zhaopeng03 songzehao]# mkdir /usr/share/java // 在各节点创建java文件夹
[root@bi-zhaopeng03 songzehao]# cp mysql-connector-java-5.1.44-bin.jar /usr/share/java/mysql-connector-java.jar //将mysql-connector-java-5.1.44-bin.jar拷贝到/usr/share/java路径下并重命名为mysql-connector-java.jar
创建数据库
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
grant all on *.* to root@"%" Identified by "123456";
安装Cloudera-Manager
1.解压cm tar包到所有服务器指定目录(或者在主节点解压好,然后通过scp到各个节点同一目录下)
[root@bi-zhaopeng03 songzehao]# mkdir /opt/cloudera-manager
[root@bi-zhaopeng03 songzehao]# tar -axvf cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz -C /opt/cloudera-manager
2.创建cloudera-scm用户(所有节点)
[root@bi-zhaopeng03 songzehao]# useradd --system --home=/opt/cloudera-manager/cm-5.7.2/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
3.在主节点创建cloudera-manager-server的本地元数据保存目录
[root@bi-zhaopeng03 songzehao]# mkdir /var/cloudera-scm-server
[root@bi-zhaopeng03 songzehao]# chown cloudera-scm:cloudera-scm /var/cloudera-scm-server
[root@bi-zhaopeng03 songzehao]# chown cloudera-scm:cloudera-scm /opt/cloudera-manager
4.配置从节点cloudera-manger-agent指向主节点服务器
[root@bi-zhaopeng04 ~]# vi /opt/cloudera-manager/cm-5.7.2/etc/cloudera-scm-agent/config.ini
将server_host改为CMS所在的主机名即b3
# Hostname of the CM server.
server_host=b3
5.主节点中创建parcel-repo仓库目录
[root@bi-zhaopeng03 songzehao]# mkdir -p /opt/cloudera/parcel-repo
[root@bi-zhaopeng03 songzehao]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
[root@bi-zhaopeng03 songzehao]# cp CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha manifest.json /opt/cloudera/parcel-repo
注意:其中CDH-5.7.2-1.cdh5.7.2.p0.18-el5.parcel.sha1 后缀要把1去掉
6.创建parcels目录(所有节点)
[root@bi-zhaopeng03 songzehao]# mkdir -p /opt/cloudera/parcels
[root@bi-zhaopeng03 songzehao]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
解释:Clouder-Manager将CDHs从主节点的/opt/cloudera/parcel-repo目录中抽取出来,分发解压激活到各个节点的/opt/cloudera/parcels目录中
7.初始脚本配置数据库scm_prepare_database.sh(在主节点上)
[root@bi-zhaopeng03 songzehao]# /opt/cloudera-manager/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql -hb3 -uroot -p123456 --scm-host b3 scmdbn scmdbu scmdbp
说明:这个脚本就是用来创建和配置CMS需要的数据库的脚本。各参数是指:
mysql:数据库用的是mysql,如果安装过程中用的oracle,那么该参数就应该改为oracle。
-b3:数据库建立在b3主机上面,也就是主节点上面。
-uroot:root身份运行mysql。-123456:mysql的root密码是***。
--scm-host b3:CMS的主机,一般是和mysql安装的主机是在同一个主机上,最后三个参数是:数据库名,数据库用户名,数据库密码。
此处执行可能会报权限不够的错,参考以下或另行处理即可
update user set PASSWORD=PASSWORD('123456') where user='root';
GRANT ALL PRIVILEGES ON *.* TO 'root'@'b3' IDENTIFIED BY '123456' WITH GRANT OPTION;
FLUSH PRIVILEGES;
此处执行可能报以下错:
[root@bi-zhaopeng03 ~]# /opt/cloudera-manager/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql -hb3 -uroot -p123456 --scm-host b3 scmdbn scmdbu scmdbp
JAVA_HOME=/usr/local/java
Verifying that we can write to /opt/cloudera-manager/cm-5.7.2/etc/cloudera-scm-server
2019-06-05 11:21:14,867 [main] ERROR com.cloudera.enterprise.dbutil.DbProvisioner - Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://b3/?useUnicode=true&characterEncoding=UTF-8'
java.sql.SQLException: Can't create database 'scmdbn'; database exists
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2491)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2449)
at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:845)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:745)
at com.cloudera.enterprise.dbutil.DbProvisioner.executeSql(DbProvisioner.java:286)
at com.cloudera.enterprise.dbutil.DbProvisioner.doMain(DbProvisioner.java:95)
at com.cloudera.enterprise.dbutil.DbProvisioner.main(DbProvisioner.java:110)
2019-06-05 11:21:14,870 [main] ERROR com.cloudera.enterprise.dbutil.DbProvisioner - Stack Trace:
java.sql.SQLException: Can't create database 'scmdbn'; database exists
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2491)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2449)
at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:845)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:745)
at com.cloudera.enterprise.dbutil.DbProvisioner.executeSql(DbProvisioner.java:286)
at com.cloudera.enterprise.dbutil.DbProvisioner.doMain(DbProvisioner.java:95)
at com.cloudera.enterprise.dbutil.DbProvisioner.main(DbProvisioner.java:110)
--> Error 1, giving up (use --force if you wish to ignore the error)
可尝试drop掉数据库scmdbn
drop database if exists scmdbn;
也可尝试参考此解决方案:安装CDH后启动cloudera-scm-server遇到问题,求帮忙,求指教-CSDN论坛
重新执行初始化CM5数据库命令
/opt/cloudera-manager/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql -hlocalhost -uroot -p123456 --scm-host localhost scmdbn scmdbu scmdbp
再次执行 /opt/cm-5.6.1/etc/init.d/cloudera-scm-server start 启动服务端。
再次执行 /opt/cm-5.6.1/etc/init.d/cloudera-scm-agent start 启动客户端。
8.启动主节点
[root@bi-zhaopeng03 songzehao]# cp /opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
[root@bi-zhaopeng03 songzehao]# chkconfig cloudera-scm-server on
[root@bi-zhaopeng03 songzehao]# vi /etc/init.d/cloudera-scm-server
CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}改为=/opt/cloudera-manager/cm-5.7.2/etc/default
[root@bi-zhaopeng03 songzehao]# service cloudera-scm-server start
//同时为了保证在每次服务器重启的时候都能启动cloudera-scm-server,应该在开机启动脚本/etc/rc.local中加入命令:service cloudera-scm-server restart
9.启动cloudera-scm-agent(所有节点)
[root@bi-zhaopeng03 songzehao]# mkdir /opt/cloudera-manager/cm-5.7.2/run/cloudera-scm-agent
[root@bi-zhaopeng03 songzehao]# cp /opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
[root@bi-zhaopeng03 songzehao]# chkconfig cloudera-scm-agent on
[root@bi-zhaopeng03 songzehao]# vi /etc/init.d/cloudera-scm-agent
CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}改为=/opt/cloudera-manager/cm-5.7.2/etc/default
[root@bi-zhaopeng03 songzehao]# service cloudera-scm-agent start
//同时为了保证在每次服务器重启的时候都能启动cloudera-scm-agent,应该在开机启动脚本/etc/rc.local中加入命令:service cloudera-scm-agent restart
在浏览器安装CDH
面对一个陌生环境的搭建,前面肯定有各种想到想不到查到查不到的问题,要把握住两点,
- 要相信所有问题最终都能被解决掉;
- 细心查看日志(/opt/cloudera-manager/cm-5.7.2/log/cloudera-scm-server/ cloudera-scm-server.log)和数据库脚本(/opt/cloudera-manager/cm-5.7.2/lib64/cmf/service/common/cloudera-config.sh)内容,分析出错原因。
等待主节点完成启动就在浏览器中进行操作了,进入b3:7180 默认使用admin admin登录,在浏览器中使用操作安装,可直接参考CDH5.7Hadoop集群搭建(离线版) - 数据民工张三省 - 博客园与Cloudera Manager 5.9 和 CDH 5.9 离线安装指南及个人采坑填坑记 - 破击手 - 博客园
(1)注意在【群集设置 - 选择您要再群集上安装的CDH5服务】步骤里,选择要安装的服务组合,量力而行,我的机器配置较低,所以暂时只选择了“核心Hadoop”组合,之后需要其他组件另行安装即可。
(2)注意在【群集安装 - 检查主机正确性】步骤里,若报错“检查器在所有主机上运行失败.”或“该群集的主机上未安装 Java”,查看server日志,应该会报错“JAVA_HOME is not set and could not be found”。这时候,我们的jdk肯定是正常安装了,所以去查看/opt/cloudera-manager/cm-5.7.2/lib64/cmf/service/common/cloudera-config.sh,可以发现它认定的JAVA8_HOME中不包含“/usr/local/java”,因此一种方式是软链接到它指定的目录,另一种是在它的集合中加进去“/usr/local/java”,如下:
local JAVA8_HOME_CANDIDATES=(
'/usr/local/java'
'/usr/java/jdk1.8'
'/usr/java/jre1.8'
'/usr/lib/jvm/j2sdk1.8-oracle'
'/usr/lib/jvm/j2sdk1.8-oracle/jre'
'/usr/lib/jvm/java-8-oracle'
)
最终此步骤成功
(3)注意在【群集设置 - 数据库】步骤里,若报错“A package was not selected.”,如下:
莫名其妙,日志也没有追加新的信息,返回几步重新再点过来就好了。
安装成功
- hadoop配置文件目录: /etc/hadoop/conf(包括core-site.xml,hdfs-site.xml等)
- hdfs文件系统地址:cat /etc/hadoop/conf/core-site.xml,查找fs.defaultFS = hdfs://bi-zhaopeng03:8020
- cloudera-manager的web管理地址:主节点ip:7180
- hdfs的web管理地址:主节点ip:50070
- yarn的web管理地址:主节点ip:8088
- hue的web管理地址:主节点ip:8888
测试工作
在集群的一台机器上执行以下模拟圆周率Pi(π)的示例程序:
sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100
通过YARN的Web管理界面(http://10.1.4.18:8088)也可以看到MapReduce的执行状态:
查看MapReduce应用程序的完整日志:
[root@bi-zhaopeng04 ~]# sudo -u hdfs yarn logs -applicationId application_1559721784881_0001
MapReduce执行过程中终端的输出如下:
[root@bi-zhaopeng04 ~]# sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100
Number of Maps = 10
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
19/06/06 14:08:23 INFO client.RMProxy: Connecting to ResourceManager at bi-zhaopeng03/10.1.4.18:8032
19/06/06 14:08:23 INFO input.FileInputFormat: Total input paths to process : 10
19/06/06 14:08:23 INFO mapreduce.JobSubmitter: number of splits:10
19/06/06 14:08:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1559721784881_0001
19/06/06 14:08:24 INFO impl.YarnClientImpl: Submitted application application_1559721784881_0001
19/06/06 14:08:24 INFO mapreduce.Job: The url to track the job: http://bi-zhaopeng03:8088/proxy/application_1559721784881_0001/
19/06/06 14:08:24 INFO mapreduce.Job: Running job: job_1559721784881_0001
19/06/06 14:08:29 INFO mapreduce.Job: Job job_1559721784881_0001 running in uber mode : false
19/06/06 14:08:29 INFO mapreduce.Job: map 0% reduce 0%
19/06/06 14:08:34 INFO mapreduce.Job: map 10% reduce 0%
19/06/06 14:08:37 INFO mapreduce.Job: map 20% reduce 0%
19/06/06 14:08:40 INFO mapreduce.Job: map 30% reduce 0%
19/06/06 14:08:43 INFO mapreduce.Job: map 40% reduce 0%
19/06/06 14:08:46 INFO mapreduce.Job: map 50% reduce 0%
19/06/06 14:08:49 INFO mapreduce.Job: map 60% reduce 0%
19/06/06 14:08:52 INFO mapreduce.Job: map 70% reduce 0%
19/06/06 14:08:55 INFO mapreduce.Job: map 80% reduce 0%
19/06/06 14:08:58 INFO mapreduce.Job: map 90% reduce 0%
19/06/06 14:09:06 INFO mapreduce.Job: map 90% reduce 30%
19/06/06 14:13:30 INFO mapreduce.Job: map 90% reduce 0%
19/06/06 14:13:33 INFO mapreduce.Job: map 100% reduce 0%
19/06/06 14:13:38 INFO mapreduce.Job: map 100% reduce 100%
19/06/06 14:13:38 INFO mapreduce.Job: Job job_1559721784881_0001 completed successfully
19/06/06 14:13:38 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=91
FILE: Number of bytes written=1310718
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2680
HDFS: Number of bytes written=215
HDFS: Number of read operations=43
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Job Counters
Killed reduce tasks=1
Launched map tasks=10
Launched reduce tasks=2
Data-local map tasks=10
Total time spent by all maps in occupied slots (ms)=17328
Total time spent by all reduces in occupied slots (ms)=272280
Total time spent by all map tasks (ms)=17328
Total time spent by all reduce tasks (ms)=272280
Total vcore-seconds taken by all map tasks=17328
Total vcore-seconds taken by all reduce tasks=272280
Total megabyte-seconds taken by all map tasks=17743872
Total megabyte-seconds taken by all reduce tasks=278814720
Map-Reduce Framework
Map input records=10
Map output records=20
Map output bytes=180
Map output materialized bytes=340
Input split bytes=1500
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=340
Reduce input records=20
Reduce output records=0
Spilled Records=40
Shuffled Maps =10
Failed Shuffles=0
Merged Map outputs=10
GC time elapsed (ms)=589
CPU time spent (ms)=5390
Physical memory (bytes) snapshot=4883341312
Virtual memory (bytes) snapshot=31219437568
Total committed heap usage (bytes)=4787273728
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1180
File Output Format Counters
Bytes Written=97
Job Finished in 315.215 seconds
Estimated value of Pi is 3.14800000000000000000
机器比较慢,执行了315s,不过结果也是成功计算了出来,说明我们的CDH5已搭建成功。