Hadoop强大性能测试
环境说明：本机window10 64位，内存16g，vmware12 pro，centos7，hadoop-2.6.4
体会了hadoop的高可扩展，高可用性，故障恢复，以及HDFS文件存储分块，MapReduce快速计算。
一、使用虚拟机搭建hadoop集群搭建
搭建说明:
192.168.136.128 hadoop1 nameNode ResourceManage
192.168.136.129 hadoop2 dataNode NodeManager
192.168.136.130 hadoop3 dataNode NodeManager
1.准备Linux环境
1.0先将虚拟机的网络模式选为NAT
1.1修改主机名
vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=hadoop1 ###
或者centos7,分别在三台机子上执行
hostnamectl set-hostname hadoop1
hostnamectl set-hostname hadoop2
hostnamectl set-hostname hadoop3
1.2修改IP
两种方式：
第一种：通过Linux图形界面进行修改（强烈推荐）
进入Linux图形界面 -> 右键点击右上方的两个小电脑 -> 点击Edit connections -> 选中当前网络System eth0 -> 点击edit按钮 -> 选择IPv4 -> method选择为manual -> 点击add按钮 -> 添加IP：192.168.1.101 子网掩码：255.255.255.0 网关：192.168.1.1 -> apply
第二种：修改配置文件方式（屌丝程序猿专用）
vim /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
BOOTPROTO="static" ###
HWADDR="00:0C:29:3C:BF:E7"
IPV6INIT="yes"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"
UUID="ce22eeca-ecde-4536-8cc2-ef0dc36d4a8c"
IPADDR="192.168.136.128" ###
NETMASK="255.255.255.0" ###
GATEWAY="192.168.136.1" ###
1.3修改主机名和IP的映射关系
vim /etc/hosts
192.168.136.128 hadoop1
192.168.136.129 hadoop2
192.168.136.130 hadoop3
1.4关闭防火墙centos6.4
#查看防火墙状态
service iptables status
#关闭防火墙
service iptables stop
#查看防火墙开机启动状态
chkconfig iptables --list
#关闭防火墙开机启动
chkconfig iptables off
centos7
systemctl status firewalld.service
systemctl stop firewalld.service //下次启动还开启
systemctl disable firewalld.service //下次启动也是关闭防火墙
1.5 修改sudo
su root
vim /etc/sudoers
给hadoop用户添加执行的权限
关闭linux服务器的图形界面：
vi /etc/inittab
1.5重启Linux
reboot
2.安装JDK
2.1上传alt+p 后出现sftp窗口，然后put d:\xxx\yy\ll\jdk-7u_65-i585.tar.gz
2.2解压jdk
#创建文件夹
mkdir /home/hadoop/app
#解压
tar -zxvf jdk-7u55-linux-i586.tar.gz -C /home/hadoop/app
2.3将java添加到环境变量中
vim /etc/profile
#在文件最后添加
export JAVA_HOME=/home/hadoop/app/jdk-7u_65-i585
export PATH=$PATH:$JAVA_HOME/bin
#刷新配置
source /etc/profile
3.安装hadoop2.6.4
先上传hadoop的安装包到服务器上去/home/hadoop/
注意：hadoop2.x的配置文件$HADOOP_HOME/etc/hadoop
伪分布式需要修改5个配置文件
3.1配置hadoop
第一个：hadoop-env.sh
vim hadoop-env.sh
#第27行
export JAVA_HOME=/usr/java/jdk1.7.0_65
第二个：core-site.xml
<!-- 指定HADOOP所使用的文件系统schema（URI），HDFS的老大（NameNode）的地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop1:9000</value>
</property>
<!-- 指定hadoop运行时产生文件的存储目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoop-2.6.4/tmp</value>
</property>
第三个：hdfs-site.xml
<!-- 指定HDFS副本的数量 -->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>hadoop1:50090</value>
</property>
第四个：mapred-site.xml (mv mapred-site.xml.template mapred-site.xml)
mv mapred-site.xml.template mapred-site.xml
vim mapred-site.xml
<!-- 指定mr运行在yarn上 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
第五个：yarn-site.xml
<!-- 指定YARN的老大（ResourceManager）的地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop1</value>
</property>
<!-- reducer获取数据的方式 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
3.2将hadoop添加到环境变量
vim /etc/proflie
export JAVA_HOME=/usr/java/jdk1.7.0_65
export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.4
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
source /etc/profile
3.3格式化namenode（是对namenode进行初始化）
hdfs namenode -format (hadoop namenode -format)
3.4启动hadoop
先启动HDFS
sbin/start-dfs.sh
再启动YARN
sbin/start-yarn.sh
3.5验证是否启动成功
使用jps命令验证
hadoop1
27408 NameNode
28218 Jps
27643 SecondaryNameNode
28066 NodeManager
27803 ResourceManager
27512 DataNode
hadoop2
1733 NodeManager
1382 DataNode
1861 Jps
hadoop3
1684 NodeManager
1867 Jps
1355 DataNode
http://192.168.136.128:50070 （HDFS管理界面）
http://192.168.136.128:8088 （MR管理界面）
http://192.168.136.128:50070/explorer.html#/ (hdfs查看上传文件)
4.配置ssh免登陆
#生成ssh免登陆密钥
#进入到我的home目录 hadoop1
cd ~/.ssh
ssh-keygen -t rsa （四个回车）
执行完这个命令后，会生成两个文件id_rsa（私钥）、id_rsa.pub（公钥）
将公钥拷贝到要免密登陆的目标机器上
ssh-copy-id localhost
---------------------------
ssh免登陆：
生成key:
ssh-keygen
复制从A复制到B上:
ssh-copy-id hadoop1
ssh-copy-id hadoop2
ssh-copy-id hadoop3
验证：
ssh localhost/exit，ps -e|grep ssh
ssh A #在B中执行
二、Hdaoop HDFS api 编写（难点在于window环境搭建，按照文中提供的链接安装window hadoop环境，亲测有效）
package com.skymesh.hadoop.hdfs;
import org.apache.commons.io.IOUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.fs.permission.FsPermission;
import java.io.DataOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.Iterator;
import java.util.Map;
/**
* 1、按照hadoop-2.6.4搭建手册安装好hadoop集群
* window10 安装hadoop：https://blog.csdn.net/songhaifengshuaige/article/details/79575308
* window安装hadoop,hadoop找不到JAVA_HOME,java不能放在有空格的路径中，eg C:\Program Files\ C:\Program Files (x86),重新复制java安装到到没有空格的目录下
* window10 hadoop工具：https://github.com/steveloughran/winutils
* window10 用户名称和hadoop集群用户权限问题在hadoop集群配置
*/
public class HdfsClient {
static FileSystem fileSystem=null;
public static void main(String[] args) throws IOException, URISyntaxException, InterruptedException {
HdfsClient hdfsClient = new HdfsClient();
hdfsClient.init2();
hdfsClient.uploadFileToHadoop();
//hdfsClient.deletFile();
hdfsClient.queryFile();
hdfsClient.downLoadFile();
}
public void init() throws IOException {
Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://192.168.136.128:9000");
fileSystem = FileSystem.get(conf);
System.out.println("连接"+"hdfs://192.168.136.128:9000"+"成功");
}
public void init2() throws URISyntaxException, IOException, InterruptedException {
Configuration conf = new Configuration();
conf.set("fs.replication","3");
//设置用户
fileSystem = FileSystem.get(new URI("hdfs://192.168.136.128:9000"),conf,"root");
}
/**
* 上传本地文件到Hadoop系统
* 如果系统中存在再次上传同样的文件名称会覆盖原来文件
* @throws IOException
*/
public void uploadFileToHadoop() throws IOException {
Path from = new Path("C:\\Users\\admin\\Desktop\\word2.txt");
Path to = new Path("/wordcount/input/word2.txt");
fileSystem.copyFromLocalFile(from,to);
System.out.println("上传本地文件到hadoop成功");
}
/**
* 删除文件
* @throws IOException
*/
public void deletFile() throws IOException {
Path filePath = new Path("/readme22.txt");
fileSystem.delete(filePath,true);
System.out.println("删除文件："+"/readme22.txt");
}
/**
* 查询文件系统文件
* @throws IOException
*/
public void queryFile() throws IOException {
Path filePath = new Path("/file/我的文件.txt");
Path homeDirectory = fileSystem.getHomeDirectory();
System.out.println(homeDirectory.getName());
fileSystem.createNewFile(filePath);
System.out.println("创建文件");
boolean exists = fileSystem.exists(filePath);
System.out.println("文件/file/我的文件.txt是否存在:"+exists);
Path rootPath = new Path("/");
RemoteIterator<LocatedFileStatus> locatedFileStatusRemoteIterator = fileSystem.listFiles(rootPath, true);
while (locatedFileStatusRemoteIterator.hasNext()){
LocatedFileStatus next = locatedFileStatusRemoteIterator.next();
long blockSize = next.getBlockSize();
String group = next.getGroup();
long len = next.getLen();
String owner = next.getOwner();
short replication = next.getReplication();
FsPermission permission = next.getPermission();
boolean isfile = next.isFile();
Path path = next.getPath();
System.out.println("是否文件："+isfile+"\t 副本数量："+replication+"\t 拥有人："
+owner+"\t 文件大小："+len+"\t 文件所属组："+group+"\t blockSize:"+blockSize+"\t 文件名称："+path.getName());
}
}
public void queryConfigKey(){
Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://192.168.136.128:9000");
Iterator<Map.Entry<String, String>> iterator = conf.iterator();
while (iterator.hasNext()){
Map.Entry<String, String> entry = iterator.next();
System.out.println(entry.getKey()+"="+entry.getValue());
}
}
/**
* 从hadoop上拷贝数据到本地
* @throws IOException
*/
public void downLoadFile() throws IOException {
FSDataInputStream dataInput = fileSystem.open(new Path("/readme.txt"));
DataOutputStream out = new DataOutputStream(new FileOutputStream(new File("C:\\Users\\admin\\Desktop\\readme77.txt")));
IOUtils.copy(dataInput,out);
System.out.println("下载文件：");
}
}
三、mapreduce 体验快速计算（本机生成了10000000个随机单词，文件72M，mapreduce计算时间8s）
1、生成随机单词，没有优化大概需要10分钟
package com.skymesh.mapreduce;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.Random;
public class CreateWord {
public static void main(String[] args) throws IOException {
File file = new File("C:\\Users\\admin\\Desktop\\word2.txt");
FileOutputStream outputStream = new FileOutputStream(file);
char[] charArr = {'A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z',
'a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z'};
Random random = new Random();
for (int i=0;i<10000000;i++){
int n = random.nextInt(6)+4;
for (int j=n;j>0;j--){
int index = random.nextInt(51);
outputStream.write(charArr[index]);
}
outputStream.write(" ".getBytes());
if (i%40==0){
outputStream.write("\n".getBytes());
}
}
outputStream.close();
}
}
2、mapreduce代码计算代码
package com.skymesh.mapreduce;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import java.io.IOException;
public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
/**
* map阶段的业务逻辑就写在自定义的map()方法中
* maptask会对每一行输入数据调用一次我们自定义的map()方法
*/
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
//将maptask传给我们的文本内容先转换成String
String line = value.toString();
//根据空格将这一行切分成单词
String[] words = line.split(" ");
//将单词输出为<单词，1>
for(String word:words){
//将单词作为key，将次数1作为value，以便于后续的数据分发，可以根据单词分发，以便于相同单词会到相同的reduce task
context.write(new Text(word), new IntWritable(1));
}
}
}
package com.skymesh.mapreduce;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
import java.io.IOException;
public class WordCountReduce extends Reducer<Text, IntWritable, Text, IntWritable> {
/**
* <angelababy,1><angelababy,1><angelababy,1><angelababy,1><angelababy,1>
* <hello,1><hello,1><hello,1><hello,1><hello,1><hello,1>
* <banana,1><banana,1><banana,1><banana,1><banana,1><banana,1>
* 入参key，是一组相同单词kv对的key
*/
@Override
protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws InterruptedException, IOException {
int count=0;
/*Iterator<IntWritable> iterator = values.iterator();
while(iterator.hasNext()){
count += iterator.next().get();
}*/
for(IntWritable value:values){
count += value.get();
}
context.write(key, new IntWritable(count));
}
}
package com.skymesh.mapreduce;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* 相当于一个yarn集群的客户端
* 需要在此封装我们的mr程序的相关运行参数，指定jar包
* 最后提交给yarn
* @author
*
*/
public class WordCountDriver {
public static void main(String[] args) throws Exception {
if (args == null || args.length == 0) {
args = new String[2];
args[0] = "hdfs://192.168.136.128:9000/wordcount/input";
args[1] = "hdfs://192.168.136.128:9000/wordcount/output3";
}
Configuration conf = new Configuration();
//设置的没有用! ??????
// conf.set("HADOOP_USER_NAME", "hadoop");
// conf.set("dfs.permissions.enabled", "false");
/*conf.set("mapreduce.framework.name", "yarn");
conf.set("yarn.resoucemanager.hostname", "mini1");*/
Job job = Job.getInstance(conf);
/*job.setJar("/home/hadoop/wc.jar");*/
//指定本程序的jar包所在的本地路径
job.setJarByClass(WordCountDriver.class);
//指定本业务job要使用的mapper/Reducer业务类
job.setMapperClass(WordCountMapper.class);
job.setReducerClass(WordCountReduce.class);
//指定mapper输出数据的kv类型
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
//指定最终输出的数据的kv类型
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
//指定job的输入原始文件所在目录
FileInputFormat.setInputPaths(job, new Path(args[0]));
//指定job的输出结果所在目录
FileOutputFormat.setOutputPath(job, new Path(args[1]));
//将job中配置的相关参数，以及job所用的java类所在的jar包，提交给yarn去运行
/*job.submit();*/
boolean res = job.waitForCompletion(true);
System.exit(res?0:1);
}
}
日志记录
"C:\Program Files\Java\jdk1.8.0_91\bin\java.exe" "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2019.3.4\lib\idea_rt.jar=64150:C:\Program Files\JetBrains\IntelliJ IDEA 2019.3.4\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\Java\jdk1.8.0_91\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\ext\cldrdata.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\ext\jaccess.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\ext\jfxrt.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\ext\nashorn.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\ext\sunpkcs11.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\jce.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\jfxswt.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_91\jre\lib\rt.jar;D:\Jcould\hadoop\out\production\hadoop;C:\Users\admin\.m2\repository\junit\junit\4.12\junit-4.12.jar;C:\Users\admin\.m2\repository\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar;D:\Jcould\hadoop\web\WEB-INF\lib\xz-1.0.jar;D:\Jcould\hadoop\web\WEB-INF\lib\asm-3.2.jar;D:\Jcould\hadoop\web\WEB-INF\lib\guice-3.0.jar;D:\Jcould\hadoop\web\WEB-INF\lib\avro-1.7.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\gson-2.2.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\junit-4.11.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jsch-0.1.42.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jsp-api-2.1.jar;D:\Jcould\hadoop\web\WEB-INF\lib\xmlenc-0.52.jar;D:\Jcould\hadoop\web\WEB-INF\lib\guava-11.0.2.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jets3t-0.9.0.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jettison-1.1.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jetty-6.1.26.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jline-0.9.94.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jsr305-1.3.9.jar;D:\Jcould\hadoop\web\WEB-INF\lib\log4j-1.2.17.jar;D:\Jcould\hadoop\web\WEB-INF\lib\paranamer-2.3.jar;D:\Jcould\hadoop\web\WEB-INF\lib\activation-1.1.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-el-1.0.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-io-2.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\httpcore-4.2.5.jar;D:\Jcould\hadoop\web\WEB-INF\lib\javax.inject-1.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jaxb-api-2.2.2.jar;D:\Jcould\hadoop\web\WEB-INF\lib\stax-api-1.0-2.jar;D:\Jcould\hadoop\web\WEB-INF\lib\aopalliance-1.0.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-cli-1.2.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-net-3.1.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jersey-core-1.9.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jersey-json-1.9.jar;D:\Jcould\hadoop\web\WEB-INF\lib\servlet-api-2.5.jar;D:\Jcould\hadoop\web\WEB-INF\lib\slf4j-api-1.7.5.jar;D:\Jcould\hadoop\web\WEB-INF\lib\xml-apis-1.3.04.jar;D:\Jcould\hadoop\web\WEB-INF\lib\zookeeper-3.4.6.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-lang-2.6.jar;D:\Jcould\hadoop\web\WEB-INF\lib\httpclient-4.2.5.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jersey-guice-1.9.jar;D:\Jcould\hadoop\web\WEB-INF\lib\xercesImpl-2.9.1.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-codec-1.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\guice-servlet-3.0.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-auth-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-hdfs-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hamcrest-core-1.3.jar;D:\Jcould\hadoop\web\WEB-INF\lib\htrace-core-3.0.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jackson-xc-1.9.13.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jaxb-impl-2.2.3-1.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jersey-client-1.9.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jersey-server-1.9.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jetty-util-6.1.26.jar;D:\Jcould\hadoop\web\WEB-INF\lib\mockito-all-1.8.5.jar;D:\Jcould\hadoop\web\WEB-INF\lib\netty-3.6.2.Final.jar;D:\Jcould\hadoop\web\WEB-INF\lib\api-util-1.0.0-M20.jar;D:\Jcould\hadoop\web\WEB-INF\lib\leveldbjni-all-1.8.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-math3-3.1.1.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-common-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\java-xmlbuilder-0.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\protobuf-java-2.5.0.jar;D:\Jcould\hadoop\web\WEB-INF\lib\slf4j-log4j12-1.7.5.jar;D:\Jcould\hadoop\web\WEB-INF\lib\snappy-java-1.0.4.1.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-digester-1.8.jar;D:\Jcould\hadoop\web\WEB-INF\lib\curator-client-2.6.0.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jackson-jaxrs-1.9.13.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-daemon-1.0.13.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-logging-1.1.3.jar;D:\Jcould\hadoop\web\WEB-INF\lib\curator-recipes-2.6.0.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-api-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jasper-runtime-5.5.23.jar;D:\Jcould\hadoop\web\WEB-INF\lib\api-asn1-api-1.0.0-M20.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-compress-1.4.1.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-httpclient-3.1.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jasper-compiler-5.5.23.jar;D:\Jcould\hadoop\web\WEB-INF\lib\apacheds-i18n-2.0.0-M15.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-beanutils-1.7.0.jar;D:\Jcould\hadoop\web\WEB-INF\lib\curator-framework-2.6.0.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jackson-core-asl-1.9.13.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-annotations-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-client-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-common-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-collections-3.2.2.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-configuration-1.6.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-hdfs-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\jackson-mapper-asl-1.9.13.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-registry-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\commons-beanutils-core-1.8.0.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-api-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-tests-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-examples-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-common-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-hs-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-client-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-common-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-app-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-core-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-web-proxy-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-common-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-nodemanager-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-shuffle-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-client-2.6.4-test-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-common-2.6.4-test-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-tests-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-jobclient-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-examples-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-common-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-hs-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-hs-plugins-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-resourcemanager-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-app-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-core-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-web-proxy-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-tests-2.6.4-test-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-common-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-common-2.6.4-test-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-nodemanager-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-jobclient-2.6.4-tests.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-shuffle-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-jobclient-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-applications-distributedshell-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-web-proxy-2.6.4-test-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-mapreduce-client-hs-plugins-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-resourcemanager-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-nodemanager-2.6.4-test-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-applicationhistoryservice-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-applications-unmanaged-am-launcher-2.6.4.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-resourcemanager-2.6.4-test-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-applications-distributedshell-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-applicationhistoryservice-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-applications-distributedshell-2.6.4-test-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-applications-unmanaged-am-launcher-2.6.4-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-server-applicationhistoryservice-2.6.4-test-sources.jar;D:\Jcould\hadoop\web\WEB-INF\lib\hadoop-yarn-applications-unmanaged-am-launcher-2.6.4-test-sources.jar" com.skymesh.mapreduce.WordCountDriver
2021-02-16 11:30:22,966 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-02-16 11:30:23,841 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1129)) - session.id is deprecated. Instead, use dfs.metrics.session-id
2021-02-16 11:30:23,841 INFO [main] jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId=
2021-02-16 11:30:24,089 WARN [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadFiles(64)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-02-16 11:30:24,120 WARN [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadFiles(171)) - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2021-02-16 11:30:24,167 INFO [main] input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 2
2021-02-16 11:30:24,330 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(199)) - number of splits:2
2021-02-16 11:30:24,468 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(288)) - Submitting tokens for job: job_local1656217725_0001
2021-02-16 11:30:24,601 INFO [main] mapreduce.Job (Job.java:submit(1301)) - The url to track the job: http://localhost:8080/
2021-02-16 11:30:24,602 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1346)) - Running job: job_local1656217725_0001
2021-02-16 11:30:24,602 INFO [Thread-13] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2021-02-16 11:30:24,607 INFO [Thread-13] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2021-02-16 11:30:24,641 INFO [Thread-13] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2021-02-16 11:30:24,641 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1656217725_0001_m_000000_0
2021-02-16 11:30:24,658 INFO [LocalJobRunner Map Task Executor #0] util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2021-02-16 11:30:24,689 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@15daadad
2021-02-16 11:30:24,692 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(753)) - Processing split: hdfs://192.168.136.128:9000/wordcount/input/word2.txt:0+74992653
2021-02-16 11:30:24,721 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2021-02-16 11:30:24,721 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2021-02-16 11:30:24,721 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2021-02-16 11:30:24,721 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2021-02-16 11:30:24,721 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2021-02-16 11:30:24,723 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-02-16 11:30:25,604 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - Job job_local1656217725_0001 running in uber mode : false
2021-02-16 11:30:25,604 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1374)) - map 0% reduce 0%
2021-02-16 11:30:31,859 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map > map
2021-02-16 11:30:31,905 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1374)) - map 33% reduce 0%
2021-02-16 11:30:31,941 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:startSpill(1564)) - Spilling map output
2021-02-16 11:30:31,941 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:startSpill(1565)) - bufstart = 0; bufend = 35076837; bufvoid = 104857600
2021-02-16 11:30:31,941 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:startSpill(1567)) - kvstart = 26214396(104857584); kvend = 14012092(56048368); length = 12202305/6553600
2021-02-16 11:30:31,941 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 45562594 kvi 11390644(45562576)
2021-02-16 11:30:34,859 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map > map
2021-02-16 11:30:36,914 INFO [SpillThread] mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2021-02-16 11:30:36,915 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:resetSpill(1219)) - (RESET) equator 45562594 kv 11390644(45562576) kvi 8769216(35076864)
2021-02-16 11:30:37,498 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:startSpill(1564)) - Spilling map output
2021-02-16 11:30:37,498 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:startSpill(1565)) - bufstart = 45562594; bufend = 80638448; bufvoid = 104857600
2021-02-16 11:30:37,498 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:startSpill(1567)) - kvstart = 11390644(45562576); kvend = 25402492(101609968); length = 12202553/6553600
2021-02-16 11:30:37,498 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 91124200 kvi 22781044(91124176)
2021-02-16 11:30:37,860 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map > map
2021-02-16 11:30:40,861 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map > map
2021-02-16 11:30:42,237 INFO [SpillThread] mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 1
2021-02-16 11:30:42,237 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:resetSpill(1219)) - (RESET) equator 91124200 kv 22781044(91124176) kvi 20159616(80638464)
2021-02-16 11:30:42,807 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:startSpill(1564)) - Spilling map output
2021-02-16 11:30:42,807 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:startSpill(1565)) - bufstart = 91124200; bufend = 21347737; bufvoid = 104857600
2021-02-16 11:30:42,807 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:startSpill(1567)) - kvstart = 22781044(91124176); kvend = 10579812(42319248); length = 12201233/6553600
2021-02-16 11:30:42,807 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 31833484 kvi 7958364(31833456)
2021-02-16 11:30:43,862 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map > map
2021-02-16 11:30:46,862 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map > map
2021-02-16 11:30:47,550 INFO [SpillThread] mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 2
2021-02-16 11:30:47,550 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:resetSpill(1219)) - (RESET) equator 31833484 kv 7958364(31833456) kvi 5336940(21347760)
2021-02-16 11:30:47,663 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map > map
2021-02-16 11:30:47,663 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2021-02-16 11:30:47,663 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2021-02-16 11:30:47,663 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 31833484; bufend = 41592309; bufvoid = 104857600
2021-02-16 11:30:47,663 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 7958364(31833456); kvend = 4564468(18257872); length = 3393897/6553600
2021-02-16 11:30:48,714 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 3
2021-02-16 11:30:48,878 INFO [LocalJobRunner Map Task Executor #0] mapred.Merger (Merger.java:merge(597)) - Merging 4 sorted segments
2021-02-16 11:30:48,881 INFO [LocalJobRunner Map Task Executor #0] mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 4 segments left of total size: 134992641 bytes
2021-02-16 11:30:49,862 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map > sort >
2021-02-16 11:30:49,920 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1374)) - map 38% reduce 0%
2021-02-16 11:30:52,863 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map > sort >
2021-02-16 11:30:52,926 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1374)) - map 50% reduce 0%
2021-02-16 11:30:52,946 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(1001)) - Task:attempt_local1656217725_0001_m_000000_0 is done. And is in the process of committing
2021-02-16 11:30:52,950 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map > sort
2021-02-16 11:30:52,950 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1656217725_0001_m_000000_0' done.
2021-02-16 11:30:52,950 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1656217725_0001_m_000000_0
2021-02-16 11:30:52,950 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1656217725_0001_m_000001_0
2021-02-16 11:30:52,954 INFO [LocalJobRunner Map Task Executor #0] util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2021-02-16 11:30:52,985 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@5668085f
2021-02-16 11:30:52,985 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(753)) - Processing split: hdfs://192.168.136.128:9000/wordcount/input/word.txt:0+73
2021-02-16 11:30:52,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2021-02-16 11:30:52,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2021-02-16 11:30:52,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2021-02-16 11:30:52,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2021-02-16 11:30:52,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2021-02-16 11:30:52,993 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-02-16 11:30:53,410 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2021-02-16 11:30:53,410 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2021-02-16 11:30:53,410 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2021-02-16 11:30:53,410 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 125; bufvoid = 104857600
2021-02-16 11:30:53,410 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214348(104857392); length = 49/6553600
2021-02-16 11:30:53,414 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2021-02-16 11:30:53,414 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(1001)) - Task:attempt_local1656217725_0001_m_000001_0 is done. And is in the process of committing
2021-02-16 11:30:53,418 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - map
2021-02-16 11:30:53,418 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1656217725_0001_m_000001_0' done.
2021-02-16 11:30:53,418 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1656217725_0001_m_000001_0
2021-02-16 11:30:53,418 INFO [Thread-13] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2021-02-16 11:30:53,418 INFO [Thread-13] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2021-02-16 11:30:53,418 INFO [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1656217725_0001_r_000000_0
2021-02-16 11:30:53,426 INFO [pool-6-thread-1] util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2021-02-16 11:30:53,461 INFO [pool-6-thread-1] mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@271e53a1
2021-02-16 11:30:53,465 INFO [pool-6-thread-1] mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7ca1467d
2021-02-16 11:30:53,473 INFO [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(197)) - MergerManager: memoryLimit=2660761600, maxSingleShuffleLimit=665190400, mergeThreshold=1756102784, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2021-02-16 11:30:53,473 INFO [EventFetcher for fetching Map Completion Events] reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1656217725_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2021-02-16 11:30:53,492 INFO [localfetcher#1] reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local1656217725_0001_m_000001_0 decomp: 153 len: 157 to MEMORY
2021-02-16 11:30:53,492 INFO [localfetcher#1] reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 153 bytes from map-output for attempt_local1656217725_0001_m_000001_0
2021-02-16 11:30:53,492 INFO [localfetcher#1] reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(315)) - closeInMemoryFile -> map-output of size: 153, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->153
2021-02-16 11:30:53,527 INFO [localfetcher#1] reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local1656217725_0001_m_000000_0 decomp: 134992655 len: 134992659 to MEMORY
2021-02-16 11:30:53,606 INFO [localfetcher#1] reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 134992655 bytes from map-output for attempt_local1656217725_0001_m_000000_0
2021-02-16 11:30:53,606 INFO [localfetcher#1] reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(315)) - closeInMemoryFile -> map-output of size: 134992655, inMemoryMapOutputs.size() -> 2, commitMemory -> 153, usedMemory ->134992808
2021-02-16 11:30:53,606 INFO [EventFetcher for fetching Map Completion Events] reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2021-02-16 11:30:53,606 INFO [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2021-02-16 11:30:53,609 INFO [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(687)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2021-02-16 11:30:53,609 INFO [pool-6-thread-1] mapred.Merger (Merger.java:merge(597)) - Merging 2 sorted segments
2021-02-16 11:30:53,609 INFO [pool-6-thread-1] mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 2 segments left of total size: 134992798 bytes
2021-02-16 11:30:53,930 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1374)) - map 100% reduce 0%
2021-02-16 11:30:56,045 INFO [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(754)) - Merged 2 segments, 134992808 bytes to disk to satisfy reduce memory limit
2021-02-16 11:30:56,045 INFO [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(784)) - Merging 1 files, 134992810 bytes from disk
2021-02-16 11:30:56,045 INFO [pool-6-thread-1] reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(799)) - Merging 0 segments, 0 bytes from memory into reduce
2021-02-16 11:30:56,045 INFO [pool-6-thread-1] mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2021-02-16 11:30:56,045 INFO [pool-6-thread-1] mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 134992803 bytes
2021-02-16 11:30:56,045 INFO [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2021-02-16 11:30:56,057 INFO [pool-6-thread-1] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1129)) - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2021-02-16 11:30:59,429 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2021-02-16 11:30:59,992 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1374)) - map 100% reduce 88%
2021-02-16 11:31:02,429 INFO [communication thread] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2021-02-16 11:31:02,994 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1374)) - map 100% reduce 97%
2021-02-16 11:31:04,682 INFO [pool-6-thread-1] mapred.Task (Task.java:done(1001)) - Task:attempt_local1656217725_0001_r_000000_0 is done. And is in the process of committing
2021-02-16 11:31:04,683 INFO [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2021-02-16 11:31:04,683 INFO [pool-6-thread-1] mapred.Task (Task.java:commit(1162)) - Task attempt_local1656217725_0001_r_000000_0 is allowed to commit now
2021-02-16 11:31:04,689 INFO [pool-6-thread-1] output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1656217725_0001_r_000000_0' to hdfs://192.168.136.128:9000/wordcount/output3/_temporary/0/task_local1656217725_0001_r_000000
2021-02-16 11:31:04,690 INFO [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2021-02-16 11:31:04,690 INFO [pool-6-thread-1] mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1656217725_0001_r_000000_0' done.
2021-02-16 11:31:04,690 INFO [pool-6-thread-1] mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1656217725_0001_r_000000_0
2021-02-16 11:31:04,690 INFO [Thread-13] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2021-02-16 11:31:04,994 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1374)) - map 100% reduce 100%
2021-02-16 11:31:04,994 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1385)) - Job job_local1656217725_0001 completed successfully
2021-02-16 11:31:05,001 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1392)) - Counters: 38
File System Counters
FILE: Number of bytes read=674965183
FILE: Number of bytes written=945716056
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=224978105
HDFS: Number of bytes written=93628549
HDFS: Number of read operations=25
HDFS: Number of large read operations=0
HDFS: Number of write operations=5
Map-Reduce Framework
Map input records=4
Map output records=10000013
Map output bytes=114992778
Map output materialized bytes=134992816
Input split bytes=235
Combine input records=0
Combine output records=0
Reduce input groups=9805716
Reduce shuffle bytes=134992816
Reduce input records=10000013
Reduce output records=9805716
Spilled Records=30000026
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=5512
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=5407506432
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=74992726
File Output Format Counters
Bytes Written=93628549
Process finished with exit code 0
5万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
