使用命令shell和MyEclipse两种方式执行Hadoop的WordCount程序

最新推荐文章于 2024-04-19 20:08:17 发布

武睿傲雪

最新推荐文章于 2024-04-19 20:08:17 发布

阅读量2.9k

点赞数 1

分类专栏： hadoop学习笔记文章标签： hadoop wordcount MyEclipse 命令行 hdfs

本文链接：https://blog.csdn.net/wuruiaoxue/article/details/52736436

版权

hadoop学习笔记专栏收录该内容

15 篇文章 1 订阅

订阅专栏

大家初次接触编程时学习的无论时哪种语言，看到的第一个示例程序都是“Hello World”。在Hadoop中也有一个类似于Hello World的程序。这就是WordCount.下文会结合这个程序通过两种方式具体讲解与MapReduce程序有关的所有类。（关于WordCount的程序网上已经有很多了，我这里说一些网上没有的。）

通过MyEclipse调用Hadoop的JAR类库实现WordCount:
WordCount程序主要包括Map类 Reduce类和主函数部分。
（1）首先安装好MyEclipse.我这里使用的时MyEclipse 2015 stable 2.0 如图所示：

MyEclipse在使用时是需要破解，网上有相关教程，这里就不具体说明了。
（2）安装好的MyEclipse会发现没有菜单栏，我们可以通过如下代码所示的方式执行shell命令。（这也是从网上找到的）
在桌面上新建一个MyEclipse.sh文件

#!/bin/bash
cd /home/dtw/MyEclipse2015
export UBUNTU_MENUPROXY=0
nohup ./myeclipse &
rm ./nohup.out

/home/dtw/MyEclipse2015是我的MyEclipse可执行文件的路径，替换就可以了。
其他我这里就不赘述了，网上都有。
（3）下载好Hadoop源文件。
我这里使用的是hadoop-2.7.1,这个版本算比较高的，比较高的版本一般许多命令都是最新的需要重新对命令进行学习。
（4）将Hadoop需要使用的JAR类库从这个目录下面获取到。
如图所示：
这里写图片描述
通过使用Ctrl+F命令找到如下的JAR包

这些包在后面的引用包中会使用到在执行过程中如果出现哪个类没有，这时候就要注意了看一下你的包有没有被引入进来，引入的方法网上都有主要通过buildpath加进来，这里就不赘述了。
（5）添加Hadoop插件，网上有教程，这里不赘述了，添加后在如图所示的位置选择Map/Reduce之后就是Hadoop的配置了，因为这里重点不是介绍这个的所以就不详述了。如图所示：
这里写图片描述
（6）在MyEclipse中File->New->Java Project 这和Windows下的MyEclipse是一样的。因为需要主函数执行程序所以使用Java Project.添加一个cn.edu.ruc.cloudcomputing.book.chapter03的包名，然后在包下面新建一个WordCount类。如图所示：

接下来就是代码部分了，我先贴代码然后再详细说明。
WordCount.java

package cn.edu.ruc.cloudcomputing.book.chapter03;
import java.io.IOException;
import java.util.*;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
public class WordCount {
 public static class Map extends MapReduceBase implements Mapper<LongWritable,Text,Text,IntWritable>
 {
     private final static IntWritable one=new IntWritable(1);
     private Text word=new Text();
    @Override
    public void map(LongWritable key, Text value,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        String line=value.toString();
        StringTokenizer tokenizer=new StringTokenizer(line);
        while(tokenizer.hasMoreElements())
        {
            word.set(tokenizer.nextToken());
            output.collect(word, one);
        }
    }

 }
 public static class Reduce extends MapReduceBase implements Reducer<Text,IntWritable,Text,IntWritable>
 {

    @Override
    public void reduce(Text key, Iterator<IntWritable> values,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        int sum=0;
        while(values.hasNext())
        {
            sum+=values.next().get();
        }
        output.collect(key, new IntWritable(sum));
    }

 }
 public static void main(String[] args) throws Exception
 {
     JobConf conf=new JobConf(WordCount.class);
     conf.setJobName("wordcount");

     conf.setOutputKeyClass(Text.class);
     conf.setOutputValueClass(IntWritable.class);

     conf.setMapperClass(Map.class);
     conf.setReducerClass(Reduce.class);

     conf.setInputFormat(TextInputFormat.class);
     conf.setOutputFormat(TextOutputFormat.class);

     FileInputFormat.setInputPaths(conf, new Path(args[0]));
     FileOutputFormat.setOutputPath(conf, new Path(args[1]));

     JobClient.runJob(conf);
 }
}

代码分为三个部分：Map静态类 Reduce静态类主函数Main
Map类：

public static class Map extends MapReduceBase implements Mapper<LongWritable,Text,Text,IntWritable>
 {
     private final static IntWritable one=new IntWritable(1);
     private Text word=new Text();
    @Override
    public void map(LongWritable key, Text value,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        String line=value.toString();
        StringTokenizer tokenizer=new StringTokenizer(line);
        while(tokenizer.hasMoreElements())
        {
            word.set(tokenizer.nextToken());
            output.collect(word, one);
        }
    }

 }

我们可以看到Map类继承MapReduceBase类并实现了Mapper

public abstract void map(K1, V1, org.apache.hadoop.mapred.OutputCollector<K2, V2>, org.apache.hadoop.mapred.Reporter) throws java.io.IOException;

这就是前面代码中需要包含代码实现的原因，因为继承这个接口需要实现这个接口中所有方法。

同样对于Reducer接口，下面是Reducer接口通过反编译工具得到的类的细节：
Reduce类

public static class Reduce extends MapReduceBase implements Reducer<Text,IntWritable,Text,IntWritable>
 {

    @Override
    public void reduce(Text key, Iterator<IntWritable> values,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        int sum=0;
        while(values.hasNext())
        {
            sum+=values.next().get();
        }
        output.collect(key, new IntWritable(sum));
    }

 }

Reducer接口：
这里写图片描述
可以看出Reducer接口继承自org.apache.hadoop.mapred.JobConfigurable, org.apache.hadoop.io.Closeable，需要实现这个方法：

public abstract void reduce(K2, java.util.Iterator<V2>, org.apache.hadoop.mapred.OutputCollector<K3, V3>, org.apache.hadoop.mapred.Reporter) throws java.io.IOException;

理由如前述。
Main函数：

public static void main(String[] args) throws Exception
 {
     JobConf conf=new JobConf(WordCount.class);
     conf.setJobName("wordcount");

     conf.setOutputKeyClass(Text.class);
     conf.setOutputValueClass(IntWritable.class);

     conf.setMapperClass(Map.class);
     conf.setReducerClass(Reduce.class);

     conf.setInputFormat(TextInputFormat.class);
     conf.setOutputFormat(TextOutputFormat.class);

     FileInputFormat.setInputPaths(conf, new Path(args[0]));
     FileOutputFormat.setOutputPath(conf, new Path(args[1]));

     JobClient.runJob(conf);
 }

最后主函数调用Jobconf类来对MapReduce Jobjin进行初始化，然后调用SetJobName()方法命名这个Job.对Job进行合理命名有助于更快地找到Job,以便在JobTracker和TaskTracker的页面对其进行监视。对于HDFS和MapReduce的原理介绍在相关书籍上有详述这里就不赘述了。
（7）设置输入路径input/file01,input/file02 输出路径output
在主函数的args给出来通过空格隔开，需要是完整路径，具体给出方式，如图所示：
这里写图片描述
选中WordCount右键Run As之后有个Run Configurations,在Arguments中进行设置。
(8）执行WordCount类既可以通过Run as Java Application也可以Run as Run on Hadoop.执行完就会生成一个output文件夹，文件夹里面有两个文件，如图所示：
控制台中的内容：

2016-10-05 00:30:55,946 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.connection.maxidletime
2016-10-05 00:30:55,946 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.combine.progress.records
2016-10-05 00:30:55,947 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.output.key.class
2016-10-05 00:30:55,947 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.persist.jobstatus.hours
2016-10-05 00:30:55,947 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.address
2016-10-05 00:30:55,948 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.ha.automatic-failover.embedded
2016-10-05 00:30:55,948 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.recovery.store.fs.uri
2016-10-05 00:30:55,948 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.zk-state-store.parent-path
2016-10-05 00:30:55,949 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.job.task.listener.thread-count
2016-10-05 00:30:55,949 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.running.map.limit
2016-10-05 00:30:55,949 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.zookeeper.parent-znode
2016-10-05 00:30:55,949 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.container-executor.class
2016-10-05 00:30:55,950 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.skip.checksum.errors
2016-10-05 00:30:55,950 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.user.name
2016-10-05 00:30:55,950 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.scheduler.client.thread-count
2016-10-05 00:30:55,951 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.recovery.dir
2016-10-05 00:30:55,951 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.emit-timeline-data
2016-10-05 00:30:55,951 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.http.authentication.kerberos.principal
2016-10-05 00:30:55,951 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.log.level
2016-10-05 00:30:55,952 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern
2016-10-05 00:30:55,952 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3.maxRetries
2016-10-05 00:30:55,952 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.resourcemanager.minimum.version
2016-10-05 00:30:55,953 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.kerberos.kinit.command
2016-10-05 00:30:55,953 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.log-aggregation.retain-check-interval-seconds
2016-10-05 00:30:55,954 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.process-kill-wait.ms
2016-10-05 00:30:55,954 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.cgroups.mount
2016-10-05 00:30:55,954 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.working.dir
2016-10-05 00:30:55,954 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.fast.buffer.size
2016-10-05 00:30:55,955 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.handler.count
2016-10-05 00:30:55,955 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.admin.address
2016-10-05 00:30:55,955 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.client-am.ipc.max-retries
2016-10-05 00:30:55,956 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.nm.uploader.replication.factor
2016-10-05 00:30:55,956 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.util.hash.type
2016-10-05 00:30:55,956 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.seqfile.lazydecompress
2016-10-05 00:30:55,957 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.disk-health-checker.min-healthy-disks
2016-10-05 00:30:55,957 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.client.max-cached-nodemanagers-proxies
2016-10-05 00:30:55,957 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.maxtaskfailures.per.tracker
2016-10-05 00:30:55,957 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.healthchecker.script.timeout
2016-10-05 00:30:55,958 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.fs.state-store.retry-interval-ms
2016-10-05 00:30:55,960 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.group.mapping.ldap.search.attr.group.name
2016-10-05 00:30:55,960 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.df.interval
2016-10-05 00:30:55,960 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.crypto.buffer.size
2016-10-05 00:30:55,961 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.reduce.shuffle.consumer.plugin.class
2016-10-05 00:30:55,961 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.address
2016-10-05 00:30:55,961 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.tasks.sleeptimebeforesigkill
2016-10-05 00:30:55,962 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.acl-view-job
2016-10-05 00:30:55,962 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.job.committer.cancel-timeout
2016-10-05 00:30:55,962 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.connection-keep-alive.enable
2016-10-05 00:30:55,964 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.minicluster.fixed.ports
2016-10-05 00:30:55,964 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.admin.acl
2016-10-05 00:30:55,964 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.scheduler.address
2016-10-05 00:30:55,965 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.http.address
2016-10-05 00:30:55,965 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.group.mapping.ldap.ssl
2016-10-05 00:30:55,965 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.memory.mb
2016-10-05 00:30:55,965 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.client.best-effort
2016-10-05 00:30:55,966 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.merge.progress.records
2016-10-05 00:30:55,966 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size
2016-10-05 00:30:55,966 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.threads.keepalivetime
2016-10-05 00:30:55,967 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.recovery.enabled
2016-10-05 00:30:55,967 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for net.topology.script.number.args
2016-10-05 00:30:55,990 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.local.clientfactory.class.name
2016-10-05 00:30:55,991 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.threads.core
2016-10-05 00:30:55,991 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.group.mapping.ldap.directory.search.timeout
2016-10-05 00:30:55,997 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.native.lib.available
2016-10-05 00:30:55,997 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.disk-health-checker.interval-ms
2016-10-05 00:30:55,998 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs
2016-10-05 00:30:55,998 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.connection.ssl.enabled
2016-10-05 00:30:55,998 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.webapp.address
2016-10-05 00:30:55,999 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.resource-tracker.client.thread-count
2016-10-05 00:30:55,999 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.client.failover-retries
2016-10-05 00:30:55,999 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.jaas.context
2016-10-05 00:30:56,000 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.store.class
2016-10-05 00:30:56,000 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.zk-timeout-ms
2016-10-05 00:30:56,000 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.aux-services.mapreduce_shuffle.class
2016-10-05 00:30:56,001 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.health-monitor.rpc-timeout.ms
2016-10-05 00:30:56,001 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.markreset.buffer.percent
2016-10-05 00:30:56,001 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size
2016-10-05 00:30:56,002 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.client.thread-count
2016-10-05 00:30:56,002 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.health-checker.script.timeout-ms
2016-10-05 00:30:56,002 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for file.bytes-per-checksum
2016-10-05 00:30:56,003 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.map.index.skip
2016-10-05 00:30:56,003 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.client-server.address
2016-10-05 00:30:56,003 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.client.retry-interval-ms
2016-10-05 00:30:56,004 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.timeout
2016-10-05 00:30:56,004 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.cpu.vcores
2016-10-05 00:30:56,005 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ftp.blocksize
2016-10-05 00:30:56,005 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.container-manager.thread-count
2016-10-05 00:30:56,005 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.server.listen.queue.size
2016-10-05 00:30:56,006 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.hostname.verifier
2016-10-05 00:30:56,006 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.ha.automatic-failover.enabled
2016-10-05 00:30:56,006 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.dns.interface
2016-10-05 00:30:56,007 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapred.reducer.class
2016-10-05 00:30:56,007 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.group.mapping.ldap.search.attr.member
2016-10-05 00:30:56,007 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.outofband.heartbeat
2016-10-05 00:30:56,007 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.impl
2016-10-05 00:30:56,008 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.userlog.retain.hours
2016-10-05 00:30:56,008 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.system.acls
2016-10-05 00:30:56,009 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.name
2016-10-05 00:30:56,009 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.resource.memory-mb
2016-10-05 00:30:56,009 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.shuffle.log.separate
2016-10-05 00:30:56,010 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds
2016-10-05 00:30:56,010 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.keystores.factory.class
2016-10-05 00:30:56,010 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.zk.retry.ceiling.ms
2016-10-05 00:30:56,011 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.http.policy
2016-10-05 00:30:56,011 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.windows-container.memory-limit.enabled
2016-10-05 00:30:56,012 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.root-dir
2016-10-05 00:30:56,012 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.AbstractFileSystem.har.impl
2016-10-05 00:30:56,012 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.random.device.file.path
2016-10-05 00:30:56,013 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.maxattempts
2016-10-05 00:30:56,013 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.webapp.address
2016-10-05 00:30:56,013 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.require.client.cert
2016-10-05 00:30:56,014 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ftp.client-write-packet-size
2016-10-05 00:30:56,014 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.cleaner.enable
2016-10-05 00:30:56,014 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.du.interval
2016-10-05 00:30:56,015 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.retry-delay.max.ms
2016-10-05 00:30:56,015 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.profile.reduces
2016-10-05 00:30:56,015 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.health-monitor.connect-retry-interval.ms
2016-10-05 00:30:56,016 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.jobhistory.task.numberprogresssplits
2016-10-05 00:30:56,016 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.ftp.host.port
2016-10-05 00:30:56,016 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.speculative
2016-10-05 00:30:56,016 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.client.submit.file.replication
2016-10-05 00:30:56,017 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3native.blocksize
2016-10-05 00:30:56,017 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.ubertask.maxmaps
2016-10-05 00:30:56,017 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.cluster.acls.enabled
2016-10-05 00:30:56,018 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.app-checker.class
2016-10-05 00:30:56,018 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.uid.cache.secs
2016-10-05 00:30:56,018 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.localizer.fetch.thread-count
2016-10-05 00:30:56,019 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.trash.checkpoint.interval
2016-10-05 00:30:56,019 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for map.sort.class
2016-10-05 00:30:56,019 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapred.queue.default.acl-administer-jobs
2016-10-05 00:30:56,020 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.staging-dir
2016-10-05 00:30:56,020 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.connect.timeout
2016-10-05 00:30:56,020 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.AbstractFileSystem.file.impl
2016-10-05 00:30:56,021 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.env-whitelist
2016-10-05 00:30:56,021 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.zk.retry.interval.ms
2016-10-05 00:30:56,021 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage
2016-10-05 00:30:56,021 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.keytab
2016-10-05 00:30:56,022 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.reduces
2016-10-05 00:30:56,022 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.recovery.store.class
2016-10-05 00:30:56,023 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.complete.cancel.delegation.tokens
2016-10-05 00:30:56,023 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.group.mapping.ldap.search.filter.user
2016-10-05 00:30:56,023 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapred.output.format.class
2016-10-05 00:30:56,024 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.sleep-delay-before-sigkill.ms
2016-10-05 00:30:56,024 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.joblist.cache.size
2016-10-05 00:30:56,024 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.recovery.store.leveldb.path
2016-10-05 00:30:56,024 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.healthchecker.interval
2016-10-05 00:30:56,025 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.heartbeats.in.second
2016-10-05 00:30:56,025 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.persist.jobstatus.dir
2016-10-05 00:30:56,025 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.rpc.protection
2016-10-05 00:30:56,026 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.container.log.backups
2016-10-05 00:30:56,026 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ftp.stream-buffer-size
2016-10-05 00:30:56,026 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.address
2016-10-05 00:30:56,027 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.recovery.enabled
2016-10-05 00:30:56,027 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.input.fileinputformat.numinputfiles
2016-10-05 00:30:56,027 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.admin.client.thread-count
2016-10-05 00:30:56,028 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.groups.negative-cache.secs
2016-10-05 00:30:56,028 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for file.client-write-packet-size
2016-10-05 00:30:56,028 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.http.authentication.simple.anonymous.allowed
2016-10-05 00:30:56,028 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.leveldb-timeline-store.path
2016-10-05 00:30:56,029 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.proxy-user-privileges.enabled
2016-10-05 00:30:56,029 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.log.retain-seconds
2016-10-05 00:30:56,029 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms
2016-10-05 00:30:56,030 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.zk.session.timeout.ms
2016-10-05 00:30:56,030 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.failover-controller.cli-check.rpc-timeout.ms
2016-10-05 00:30:56,030 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.instrumentation
2016-10-05 00:30:56,031 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.cgroups.hierarchy
2016-10-05 00:30:56,031 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.groups.cache.warn.after.ms
2016-10-05 00:30:56,031 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.fetch.retry.timeout-ms
2016-10-05 00:30:56,032 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.client.thread-count
2016-10-05 00:30:56,032 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.mapfile.bloom.size
2016-10-05 00:30:56,032 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.work-preserving-recovery.enabled
2016-10-05 00:30:56,033 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.docker-container-executor.exec-name
2016-10-05 00:30:56,033 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.zk-num-retries
2016-10-05 00:30:56,033 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for dfs.ha.fencing.ssh.connect-timeout
2016-10-05 00:30:56,033 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.zk.root
2016-10-05 00:30:56,034 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3.bytes-per-checksum
2016-10-05 00:30:56,034 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.container.log.limit.kb
2016-10-05 00:30:56,034 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.automatic.close
2016-10-05 00:30:56,035 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.admin.thread-count
2016-10-05 00:30:56,035 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.trash.interval
2016-10-05 00:30:56,035 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.ttl-ms
2016-10-05 00:30:56,035 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.authentication
2016-10-05 00:30:56,036 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.defaultFS
2016-10-05 00:30:56,036 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.connect.max.retries
2016-10-05 00:30:56,037 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.server.conf
2016-10-05 00:30:56,037 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.delayed.delegation-token.removal-interval-ms
2016-10-05 00:30:56,037 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.taskscheduler
2016-10-05 00:30:56,037 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.store-class
2016-10-05 00:30:56,038 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.output.fileoutputformat.compress
2016-10-05 00:30:56,047 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.am.liveness-monitor.expiry-interval-ms
2016-10-05 00:30:56,047 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for net.topology.node.switch.mapping.impl
2016-10-05 00:30:56,047 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.swift.impl
2016-10-05 00:30:56,047 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.fast.upload
2016-10-05 00:30:56,048 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.fetch.retry.enabled
2016-10-05 00:30:56,048 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.max.split.locations
2016-10-05 00:30:56,048 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.counters.max
2016-10-05 00:30:56,049 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.address
2016-10-05 00:30:56,049 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.windows-container.cpu-limit.enabled
2016-10-05 00:30:56,049 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.map.index.interval
2016-10-05 00:30:56,049 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.shuffle.log.limit.kb
2016-10-05 00:30:56,050 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapred.child.java.opts
2016-10-05 00:30:56,050 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.local.dir.minspacestart
2016-10-05 00:30:56,050 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.client.progressmonitor.pollinterval
2016-10-05 00:30:56,051 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.profile.map.params
2016-10-05 00:30:56,051 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.queuename
2016-10-05 00:30:56,051 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.tasktracker.maxblacklists
2016-10-05 00:30:56,051 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.connection.establish.timeout
2016-10-05 00:30:56,052 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.zk.quorum
2016-10-05 00:30:56,052 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts
2016-10-05 00:30:56,052 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.localizer.address
2016-10-05 00:30:56,053 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.mapfile.bloom.error.rate
2016-10-05 00:30:56,053 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.delete.thread-count
2016-10-05 00:30:56,053 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.split.metainfo.maxsize
2016-10-05 00:30:56,053 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.scheduler.maximum-allocation-vcores
2016-10-05 00:30:56,054 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapred.mapper.new-api
2016-10-05 00:30:56,054 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.dir
2016-10-05 00:30:56,054 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.resource.mb
2016-10-05 00:30:56,055 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapred.input.format.class
2016-10-05 00:30:56,055 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.reducer.preempt.delay.sec
2016-10-05 00:30:56,055 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.output.compress.codec
2016-10-05 00:30:56,056 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb
2016-10-05 00:30:56,056 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.connection.maximum
2016-10-05 00:30:56,056 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.log.level
2016-10-05 00:30:56,056 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.seqfile.compress.blocksize
2016-10-05 00:30:56,057 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.taskcontroller
2016-10-05 00:30:56,057 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.groups.cache.secs
2016-10-05 00:30:56,057 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.client-server.thread-count
2016-10-05 00:30:56,058 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.input.lineinputformat.linespermap
2016-10-05 00:30:56,058 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.end-notification.max.attempts
2016-10-05 00:30:56,058 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user
2016-10-05 00:30:56,058 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.uploader.server.address
2016-10-05 00:30:56,059 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.webapp.address
2016-10-05 00:30:56,059 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.recovery.enable
2016-10-05 00:30:56,059 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.submithostname
2016-10-05 00:30:56,059 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.expire.trackers.interval
2016-10-05 00:30:56,060 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.webapp.address
2016-10-05 00:30:56,060 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.nm.uploader.thread-count
2016-10-05 00:30:56,060 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.kms.client.encrypted.key.cache.num.refill.threads
2016-10-05 00:30:56,062 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.health-checker.interval-ms
2016-10-05 00:30:56,062 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.loadedjobs.cache.size
2016-10-05 00:30:56,063 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.map.output.collector.class
2016-10-05 00:30:56,063 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.authorization
2016-10-05 00:30:56,063 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.am.max-attempts
2016-10-05 00:30:56,063 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.ftp.host
2016-10-05 00:30:56,064 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.attempts.maximum
2016-10-05 00:30:56,064 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms
2016-10-05 00:30:56,064 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.ifile.readahead
2016-10-05 00:30:56,064 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.scheduler.monitor.enable
2016-10-05 00:30:56,065 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.zk-retry-interval-ms
2016-10-05 00:30:56,065 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.zookeeper.session-timeout.ms
2016-10-05 00:30:56,065 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.taskmemorymanager.monitoringinterval
2016-10-05 00:30:56,066 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.parallelcopies
2016-10-05 00:30:56,066 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.output.value.class
2016-10-05 00:30:56,066 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.skip.maxrecords
2016-10-05 00:30:56,067 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.webapp.https.address
2016-10-05 00:30:56,067 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.read.timeout
2016-10-05 00:30:56,067 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.remote-app-log-dir-suffix
2016-10-05 00:30:56,067 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.instrumentation
2016-10-05 00:30:56,068 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.output.fileoutputformat.compress.codec
2016-10-05 00:30:56,068 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.connect.retry.interval
2016-10-05 00:30:56,068 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.speculative
2016-10-05 00:30:56,069 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.keytab
2016-10-05 00:30:56,069 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.datestring.cache.size
2016-10-05 00:30:56,069 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for file.blocksize
2016-10-05 00:30:56,069 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.admin.address
2016-10-05 00:30:56,070 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.cpu.vcores
2016-10-05 00:30:56,070 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled
2016-10-05 00:30:56,070 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.resource-tracker.address
2016-10-05 00:30:56,070 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.configuration.provider-class
2016-10-05 00:30:56,071 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.local.dir.minspacekill
2016-10-05 00:30:56,071 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.staging.root.dir
2016-10-05 00:30:56,071 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.retiredjobs.cache.size
2016-10-05 00:30:56,072 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.rm.enabled
2016-10-05 00:30:56,072 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.connect.max.retries.on.timeouts
2016-10-05 00:30:56,072 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.crypto.codec.classes.aes.ctr.nopadding
2016-10-05 00:30:56,072 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.zookeeper.acl
2016-10-05 00:30:56,073 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.local-dirs
2016-10-05 00:30:56,073 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.app-submission.cross-platform
2016-10-05 00:30:56,073 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.connect.timeout
2016-10-05 00:30:56,074 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for rpc.metrics.quantile.enable
2016-10-05 00:30:56,074 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.recovery.enabled
2016-10-05 00:30:56,074 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.leveldb-state-store.path
2016-10-05 00:30:56,074 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.end-notification.retry.attempts
2016-10-05 00:30:56,075 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.system.dir
2016-10-05 00:30:56,075 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.admin-env
2016-10-05 00:30:56,075 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.log-aggregation.retain-seconds
2016-10-05 00:30:56,075 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.jobhistory.block.size
2016-10-05 00:30:56,076 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.indexcache.mb
2016-10-05 00:30:56,076 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.handler-thread-count
2016-10-05 00:30:56,076 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.hostname
2016-10-05 00:30:56,077 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for net.topology.impl
2016-10-05 00:30:56,077 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.multipart.purge.age
2016-10-05 00:30:56,077 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.threads.max
2016-10-05 00:30:56,077 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.java.secure.random.algorithm
2016-10-05 00:30:56,078 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.container-monitor.interval-ms
2016-10-05 00:30:56,078 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.multipart.threshold
2016-10-05 00:30:56,078 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.reduce.slowstart.completedmaps
2016-10-05 00:30:56,079 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapred.reducer.new-api
2016-10-05 00:30:56,079 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.instrumentation.requires.admin
2016-10-05 00:30:56,079 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.compression.codec.bzip2.library
2016-10-05 00:30:56,079 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.http.authentication.signature.secret.file
2016-10-05 00:30:56,080 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.secure
2016-10-05 00:30:56,080 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.input.fileinputformat.inputdir
2016-10-05 00:30:56,080 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.localizer.cache.target-size-mb
2016-10-05 00:30:56,081 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.maxattempts
2016-10-05 00:30:56,081 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3native.replication
2016-10-05 00:30:56,081 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.skip.proc.count.autoincr
2016-10-05 00:30:56,081 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for file.replication
2016-10-05 00:30:56,082 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.cleaner.interval-ms
2016-10-05 00:30:56,082 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.idlethreshold
2016-10-05 00:30:56,082 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.tmp.dir
2016-10-05 00:30:56,082 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.store.class
2016-10-05 00:30:56,083 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.restart.recover
2016-10-05 00:30:56,083 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.address
2016-10-05 00:30:56,083 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.cluster.local.dir
2016-10-05 00:30:56,084 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.speculative.retry-after-no-speculate
2016-10-05 00:30:56,084 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.client.nodemanager-client-async.thread-pool-max-size
2016-10-05 00:30:56,084 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.state-store-class
2016-10-05 00:30:56,084 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.resource.cpu-vcores
2016-10-05 00:30:56,085 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.nested-level
2016-10-05 00:30:56,085 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3.buffer.dir
2016-10-05 00:30:56,085 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.remote-app-log-dir
2016-10-05 00:30:56,085 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.memory.mb
2016-10-05 00:30:56,086 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.client.job.max-retries
2016-10-05 00:30:56,086 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.work.around.non.threadsafe.getpwuid
2016-10-05 00:30:56,086 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.profile.reduce.params
2016-10-05 00:30:56,087 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.client.max-retries
2016-10-05 00:30:56,087 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.transfer.buffer.size
2016-10-05 00:30:56,087 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.persist.jobstatus.active
2016-10-05 00:30:56,087 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.log-dirs
2016-10-05 00:30:56,088 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs
2016-10-05 00:30:56,088 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.health-monitor.sleep-after-disconnect.ms
2016-10-05 00:30:56,088 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.fs.state-store.uri
2016-10-05 00:30:56,089 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.rpc.socket.factory.class.default
2016-10-05 00:30:56,089 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.keytab
2016-10-05 00:30:56,089 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.AbstractFileSystem.ftp.impl
2016-10-05 00:30:56,089 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.leveldb-state-store.path
2016-10-05 00:30:56,090 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.profile
2016-10-05 00:30:56,090 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.move.interval-ms
2016-10-05 00:30:56,090 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.max.total.tasks
2016-10-05 00:30:56,091 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.kms.client.encrypted.key.cache.size
2016-10-05 00:30:56,091 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.http.policy
2016-10-05 00:30:56,091 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.intermediate-done-dir
2016-10-05 00:30:56,091 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.hard-kill-timeout-ms
2016-10-05 00:30:56,092 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.skip.proc.count.autoincr
2016-10-05 00:30:56,092 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.AbstractFileSystem.viewfs.impl
2016-10-05 00:30:56,092 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.speculative.slowtaskthreshold
2016-10-05 00:30:56,092 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled
2016-10-05 00:30:56,093 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3native.stream-buffer-size
2016-10-05 00:30:56,093 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.delete.debug-delay-sec
2016-10-05 00:30:56,093 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3n.multipart.uploads.block.size
2016-10-05 00:30:56,094 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.ifile.readahead.bytes
2016-10-05 00:30:56,094 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.scheduler.maximum-allocation-mb
2016-10-05 00:30:56,094 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.fallback-to-simple-auth-allowed
2016-10-05 00:30:56,094 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.har.impl.disable.cache
2016-10-05 00:30:56,095 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.leveldb-timeline-store.read-cache-size
2016-10-05 00:30:56,095 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3native.bytes-per-checksum
2016-10-05 00:30:56,095 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.hostname
2016-10-05 00:30:56,096 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.committer.setup.cleanup.needed
2016-10-05 00:30:56,096 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms
2016-10-05 00:30:56,096 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.paging.maximum
2016-10-05 00:30:56,096 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.client.nodemanager-connect.retry-interval-ms
2016-10-05 00:30:56,097 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.log-aggregation.compression-type
2016-10-05 00:30:56,097 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.job.committer.commit-window
2016-10-05 00:30:56,097 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.http.authentication.type
2016-10-05 00:30:56,098 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.submithostaddress
2016-10-05 00:30:56,098 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.vmem-check-enabled
2016-10-05 00:30:56,098 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.jetty.logs.serve.aliases
2016-10-05 00:30:56,098 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.failover-controller.graceful-fence.rpc-timeout.ms
2016-10-05 00:30:56,099 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.input.buffer.percent
2016-10-05 00:30:56,099 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.running.reduce.limit
2016-10-05 00:30:56,099 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.io.sort.mb
2016-10-05 00:30:56,100 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.merge.inmem.threshold
2016-10-05 00:30:56,100 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.client.application-client-protocol.poll-interval-ms
2016-10-05 00:30:56,100 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.kms.client.authentication.retry-count
2016-10-05 00:30:56,100 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.connect.max-wait.ms
2016-10-05 00:30:56,101 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.enabled
2016-10-05 00:30:56,101 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users
2016-10-05 00:30:56,101 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.client.conf
2016-10-05 00:30:56,101 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.container.liveness-monitor.interval-ms
2016-10-05 00:30:56,102 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.vmem-pmem-ratio
2016-10-05 00:30:56,102 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.client.completion.pollinterval
2016-10-05 00:30:56,102 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.client.max-retries
2016-10-05 00:30:56,103 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.enabled
2016-10-05 00:30:56,103 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.AbstractFileSystem.hdfs.impl
2016-10-05 00:30:56,103 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.client.resolve.remote.symlinks
2016-10-05 00:30:56,103 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.reduce.tasks.maximum
2016-10-05 00:30:56,104 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.hostname
2016-10-05 00:30:56,104 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.input.buffer.percent
2016-10-05 00:30:56,104 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.cleaner.resource-sleep-ms
2016-10-05 00:30:56,105 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.multipart.purge
2016-10-05 00:30:56,105 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.command-opts
2016-10-05 00:30:56,105 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.cleaner.period-mins
2016-10-05 00:30:56,106 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.webapp.https.address
2016-10-05 00:30:56,106 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.ssl.file.buffer.size
2016-10-05 00:30:56,106 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.maxtasks.perjob
2016-10-05 00:30:56,106 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.userlog.limit.kb
2016-10-05 00:30:56,107 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.kms.client.encrypted.key.cache.low-watermark
2016-10-05 00:30:56,107 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.buffer.dir
2016-10-05 00:30:56,107 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3.client-write-packet-size
2016-10-05 00:30:56,108 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.user.group.static.mapping.overrides
2016-10-05 00:30:56,108 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.max.threads
2016-10-05 00:30:56,108 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.kms.client.encrypted.key.cache.expiry
2016-10-05 00:30:56,108 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.maps
2016-10-05 00:30:56,109 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.enabled.protocols
2016-10-05 00:30:56,109 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.output.compress
2016-10-05 00:30:56,109 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3.blocksize
2016-10-05 00:30:56,110 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.pmem-check-enabled
2016-10-05 00:30:56,110 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.scheduler.minimum-allocation-mb
2016-10-05 00:30:56,110 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.http-authentication.type
2016-10-05 00:30:56,111 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.max-age-ms
2016-10-05 00:30:56,111 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ftp.replication
2016-10-05 00:30:56,111 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.input.fileinputformat.split.minsize
2016-10-05 00:30:56,111 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3n.block.size
2016-10-05 00:30:56,112 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.token.tracking.ids.enabled
2016-10-05 00:30:56,112 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.shuffle.log.backups
2016-10-05 00:30:56,112 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.ipc.rpc.class
2016-10-05 00:30:56,113 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.localizer.cache.cleanup.interval-ms
2016-10-05 00:30:56,113 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.http.staticuser.user
2016-10-05 00:30:56,113 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.move.thread-count
2016-10-05 00:30:56,113 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.jvm.numtasks
2016-10-05 00:30:56,114 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.multipart.size
2016-10-05 00:30:56,114 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.profile.maps
2016-10-05 00:30:56,114 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapred.mapper.class
2016-10-05 00:30:56,115 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.port
2016-10-05 00:30:56,115 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.nodemanager.minimum.version
2016-10-05 00:30:56,115 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.connection-keep-alive.timeout
2016-10-05 00:30:56,115 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.merge.percent
2016-10-05 00:30:56,116 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.http.address
2016-10-05 00:30:56,116 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.amlauncher.thread-count
2016-10-05 00:30:56,116 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.connect.retry-interval.ms
2016-10-05 00:30:56,116 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.skip.start.attempts
2016-10-05 00:30:56,117 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.scheduler.minimum-allocation-vcores
2016-10-05 00:30:56,117 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.io.sort.factor
2016-10-05 00:30:56,117 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.speculative.retry-after-speculate
2016-10-05 00:30:56,117 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for nfs.exports.allowed.hosts
2016-10-05 00:30:56,118 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for tfile.fs.input.buffer.size
2016-10-05 00:30:56,118 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.server.max.connections
2016-10-05 00:30:56,118 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for tfile.io.chunk.size
2016-10-05 00:30:56,118 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3.block.size
2016-10-05 00:30:56,119 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3n.multipart.copy.block.size
2016-10-05 00:30:56,119 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.speculative.speculative-cap-total-tasks
2016-10-05 00:30:56,119 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.serializations
2016-10-05 00:30:56,119 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.max-completed-applications
2016-10-05 00:30:56,120 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.principal
2016-10-05 00:30:56,120 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.output.fileoutputformat.outputdir
2016-10-05 00:30:56,120 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.ha.automatic-failover.zk-base-path
2016-10-05 00:30:56,121 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.fetch.retry.interval-ms
2016-10-05 00:30:56,121 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.end-notification.retry.interval
2016-10-05 00:30:56,121 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.speculative.minimum-allowed-tasks
2016-10-05 00:30:56,121 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3n.multipart.uploads.enabled
2016-10-05 00:30:56,122 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.seqfile.sorter.recordlimit
2016-10-05 00:30:56,122 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3native.client-write-packet-size
2016-10-05 00:30:56,122 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.group.mapping
2016-10-05 00:30:56,122 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.nodemanager-connect-retries
2016-10-05 00:30:56,123 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ftp.bytes-per-checksum
2016-10-05 00:30:56,123 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.store.in-memory.check-period-mins
2016-10-05 00:30:56,123 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.connection.timeout
2016-10-05 00:30:56,124 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.end-notification.max.retry.interval
2016-10-05 00:30:56,124 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.acl.enable
2016-10-05 00:30:56,124 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nm.liveness-monitor.expiry-interval-ms
2016-10-05 00:30:56,124 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.enabled
2016-10-05 00:30:56,125 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.input.fileinputformat.list-status.num-threads
2016-10-05 00:30:56,125 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.map.tasks.maximum
2016-10-05 00:30:56,125 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.ttl-enable
2016-10-05 00:30:56,125 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.resources-handler.class
2016-10-05 00:30:56,126 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.state-store.max-completed-applications
2016-10-05 00:30:56,126 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.classloader
2016-10-05 00:30:56,126 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size
2016-10-05 00:30:56,126 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.store.in-memory.initial-delay-mins
2016-10-05 00:30:56,127 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.hdfs-servers
2016-10-05 00:30:56,127 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.dns.nameserver
2016-10-05 00:30:56,127 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.ubertask.maxreduces
2016-10-05 00:30:56,130 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.ssl.enabled
2016-10-05 00:30:56,130 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.log-aggregation-enable
2016-10-05 00:30:56,131 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.report.address
2016-10-05 00:30:56,131 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.http.threads
2016-10-05 00:30:56,131 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for tfile.fs.output.buffer.size
2016-10-05 00:30:56,132 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.permissions.umask-mode
2016-10-05 00:30:56,132 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.checksum.algo.impl
2016-10-05 00:30:56,132 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.am.max-attempts
2016-10-05 00:30:56,133 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.resource.percentage-physical-cpu-limit
2016-10-05 00:30:56,133 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.bytes.per.checksum
2016-10-05 00:30:56,133 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.failover-controller.graceful-fence.connection.retries
2016-10-05 00:30:56,133 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.speculative.speculative-cap-running-tasks
2016-10-05 00:30:56,134 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.cleaner.initial-delay-mins
2016-10-05 00:30:56,134 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.resource.cpu-vcores
2016-10-05 00:30:56,134 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.common.configuration.version
2016-10-05 00:30:56,135 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.ubertask.enable
2016-10-05 00:30:56,135 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.acl-modify-job
2016-10-05 00:30:56,135 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.seqfile.local.dir
2016-10-05 00:30:56,135 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3.sleepTimeSeconds
2016-10-05 00:30:56,136 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.system-metrics-publisher.enabled
2016-10-05 00:30:56,136 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.client.output.filter
2016-10-05 00:30:56,392 [org.apache.hadoop.mapreduce.Job]-[INFO] The url to track the job: http://localhost:8080/
2016-10-05 00:30:56,398 [org.apache.hadoop.mapreduce.Job]-[INFO] Running job: job_local1801377718_0001
2016-10-05 00:30:56,402 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-05 00:30:56,403 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-05 00:30:56,407 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] OutputCommitter set in config null
2016-10-05 00:30:56,411 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2016-10-05 00:30:56,426 [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter]-[INFO] File Output Committer Algorithm version is 1
2016-10-05 00:30:56,535 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[DEBUG] Starting mapper thread pool executor.
2016-10-05 00:30:56,536 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[DEBUG] Max local threads: 1
2016-10-05 00:30:56,536 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[DEBUG] Map tasks to process: 1
2016-10-05 00:30:56,553 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] Waiting for map tasks
2016-10-05 00:30:56,568 [org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable]-[INFO] Starting task: attempt_local1801377718_0001_m_000000_0
2016-10-05 00:30:56,603 [org.apache.hadoop.mapred.SortedRanges$SkipRangeIterator]-[DEBUG] currentIndex 0   0:0
2016-10-05 00:30:56,640 [org.apache.hadoop.mapred.LocalJobRunner]-[DEBUG] mapreduce.cluster.local.dir for child : /tmp/hadoop-dtw/mapred/local/localRunner//dtw/jobcache/job_local1801377718_0001/attempt_local1801377718_0001_m_000000_0
2016-10-05 00:30:56,649 [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter]-[INFO] File Output Committer Algorithm version is 1
2016-10-05 00:30:56,698 [org.apache.hadoop.mapred.Task]-[INFO]  Using ResourceCalculatorProcessTree : [ ]
2016-10-05 00:30:56,723 [org.apache.hadoop.mapred.MapTask]-[INFO] Processing split: file:/home/dtw/hadoop-2.7.1/input/file01:0+86
2016-10-05 00:30:56,741 [org.apache.hadoop.mapred.MapTask]-[INFO] numReduceTasks: 1
2016-10-05 00:30:56,750 [org.apache.hadoop.mapred.MapTask]-[DEBUG] Trying map output collector class: org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-10-05 00:30:56,807 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] (EQUATOR) 0 kvi 26214396(104857584)
2016-10-05 00:30:56,807 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] mapreduce.task.io.sort.mb: 100
2016-10-05 00:30:56,808 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] soft limit at 83886080
2016-10-05 00:30:56,808 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] bufstart = 0; bufvoid = 104857600
2016-10-05 00:30:56,808 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] kvstart = 26214396; length = 6553600
2016-10-05 00:30:56,818 [org.apache.hadoop.mapred.MapTask]-[INFO] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-10-05 00:30:56,833 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] 
2016-10-05 00:30:56,834 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] Starting flush of map output
2016-10-05 00:30:56,834 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] Spilling map output
2016-10-05 00:30:56,835 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] bufstart = 0; bufend = 150; bufvoid = 104857600
2016-10-05 00:30:56,835 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] kvstart = 26214396(104857584); kvend = 26214336(104857344); length = 61/6553600
2016-10-05 00:30:56,860 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] Finished spill 0
2016-10-05 00:30:56,868 [org.apache.hadoop.mapred.Task]-[INFO] Task:attempt_local1801377718_0001_m_000000_0 is done. And is in the process of committing
2016-10-05 00:30:56,887 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] file:/home/dtw/hadoop-2.7.1/input/file01:0+86
2016-10-05 00:30:56,888 [org.apache.hadoop.mapred.Task]-[INFO] Task 'attempt_local1801377718_0001_m_000000_0' done.
2016-10-05 00:30:56,888 [org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable]-[INFO] Finishing task: attempt_local1801377718_0001_m_000000_0
2016-10-05 00:30:56,889 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] map task executor complete.
2016-10-05 00:30:56,903 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[DEBUG] Starting reduce thread pool executor.
2016-10-05 00:30:56,904 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[DEBUG] Max local threads: 1
2016-10-05 00:30:56,904 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[DEBUG] Reduce tasks to process: 1
2016-10-05 00:30:56,905 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] Waiting for reduce tasks
2016-10-05 00:30:56,906 [org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable]-[INFO] Starting task: attempt_local1801377718_0001_r_000000_0
2016-10-05 00:30:56,912 [org.apache.hadoop.mapred.SortedRanges$SkipRangeIterator]-[DEBUG] currentIndex 0   0:0
2016-10-05 00:30:56,920 [org.apache.hadoop.mapred.LocalJobRunner]-[DEBUG] mapreduce.cluster.local.dir for child : /tmp/hadoop-dtw/mapred/local/localRunner//dtw/jobcache/job_local1801377718_0001/attempt_local1801377718_0001_r_000000_0
2016-10-05 00:30:56,923 [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter]-[INFO] File Output Committer Algorithm version is 1
2016-10-05 00:30:56,925 [org.apache.hadoop.mapred.Task]-[INFO]  Using ResourceCalculatorProcessTree : [ ]
2016-10-05 00:30:56,937 [org.apache.hadoop.mapred.ReduceTask]-[INFO] Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@2df4dede
2016-10-05 00:30:56,968 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[INFO] MergerManager: memoryLimit=1109078784, maxSingleShuffleLimit=277269696, mergeThreshold=731992000, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2016-10-05 00:30:56,980 [org.apache.hadoop.mapreduce.task.reduce.EventFetcher]-[INFO] attempt_local1801377718_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2016-10-05 00:30:56,985 [org.apache.hadoop.mapreduce.task.reduce.EventFetcher]-[DEBUG] Got 0 map completion events from 0
2016-10-05 00:30:56,985 [org.apache.hadoop.mapreduce.task.reduce.EventFetcher]-[DEBUG] GetMapEventsThread about to sleep for 1000
2016-10-05 00:30:56,991 [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher]-[DEBUG] LocalFetcher 1 going to fetch: attempt_local1801377718_0001_m_000000_0
2016-10-05 00:30:57,036 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[DEBUG] attempt_local1801377718_0001_m_000000_0: Proceeding with shuffle since usedMemory (0) is lesser than memoryLimit (1109078784).CommitMemory is (0)
2016-10-05 00:30:57,043 [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher]-[INFO] localfetcher#1 about to shuffle output of map attempt_local1801377718_0001_m_000000_0 decomp: 184 len: 188 to MEMORY
2016-10-05 00:30:57,052 [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput]-[INFO] Read 184 bytes from map-output for attempt_local1801377718_0001_m_000000_0
2016-10-05 00:30:57,057 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[INFO] closeInMemoryFile -> map-output of size: 184, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->184
2016-10-05 00:30:57,060 [org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl]-[DEBUG] map attempt_local1801377718_0001_m_000000_0 done 1 / 1 copied.
2016-10-05 00:30:57,060 [org.apache.hadoop.mapreduce.task.reduce.EventFetcher]-[INFO] EventFetcher is interrupted.. Returning
2016-10-05 00:30:57,063 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] 1 / 1 copied.
2016-10-05 00:30:57,063 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[INFO] finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-10-05 00:30:57,076 [org.apache.hadoop.mapred.Merger$MergeQueue]-[INFO] Merging 1 sorted segments
2016-10-05 00:30:57,076 [org.apache.hadoop.mapred.Merger$MergeQueue]-[INFO] Down to the last merge-pass, with 1 segments left of total size: 178 bytes
2016-10-05 00:30:57,080 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[INFO] Merged 1 segments, 184 bytes to disk to satisfy reduce memory limit
2016-10-05 00:30:57,080 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[DEBUG] Disk file: /tmp/hadoop-dtw/mapred/local/localRunner/dtw/jobcache/job_local1801377718_0001/attempt_local1801377718_0001_r_000000_0/output/map_0.out.merged Length is 188
2016-10-05 00:30:57,081 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[INFO] Merging 1 files, 188 bytes from disk
2016-10-05 00:30:57,082 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[INFO] Merging 0 segments, 0 bytes from memory into reduce
2016-10-05 00:30:57,083 [org.apache.hadoop.mapred.Merger$MergeQueue]-[INFO] Merging 1 sorted segments
2016-10-05 00:30:57,084 [org.apache.hadoop.mapred.Merger$MergeQueue]-[INFO] Down to the last merge-pass, with 1 segments left of total size: 178 bytes
2016-10-05 00:30:57,086 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] 1 / 1 copied.
2016-10-05 00:30:57,106 [org.apache.hadoop.mapred.Task]-[INFO] Task:attempt_local1801377718_0001_r_000000_0 is done. And is in the process of committing
2016-10-05 00:30:57,110 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] 1 / 1 copied.
2016-10-05 00:30:57,110 [org.apache.hadoop.mapred.Task]-[INFO] Task attempt_local1801377718_0001_r_000000_0 is allowed to commit now
2016-10-05 00:30:57,112 [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter]-[INFO] Saved output of task 'attempt_local1801377718_0001_r_000000_0' to file:/home/dtw/hadoop-2.7.1/output/_temporary/0/task_local1801377718_0001_r_000000
2016-10-05 00:30:57,114 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] reduce > reduce
2016-10-05 00:30:57,115 [org.apache.hadoop.mapred.Task]-[INFO] Task 'attempt_local1801377718_0001_r_000000_0' done.
2016-10-05 00:30:57,115 [org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable]-[INFO] Finishing task: attempt_local1801377718_0001_r_000000_0
2016-10-05 00:30:57,115 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] reduce task executor complete.
2016-10-05 00:30:57,122 [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter]-[DEBUG] Merging data from DeprecatedRawLocalFileStatus{path=file:/home/dtw/hadoop-2.7.1/output/_temporary/0/task_local1801377718_0001_r_000000; isDirectory=true; modification_time=1475598657000; access_time=0; owner=; group=; permission=rwxrwxrwx; isSymlink=false} to file:/home/dtw/hadoop-2.7.1/output
2016-10-05 00:30:57,123 [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter]-[DEBUG] Merging data from DeprecatedRawLocalFileStatus{path=file:/home/dtw/hadoop-2.7.1/output/_temporary/0/task_local1801377718_0001_r_000000/part-00000; isDirectory=false; length=49; replication=1; blocksize=33554432; modification_time=1475598657000; access_time=0; owner=; group=; permission=rw-rw-rw-; isSymlink=false} to file:/home/dtw/hadoop-2.7.1/output/part-00000
2016-10-05 00:30:57,183 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:331)
2016-10-05 00:30:57,404 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-05 00:30:57,405 [org.apache.hadoop.mapreduce.Job]-[INFO] Job job_local1801377718_0001 running in uber mode : false
2016-10-05 00:30:57,410 [org.apache.hadoop.mapreduce.Job]-[INFO]  map 100% reduce 100%
2016-10-05 00:30:57,412 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:670)
2016-10-05 00:30:57,413 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-05 00:30:57,414 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-05 00:30:57,415 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:670)
2016-10-05 00:30:57,415 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-05 00:30:57,416 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-05 00:30:57,416 [org.apache.hadoop.mapreduce.Job]-[INFO] Job job_local1801377718_0001 completed successfully
2016-10-05 00:30:57,417 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.getCounters(Job.java:758)
2016-10-05 00:30:57,450 [org.apache.hadoop.mapreduce.Job]-[INFO] Counters: 30
    File System Counters
        FILE: Number of bytes read=872
        FILE: Number of bytes written=409041
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
    Map-Reduce Framework
        Map input records=5
        Map output records=16
        Map output bytes=150
        Map output materialized bytes=188
        Input split bytes=92
        Combine input records=0
        Combine output records=0
        Reduce input groups=7
        Reduce shuffle bytes=188
        Reduce input records=16
        Reduce output records=7
        Spilled Records=32
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=0
        Total committed heap usage (bytes)=482344960
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=86
    File Output Format Counters 
        Bytes Written=61

input文件中的内容：
file01中的内容：

Hello World Bye World
Hello Hadoop Bye Hadoop
Bye Hadoop Hello Hadoop
yes me
good yes

file02中的内容：

Hello World Bye World
Hello Hadoop Bye Hadoop
Bye Hadoop Hello Hadoop
yes me
catch ese

执行后得到output中的内容：
包含两个文件_SUCCESS和part-00000文件
其中part-00000是最终结果，内容为：

Bye 3
Hadoop  4
Hello   3
World   2
good    1
me  1
yes 2

至此MyEclipse部分就讲解结束了。

上面是旧版本的API

从0.20.2版本开始，Hadoop提供一个新的API。新的API是在org.apache.hadoop.mapreduce中的，旧版的API则是在org.apache.hadoop.mapred中。新的API不兼容旧的API，WordCount程序用新的API重写如下：

package cn.edu.ruc.cloudcomputing.book.chapter03;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.*;
import org.apache.hadoop.mapreduce.lib.output.*;
import org.apache.hadoop.util.*;
public class WordCount extends Configured implements Tool {

    public static class Map extends Mapper<LongWritable,Text,Text,IntWritable>
    {
        private final static IntWritable one=new IntWritable(1);
        private Text word=new Text();
        protected void map(LongWritable key, Text value,Context context) throws java.io.IOException, java.lang.InterruptedException
        {
            String line=value.toString();
            StringTokenizer tokenizer=new StringTokenizer(line);
            while(tokenizer.hasMoreElements())
            {
                word.set(tokenizer.nextToken());
                context.write(word, one);
            }
        }

    }

    public static class Reduce extends Reducer<Text,IntWritable,Text,IntWritable >
    {
        protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws java.io.IOException, java.lang.InterruptedException
        {
            int sum=0;
            for(IntWritable val:values)
            {
                sum+=val.get();
            }
            context.write(key, new IntWritable(sum));
        }

    }
    @Override
    public int run(String[] args) throws Exception {
        Job job=new Job(getConf());
        job.setJarByClass(WordCount.class);
        job.setJobName("wordcount");

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        job.setMapperClass(Map.class);
        job.setReducerClass(Reduce.class);

        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job,new Path( args[1]));

        boolean success=job.waitForCompletion(true);
        return success?0:1;
    }
    public static void main(String[] args) throws Exception {
        int ret=ToolRunner.run(new WordCount(), args);
        System.exit(ret);

    }


}

新旧API的区别在网上都有，我这里对函数介绍一下。
在新的API中Mapper和Reducer已经不是接口而是抽象类。而且Map函数与Reduce函数也已经不再实现Mapper和Reducer接口，而是继承Mapper和Reducer抽象类。这样做更容易扩展，因为添加方法到抽象类中更容易。
Mapper类通过反编译工具得到如图所示：
这里写图片描述
我们可以看出Mapper要重写的方法：

protected void map(KEYIN, VALUEIN, org.apache.hadoop.mapreduce.Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context) throws java.io.IOException, java.lang.InterruptedException;

Map类：

public static class Map extends Mapper<LongWritable,Text,Text,IntWritable>
    {
        private final static IntWritable one=new IntWritable(1);
        private Text word=new Text();
        protected void map(LongWritable key, Text value,Context context) throws java.io.IOException, java.lang.InterruptedException
        {
            String line=value.toString();
            StringTokenizer tokenizer=new StringTokenizer(line);
            while(tokenizer.hasMoreElements())
            {
                word.set(tokenizer.nextToken());
                context.write(word, one);
            }
        }

    }

同样Reducer类通过反编译工具得到如图所示的类内部结构：
这里写图片描述
我们可以看出要重写的reduce方法：

protected void reduce(KEYIN, java.lang.Iterable<VALUEIN>, org.apache.hadoop.mapreduce.Reducer<KEYIN, VALUEIN, KEYOUT, VALUEOUT>.Context) throws java.io.IOException, java.lang.InterruptedException;

Reduce类：

public static class Reduce extends Reducer<Text,IntWritable,Text,IntWritable >
    {
        protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws java.io.IOException, java.lang.InterruptedException
        {
            int sum=0;
            for(IntWritable val:values)
            {
                sum+=val.get();
            }
            context.write(key, new IntWritable(sum));
        }

    }

新的API中更广泛地使用了context对象，并使用MapContext进行MapReduce间的通信，MapContext同时充当OutputCollector和Reporter的角色。
WordCount继承了Configured类
通过反编译工具得到Configured类结构：
这里写图片描述
getConf方法在后面的Job类会使用到。
WordCount实现了Tool类
通过反编译工具得到Tool接口结构:

可以看出WordCount类需要实现run方法:

@Override
    public int run(String[] args) throws Exception {
        Job job=new Job(getConf());
        job.setJarByClass(WordCount.class);
        job.setJobName("wordcount");

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        job.setMapperClass(Map.class);
        job.setReducerClass(Reduce.class);

        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job,new Path( args[1]));

        boolean success=job.waitForCompletion(true);
        return success?0:1;
    }

最后就是主函数了。
Main函数：

public static void main(String[] args) throws Exception {
        int ret=ToolRunner.run(new WordCount(), args);
        System.exit(ret);

    }

输入输出和前面的完全一样，得到的控制台的结果如下所示：

2016-10-07 01:38:40,640 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.http.authentication.token.validity
2016-10-07 01:38:40,640 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.group.mapping.ldap.search.filter.group
2016-10-07 01:38:40,640 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.scheduler.monitor.policies
2016-10-07 01:38:40,641 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.zk.connection.timeout.ms
2016-10-07 01:38:40,641 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.crypto.cipher.suite
2016-10-07 01:38:40,641 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.http-authentication.simple.anonymous.allowed
2016-10-07 01:38:40,641 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.fs.state-store.retry-policy-spec
2016-10-07 01:38:40,641 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.profile.params
2016-10-07 01:38:40,642 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.admin.acl
2016-10-07 01:38:40,642 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.local-cache.max-files-per-directory
2016-10-07 01:38:40,642 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.client.failover-retries-on-socket-timeouts
2016-10-07 01:38:40,642 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.containerlauncher.threadpool-initial-size
2016-10-07 01:38:40,643 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.nodemanagers.heartbeat-interval-ms
2016-10-07 01:38:40,643 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.client.failover-proxy-provider
2016-10-07 01:38:40,643 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.fs.state-store.num-retries
2016-10-07 01:38:40,644 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.sort.spill.percent
2016-10-07 01:38:40,644 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for file.stream-buffer-size
2016-10-07 01:38:40,644 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.connection.maxidletime
2016-10-07 01:38:40,644 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.combine.progress.records
2016-10-07 01:38:40,645 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.output.key.class
2016-10-07 01:38:40,645 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.persist.jobstatus.hours
2016-10-07 01:38:40,645 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.address
2016-10-07 01:38:40,645 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.ha.automatic-failover.embedded
2016-10-07 01:38:40,646 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.recovery.store.fs.uri
2016-10-07 01:38:40,646 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.zk-state-store.parent-path
2016-10-07 01:38:40,646 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.job.task.listener.thread-count
2016-10-07 01:38:40,646 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.running.map.limit
2016-10-07 01:38:40,647 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.zookeeper.parent-znode
2016-10-07 01:38:40,647 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.container-executor.class
2016-10-07 01:38:40,647 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.skip.checksum.errors
2016-10-07 01:38:40,647 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.user.name
2016-10-07 01:38:40,648 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.scheduler.client.thread-count
2016-10-07 01:38:40,648 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.recovery.dir
2016-10-07 01:38:40,648 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.emit-timeline-data
2016-10-07 01:38:40,648 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.http.authentication.kerberos.principal
2016-10-07 01:38:40,649 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.log.level
2016-10-07 01:38:40,649 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern
2016-10-07 01:38:40,649 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3.maxRetries
2016-10-07 01:38:40,649 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.resourcemanager.minimum.version
2016-10-07 01:38:40,650 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.kerberos.kinit.command
2016-10-07 01:38:40,650 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.log-aggregation.retain-check-interval-seconds
2016-10-07 01:38:40,650 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.process-kill-wait.ms
2016-10-07 01:38:40,650 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.cgroups.mount
2016-10-07 01:38:40,651 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.working.dir
2016-10-07 01:38:40,651 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.fast.buffer.size
2016-10-07 01:38:40,651 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.handler.count
2016-10-07 01:38:40,651 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.admin.address
2016-10-07 01:38:40,652 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.client-am.ipc.max-retries
2016-10-07 01:38:40,653 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.nm.uploader.replication.factor
2016-10-07 01:38:40,653 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.util.hash.type
2016-10-07 01:38:40,653 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.seqfile.lazydecompress
2016-10-07 01:38:40,654 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.disk-health-checker.min-healthy-disks
2016-10-07 01:38:40,654 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.client.max-cached-nodemanagers-proxies
2016-10-07 01:38:40,654 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.maxtaskfailures.per.tracker
2016-10-07 01:38:40,654 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.healthchecker.script.timeout
2016-10-07 01:38:40,655 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.fs.state-store.retry-interval-ms
2016-10-07 01:38:40,655 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.group.mapping.ldap.search.attr.group.name
2016-10-07 01:38:40,655 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.df.interval
2016-10-07 01:38:40,656 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.crypto.buffer.size
2016-10-07 01:38:40,656 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.reduce.class
2016-10-07 01:38:40,656 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.map.class
2016-10-07 01:38:40,656 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.reduce.shuffle.consumer.plugin.class
2016-10-07 01:38:40,657 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.address
2016-10-07 01:38:40,657 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.tasks.sleeptimebeforesigkill
2016-10-07 01:38:40,657 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.acl-view-job
2016-10-07 01:38:40,657 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.job.committer.cancel-timeout
2016-10-07 01:38:40,658 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.connection-keep-alive.enable
2016-10-07 01:38:40,658 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.minicluster.fixed.ports
2016-10-07 01:38:40,658 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.admin.acl
2016-10-07 01:38:40,659 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.scheduler.address
2016-10-07 01:38:40,659 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.http.address
2016-10-07 01:38:40,659 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.group.mapping.ldap.ssl
2016-10-07 01:38:40,659 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.memory.mb
2016-10-07 01:38:40,660 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.client.best-effort
2016-10-07 01:38:40,660 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.merge.progress.records
2016-10-07 01:38:40,660 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size
2016-10-07 01:38:40,660 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.threads.keepalivetime
2016-10-07 01:38:40,661 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.recovery.enabled
2016-10-07 01:38:40,661 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for net.topology.script.number.args
2016-10-07 01:38:40,661 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.local.clientfactory.class.name
2016-10-07 01:38:40,661 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.threads.core
2016-10-07 01:38:40,662 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.group.mapping.ldap.directory.search.timeout
2016-10-07 01:38:40,662 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.native.lib.available
2016-10-07 01:38:40,662 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.disk-health-checker.interval-ms
2016-10-07 01:38:40,662 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs
2016-10-07 01:38:40,663 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.connection.ssl.enabled
2016-10-07 01:38:40,663 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.webapp.address
2016-10-07 01:38:40,663 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.resource-tracker.client.thread-count
2016-10-07 01:38:40,663 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.client.failover-retries
2016-10-07 01:38:40,664 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.jaas.context
2016-10-07 01:38:40,664 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.store.class
2016-10-07 01:38:40,664 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.zk-timeout-ms
2016-10-07 01:38:40,664 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.aux-services.mapreduce_shuffle.class
2016-10-07 01:38:40,665 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.health-monitor.rpc-timeout.ms
2016-10-07 01:38:40,665 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.markreset.buffer.percent
2016-10-07 01:38:40,665 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size
2016-10-07 01:38:40,665 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.client.thread-count
2016-10-07 01:38:40,666 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.health-checker.script.timeout-ms
2016-10-07 01:38:40,666 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for file.bytes-per-checksum
2016-10-07 01:38:40,666 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.map.index.skip
2016-10-07 01:38:40,666 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.client-server.address
2016-10-07 01:38:40,667 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.client.retry-interval-ms
2016-10-07 01:38:40,667 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.timeout
2016-10-07 01:38:40,667 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.cpu.vcores
2016-10-07 01:38:40,668 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ftp.blocksize
2016-10-07 01:38:40,668 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.container-manager.thread-count
2016-10-07 01:38:40,668 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.server.listen.queue.size
2016-10-07 01:38:40,668 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.hostname.verifier
2016-10-07 01:38:40,669 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.ha.automatic-failover.enabled
2016-10-07 01:38:40,669 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.dns.interface
2016-10-07 01:38:40,669 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.group.mapping.ldap.search.attr.member
2016-10-07 01:38:40,669 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.outofband.heartbeat
2016-10-07 01:38:40,670 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.impl
2016-10-07 01:38:40,670 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.userlog.retain.hours
2016-10-07 01:38:40,670 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.system.acls
2016-10-07 01:38:40,671 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.name
2016-10-07 01:38:40,671 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.resource.memory-mb
2016-10-07 01:38:40,671 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.shuffle.log.separate
2016-10-07 01:38:40,671 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds
2016-10-07 01:38:40,672 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.keystores.factory.class
2016-10-07 01:38:40,672 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.zk.retry.ceiling.ms
2016-10-07 01:38:40,672 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.http.policy
2016-10-07 01:38:40,672 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.windows-container.memory-limit.enabled
2016-10-07 01:38:40,673 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.root-dir
2016-10-07 01:38:40,673 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.AbstractFileSystem.har.impl
2016-10-07 01:38:40,673 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.random.device.file.path
2016-10-07 01:38:40,673 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.maxattempts
2016-10-07 01:38:40,674 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.webapp.address
2016-10-07 01:38:40,674 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.require.client.cert
2016-10-07 01:38:40,674 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ftp.client-write-packet-size
2016-10-07 01:38:40,675 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.cleaner.enable
2016-10-07 01:38:40,675 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.du.interval
2016-10-07 01:38:40,675 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.retry-delay.max.ms
2016-10-07 01:38:40,675 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.profile.reduces
2016-10-07 01:38:40,676 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.health-monitor.connect-retry-interval.ms
2016-10-07 01:38:40,676 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.jobhistory.task.numberprogresssplits
2016-10-07 01:38:40,676 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.ftp.host.port
2016-10-07 01:38:40,677 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.speculative
2016-10-07 01:38:40,677 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.client.submit.file.replication
2016-10-07 01:38:40,677 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3native.blocksize
2016-10-07 01:38:40,677 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.ubertask.maxmaps
2016-10-07 01:38:40,678 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.cluster.acls.enabled
2016-10-07 01:38:40,678 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.app-checker.class
2016-10-07 01:38:40,678 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.uid.cache.secs
2016-10-07 01:38:40,678 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.localizer.fetch.thread-count
2016-10-07 01:38:40,679 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.trash.checkpoint.interval
2016-10-07 01:38:40,679 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for map.sort.class
2016-10-07 01:38:40,679 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapred.queue.default.acl-administer-jobs
2016-10-07 01:38:40,680 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.staging-dir
2016-10-07 01:38:40,680 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.connect.timeout
2016-10-07 01:38:40,680 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.AbstractFileSystem.file.impl
2016-10-07 01:38:40,680 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.env-whitelist
2016-10-07 01:38:40,681 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.zk.retry.interval.ms
2016-10-07 01:38:40,681 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage
2016-10-07 01:38:40,681 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.keytab
2016-10-07 01:38:40,681 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.reduces
2016-10-07 01:38:40,682 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.recovery.store.class
2016-10-07 01:38:40,682 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.complete.cancel.delegation.tokens
2016-10-07 01:38:40,682 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.group.mapping.ldap.search.filter.user
2016-10-07 01:38:40,682 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.sleep-delay-before-sigkill.ms
2016-10-07 01:38:40,683 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.joblist.cache.size
2016-10-07 01:38:40,683 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.outputformat.class
2016-10-07 01:38:40,683 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.recovery.store.leveldb.path
2016-10-07 01:38:40,683 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.healthchecker.interval
2016-10-07 01:38:40,684 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.heartbeats.in.second
2016-10-07 01:38:40,684 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.persist.jobstatus.dir
2016-10-07 01:38:40,684 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.rpc.protection
2016-10-07 01:38:40,685 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.container.log.backups
2016-10-07 01:38:40,685 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ftp.stream-buffer-size
2016-10-07 01:38:40,685 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.address
2016-10-07 01:38:40,685 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.recovery.enabled
2016-10-07 01:38:40,686 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.input.fileinputformat.numinputfiles
2016-10-07 01:38:40,686 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.admin.client.thread-count
2016-10-07 01:38:40,686 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.groups.negative-cache.secs
2016-10-07 01:38:40,686 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for file.client-write-packet-size
2016-10-07 01:38:40,687 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.http.authentication.simple.anonymous.allowed
2016-10-07 01:38:40,687 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.leveldb-timeline-store.path
2016-10-07 01:38:40,687 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.proxy-user-privileges.enabled
2016-10-07 01:38:40,687 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.log.retain-seconds
2016-10-07 01:38:40,688 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms
2016-10-07 01:38:40,688 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.zk.session.timeout.ms
2016-10-07 01:38:40,688 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.failover-controller.cli-check.rpc-timeout.ms
2016-10-07 01:38:40,688 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.instrumentation
2016-10-07 01:38:40,689 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.cgroups.hierarchy
2016-10-07 01:38:40,689 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.groups.cache.warn.after.ms
2016-10-07 01:38:40,689 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.fetch.retry.timeout-ms
2016-10-07 01:38:40,689 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.client.thread-count
2016-10-07 01:38:40,690 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.mapfile.bloom.size
2016-10-07 01:38:40,690 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.work-preserving-recovery.enabled
2016-10-07 01:38:40,690 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.docker-container-executor.exec-name
2016-10-07 01:38:40,690 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.zk-num-retries
2016-10-07 01:38:40,691 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for dfs.ha.fencing.ssh.connect-timeout
2016-10-07 01:38:40,691 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.zk.root
2016-10-07 01:38:40,691 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3.bytes-per-checksum
2016-10-07 01:38:40,691 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.container.log.limit.kb
2016-10-07 01:38:40,692 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.automatic.close
2016-10-07 01:38:40,692 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.admin.thread-count
2016-10-07 01:38:40,692 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.trash.interval
2016-10-07 01:38:40,692 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.ttl-ms
2016-10-07 01:38:40,693 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.authentication
2016-10-07 01:38:40,693 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.defaultFS
2016-10-07 01:38:40,693 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.connect.max.retries
2016-10-07 01:38:40,694 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.server.conf
2016-10-07 01:38:40,694 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.delayed.delegation-token.removal-interval-ms
2016-10-07 01:38:40,694 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.taskscheduler
2016-10-07 01:38:40,695 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.store-class
2016-10-07 01:38:40,695 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.output.fileoutputformat.compress
2016-10-07 01:38:40,695 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.am.liveness-monitor.expiry-interval-ms
2016-10-07 01:38:40,695 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for net.topology.node.switch.mapping.impl
2016-10-07 01:38:40,696 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.swift.impl
2016-10-07 01:38:40,696 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.fast.upload
2016-10-07 01:38:40,696 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.fetch.retry.enabled
2016-10-07 01:38:40,697 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.max.split.locations
2016-10-07 01:38:40,697 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.counters.max
2016-10-07 01:38:40,697 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.address
2016-10-07 01:38:40,697 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.windows-container.cpu-limit.enabled
2016-10-07 01:38:40,698 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.map.index.interval
2016-10-07 01:38:40,698 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.shuffle.log.limit.kb
2016-10-07 01:38:40,698 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapred.child.java.opts
2016-10-07 01:38:40,698 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.local.dir.minspacestart
2016-10-07 01:38:40,699 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.client.progressmonitor.pollinterval
2016-10-07 01:38:40,699 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.profile.map.params
2016-10-07 01:38:40,699 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.queuename
2016-10-07 01:38:40,700 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.tasktracker.maxblacklists
2016-10-07 01:38:40,700 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.connection.establish.timeout
2016-10-07 01:38:40,700 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.zk.quorum
2016-10-07 01:38:40,700 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts
2016-10-07 01:38:40,701 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.localizer.address
2016-10-07 01:38:40,701 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.mapfile.bloom.error.rate
2016-10-07 01:38:40,701 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.delete.thread-count
2016-10-07 01:38:40,701 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.split.metainfo.maxsize
2016-10-07 01:38:40,702 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.scheduler.maximum-allocation-vcores
2016-10-07 01:38:40,702 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapred.mapper.new-api
2016-10-07 01:38:40,702 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.dir
2016-10-07 01:38:40,702 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.resource.mb
2016-10-07 01:38:40,703 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.reducer.preempt.delay.sec
2016-10-07 01:38:40,703 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.output.compress.codec
2016-10-07 01:38:40,703 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb
2016-10-07 01:38:40,738 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.connection.maximum
2016-10-07 01:38:40,738 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.log.level
2016-10-07 01:38:40,739 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.seqfile.compress.blocksize
2016-10-07 01:38:40,739 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.taskcontroller
2016-10-07 01:38:40,739 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.groups.cache.secs
2016-10-07 01:38:40,740 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.client-server.thread-count
2016-10-07 01:38:40,740 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.input.lineinputformat.linespermap
2016-10-07 01:38:40,740 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.end-notification.max.attempts
2016-10-07 01:38:40,740 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user
2016-10-07 01:38:40,740 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.uploader.server.address
2016-10-07 01:38:40,741 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.webapp.address
2016-10-07 01:38:40,741 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.recovery.enable
2016-10-07 01:38:40,741 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.submithostname
2016-10-07 01:38:40,741 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.expire.trackers.interval
2016-10-07 01:38:40,742 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.webapp.address
2016-10-07 01:38:40,742 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.nm.uploader.thread-count
2016-10-07 01:38:40,742 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.kms.client.encrypted.key.cache.num.refill.threads
2016-10-07 01:38:40,742 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.health-checker.interval-ms
2016-10-07 01:38:40,743 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.loadedjobs.cache.size
2016-10-07 01:38:40,743 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.map.output.collector.class
2016-10-07 01:38:40,743 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.authorization
2016-10-07 01:38:40,743 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.am.max-attempts
2016-10-07 01:38:40,744 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.ftp.host
2016-10-07 01:38:40,744 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.attempts.maximum
2016-10-07 01:38:40,744 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms
2016-10-07 01:38:40,744 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.ifile.readahead
2016-10-07 01:38:40,745 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.scheduler.monitor.enable
2016-10-07 01:38:40,745 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.zk-retry-interval-ms
2016-10-07 01:38:40,745 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.zookeeper.session-timeout.ms
2016-10-07 01:38:40,745 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.taskmemorymanager.monitoringinterval
2016-10-07 01:38:40,746 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.parallelcopies
2016-10-07 01:38:40,746 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.output.value.class
2016-10-07 01:38:40,746 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.skip.maxrecords
2016-10-07 01:38:40,746 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.webapp.https.address
2016-10-07 01:38:40,747 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.read.timeout
2016-10-07 01:38:40,747 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.remote-app-log-dir-suffix
2016-10-07 01:38:40,747 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.instrumentation
2016-10-07 01:38:40,752 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.output.fileoutputformat.compress.codec
2016-10-07 01:38:40,752 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.connect.retry.interval
2016-10-07 01:38:40,752 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.speculative
2016-10-07 01:38:40,757 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.keytab
2016-10-07 01:38:40,758 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.datestring.cache.size
2016-10-07 01:38:40,758 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for file.blocksize
2016-10-07 01:38:40,758 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.admin.address
2016-10-07 01:38:40,759 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.cpu.vcores
2016-10-07 01:38:40,759 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled
2016-10-07 01:38:40,759 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.resource-tracker.address
2016-10-07 01:38:40,759 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.configuration.provider-class
2016-10-07 01:38:40,760 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.local.dir.minspacekill
2016-10-07 01:38:40,760 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.staging.root.dir
2016-10-07 01:38:40,760 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.retiredjobs.cache.size
2016-10-07 01:38:40,761 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.rm.enabled
2016-10-07 01:38:40,761 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.connect.max.retries.on.timeouts
2016-10-07 01:38:40,761 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.crypto.codec.classes.aes.ctr.nopadding
2016-10-07 01:38:40,761 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.zookeeper.acl
2016-10-07 01:38:40,762 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.local-dirs
2016-10-07 01:38:40,762 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.app-submission.cross-platform
2016-10-07 01:38:40,762 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.connect.timeout
2016-10-07 01:38:40,762 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for rpc.metrics.quantile.enable
2016-10-07 01:38:40,763 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.recovery.enabled
2016-10-07 01:38:40,763 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.leveldb-state-store.path
2016-10-07 01:38:40,763 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.end-notification.retry.attempts
2016-10-07 01:38:40,763 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.system.dir
2016-10-07 01:38:40,764 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.admin-env
2016-10-07 01:38:40,764 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.log-aggregation.retain-seconds
2016-10-07 01:38:40,764 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.jobhistory.block.size
2016-10-07 01:38:40,764 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.indexcache.mb
2016-10-07 01:38:40,765 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.handler-thread-count
2016-10-07 01:38:40,765 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.hostname
2016-10-07 01:38:40,765 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for net.topology.impl
2016-10-07 01:38:40,765 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.multipart.purge.age
2016-10-07 01:38:40,766 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.threads.max
2016-10-07 01:38:40,766 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.java.secure.random.algorithm
2016-10-07 01:38:40,766 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.container-monitor.interval-ms
2016-10-07 01:38:40,766 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.multipart.threshold
2016-10-07 01:38:40,768 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.reduce.slowstart.completedmaps
2016-10-07 01:38:40,768 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapred.reducer.new-api
2016-10-07 01:38:40,768 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.instrumentation.requires.admin
2016-10-07 01:38:40,769 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.compression.codec.bzip2.library
2016-10-07 01:38:40,769 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.http.authentication.signature.secret.file
2016-10-07 01:38:40,769 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.registry.secure
2016-10-07 01:38:40,769 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.input.fileinputformat.inputdir
2016-10-07 01:38:40,770 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.localizer.cache.target-size-mb
2016-10-07 01:38:40,770 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.maxattempts
2016-10-07 01:38:40,771 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3native.replication
2016-10-07 01:38:40,771 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.skip.proc.count.autoincr
2016-10-07 01:38:40,771 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for file.replication
2016-10-07 01:38:40,771 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.cleaner.interval-ms
2016-10-07 01:38:40,772 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.idlethreshold
2016-10-07 01:38:40,772 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.tmp.dir
2016-10-07 01:38:40,772 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.store.class
2016-10-07 01:38:40,773 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.restart.recover
2016-10-07 01:38:40,773 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.address
2016-10-07 01:38:40,773 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.cluster.local.dir
2016-10-07 01:38:40,773 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.speculative.retry-after-no-speculate
2016-10-07 01:38:40,774 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.client.nodemanager-client-async.thread-pool-max-size
2016-10-07 01:38:40,774 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.state-store-class
2016-10-07 01:38:40,774 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.resource.cpu-vcores
2016-10-07 01:38:40,775 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.inputformat.class
2016-10-07 01:38:40,775 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.nested-level
2016-10-07 01:38:40,775 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3.buffer.dir
2016-10-07 01:38:40,775 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.remote-app-log-dir
2016-10-07 01:38:40,776 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.memory.mb
2016-10-07 01:38:40,776 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.client.job.max-retries
2016-10-07 01:38:40,776 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.work.around.non.threadsafe.getpwuid
2016-10-07 01:38:40,777 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.profile.reduce.params
2016-10-07 01:38:40,777 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.client.max-retries
2016-10-07 01:38:40,777 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.transfer.buffer.size
2016-10-07 01:38:40,777 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.persist.jobstatus.active
2016-10-07 01:38:40,778 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.log-dirs
2016-10-07 01:38:40,778 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs
2016-10-07 01:38:40,778 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.health-monitor.sleep-after-disconnect.ms
2016-10-07 01:38:40,778 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.fs.state-store.uri
2016-10-07 01:38:40,779 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.rpc.socket.factory.class.default
2016-10-07 01:38:40,779 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.keytab
2016-10-07 01:38:40,779 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.AbstractFileSystem.ftp.impl
2016-10-07 01:38:40,780 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.leveldb-state-store.path
2016-10-07 01:38:40,781 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.profile
2016-10-07 01:38:40,781 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.move.interval-ms
2016-10-07 01:38:40,781 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.max.total.tasks
2016-10-07 01:38:40,784 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.kms.client.encrypted.key.cache.size
2016-10-07 01:38:40,784 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.http.policy
2016-10-07 01:38:40,784 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.intermediate-done-dir
2016-10-07 01:38:40,785 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.hard-kill-timeout-ms
2016-10-07 01:38:40,785 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.skip.proc.count.autoincr
2016-10-07 01:38:40,785 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.AbstractFileSystem.viewfs.impl
2016-10-07 01:38:40,785 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.speculative.slowtaskthreshold
2016-10-07 01:38:40,785 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled
2016-10-07 01:38:40,786 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3native.stream-buffer-size
2016-10-07 01:38:40,786 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.delete.debug-delay-sec
2016-10-07 01:38:40,786 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3n.multipart.uploads.block.size
2016-10-07 01:38:40,786 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.ifile.readahead.bytes
2016-10-07 01:38:40,787 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.scheduler.maximum-allocation-mb
2016-10-07 01:38:40,787 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.client.fallback-to-simple-auth-allowed
2016-10-07 01:38:40,787 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.har.impl.disable.cache
2016-10-07 01:38:40,787 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.leveldb-timeline-store.read-cache-size
2016-10-07 01:38:40,788 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3native.bytes-per-checksum
2016-10-07 01:38:40,788 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.hostname
2016-10-07 01:38:40,788 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.committer.setup.cleanup.needed
2016-10-07 01:38:40,788 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms
2016-10-07 01:38:40,789 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.paging.maximum
2016-10-07 01:38:40,789 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.client.nodemanager-connect.retry-interval-ms
2016-10-07 01:38:40,789 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.log-aggregation.compression-type
2016-10-07 01:38:40,789 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.job.committer.commit-window
2016-10-07 01:38:40,789 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.http.authentication.type
2016-10-07 01:38:40,790 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.submithostaddress
2016-10-07 01:38:40,790 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.vmem-check-enabled
2016-10-07 01:38:40,790 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.jetty.logs.serve.aliases
2016-10-07 01:38:40,790 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.failover-controller.graceful-fence.rpc-timeout.ms
2016-10-07 01:38:40,791 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.input.buffer.percent
2016-10-07 01:38:40,791 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.running.reduce.limit
2016-10-07 01:38:40,791 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.io.sort.mb
2016-10-07 01:38:40,791 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.merge.inmem.threshold
2016-10-07 01:38:40,792 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.client.application-client-protocol.poll-interval-ms
2016-10-07 01:38:40,792 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.kms.client.authentication.retry-count
2016-10-07 01:38:40,792 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.connect.max-wait.ms
2016-10-07 01:38:40,793 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.enabled
2016-10-07 01:38:40,793 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users
2016-10-07 01:38:40,793 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.client.conf
2016-10-07 01:38:40,793 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.container.liveness-monitor.interval-ms
2016-10-07 01:38:40,794 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.vmem-pmem-ratio
2016-10-07 01:38:40,794 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.client.completion.pollinterval
2016-10-07 01:38:40,794 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.client.max-retries
2016-10-07 01:38:40,795 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.enabled
2016-10-07 01:38:40,795 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.AbstractFileSystem.hdfs.impl
2016-10-07 01:38:40,795 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.client.resolve.remote.symlinks
2016-10-07 01:38:40,795 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.reduce.tasks.maximum
2016-10-07 01:38:40,796 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.client.genericoptionsparser.used
2016-10-07 01:38:40,796 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.hostname
2016-10-07 01:38:40,796 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.input.buffer.percent
2016-10-07 01:38:40,796 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.cleaner.resource-sleep-ms
2016-10-07 01:38:40,797 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.multipart.purge
2016-10-07 01:38:40,797 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.command-opts
2016-10-07 01:38:40,797 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.cleaner.period-mins
2016-10-07 01:38:40,798 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.webapp.https.address
2016-10-07 01:38:40,798 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.ssl.file.buffer.size
2016-10-07 01:38:40,798 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.maxtasks.perjob
2016-10-07 01:38:40,798 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.userlog.limit.kb
2016-10-07 01:38:40,799 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.kms.client.encrypted.key.cache.low-watermark
2016-10-07 01:38:40,799 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.buffer.dir
2016-10-07 01:38:40,799 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3.client-write-packet-size
2016-10-07 01:38:40,799 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.user.group.static.mapping.overrides
2016-10-07 01:38:40,799 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.max.threads
2016-10-07 01:38:40,800 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.kms.client.encrypted.key.cache.expiry
2016-10-07 01:38:40,800 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.maps
2016-10-07 01:38:40,800 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.ssl.enabled.protocols
2016-10-07 01:38:40,800 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.map.output.compress
2016-10-07 01:38:40,801 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3.blocksize
2016-10-07 01:38:40,801 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.pmem-check-enabled
2016-10-07 01:38:40,801 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.scheduler.minimum-allocation-mb
2016-10-07 01:38:40,801 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.http-authentication.type
2016-10-07 01:38:40,801 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.max-age-ms
2016-10-07 01:38:40,802 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ftp.replication
2016-10-07 01:38:40,802 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.input.fileinputformat.split.minsize
2016-10-07 01:38:40,802 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3n.block.size
2016-10-07 01:38:40,802 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.token.tracking.ids.enabled
2016-10-07 01:38:40,803 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.shuffle.log.backups
2016-10-07 01:38:40,803 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.ipc.rpc.class
2016-10-07 01:38:40,803 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.localizer.cache.cleanup.interval-ms
2016-10-07 01:38:40,803 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.http.staticuser.user
2016-10-07 01:38:40,804 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.move.thread-count
2016-10-07 01:38:40,804 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.jvm.numtasks
2016-10-07 01:38:40,804 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.multipart.size
2016-10-07 01:38:40,805 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.profile.maps
2016-10-07 01:38:40,805 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.port
2016-10-07 01:38:40,805 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.nodemanager.minimum.version
2016-10-07 01:38:40,806 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.connection-keep-alive.timeout
2016-10-07 01:38:40,806 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.merge.percent
2016-10-07 01:38:40,806 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobtracker.http.address
2016-10-07 01:38:40,806 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.amlauncher.thread-count
2016-10-07 01:38:40,807 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.connect.retry-interval.ms
2016-10-07 01:38:40,807 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.skip.start.attempts
2016-10-07 01:38:40,807 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.scheduler.minimum-allocation-vcores
2016-10-07 01:38:40,807 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.task.io.sort.factor
2016-10-07 01:38:40,808 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.speculative.retry-after-speculate
2016-10-07 01:38:40,808 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for nfs.exports.allowed.hosts
2016-10-07 01:38:40,808 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for tfile.fs.input.buffer.size
2016-10-07 01:38:40,808 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ipc.server.max.connections
2016-10-07 01:38:40,809 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for tfile.io.chunk.size
2016-10-07 01:38:40,810 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3.block.size
2016-10-07 01:38:40,812 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3n.multipart.copy.block.size
2016-10-07 01:38:40,812 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.speculative.speculative-cap-total-tasks
2016-10-07 01:38:40,822 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.serializations
2016-10-07 01:38:40,823 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.max-completed-applications
2016-10-07 01:38:40,823 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.jobhistory.principal
2016-10-07 01:38:40,824 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.output.fileoutputformat.outputdir
2016-10-07 01:38:40,824 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.ha.automatic-failover.zk-base-path
2016-10-07 01:38:40,824 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.reduce.shuffle.fetch.retry.interval-ms
2016-10-07 01:38:40,825 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.end-notification.retry.interval
2016-10-07 01:38:40,825 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.speculative.minimum-allowed-tasks
2016-10-07 01:38:40,825 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3n.multipart.uploads.enabled
2016-10-07 01:38:40,825 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.seqfile.sorter.recordlimit
2016-10-07 01:38:40,826 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for s3native.client-write-packet-size
2016-10-07 01:38:40,826 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.security.group.mapping
2016-10-07 01:38:40,826 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.nodemanager-connect-retries
2016-10-07 01:38:40,826 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ftp.bytes-per-checksum
2016-10-07 01:38:40,827 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.store.in-memory.check-period-mins
2016-10-07 01:38:40,827 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3a.connection.timeout
2016-10-07 01:38:40,827 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.end-notification.max.retry.interval
2016-10-07 01:38:40,827 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.acl.enable
2016-10-07 01:38:40,828 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nm.liveness-monitor.expiry-interval-ms
2016-10-07 01:38:40,828 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.enabled
2016-10-07 01:38:40,828 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.input.fileinputformat.list-status.num-threads
2016-10-07 01:38:40,829 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.map.tasks.maximum
2016-10-07 01:38:40,829 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.ttl-enable
2016-10-07 01:38:40,829 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.linux-container-executor.resources-handler.class
2016-10-07 01:38:40,829 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.state-store.max-completed-applications
2016-10-07 01:38:40,829 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.classloader
2016-10-07 01:38:40,830 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size
2016-10-07 01:38:40,830 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.store.in-memory.initial-delay-mins
2016-10-07 01:38:40,831 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.hdfs-servers
2016-10-07 01:38:40,832 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.dns.nameserver
2016-10-07 01:38:40,832 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.ubertask.maxreduces
2016-10-07 01:38:40,833 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.shuffle.ssl.enabled
2016-10-07 01:38:40,833 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.log-aggregation-enable
2016-10-07 01:38:40,833 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.report.address
2016-10-07 01:38:40,834 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.tasktracker.http.threads
2016-10-07 01:38:40,834 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for tfile.fs.output.buffer.size
2016-10-07 01:38:40,834 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.permissions.umask-mode
2016-10-07 01:38:40,834 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.checksum.algo.impl
2016-10-07 01:38:40,835 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.am.max-attempts
2016-10-07 01:38:40,835 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.nodemanager.resource.percentage-physical-cpu-limit
2016-10-07 01:38:40,835 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.bytes.per.checksum
2016-10-07 01:38:40,835 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for ha.failover-controller.graceful-fence.connection.retries
2016-10-07 01:38:40,836 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.speculative.speculative-cap-running-tasks
2016-10-07 01:38:40,836 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.sharedcache.cleaner.initial-delay-mins
2016-10-07 01:38:40,836 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.app.mapreduce.am.resource.cpu-vcores
2016-10-07 01:38:40,836 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for hadoop.common.configuration.version
2016-10-07 01:38:40,837 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.ubertask.enable
2016-10-07 01:38:40,837 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.job.acl-modify-job
2016-10-07 01:38:40,837 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for io.seqfile.local.dir
2016-10-07 01:38:40,837 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for fs.s3.sleepTimeSeconds
2016-10-07 01:38:40,838 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for yarn.resourcemanager.system-metrics-publisher.enabled
2016-10-07 01:38:40,838 [org.apache.hadoop.conf.Configuration]-[DEBUG] Handling deprecation for mapreduce.client.output.filter
2016-10-07 01:38:41,060 [org.apache.hadoop.mapreduce.Job]-[INFO] The url to track the job: http://localhost:8080/
2016-10-07 01:38:41,062 [org.apache.hadoop.mapreduce.Job]-[INFO] Running job: job_local1365548564_0001
2016-10-07 01:38:41,064 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] OutputCommitter set in config null
2016-10-07 01:38:41,070 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-07 01:38:41,071 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-07 01:38:41,085 [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter]-[INFO] File Output Committer Algorithm version is 1
2016-10-07 01:38:41,090 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2016-10-07 01:38:41,165 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[DEBUG] Starting mapper thread pool executor.
2016-10-07 01:38:41,166 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[DEBUG] Max local threads: 1
2016-10-07 01:38:41,166 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[DEBUG] Map tasks to process: 1
2016-10-07 01:38:41,171 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] Waiting for map tasks
2016-10-07 01:38:41,172 [org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable]-[INFO] Starting task: attempt_local1365548564_0001_m_000000_0
2016-10-07 01:38:41,193 [org.apache.hadoop.mapred.SortedRanges$SkipRangeIterator]-[DEBUG] currentIndex 0   0:0
2016-10-07 01:38:41,224 [org.apache.hadoop.mapred.LocalJobRunner]-[DEBUG] mapreduce.cluster.local.dir for child : /tmp/hadoop-dtw/mapred/local/localRunner//dtw/jobcache/job_local1365548564_0001/attempt_local1365548564_0001_m_000000_0
2016-10-07 01:38:41,230 [org.apache.hadoop.mapred.Task]-[DEBUG] using new api for output committer
2016-10-07 01:38:41,231 [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter]-[INFO] File Output Committer Algorithm version is 1
2016-10-07 01:38:41,284 [org.apache.hadoop.mapred.Task]-[INFO]  Using ResourceCalculatorProcessTree : [ ]
2016-10-07 01:38:41,299 [org.apache.hadoop.mapred.MapTask]-[INFO] Processing split: file:/home/dtw/hadoop-2.7.1/input/file01:0+86
2016-10-07 01:38:41,330 [org.apache.hadoop.mapred.MapTask]-[DEBUG] Trying map output collector class: org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-10-07 01:38:41,399 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] (EQUATOR) 0 kvi 26214396(104857584)
2016-10-07 01:38:41,400 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] mapreduce.task.io.sort.mb: 100
2016-10-07 01:38:41,400 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] soft limit at 83886080
2016-10-07 01:38:41,401 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] bufstart = 0; bufvoid = 104857600
2016-10-07 01:38:41,401 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] kvstart = 26214396; length = 6553600
2016-10-07 01:38:41,410 [org.apache.hadoop.mapred.MapTask]-[INFO] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-10-07 01:38:41,429 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] 
2016-10-07 01:38:41,430 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] Starting flush of map output
2016-10-07 01:38:41,430 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] Spilling map output
2016-10-07 01:38:41,430 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] bufstart = 0; bufend = 150; bufvoid = 104857600
2016-10-07 01:38:41,430 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] kvstart = 26214396(104857584); kvend = 26214336(104857344); length = 61/6553600
2016-10-07 01:38:41,452 [org.apache.hadoop.mapred.MapTask$MapOutputBuffer]-[INFO] Finished spill 0
2016-10-07 01:38:41,459 [org.apache.hadoop.mapred.Task]-[INFO] Task:attempt_local1365548564_0001_m_000000_0 is done. And is in the process of committing
2016-10-07 01:38:41,475 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] map
2016-10-07 01:38:41,476 [org.apache.hadoop.mapred.Task]-[INFO] Task 'attempt_local1365548564_0001_m_000000_0' done.
2016-10-07 01:38:41,476 [org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable]-[INFO] Finishing task: attempt_local1365548564_0001_m_000000_0
2016-10-07 01:38:41,477 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] map task executor complete.
2016-10-07 01:38:41,483 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[DEBUG] Starting reduce thread pool executor.
2016-10-07 01:38:41,484 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[DEBUG] Max local threads: 1
2016-10-07 01:38:41,484 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[DEBUG] Reduce tasks to process: 1
2016-10-07 01:38:41,485 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] Waiting for reduce tasks
2016-10-07 01:38:41,492 [org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable]-[INFO] Starting task: attempt_local1365548564_0001_r_000000_0
2016-10-07 01:38:41,498 [org.apache.hadoop.mapred.SortedRanges$SkipRangeIterator]-[DEBUG] currentIndex 0   0:0
2016-10-07 01:38:41,533 [org.apache.hadoop.mapred.LocalJobRunner]-[DEBUG] mapreduce.cluster.local.dir for child : /tmp/hadoop-dtw/mapred/local/localRunner//dtw/jobcache/job_local1365548564_0001/attempt_local1365548564_0001_r_000000_0
2016-10-07 01:38:41,536 [org.apache.hadoop.mapred.Task]-[DEBUG] using new api for output committer
2016-10-07 01:38:41,536 [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter]-[INFO] File Output Committer Algorithm version is 1
2016-10-07 01:38:41,538 [org.apache.hadoop.mapred.Task]-[INFO]  Using ResourceCalculatorProcessTree : [ ]
2016-10-07 01:38:41,544 [org.apache.hadoop.mapred.ReduceTask]-[INFO] Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@68d2f9f
2016-10-07 01:38:41,572 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[INFO] MergerManager: memoryLimit=1109078784, maxSingleShuffleLimit=277269696, mergeThreshold=731992000, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2016-10-07 01:38:41,577 [org.apache.hadoop.mapreduce.task.reduce.EventFetcher]-[INFO] attempt_local1365548564_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2016-10-07 01:38:41,583 [org.apache.hadoop.mapreduce.task.reduce.EventFetcher]-[DEBUG] Got 0 map completion events from 0
2016-10-07 01:38:41,583 [org.apache.hadoop.mapreduce.task.reduce.EventFetcher]-[DEBUG] GetMapEventsThread about to sleep for 1000
2016-10-07 01:38:41,593 [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher]-[DEBUG] LocalFetcher 1 going to fetch: attempt_local1365548564_0001_m_000000_0
2016-10-07 01:38:41,632 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[DEBUG] attempt_local1365548564_0001_m_000000_0: Proceeding with shuffle since usedMemory (0) is lesser than memoryLimit (1109078784).CommitMemory is (0)
2016-10-07 01:38:41,636 [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher]-[INFO] localfetcher#1 about to shuffle output of map attempt_local1365548564_0001_m_000000_0 decomp: 184 len: 188 to MEMORY
2016-10-07 01:38:41,644 [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput]-[INFO] Read 184 bytes from map-output for attempt_local1365548564_0001_m_000000_0
2016-10-07 01:38:41,646 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[INFO] closeInMemoryFile -> map-output of size: 184, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->184
2016-10-07 01:38:41,648 [org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl]-[DEBUG] map attempt_local1365548564_0001_m_000000_0 done 1 / 1 copied.
2016-10-07 01:38:41,649 [org.apache.hadoop.mapreduce.task.reduce.EventFetcher]-[INFO] EventFetcher is interrupted.. Returning
2016-10-07 01:38:41,652 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] 1 / 1 copied.
2016-10-07 01:38:41,653 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[INFO] finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-10-07 01:38:41,668 [org.apache.hadoop.mapred.Merger$MergeQueue]-[INFO] Merging 1 sorted segments
2016-10-07 01:38:41,668 [org.apache.hadoop.mapred.Merger$MergeQueue]-[INFO] Down to the last merge-pass, with 1 segments left of total size: 178 bytes
2016-10-07 01:38:41,672 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[INFO] Merged 1 segments, 184 bytes to disk to satisfy reduce memory limit
2016-10-07 01:38:41,673 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[DEBUG] Disk file: /tmp/hadoop-dtw/mapred/local/localRunner/dtw/jobcache/job_local1365548564_0001/attempt_local1365548564_0001_r_000000_0/output/map_0.out.merged Length is 188
2016-10-07 01:38:41,673 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[INFO] Merging 1 files, 188 bytes from disk
2016-10-07 01:38:41,675 [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl]-[INFO] Merging 0 segments, 0 bytes from memory into reduce
2016-10-07 01:38:41,675 [org.apache.hadoop.mapred.Merger$MergeQueue]-[INFO] Merging 1 sorted segments
2016-10-07 01:38:41,676 [org.apache.hadoop.mapred.Merger$MergeQueue]-[INFO] Down to the last merge-pass, with 1 segments left of total size: 178 bytes
2016-10-07 01:38:41,678 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] 1 / 1 copied.
2016-10-07 01:38:41,716 [org.apache.hadoop.conf.Configuration]-[INFO] mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2016-10-07 01:38:41,732 [org.apache.hadoop.mapred.Task]-[INFO] Task:attempt_local1365548564_0001_r_000000_0 is done. And is in the process of committing
2016-10-07 01:38:41,735 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] 1 / 1 copied.
2016-10-07 01:38:41,736 [org.apache.hadoop.mapred.Task]-[INFO] Task attempt_local1365548564_0001_r_000000_0 is allowed to commit now
2016-10-07 01:38:41,739 [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter]-[INFO] Saved output of task 'attempt_local1365548564_0001_r_000000_0' to file:/home/dtw/hadoop-2.7.1/output/_temporary/0/task_local1365548564_0001_r_000000
2016-10-07 01:38:41,741 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] reduce > reduce
2016-10-07 01:38:41,741 [org.apache.hadoop.mapred.Task]-[INFO] Task 'attempt_local1365548564_0001_r_000000_0' done.
2016-10-07 01:38:41,742 [org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable]-[INFO] Finishing task: attempt_local1365548564_0001_r_000000_0
2016-10-07 01:38:41,742 [org.apache.hadoop.mapred.LocalJobRunner$Job]-[INFO] reduce task executor complete.
2016-10-07 01:38:41,751 [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter]-[DEBUG] Merging data from DeprecatedRawLocalFileStatus{path=file:/home/dtw/hadoop-2.7.1/output/_temporary/0/task_local1365548564_0001_r_000000; isDirectory=true; modification_time=1475775521000; access_time=0; owner=; group=; permission=rwxrwxrwx; isSymlink=false} to file:/home/dtw/hadoop-2.7.1/output
2016-10-07 01:38:41,752 [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter]-[DEBUG] Merging data from DeprecatedRawLocalFileStatus{path=file:/home/dtw/hadoop-2.7.1/output/_temporary/0/task_local1365548564_0001_r_000000/part-r-00000; isDirectory=false; length=49; replication=1; blocksize=33554432; modification_time=1475775521000; access_time=0; owner=; group=; permission=rw-rw-rw-; isSymlink=false} to file:/home/dtw/hadoop-2.7.1/output/part-r-00000
2016-10-07 01:38:41,773 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:331)
2016-10-07 01:38:42,072 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-07 01:38:42,073 [org.apache.hadoop.mapreduce.Job]-[INFO] Job job_local1365548564_0001 running in uber mode : false
2016-10-07 01:38:42,077 [org.apache.hadoop.mapreduce.Job]-[INFO]  map 100% reduce 100%
2016-10-07 01:38:42,078 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:670)
2016-10-07 01:38:42,079 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-07 01:38:42,079 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-07 01:38:42,080 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:670)
2016-10-07 01:38:42,081 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-07 01:38:42,081 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)
2016-10-07 01:38:42,081 [org.apache.hadoop.mapreduce.Job]-[INFO] Job job_local1365548564_0001 completed successfully
2016-10-07 01:38:42,083 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.getCounters(Job.java:758)
2016-10-07 01:38:42,119 [org.apache.hadoop.mapreduce.Job]-[INFO] Counters: 30
    File System Counters
        FILE: Number of bytes read=898
        FILE: Number of bytes written=409991
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
    Map-Reduce Framework
        Map input records=5
        Map output records=16
        Map output bytes=150
        Map output materialized bytes=188
        Input split bytes=105
        Combine input records=0
        Combine output records=0
        Reduce input groups=7
        Reduce shuffle bytes=188
        Reduce input records=16
        Reduce output records=7
        Spilled Records=32
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=26
        Total committed heap usage (bytes)=453509120
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=86
    File Output Format Counters 
        Bytes Written=61
2016-10-07 01:38:42,120 [org.apache.hadoop.security.UserGroupInformation]-[DEBUG] PrivilegedAction as:dtw (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)

output输出内容：
part-r-00000文件中内容：

Bye 3
Hadoop  4
Hello   3
World   2
good    1
me  1
yes 2

2.命令行中运行MapReduce程序
读者可以在Eclipse里运行MapReduce程序也可以在命令行中运行MapReduce程序，但是在实际应用中，还是推荐到命令行中运行程序。和前面一样，首先安装Hadoop,然后输入变异打包生成JAR程序，如果所示安装路径是～/hadoop-2.7.1。
我们可以直接使用MyEclipse编译得到.class类具体时在WordCount文件的bin目录下,如图所示：
这里写图片描述
如果将.class类打包程JAR包，网上有许多教程这里就不赘述了，打包成wordcount.jar.
接下来就是上传输入文件（输入文件是file01,file02,存放在～/hadoop-2.7.1/input）：
上面是本地文件，需要上传到hdfs文件中，本例上传到hdfs://localhost:9000/user/hdfs/input文件中
首先在hdfs上新建hdfs文件
这里写图片描述
然后在hdfs上新建input文件

最后将本地的input的两个文件上传到hdfs上

下面通过这个命令看一下是否上传成功

接下来就是运行生成的JAR文件，为了叙述方便先将生成的JAR文件放入hadoop的安装文件夹中，然后运行如下命令。
这里写图片描述
下面就是执行完得到的结果：
shell中的内容：

16/10/05 01:02:35 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
16/10/05 01:02:35 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
16/10/05 01:02:36 INFO input.FileInputFormat: Total input paths to process : 2
16/10/05 01:02:36 INFO mapreduce.JobSubmitter: number of splits:2
16/10/05 01:02:36 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local844278475_0001
16/10/05 01:02:37 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
16/10/05 01:02:37 INFO mapreduce.Job: Running job: job_local844278475_0001
16/10/05 01:02:37 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/10/05 01:02:37 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
16/10/05 01:02:37 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/10/05 01:02:37 INFO mapred.LocalJobRunner: Waiting for map tasks
16/10/05 01:02:37 INFO mapred.LocalJobRunner: Starting task: attempt_local844278475_0001_m_000000_0
16/10/05 01:02:37 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
16/10/05 01:02:37 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/10/05 01:02:37 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hdfs/input/file02:0+87
16/10/05 01:02:37 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/10/05 01:02:37 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/10/05 01:02:37 INFO mapred.MapTask: soft limit at 83886080
16/10/05 01:02:37 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/10/05 01:02:37 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/10/05 01:02:37 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/10/05 01:02:37 INFO mapred.LocalJobRunner: 
16/10/05 01:02:37 INFO mapred.MapTask: Starting flush of map output
16/10/05 01:02:37 INFO mapred.MapTask: Spilling map output
16/10/05 01:02:37 INFO mapred.MapTask: bufstart = 0; bufend = 151; bufvoid = 104857600
16/10/05 01:02:37 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214336(104857344); length = 61/6553600
16/10/05 01:02:37 INFO mapred.MapTask: Finished spill 0
16/10/05 01:02:37 INFO mapred.Task: Task:attempt_local844278475_0001_m_000000_0 is done. And is in the process of committing
16/10/05 01:02:37 INFO mapred.LocalJobRunner: map
16/10/05 01:02:37 INFO mapred.Task: Task 'attempt_local844278475_0001_m_000000_0' done.
16/10/05 01:02:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local844278475_0001_m_000000_0
16/10/05 01:02:37 INFO mapred.LocalJobRunner: Starting task: attempt_local844278475_0001_m_000001_0
16/10/05 01:02:37 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
16/10/05 01:02:37 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/10/05 01:02:37 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hdfs/input/file01:0+86
16/10/05 01:02:37 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/10/05 01:02:37 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/10/05 01:02:37 INFO mapred.MapTask: soft limit at 83886080
16/10/05 01:02:37 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/10/05 01:02:37 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/10/05 01:02:37 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/10/05 01:02:37 INFO mapred.LocalJobRunner: 
16/10/05 01:02:37 INFO mapred.MapTask: Starting flush of map output
16/10/05 01:02:37 INFO mapred.MapTask: Spilling map output
16/10/05 01:02:37 INFO mapred.MapTask: bufstart = 0; bufend = 150; bufvoid = 104857600
16/10/05 01:02:37 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214336(104857344); length = 61/6553600
16/10/05 01:02:37 INFO mapred.MapTask: Finished spill 0
16/10/05 01:02:37 INFO mapred.Task: Task:attempt_local844278475_0001_m_000001_0 is done. And is in the process of committing
16/10/05 01:02:37 INFO mapred.LocalJobRunner: map
16/10/05 01:02:37 INFO mapred.Task: Task 'attempt_local844278475_0001_m_000001_0' done.
16/10/05 01:02:38 INFO mapred.LocalJobRunner: Finishing task: attempt_local844278475_0001_m_000001_0
16/10/05 01:02:38 INFO mapred.LocalJobRunner: map task executor complete.
16/10/05 01:02:38 INFO mapred.LocalJobRunner: Waiting for reduce tasks
16/10/05 01:02:38 INFO mapred.LocalJobRunner: Starting task: attempt_local844278475_0001_r_000000_0
16/10/05 01:02:38 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
16/10/05 01:02:38 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/10/05 01:02:38 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@72650926
16/10/05 01:02:38 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=333971456, maxSingleShuffleLimit=83492864, mergeThreshold=220421168, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/10/05 01:02:38 INFO reduce.EventFetcher: attempt_local844278475_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
16/10/05 01:02:38 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local844278475_0001_m_000001_0 decomp: 79 len: 83 to MEMORY
16/10/05 01:02:38 INFO reduce.InMemoryMapOutput: Read 79 bytes from map-output for attempt_local844278475_0001_m_000001_0
16/10/05 01:02:38 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 79, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->79
16/10/05 01:02:38 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local844278475_0001_m_000000_0 decomp: 90 len: 94 to MEMORY
16/10/05 01:02:38 INFO reduce.InMemoryMapOutput: Read 90 bytes from map-output for attempt_local844278475_0001_m_000000_0
16/10/05 01:02:38 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 90, inMemoryMapOutputs.size() -> 2, commitMemory -> 79, usedMemory ->169
16/10/05 01:02:38 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/10/05 01:02:38 INFO mapred.LocalJobRunner: 2 / 2 copied.
16/10/05 01:02:38 INFO reduce.MergeManagerImpl: finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
16/10/05 01:02:38 INFO mapred.Merger: Merging 2 sorted segments
16/10/05 01:02:38 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 157 bytes
16/10/05 01:02:38 INFO reduce.MergeManagerImpl: Merged 2 segments, 169 bytes to disk to satisfy reduce memory limit
16/10/05 01:02:38 INFO reduce.MergeManagerImpl: Merging 1 files, 171 bytes from disk
16/10/05 01:02:38 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/10/05 01:02:38 INFO mapred.Merger: Merging 1 sorted segments
16/10/05 01:02:38 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 161 bytes
16/10/05 01:02:38 INFO mapred.LocalJobRunner: 2 / 2 copied.
16/10/05 01:02:38 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
16/10/05 01:02:38 INFO mapreduce.Job: Job job_local844278475_0001 running in uber mode : false
16/10/05 01:02:38 INFO mapreduce.Job:  map 100% reduce 0%
16/10/05 01:02:38 INFO mapred.Task: Task:attempt_local844278475_0001_r_000000_0 is done. And is in the process of committing
16/10/05 01:02:38 INFO mapred.LocalJobRunner: 2 / 2 copied.
16/10/05 01:02:38 INFO mapred.Task: Task attempt_local844278475_0001_r_000000_0 is allowed to commit now
16/10/05 01:02:38 INFO output.FileOutputCommitter: Saved output of task 'attempt_local844278475_0001_r_000000_0' to hdfs://localhost:9000/user/hdfs/output/_temporary/0/task_local844278475_0001_r_000000
16/10/05 01:02:38 INFO mapred.LocalJobRunner: reduce > reduce
16/10/05 01:02:38 INFO mapred.Task: Task 'attempt_local844278475_0001_r_000000_0' done.
16/10/05 01:02:38 INFO mapred.LocalJobRunner: Finishing task: attempt_local844278475_0001_r_000000_0
16/10/05 01:02:38 INFO mapred.LocalJobRunner: reduce task executor complete.
16/10/05 01:02:39 INFO mapreduce.Job:  map 100% reduce 100%
16/10/05 01:02:39 INFO mapreduce.Job: Job job_local844278475_0001 completed successfully
16/10/05 01:02:39 INFO mapreduce.Job: Counters: 35
    File System Counters
        FILE: Number of bytes read=185671
        FILE: Number of bytes written=1001765
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=433
        HDFS: Number of bytes written=63
        HDFS: Number of read operations=25
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=5
    Map-Reduce Framework
        Map input records=10
        Map output records=32
        Map output bytes=301
        Map output materialized bytes=177
        Input split bytes=218
        Combine input records=32
        Combine output records=15
        Reduce input groups=9
        Reduce shuffle bytes=177
        Reduce input records=15
        Reduce output records=9
        Spilled Records=30
        Shuffled Maps =2
        Failed Shuffles=0
        Merged Map outputs=2
        GC time elapsed (ms)=22
        Total committed heap usage (bytes)=995098624
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=173
    File Output Format Counters 
        Bytes Written=63
dtw@dtw-ThinkPad-E555:~$

如何知道有没结果呢可以通过如下命令知道：
这里写图片描述
通过图片可以看出生成output文件，文件中同样包含两个文件，如果查看内容，使用下面的命令：

至此，所有内容都已经讲完了，希望对那些初学hadoop的朋友有一些帮助，刚开始弄的时候也出现了许多问题，后来慢慢解决，最后成功实现了功能。特别是配置上面需要有一个清晰的认识，这样才能在出现错误的时候知道什么时候什么地方去解决问题。

武睿傲雪

关注

1
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
使用命令shell和MyEclipse两种方式执行Hadoop的WordCount程序

大家初次接触编程时学习的无论时哪种语言，看到的第一个示例程序都是“Hello World”。在Hadoop中也有一个类似于Hello World的程序。这就是WordCount.下文会结合这个程序通过两种方式具体讲解与MapReduce程序有关的所有类。（关于WordCount的程序网上已经有很多了，我这里说一些网上没有的。）通过MyEclipse调用Hadoop的JAR类库实现WordCount
复制链接

扫一扫