java.lang.Exception: java.io.IOException: Illegal partition for 180 180 360 (5)

故事の尾音

已于 2022-11-17 11:21:28 修改

阅读量349

点赞数

分类专栏：大数据文章标签： java 开发语言 spring

于 2022-11-17 11:20:38 首次发布

本文链接：https://blog.csdn.net/lnqvq/article/details/127900436

版权

大数据专栏收录该内容

3 篇文章 0 订阅

订阅专栏

"C:\Program Files\Java\jdk1.8.0_261\bin\java.exe" "-javaagent:D:\java\IntelliJ IDEA 2020.1.2\lib\idea_rt.jar=10099:D:\java\IntelliJ IDEA 2020.1.2\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\Java\jdk1.8.0_261\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\ext\cldrdata.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\ext\jaccess.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\ext\jfxrt.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\ext\nashorn.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\ext\sunpkcs11.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\jce.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\jfxswt.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_261\jre\lib\rt.jar;D:\21922\ideaProject1\MapReduceDemo\target\classes;C:\Users\21922\.m2\repository\org\apache\hadoop\hadoop-client\3.1.3\hadoop-client-3.1.3.jar;C:\Users\21922\.m2\repository\org\apache\hadoop\hadoop-common\3.1.3\hadoop-common-3.1.3.jar;C:\Users\21922\.m2\repository\com\google\guava\guava\27.0-jre\guava-27.0-jre.jar;C:\Users\21922\.m2\repository\com\google\guava\failureaccess\1.0\failureaccess-1.0.jar;C:\Users\21922\.m2\repository\com\google\guava\listenablefuture\9999.0-empty-to-avoid-conflict-with-guava\listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar;C:\Users\21922\.m2\repository\org\checkerframework\checker-qual\2.5.2\checker-qual-2.5.2.jar;C:\Users\21922\.m2\repository\com\google\errorprone\error_prone_annotations\2.2.0\error_prone_annotations-2.2.0.jar;C:\Users\21922\.m2\repository\com\google\j2objc\j2objc-annotations\1.1\j2objc-annotations-1.1.jar;C:\Users\21922\.m2\repository\org\codehaus\mojo\animal-sniffer-annotations\1.17\animal-sniffer-annotations-1.17.jar;C:\Users\21922\.m2\repository\commons-cli\commons-cli\1.2\commons-cli-1.2.jar;C:\Users\21922\.m2\repository\org\apache\commons\commons-math3\3.1.1\commons-math3-3.1.1.jar;C:\Users\21922\.m2\repository\org\apache\httpcomponents\httpclient\4.5.2\httpclient-4.5.2.jar;C:\Users\21922\.m2\repository\org\apache\httpcomponents\httpcore\4.4.4\httpcore-4.4.4.jar;C:\Users\21922\.m2\repository\commons-codec\commons-codec\1.11\commons-codec-1.11.jar;C:\Users\21922\.m2\repository\commons-io\commons-io\2.5\commons-io-2.5.jar;C:\Users\21922\.m2\repository\commons-net\commons-net\3.6\commons-net-3.6.jar;C:\Users\21922\.m2\repository\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar;C:\Users\21922\.m2\repository\org\eclipse\jetty\jetty-servlet\9.3.24.v20180605\jetty-servlet-9.3.24.v20180605.jar;C:\Users\21922\.m2\repository\org\eclipse\jetty\jetty-security\9.3.24.v20180605\jetty-security-9.3.24.v20180605.jar;C:\Users\21922\.m2\repository\org\eclipse\jetty\jetty-webapp\9.3.24.v20180605\jetty-webapp-9.3.24.v20180605.jar;C:\Users\21922\.m2\repository\org\eclipse\jetty\jetty-xml\9.3.24.v20180605\jetty-xml-9.3.24.v20180605.jar;C:\Users\21922\.m2\repository\javax\servlet\jsp\jsp-api\2.1\jsp-api-2.1.jar;C:\Users\21922\.m2\repository\com\sun\jersey\jersey-servlet\1.19\jersey-servlet-1.19.jar;C:\Users\21922\.m2\repository\commons-logging\commons-logging\1.1.3\commons-logging-1.1.3.jar;C:\Users\21922\.m2\repository\commons-lang\commons-lang\2.6\commons-lang-2.6.jar;C:\Users\21922\.m2\repository\commons-beanutils\commons-beanutils\1.9.3\commons-beanutils-1.9.3.jar;C:\Users\21922\.m2\repository\org\apache\commons\commons-configuration2\2.1.1\commons-configuration2-2.1.1.jar;C:\Users\21922\.m2\repository\org\apache\commons\commons-lang3\3.4\commons-lang3-3.4.jar;C:\Users\21922\.m2\repository\org\apache\avro\avro\1.7.7\avro-1.7.7.jar;C:\Users\21922\.m2\repository\org\codehaus\jackson\jackson-core-asl\1.9.13\jackson-core-asl-1.9.13.jar;C:\Users\21922\.m2\repository\org\codehaus\jackson\jackson-mapper-asl\1.9.13\jackson-mapper-asl-1.9.13.jar;C:\Users\21922\.m2\repository\com\thoughtworks\paranamer\paranamer\2.3\paranamer-2.3.jar;C:\Users\21922\.m2\repository\org\xerial\snappy\snappy-java\1.0.5\snappy-java-1.0.5.jar;C:\Users\21922\.m2\repository\com\google\re2j\re2j\1.1\re2j-1.1.jar;C:\Users\21922\.m2\repository\com\google\protobuf\protobuf-java\2.5.0\protobuf-java-2.5.0.jar;C:\Users\21922\.m2\repository\com\google\code\gson\gson\2.2.4\gson-2.2.4.jar;C:\Users\21922\.m2\repository\org\apache\hadoop\hadoop-auth\3.1.3\hadoop-auth-3.1.3.jar;C:\Users\21922\.m2\repository\com\nimbusds\nimbus-jose-jwt\4.41.1\nimbus-jose-jwt-4.41.1.jar;C:\Users\21922\.m2\repository\com\github\stephenc\jcip\jcip-annotations\1.0-1\jcip-annotations-1.0-1.jar;C:\Users\21922\.m2\repository\net\minidev\json-smart\2.3\json-smart-2.3.jar;C:\Users\21922\.m2\repository\net\minidev\accessors-smart\1.2\accessors-smart-1.2.jar;C:\Users\21922\.m2\repository\org\ow2\asm\asm\5.0.4\asm-5.0.4.jar;C:\Users\21922\.m2\repository\org\apache\curator\curator-framework\2.13.0\curator-framework-2.13.0.jar;C:\Users\21922\.m2\repository\org\apache\curator\curator-client\2.13.0\curator-client-2.13.0.jar;C:\Users\21922\.m2\repository\org\apache\curator\curator-recipes\2.13.0\curator-recipes-2.13.0.jar;C:\Users\21922\.m2\repository\com\google\code\findbugs\jsr305\3.0.0\jsr305-3.0.0.jar;C:\Users\21922\.m2\repository\org\apache\htrace\htrace-core4\4.1.0-incubating\htrace-core4-4.1.0-incubating.jar;C:\Users\21922\.m2\repository\org\apache\commons\commons-compress\1.18\commons-compress-1.18.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerb-simplekdc\1.0.1\kerb-simplekdc-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerb-client\1.0.1\kerb-client-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerby-config\1.0.1\kerby-config-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerb-core\1.0.1\kerb-core-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerby-pkix\1.0.1\kerby-pkix-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerby-asn1\1.0.1\kerby-asn1-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerby-util\1.0.1\kerby-util-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerb-common\1.0.1\kerb-common-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerb-crypto\1.0.1\kerb-crypto-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerb-util\1.0.1\kerb-util-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\token-provider\1.0.1\token-provider-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerb-admin\1.0.1\kerb-admin-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerb-server\1.0.1\kerb-server-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerb-identity\1.0.1\kerb-identity-1.0.1.jar;C:\Users\21922\.m2\repository\org\apache\kerby\kerby-xdr\1.0.1\kerby-xdr-1.0.1.jar;C:\Users\21922\.m2\repository\com\fasterxml\jackson\core\jackson-databind\2.7.8\jackson-databind-2.7.8.jar;C:\Users\21922\.m2\repository\com\fasterxml\jackson\core\jackson-core\2.7.8\jackson-core-2.7.8.jar;C:\Users\21922\.m2\repository\org\codehaus\woodstox\stax2-api\3.1.4\stax2-api-3.1.4.jar;C:\Users\21922\.m2\repository\com\fasterxml\woodstox\woodstox-core\5.0.3\woodstox-core-5.0.3.jar;C:\Users\21922\.m2\repository\org\apache\hadoop\hadoop-hdfs-client\3.1.3\hadoop-hdfs-client-3.1.3.jar;C:\Users\21922\.m2\repository\com\squareup\okhttp\okhttp\2.7.5\okhttp-2.7.5.jar;C:\Users\21922\.m2\repository\com\squareup\okio\okio\1.6.0\okio-1.6.0.jar;C:\Users\21922\.m2\repository\com\fasterxml\jackson\core\jackson-annotations\2.7.8\jackson-annotations-2.7.8.jar;C:\Users\21922\.m2\repository\org\apache\hadoop\hadoop-yarn-api\3.1.3\hadoop-yarn-api-3.1.3.jar;C:\Users\21922\.m2\repository\javax\xml\bind\jaxb-api\2.2.11\jaxb-api-2.2.11.jar;C:\Users\21922\.m2\repository\org\apache\hadoop\hadoop-yarn-client\3.1.3\hadoop-yarn-client-3.1.3.jar;C:\Users\21922\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-core\3.1.3\hadoop-mapreduce-client-core-3.1.3.jar;C:\Users\21922\.m2\repository\org\apache\hadoop\hadoop-yarn-common\3.1.3\hadoop-yarn-common-3.1.3.jar;C:\Users\21922\.m2\repository\javax\servlet\javax.servlet-api\3.1.0\javax.servlet-api-3.1.0.jar;C:\Users\21922\.m2\repository\org\eclipse\jetty\jetty-util\9.3.24.v20180605\jetty-util-9.3.24.v20180605.jar;C:\Users\21922\.m2\repository\com\sun\jersey\jersey-core\1.19\jersey-core-1.19.jar;C:\Users\21922\.m2\repository\javax\ws\rs\jsr311-api\1.1.1\jsr311-api-1.1.1.jar;C:\Users\21922\.m2\repository\com\sun\jersey\jersey-client\1.19\jersey-client-1.19.jar;C:\Users\21922\.m2\repository\com\fasterxml\jackson\module\jackson-module-jaxb-annotations\2.7.8\jackson-module-jaxb-annotations-2.7.8.jar;C:\Users\21922\.m2\repository\com\fasterxml\jackson\jaxrs\jackson-jaxrs-json-provider\2.7.8\jackson-jaxrs-json-provider-2.7.8.jar;C:\Users\21922\.m2\repository\com\fasterxml\jackson\jaxrs\jackson-jaxrs-base\2.7.8\jackson-jaxrs-base-2.7.8.jar;C:\Users\21922\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-jobclient\3.1.3\hadoop-mapreduce-client-jobclient-3.1.3.jar;C:\Users\21922\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-common\3.1.3\hadoop-mapreduce-client-common-3.1.3.jar;C:\Users\21922\.m2\repository\org\apache\hadoop\hadoop-annotations\3.1.3\hadoop-annotations-3.1.3.jar;C:\Users\21922\.m2\repository\junit\junit\4.12\junit-4.12.jar;C:\Users\21922\.m2\repository\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar;C:\Users\21922\.m2\repository\org\slf4j\slf4j-log4j12\1.7.30\slf4j-log4j12-1.7.30.jar;C:\Users\21922\.m2\repository\org\slf4j\slf4j-api\1.7.30\slf4j-api-1.7.30.jar;C:\Users\21922\.m2\repository\log4j\log4j\1.2.17\log4j-1.2.17.jar" com.atguigu.mapreduce.writableComparable.FlowDriver
2022-11-17 11:15:59,454 WARN [org.apache.hadoop.metrics2.impl.MetricsConfig] - Cannot locate configuration: tried hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties
2022-11-17 11:15:59,544 INFO [org.apache.hadoop.metrics2.impl.MetricsSystemImpl] - Scheduled Metric snapshot period at 10 second(s).
2022-11-17 11:15:59,545 INFO [org.apache.hadoop.metrics2.impl.MetricsSystemImpl] - JobTracker metrics system started
2022-11-17 11:16:00,745 WARN [org.apache.hadoop.mapreduce.JobResourceUploader] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2022-11-17 11:16:00,761 WARN [org.apache.hadoop.mapreduce.JobResourceUploader] - No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2022-11-17 11:16:00,795 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input files to process : 1
2022-11-17 11:16:00,839 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1
2022-11-17 11:16:00,945 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local423179987_0001
2022-11-17 11:16:00,947 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Executing with tokens: []
2022-11-17 11:16:01,083 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/
2022-11-17 11:16:01,085 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local423179987_0001
2022-11-17 11:16:01,085 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null
2022-11-17 11:16:01,095 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - File Output Committer Algorithm version is 2
2022-11-17 11:16:01,095 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-11-17 11:16:01,096 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2022-11-17 11:16:01,126 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks
2022-11-17 11:16:01,127 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local423179987_0001_m_000000_0
2022-11-17 11:16:01,147 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - File Output Committer Algorithm version is 2
2022-11-17 11:16:01,147 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-11-17 11:16:01,158 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2022-11-17 11:16:01,229 INFO [org.apache.hadoop.mapred.Task] -  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@7ce7983
2022-11-17 11:16:01,235 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/output/output3/part-r-00000:0+550
2022-11-17 11:16:01,315 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584)
2022-11-17 11:16:01,315 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100
2022-11-17 11:16:01,315 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
2022-11-17 11:16:01,315 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600
2022-11-17 11:16:01,315 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600
2022-11-17 11:16:01,320 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-11-17 11:16:01,324 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output
2022-11-17 11:16:01,355 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete.
2022-11-17 11:16:01,359 WARN [org.apache.hadoop.mapred.LocalJobRunner] - job_local423179987_0001
java.lang.Exception: java.io.IOException: Illegal partition for 180	180	360 (5)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.io.IOException: Illegal partition for 180	180	360 (5)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1096)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:727)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
	at com.atguigu.mapreduce.writableComparable.FlowMapper.map(FlowMapper.java:42)
	at com.atguigu.mapreduce.writableComparable.FlowMapper.map(FlowMapper.java:17)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2022-11-17 11:16:02,101 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local423179987_0001 running in uber mode : false
2022-11-17 11:16:02,102 INFO [org.apache.hadoop.mapreduce.Job] -  map 0% reduce 0%
2022-11-17 11:16:02,104 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local423179987_0001 failed with state FAILED due to: NA
2022-11-17 11:16:02,109 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 0

Process finished with exit code 0

错误原因：设置分区时返回值从0开始

@Override
    public int getPartition(FlowBean flowBean, Text text, int i) {
        //定义一个分区号变量
        int partition;

        //获取手机号前三位
        String phone = text.toString();
        String prePhone  = phone.substring(0,3);

        if("136".equals(prePhone)){
            partition=1;
        }else if("137".equals(prePhone)) {
            partition=2;
        }else if("138".equals(prePhone)){
            partition=3;
        }else if("139".equals(prePhone)){
            partition=4;
        }else
            partition=5;
        return partition;
    }

解决方案：将分区返回值从0开始返回即可

@Override
    public int getPartition(FlowBean flowBean, Text text, int i) {
        //定义一个分区号变量
        int partition;

        //获取手机号前三位
        String phone = text.toString();
        String prePhone  = phone.substring(0,3);

        if("136".equals(prePhone)){
            partition=0;
        }else if("137".equals(prePhone)) {
            partition=1;
        }else if("138".equals(prePhone)){
            partition=2;
        }else if("139".equals(prePhone)){
            partition=3;
        }else
            partition=4;
        return partition;
    }

故事の尾音

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
1
评论
java.lang.Exception: java.io.IOException: Illegal partition for 180 180 360 (5)

【代码】java.lang.Exception: java.io.IOException: Illegal partition for 180 180 360 (5)
复制链接

扫一扫