问题:
如何实现?
The command line format is:
$ hadoop org.apache.hadoop.examples.pi.DistBbp \
<b> <nThreads> <nJobs> <type> <nPart> <remoteDir> <localDir>
And the parameters are:
<b> The number of bits to skip, i.e. compute the (b+1)th position.
<nThreads> The number of working threads.
<nJobs> The number of jobs per sum.
<type> 'm' for map side job, 'r' for reduce side job, 'x' for mix type.
<nPart> The number of parts per job.
<remoteDir> Remote directory for submitting jobs.
<localDir> Local directory for storing output files.
问题1:
未指定Main-Class 的 jar 包如何在运行时指定Main-Class?
正常情况下,java
打包成jar
包需要在MANIFEST.MF
中指定Main-Class
项以便运行java -jar ***.jar
时找到对应的主类。但是如果我不想在打包成jar
包时指定这个类,而是想运行时指定想要运行的类,不知该如何做?
java -jar HelloWorld.jar org.test.HelloWorld
如上方法运行包中的org.test.HelloWorld
类时会输出jar中没有主清单属性
的错误。当然,我是没有指定Main-Class
项,因为我想运行时动态指定。
-jar
的含义就是后面跟的jar包是有main class可独立运行
你应该用-cp
/ --classpath
java -cp HelloWorld.jar org.test.HelloWorld
问题2:
conf.set("fs.defaultFS","hdfs://dev65:49100"); conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem"); conf.set("mapreduce.framework.name", "yarn"); conf.set("fs.AbstractFileSystem.hdfs.impl", "org.apache.hadoop.fs.Hdfs");
配置之后打包运行后会报错,大致讲的就是map 申请的堆空间是1356M,实际可用是512M.conf.addResource(IOUtils.toInputStream(FileUtils.readFileToString(new File("/usr/local/bigdata/hadoop/conf/yarn-site.xml"), "utf8"))); conf.addResource(IOUtils.toInputStream(FileUtils.readFileToString(new File("/usr/local/bigdata/hadoop/conf/mapred-site.xml"), "utf8")));
2.配置完以后如果直接在eclipse中跑是会报错的。提示,找不到类的错误,原因是我们的代码中含有这句话:job.setJarByClass(MapReduce.class);
这就是给出了MapReduce的class路径,通过这个路径去寻找map和reduce的内部类路径。在eclipse中跑时候是没有打包成class文件的,自然会找不到路径,报类找不到的错误。FileInputFormat.addInputPath(Job, new Path("input"));
不支持第一个参数传入JobConf,FileinputFormat 引入的包是import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
其实怀疑引入的是org.apache.hadoop.mapred.FileInputFormat org.apache.hadoop.mapred.FileOutputFormat这样我们就可以传入 JobConf类型的参数了。
Manifest-Version: 1.0Main-Class: MapReduceClass-Path:
之前没有写 Manifest-Version这一项Main-Class: MapReduce
e后面有一个空格,这样是不行的。每行的最后是不能有空格的,冒号后面有一个空格,classpath最后有一个空行1358)) - Job job_1395919365278_0034 failed with state FAILED due to: Application application_1395919365278_0034 failed 2 times due to AM Container for appattempt_1395919365278_0034_000002 exited with exitCode: 1 due to: Exception from container-launch:org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
怀疑是需要在服务器上运行才行,当然我们也可也下载hadoop2.2 eclispe插件或是网上也有自己写classloader的例子来在本地跑mR代码import java.io.File;import java.io.IOException;import java.util.StringTokenizer;import org.apache.commons.io.FileUtils;import org.apache.commons.io.IOUtils;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapred.JobClient;import org.apache.hadoop.mapred.JobConf;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;public class MapReduce { public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } this.result.set(sum); context.write(key, this.result); } } public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> { private static final IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Mapper<Object, Text, Text, IntWritable>.Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { this.word.set(itr.nextToken()); context.write(this.word, one); } } } public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException { // TODO Auto-generated method stub Configuration conf = new Configuration(); conf.set("fs.defaultFS","hdfs://dev65:49100"); conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem"); conf.set("mapreduce.framework.name", "yarn"); conf.set("fs.AbstractFileSystem.hdfs.impl", "org.apache.hadoop.fs.Hdfs"); conf.set("yarn.resourcemanager.scheduler.address", "dev65:49012");//设置调度地址 conf.set("yarn.resourcemanager.address","dev65:49011"); //设置RM 访问位置 conf.set("yarn.scheduler.maximum-allocation-mb", "512");//设置调度地址 conf.addResource(IOUtils.toInputStream(FileUtils.readFileToString(new File("/usr/local/bigdata/hadoop/conf/yarn-site.xml"), "utf8"))); conf.set("mapreduce.map.memory.mb", "256");//设置调度地址 conf.addResource(IOUtils.toInputStream(FileUtils.readFileToString(new File("/usr/local/bigdata/hadoop/conf/mapred-site.xml"), "utf8")));
conf.addResource(IOUtils.toInputStream(FileUtils.readFileToString(new File("/usr/local/bigdata/hadoop/conf/core-site.xml"), "utf8")));// JobConf jobConf = new JobConf();// JobClient.runJob(jobConf); Job job = new Job(conf,"sort"); job.setJarByClass(MapReduce.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path("input")); FileOutputFormat.setOutputPath(job, new Path("outputtest")); job.waitForCompletion(true); System.out.println("Job ended: "); }}
yarn-site.xml<?xml version="1.0"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><configuration> <property> <name>yarn.resourcemanager.address</name> <value>dev65:49011</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>dev65:49012</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>dev65:49013</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>dev65:49014</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>dev65:49015</value> </property> <property> <name>yarn.web-proxy.address</name> <value>dev65:49016</value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>512</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>512</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>1024</value> </property> <property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>30</value> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value>/ext/bigdata/hadoop/mapred/local</value> </property> <property> <name>yarn.nodemanager.log-dirs</name> <value>/ext/bigdata/hadoop/mapred/logs</value> </property> <property> <name>yarn.log-aggregation.retain-seconds</name> <value>600</value> </property> <property> <name>yarn.app.mapreduce.am.resource.mb</name> <value>64</value> </property> <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
mapred-site.xml<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.map.memory.mb</name> <value>256</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>256</value> </property> <property> <name>mapreduce.map.java.opts</name> <value>-Xmx128M</value> </property> <property> <name>mapreduce.reduce.java.opts</name> <value>-Xmx128M</value> </property> <property> <name>mapred.child.java.opts</name> <value>-Xmx128m</value> </property> <property> <name>mapreduce.task.io.sort.mb</name> <value>64</value> </property> <property> <name>yarn.app.mapreduce.am.command-opts</name> <value>-Xmx64m</value> </property> <property> <name>mapreduce.task.io.sort.factor</name> <value>100</value> </property> <property> <name>mapreduce.reduce.shuffle.parallelcopies</name> <value>10</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>dev65:10120</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>dev65:19988</value> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/user/mapred/history/tmp</value> </property> <property> <name>mapreduce.jobhistory.done-dir</name> <value>/user/mapred/history/done</value> </property></configuration>
manifes.MFManifest-Version: 1.0Main-Class: MapReduceClass-Path: /usr/local/bigdata/hbase/lib/activation-1.1.jar /usr/local/bigdata/hbase/lib/aopalliance-1.0.jar /usr/local/bigdata/hbase/lib/asm-3.1.jar /usr/local/bigdata/hbase/lib/avro-1.7.4.jar /usr/local/bigdata/hbase/lib/commons-beanutils-1.7.0.jar /usr/local/bigdata/hbase/lib/commons-beanutils-core-1.8.0.jar /usr/local/bigdata/hbase/lib/commons-cli-1.2.jar /usr/local/bigdata/hbase/lib/commons-codec-1.7.jar /usr/local/bigdata/hbase/lib/commons-collections-3.2.1.jar /usr/local/bigdata/hbase/lib/commons-compress-1.4.1.jar /usr/local/bigdata/hbase/lib/commons-configuration-1.6.jar /usr/local/bigdata/hbase/lib/commons-daemon-1.0.13.jar /usr/local/bigdata/hbase/lib/commons-digester-1.8.jar /usr/local/bigdata/hbase/lib/commons-el-1.0.jar /usr/local/bigdata/hbase/lib/commons-httpclient-3.1.jar /usr/local/bigdata/hbase/lib/commons-io-2.4.jar /usr/local/bigdata/hbase/lib/commons-lang-2.6.jar /usr/local/bigdata/hbase/lib/commons-logging-1.1.1.jar /usr/local/bigdata/hbase/lib/commons-math-2.2.jar /usr/local/bigdata/hbase/lib/commons-net-3.1.jar /usr/local/bigdata/hbase/lib/core-3.1.1.jar /usr/local/bigdata/hbase/lib/findbugs-annotations-1.3.9-1.jar /usr/local/bigdata/hbase/lib/gmbal-api-only-3.0.0-b023.jar /usr/local/bigdata/hbase/lib/grizzly-framework-2.1.2.jar /usr/local/bigdata/hbase/lib/grizzly-http-2.1.2.jar /usr/local/bigdata/hbase/lib/grizzly-http-server-2.1.2.jar /usr/local/bigdata/hbase/lib/grizzly-http-servlet-2.1.2.jar /usr/local/bigdata/hbase/lib/grizzly-rcm-2.1.2.jar /usr/local/bigdata/hbase/lib/guava-12.0.1.jar /usr/local/bigdata/hbase/lib/guice-3.0.jar /usr/local/bigdata/hbase/lib/guice-servlet-3.0.jar /usr/local/bigdata/hbase/lib/hadoop-annotations-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-auth-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-client-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-common-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-hdfs-2.2.0-tests.jar /usr/local/bigdata/hbase/lib/hadoop-hdfs-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-mapreduce-client-app-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-mapreduce-client-common-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-mapreduce-client-core-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar /usr/local/bigdata/hbase/lib/hadoop-mapreduce-client-jobclient-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-mapreduce-client-shuffle-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-yarn-api-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-yarn-client-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-yarn-common-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-yarn-server-common-2.2.0.jar /usr/local/bigdata/hbase/lib/hadoop-yarn-server-nodemanager-2.2.0.jar /usr/local/bigdata/hbase/lib/hamcrest-core-1.3.jar /usr/local/bigdata/hbase/lib/hbase-client-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-common-0.96.1.1-hadoop2-tests.jar /usr/local/bigdata/hbase/lib/hbase-common-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-examples-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-hadoop-compat-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-hadoop2-compat-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-it-0.96.1.1-hadoop2-tests.jar /usr/local/bigdata/hbase/lib/hbase-it-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-prefix-tree-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-protocol-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-server-0.96.1.1-hadoop2-tests.jar /usr/local/bigdata/hbase/lib/hbase-server-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-shell-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-testing-util-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/hbase-thrift-0.96.1.1-hadoop2.jar /usr/local/bigdata/hbase/lib/high-scale-lib-1.1.1.jar /usr/local/bigdata/hbase/lib/htrace-core-2.01.jar /usr/local/bigdata/hbase/lib/httpclient-4.1.3.jar /usr/local/bigdata/hbase/lib/httpcore-4.1.3.jar /usr/local/bigdata/hbase/lib/jackson-core-asl-1.8.8.jar /usr/local/bigdata/hbase/lib/jackson-jaxrs-1.8.8.jar /usr/local/bigdata/hbase/lib/jackson-mapper-asl-1.8.8.jar /usr/local/bigdata/hbase/lib/jackson-xc-1.8.8.jar /usr/local/bigdata/hbase/lib/jamon-runtime-2.3.1.jar /usr/local/bigdata/hbase/lib/jasper-compiler-5.5.23.jar /usr/local/bigdata/hbase/lib/jasper-runtime-5.5.23.jar /usr/local/bigdata/hbase/lib/javax.inject-1.jar /usr/local/bigdata/hbase/lib/javax.servlet-3.1.jar /usr/local/bigdata/hbase/lib/javax.servlet-api-3.0.1.jar /usr/local/bigdata/hbase/lib/jaxb-api-2.2.2.jar /usr/local/bigdata/hbase/lib/jaxb-impl-2.2.3-1.jar /usr/local/bigdata/hbase/lib/jersey-client-1.9.jar /usr/local/bigdata/hbase/lib/jersey-core-1.8.jar /usr/local/bigdata/hbase/lib/jersey-grizzly2-1.9.jar /usr/local/bigdata/hbase/lib/jersey-guice-1.9.jar /usr/local/bigdata/hbase/lib/jersey-json-1.8.jar /usr/local/bigdata/hbase/lib/jersey-server-1.8.jar /usr/local/bigdata/hbase/lib/jersey-test-framework-core-1.9.jar /usr/local/bigdata/hbase/lib/jersey-test-framework-grizzly2-1.9.jar /usr/local/bigdata/hbase/lib/jets3t-0.6.1.jar /usr/local/bigdata/hbase/lib/jettison-1.3.1.jar /usr/local/bigdata/hbase/lib/jetty-6.1.26.jar /usr/local/bigdata/hbase/lib/jetty-sslengine-6.1.26.jar /usr/local/bigdata/hbase/lib/jetty-util-6.1.26.jar /usr/local/bigdata/hbase/lib/jruby-complete-1.6.8.jar /usr/local/bigdata/hbase/lib/jsch-0.1.42.jar /usr/local/bigdata/hbase/lib/jsp-2.1-6.1.14.jar /usr/local/bigdata/hbase/lib/jsp-api-2.1-6.1.14.jar /usr/local/bigdata/hbase/lib/jsp-api-2.1.jar /usr/local/bigdata/hbase/lib/jsr305-1.3.9.jar /usr/local/bigdata/hbase/lib/junit-4.11.jar /usr/local/bigdata/hbase/lib/libthrift-0.9.0.jar /usr/local/bigdata/hbase/lib/log4j-1.2.17.jar /usr/local/bigdata/hbase/lib/management-api-3.0.0-b012.jar /usr/local/bigdata/hbase/lib/metrics-core-2.1.2.jar /usr/local/bigdata/hbase/lib/netty-3.6.6.Final.jar /usr/local/bigdata/hbase/lib/paranamer-2.3.jar /usr/local/bigdata/hbase/lib/protobuf-java-2.5.0.jar /usr/local/bigdata/hbase/lib/servlet-api-2.5-6.1.14.jar /usr/local/bigdata/hbase/lib/servlet-api-2.5.jar /usr/local/bigdata/hbase/lib/slf4j-api-1.6.4.jar /usr/local/bigdata/hbase/lib/slf4j-log4j12-1.6.4.jar /usr/local/bigdata/hbase/lib/snappy-java-1.0.4.1.jar /usr/local/bigdata/hbase/lib/stax-api-1.0.1.jar /usr/local/bigdata/hbase/lib/xmlenc-0.52.jar /usr/local/bigdata/hbase/lib/xz-1.0.jar /usr/local/bigdata/hbase/lib/zookeeper-3.4.5.jar