- 概述
- 准备条件
- 环境搭建
- 程序运行
- Q&A
- 参考
概述
上次搭建好了CDH hadoop集群环境,已经安装了Zookeeper,HDFS,Yarn,Hive,Spark组件,现在开始写一个小的MR程序测试一把。
准备条件
- 系统环境
- 操作系统 :Windows10
- 开发所需工具 : Eclipse 4.4.2, Apache-maven-3.3.3,Java-8u131,hadoop-eclipse-plugin-2.7.0.jar,hadoop-2.6.0-cdh5.11.1.tar.gz
环境搭建
- Java安装
安装jdk-8u131,配置环境变量,省略。 - Maven安装
解压缩包,配置Maven环境变量M2_HOME,省略。 - Eclipse hadoop插件安装
网上下载了hadoop-eclipse-plugin-2.7.0.jar这个hadoop-eclipse-plugin,放置到eclipse安装目录下plugins/下,重启Eclipse即可出现Hadoop MapReduce选项,如下图所示:
此处Hadoop installation directory中需要在本地配置一个hadoop环境,本人解压缩了hadoop-2.6.0-cdh5.11.1.tar.gz这个文件,同时将Hadoop installation directory指向其解压缩的位置。
下面开始配置服务器上hadoop集群的通信位置。首先Eclipse下打开Window->Open Perspective->Others… 选择Map/Reduce(一个蓝色的大象标志)。在打开的Perspective右上角选择New Hadoop Location…,弹出如下图红框所示:
配置你远程的hadoop环境下的IP,端口以及用户名等。由于本人使用的是CDH Hadoop,使用了cloudera-manager作为运维监控工具添加的Yarn,HDFS等组件服务。这里的IP和端口配置和源生的Apache Hadoop不一致。下图中:
箭头1指示的是mapred.xml中的mapreduce.jobhistory.address的端口地址,箭头2中指示的是core-site.xml中配置的fs.defaultFS的端口地址。这样配置过后点击左上角DFS Locations连接,即可查询出服务器端的HDFS文件路径,这样可以省略了许多服务器端HDFS文件操作。
可能DFS Location中访问时会提示无法访问,权限问题。此时需要更改集群hdfs-site.xml配置,关闭hadoop集群的权限校验。
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
程序运行
上面已经搭建好了本地的系统环境,下面可以开始写一个MR程序了。
1. 首先使用maven创建一个maven-quick-start骨架的项目。
2. 在pom.xml文件中添加hadoop相应的依赖jar包。具体的jar包的依赖参照cloudera官网上的maven仓库地址 https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh5_maven_repo_511x.html
附上本人的pom文件:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>hadoop.test.mrtest</groupId>
<artifactId>hadoop.test.mrtest</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<hadoop.cdh.version>2.6.0-cdh5.11.1</hadoop.cdh.version>
</properties>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-annotations</artifactId>
<version>${hadoop.cdh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.cdh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.cdh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs-nfs</artifactId>
<version>${hadoop.cdh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-api</artifactId>
<version>${hadoop.cdh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-applications-distributedshell</artifactId>
<version>${hadoop.cdh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-server-resourcemanager</artifactId>
<version>${hadoop.cdh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-applications-unmanaged-am-launcher</artifactId>
<version>${hadoop.cdh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-common</artifactId>
<version>${hadoop.cdh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<version>${hadoop.cdh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-app</artifactId>
<version>${hadoop.cdh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-hs</artifactId>
<version>${hadoop.cdh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-hs-plugins</artifactId>
<version>${hadoop.cdh.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>jdk.tools</groupId>
<artifactId>jdk.tools</artifactId>
<version>1.8</version>
<scope>system</scope>
<systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
</dependency>
</dependencies>
</project>
3. 下面开始编写MR程序,就来一段经典的WordCount.java吧。
package hadoop.test.mrtest;
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
public class WordCount {
public static class TokenizerMapper
extends Mapper<Object, Text, Text, IntWritable>{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
public static class IntSumReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
public static void main(String args[]) throws Exception {
Configuration conf = new Configuration();
conf.set("yarn.resourcemanager.scheduler.address", "linuxvnode0:8030");
String input = "hdfs://linuxvnode0:8020/user/input";
String output = "hdfs://linuxvnode0:8020/user/output";
Job job = new Job(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(input));
FileOutputFormat.setOutputPath(job,new Path(output));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
写好了上述代码后通过eclipse导出为jar包或者通过maven打包成可执行的jar包文件。将jar包上传到hadoop集群上,执行即可。
hadoop jar xxxx.jar [Main-Class]
Q&A
- 提交MR程序后,为什么JOB总处于Accepted状态?
##这是由于ResourceManager分配给Job资源时,发现可分配的资源不足导致的,这样的话JOB会长时间的处于待定状态。解决办法是修改分配资源的配置。
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2048</value>
</property>
2. cloudera-manager的组件配置文件在哪?
本人开始在/etc/hadoop/conf 目录下进行修改,但是却发现修改无效,后来从界面上进行配置的修改时发现每次修改配置都需要重启集群,一想知道应该是读取数据库中的配置表。所以连接上mysql数据库中查询Config表中可以看到配置。
参考
Yarn资源的分配:https://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
Cloudera Manager内部结构以及配置文件位置 : http://blog.csdn.net/levy_cui/article/details/51507950
Cloudera Maven库:https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh5_maven_repo_511x.html