Hadoop（mapreduce求最大值）

风玉尘

已于 2023-06-18 20:53:28 修改

阅读量68

点赞数

分类专栏： Hadoop 文章标签： hadoop mapreduce 大数据

于 2023-05-28 22:52:02 首次发布

本文链接：https://blog.csdn.net/qq_45362257/article/details/130918777

版权

Hadoop 专栏收录该内容

4 篇文章 0 订阅

订阅专栏

一、创建maven项目

1、命令创建maven项目

[WBQ@westgisB065 maven01]$ mvn archetype:generate

2、输入groupId、artifactId、version和package，groupId一般输入公司名称就可以了，而artifactId输入项目名称，package为包名

Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): 7: 
Downloading from nexus-aliyun: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/archetypes/maven-archetype-quickstart/1.1/maven-archetype-quickstart-1.1.pom
Downloaded from nexus-aliyun: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/archetypes/maven-archetype-quickstart/1.1/maven-archetype-quickstart-1.1.pom (2.0 kB at 4.9 kB/s)
Downloading from nexus-aliyun: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/archetypes/maven-archetype-bundles/4/maven-archetype-bundles-4.pom
Downloaded from nexus-aliyun: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/archetypes/maven-archetype-bundles/4/maven-archetype-bundles-4.pom (3.6 kB at 9.3 kB/s)
Downloading from nexus-aliyun: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/archetype/maven-archetype/2.0-alpha-5/maven-archetype-2.0-alpha-5.pom
Downloaded from nexus-aliyun: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/archetype/maven-archetype/2.0-alpha-5/maven-archetype-2.0-alpha-5.pom (8.7 kB at 23 kB/s)
Downloading from nexus-aliyun: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/maven-parent/16/maven-parent-16.pom
Downloaded from nexus-aliyun: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/maven-parent/16/maven-parent-16.pom (23 kB at 59 kB/s)
Downloading from nexus-aliyun: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/archetypes/maven-archetype-quickstart/1.1/maven-archetype-quickstart-1.1.jar
Downloaded from nexus-aliyun: http://maven.aliyun.com/nexus/content/groups/public/org/apache/maven/archetypes/maven-archetype-quickstart/1.1/maven-archetype-quickstart-1.1.jar (6.2 kB at 15 kB/s)
Define value for property 'groupId': edu
Define value for property 'artifactId': edu01
Define value for property 'version' 1.0-SNAPSHOT: : 
Define value for property 'package' edu: : edu02
Confirm properties configuration:
groupId: edu
artifactId: edu01
version: 1.0-SNAPSHOT
package: edu02
 Y: : 
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Old (1.x) Archetype: maven-archetype-quickstart:1.1
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: basedir, Value: /home/WBQ/code/maven01
[INFO] Parameter: package, Value: edu02
[INFO] Parameter: groupId, Value: edu
[INFO] Parameter: artifactId, Value: edu01
[INFO] Parameter: packageName, Value: edu02
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] project created from Old (1.x) Archetype in dir: /home/WBQ/code/maven01/edu01
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:20 min
[INFO] Finished at: 2023-05-28T21:55:17+08:00
[INFO] ------------------------------------------------------------------------
[WBQ@westgisB065 maven01]$

3、使用find命令查看项目目录结构，编写pom.xml文件配置

[WBQ@westgisB065 maven01]$ find
.
./edu01
./edu01/pom.xml
./edu01/src
./edu01/src/main
./edu01/src/main/java
./edu01/src/main/java/edu02
./edu01/src/main/java/edu02/App.java
./edu01/src/test
./edu01/src/test/java
./edu01/src/test/java/edu02
./edu01/src/test/java/edu02/AppTest.java
[WBQ@westgisB065 maven01]$

pom.xml文件配置如下：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>edu</groupId>
  <artifactId>edu01</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>edu01</name>
  <url>http://maven.apache.org</url>

<!--  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
-->
<properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <java.version>1.8</java.version>
        <hadoop.version>3.1.3</hadoop.version>
        <log4j.version>1.2.14</log4j.version>
        <junit.version>4.8.2</junit.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
                   </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>${log4j.version}</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifest>
                            <mainClass>example.max</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

4、在 ./edu01/src/main/java/edu02/ 目录下编写主程序（程序详细代码在文章结尾附上）

[WBQ@westgisB065 edu02]$ ll -a
total 12
drwxrwxr-x 2 WBQ WBQ   63 May 28 21:59 .
drwxrwxr-x 3 WBQ WBQ   19 May 28 21:55 ..
-rw-rw-r-- 1 WBQ WBQ 1088 May 28 21:58 max.java
-rw-rw-r-- 1 WBQ WBQ  990 May 28 21:58 MaxMap.java
-rw-rw-r-- 1 WBQ WBQ  581 May 28 21:58 MaxReduce.java
[WBQ@westgisB065 edu02]$

5、编译项目

[WBQ@westgisB065 edu01]$ mvn compile

6、打包

[WBQ@westgisB065 edu01]$ mvn install

提示信息如下则证明打包成功：

[INFO] Installing /home/WBQ/code/maven01/edu01/target/edu01-1.0-SNAPSHOT.jar to /home/WBQ/.m2/repository/edu/edu01/1.0-SNAPSHOT/edu01-1.0-SNAPSHOT.jar
[INFO] Installing /home/WBQ/code/maven01/edu01/pom.xml to /home/WBQ/.m2/repository/edu/edu01/1.0-SNAPSHOT/edu01-1.0-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  22.627 s
[INFO] Finished at: 2023-05-28T22:20:15+08:00
[INFO] ------------------------------------------------------------------------
[WBQ@westgisB065 edu01]$

7、打包成功的jar包在项目中的target目录下（edu01-1.0-SNAPSHOT.jar）：

[WBQ@westgisB065 edu01]$ cd target/
[WBQ@westgisB065 target]$ ll -a
total 8
drwxrwxr-x 9 WBQ WBQ  192 May 28 22:20 .
drwxrwxr-x 4 WBQ WBQ   46 May 28 22:18 ..
drwxrwxr-x 3 WBQ WBQ   19 May 28 22:18 classes
-rw-rw-r-- 1 WBQ WBQ 5127 May 28 22:20 edu01-1.0-SNAPSHOT.jar
drwxrwxr-x 3 WBQ WBQ   25 May 28 22:18 generated-sources
drwxrwxr-x 3 WBQ WBQ   30 May 28 22:19 generated-test-sources
drwxrwxr-x 2 WBQ WBQ   28 May 28 22:20 maven-archiver
drwxrwxr-x 3 WBQ WBQ   35 May 28 22:18 maven-status
drwxrwxr-x 2 WBQ WBQ   61 May 28 22:20 surefire-reports
drwxrwxr-x 3 WBQ WBQ   19 May 28 22:19 test-classes
[WBQ@westgisB065 target]$

二、主程序代码如下：

max.java

package edu02;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class max {
        public static void main( String[] otherArgs )throws Exception{
                 Configuration conf = new Configuration();
             otherArgs = new String[] {"hdfs://10.103.105.64:/test/input-max/*","hdfs://10.103.105.64:/test/output-max"};

             for(int i=0;i<otherArgs.length;i++){
                 System.out.println(otherArgs[i]);
             }

             Job job = Job.getInstance(conf);
             job.setJarByClass(max.class);
             job.setMapperClass(MaxMap.class);
             job.setReducerClass(MaxReduce.class);
             job.setOutputKeyClass(Text.class);
             
             FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
             FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
             job.waitForCompletion(true);
             }
}

MaxMap.java

package edu02;

import java.io.IOException;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class MaxMap extends Mapper <LongWritable,Text,Text ,DoubleWritable>{
        double Max = 0;

    protected void map(LongWritable key ,Text value, Context context)
            throws IOException , InterruptedException{
        DoubleWritable MMax = new DoubleWritable();
        String[] strs = value.toString().split(" ");
        int flag = 0;
        String id = " ";
        Text ID = new Text();
        for(String str : strs){
                if(flag%2 == 0) {
                        id = str;
                        flag++;
                }
                else {
    double score0 = Double.parseDouble(str);
                        if(score0 > Max) {
                                Max = score0;
                                MMax=new DoubleWritable(Max);
                                ID = new Text(id);
                        }
                        flag++;
                }
        }
        if(!ID.equals(null) && !MMax.equals(0))
            context.write(ID,MMax);
}
}

MaxReduce.java

package edu02;

import java.io.IOException;

import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class MaxReduce extends Reducer<Text,DoubleWritable,Text,DoubleWritable>{
        double max = 0;
        protected void reduce(Text key,Iterable<DoubleWritable> values,Context context)
                        throws IOException , InterruptedException{
        for(DoubleWritable val : values){
                if( val.get() > max) {
                        max = val.get();
                        context.write(key,new DoubleWritable(max));
                }
        }
    }
}