map reduce中使用spring容器

最新推荐文章于 2024-04-06 16:29:33 发布

san_qilang

最新推荐文章于 2024-04-06 16:29:33 发布

阅读量538

点赞数

文章标签： spring mapreduce java hadoop 大数据

本文链接：https://blog.csdn.net/san_qilang/article/details/108674212

版权

文章目录

1、mapreduce jar应用为什么要使用Spring容器
2、如何使用
参考

1、mapreduce jar应用为什么要使用Spring容器

map reduce任务是提交到yarn资源管理器上执行的，并不是单独的一个java进程或者说是web应用，但是假如我们已经在一个spring boot应用中装配了一些bean，我们想直接引用的话，这种场景下我们就需要在map reduce任务中使用spring容器了。
接下来介绍如何使用，请注意下面的步骤是必须的，但是每一步的例子并不是完整的，只列出关键内容，以供参考。

2、如何使用

2.1、pom文件中的打包插件

<plugin>  
    <groupId>org.apache.maven.plugins</groupId>  
    <artifactId>maven-shade-plugin</artifactId>  
    <version>3.2.1</version>  
    <executions>  
        <execution>  
            <phase>package</phase>  
            <goals>  
                <goal>shade</goal>  
            </goals>  
            <configuration>
            	<shadedArtifactAttached>true</shadedArtifactAttached>
            	<shadedClassifierName>with-dependencies<shadedClassifierName>
                <filters>  
                    <filter>  
                        <artifact>*:*</artifact>  
                        <excludes>  
                            <exclude>META-INF/*.SF</exclude>  
                            <exclude>META-INF/*.DSA</exclude>  
                            <exclude>META-INF/*.RSA</exclude>  
                        </excludes>  
                    </filter>  
                </filters>  
                <transformers>  
                	<!-- 
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">  
                        <mainClass>com.lcifn.Application</mainClass>  
                    </transformer>
                    -->
                    <transformer  
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">  
                        <resource>META-INF/spring.handlers</resource>  
                    </transformer>  
                    <transformer  
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">  
                        <resource>META-INF/spring.schemas</resource>  
                    </transformer>  
                </transformers>  
            </configuration>  
        </execution>  
    </executions>  
</plugin>

2.2、spring的xml装配

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
        https://www.springframework.org/schema/beans/spring-beans.xsd">
	<bean class="org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor"/>
	<bean id="zookeeperConfig" class="com.common.config.ZookeeperConfig">
		<property name="serverList" value="11.11.11.11:2181,11.11.11.11:2181"/>
		<property name="serverPrincipalKey" value="zookeeper.server.principal"/>
		<property name="defaultServerPrincical" value="zookeeper/hadoop.hadoop.com"/>
	</bean>
	<bean id="fiClientConfig" class="com.commom.config.FiClientConfig">
		<property name="home" value='#{"/".equals(systemProperties["file.separator"]) ? "/home/mate/conf/" : "D:\\\\resources\\conf\\"}'/>
		<property name="coreSite" value='#{"/".equals(systemProperties["file.separator"]) ? "/home/mate/conf/" : "D:\\\\resources\\conf\\"}core-site.xml'/>
	</bean>
	<bean id="hdfsConf" factory-bean="fiClientConfig" factory-method="getHdfsConf" />
</beans>

2.3、ApplicationContextHelper.java

import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

public class ApplicationContextHelper {

    private static ApplicationContext ctx = null;

    /**
     * 获得bean对象
     *
     * @param className
     * @return
     */
    public static Object getBean(String className) {
        return getContext().getBean(className);
    }

    /**
     * 获得应用所在上下文环境
     *
     * @return
     */
    public static ApplicationContext getContext() {
        if (ctx == null) {
            ctx = new ClassPathXmlApplicationContext("spring-commom.xml");
        }
        return ctx;
    }
}

2.4、map阶段或reduce阶段如何使用

public class ExecuteReducer extends Reducer<Text,Text,Text,Text> {
	@Override
	protected void reduce(Text key,Iterable<Text> values,Context context) throws IOException,InterrutedException {
		//...
		EntityManagerFactory factory = (EntityMangerFactory) ApplicationContextHelper.getBean("entityManagerFactory");
		//...
	}
}