多线程map

概述

• 作用
• 应用场景
• 效果
• 代码示例

作用

• 想在 map 或 reduce 里启动多线程

应用场景

• map reduce 端 cpu 型作业

效果

• 公司原来一个 cpu 型算法作业，20 台服务器 以前运行需 55 个小时左右，改用我自己改进的多线程后，同样的前提条件，只须运行 7 小时

代码示例

• 多线程map 示例代码
• 自带的 多线程map 源代码

• public class MultithreadedMapper<K1, V1, K2, V2> 
    extends Mapper<K1, V1, K2, V2> {
  
    private static final Log LOG = LogFactory.getLog(MultithreadedMapper.class);
    private Class<? extends Mapper<K1,V1,K2,V2>> mapClass;
    private Context outer;
    private List<MapRunner> runners;
  
    /**
     * The number of threads in the thread pool that will run the map function.
     * @param job the job
     * @return the number of threads
     */
    public static int getNumberOfThreads(JobContext job) {
      return job.getConfiguration().
              getInt("mapred.map.multithreadedrunner.threads", 10);
    }
  
    /**
     * Set the number of threads in the pool for running maps.
     * @param job the job to modify
     * @param threads the new number of threads
     */
    public static void setNumberOfThreads(Job job, int threads) {
      job.getConfiguration().setInt("mapred.map.multithreadedrunner.threads", 
                                    threads);
    }
  
    /**
     * Get the application's mapper class.
     * @param <K1> the map's input key type
     * @param <V1> the map's input value type
     * @param <K2> the map's output key type
     * @param <V2> the map's output value type
     * @param job the job
     * @return the mapper class to run
     */
    @SuppressWarnings("unchecked")
    public static <K1,V1,K2,V2>
    Class<Mapper<K1,V1,K2,V2>> getMapperClass(JobContext job) {
      return (Class<Mapper<K1,V1,K2,V2>>) 
           job.getConfiguration().getClass("mapred.map.multithreadedrunner.class",
                                           Mapper.class);
    }
    
    /**
     * Set the application's mapper class.
     * @param <K1> the map input key type
     * @param <V1> the map input value type
     * @param <K2> the map output key type
     * @param <V2> the map output value type
     * @param job the job to modify
     * @param cls the class to use as the mapper
     */