概述
- 作用
- 应用场景
- 效果
- 代码示例
作用
- 想在 map 或 reduce 里启动多线程
应用场景
- map reduce 端 cpu 型作业
效果
- 公司原来一个 cpu 型算法作业,20 台服务器 以前运行需 55 个小时左右,改用我自己改进的多线程后,同样的前提条件,只须运行 7 小时
代码示例
- 多线程map 示例代码
- 自带的 多线程map 源代码
public class MultithreadedMapper<K1, V1, K2, V2> extends Mapper<K1, V1, K2, V2> { private static final Log LOG = LogFactory.getLog(MultithreadedMapper.class); private Class<? extends Mapper<K1,V1,K2,V2>> mapClass; private Context outer; private List<MapRunner> runners; /** * The number of threads in the thread pool that will run the map function. * @param job the job * @return the number of threads */ public static int getNumberOfThreads(JobContext job) { return job.getConfiguration(). getInt("mapred.map.multithreadedrunner.threads", 10); } /** * Set the number of threads in the pool for running maps. * @param job the job to modify * @param threads the new number of threads */ public static void setNumberOfThreads(Job job, int threads) { job.getConfiguration().setInt("mapred.map.multithreadedrunner.threads", threads); } /** * Get the application's mapper class. * @param <K1> the map's input key type * @param <V1> the map's input value type * @param <K2> the map's output key type * @param <V2> the map's output value type * @param job the job * @return the mapper class to run */ @SuppressWarnings("unchecked") public static <K1,V1,K2,V2> Class<Mapper<K1,V1,K2,V2>> getMapperClass(JobContext job) { return (Class<Mapper<K1,V1,K2,V2>>) job.getConfiguration().getClass("mapred.map.multithreadedrunner.class", Mapper.class); } /** * Set the application's mapper class. * @param <K1> the map input key type * @param <V1> the map input value type * @param <K2> the map output key type * @param <V2> the map output value type * @param job the job to modify * @param cls the class to use as the mapper */