场景:
mysql有压力测试,会利用压测工具,mock一大批数据,但是hbase没有提供相应的功能,此时我们可以自己写一个mock工具,以满足我们的测试需求。
分析:
我们看下Mapper的源码:
public void run(Context context) throws IOException, InterruptedException {
setup(context);
while (context.nextKeyValue()) {
map(context.getCurrentKey(), context.getCurrentValue(), context);
}
cleanup(context);
}
我们发现如果打算让map多次执行,那么只需要在这边下功夫了
思路:
1.设置一个空的输入文件
2.通过context.getConfiguration().getLong()获取自己指定的long类型的参数
3.通过context.getConfiguration().get()获取要mock的表名,列如(E:E格式),值的产生方式支持number和string两种格式
4.覆盖mapper的run方法,循环执行指定的次数
简要代码如下:
Job:
private static void runJob() {
String outputTableName = "A";
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.master", XXX);
conf.set("hbase.zookeeper.quorum", XXX);
conf.set("hbase.cluster.distributed", "true");
conf.set(TableOutputFormat.OUTPUT_TABLE, outputTableName);
try {
Job job = new Job(conf, "DataMockTask");
FileSystem hdfs = FileSystem.get(conf);
Path paths = new Path("./tmp/test");
if (!hdfs.exists(paths)) {
hdfs.create(paths);
}
FileInputFormat.setInputPaths(job, paths);
job.setInputFormatClass(TextInputFormat.class);
job.setJarByClass(DataMockTask.class);
job.setMapOutputKeyClass(NullWritable.class);
job.setMapOutputValueClass(Put.class);
job.setMapperClass(DataMockMapper.class);
job.setOutputFormatClass(TableOutputFormat.class);
job.setNumReduceTasks(0);
job.waitForCompletion(true);
} catch (Throwable e) {
throw new RuntimeException("Run DataFormatTask error! ", e);
} finally {
HConnectionManager.deleteConnection(conf, true);
}
}
Mapper:
@Override
public void run(Context context) throws IOException, InterruptedException {
setup(context);
long number = context.getConfiguration().getLong(HadoopConstants.DATA_NUM_KEY, 10000l);
for (long i = 0; i < number; i++) {
map(NullWritable.get(), NullWritable.get(), context);
}
cleanup(context);
}
@Override
protected void map(NullWritable key, NullWritable value, Context context) throws IOException,
InterruptedException {
String uid = UUID.randomUUID().toString();
Put put = new Put(Bytes.toBytes(uid));
put.add(Bytes.toBytes("E"), Bytes.toBytes("E"), Bytes.toBytes(System.currentTimeMillis()));
context.write(NullWritable.get(), put);
}