一.上篇博客有讲述从flume把数据写到Kafka的内容,但是当我们有多次重复代码要使用的时候,这个时候可以引入一种设计模式,把相同的代码通过接口的方式来封装起来,这样以后修改的时候,只需调用该接口实现我们需要的逻辑方法,即可
二.这里我们引用上次博客的代码来做案列分析
实现从Kafka导入到hbase中:
public class userAttendTohb {
public static void main(String[] args) {
//Kafka消费端属性 第一部分
Properties prop=new Properties();
prop.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,"192.168.195.20:9092");
prop.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
prop.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);
prop.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG,30000);
prop.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG,false);
prop.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG,1000);
prop.put(ConsumerConfig.GROUP_ID_CONFIG,"user");
prop.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(prop);
consumer.subscribe(Collections.singleton("user_friends"));
//配置hbase信息 连接hbase数据库 第二部分
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir","hdfs://192.168.195.20:9000/hbase");
conf.set("hbase.zookeeper.quorum","192.168.195.20");
conf.set("hbase.zookeeper.property.clientPort","2181");
try {
Connection connection = ConnectionFactory.createConnection(conf);
Table eventAttendTable =
connection.getTable(TableName.valueOf("events_db:user_friend"));
//消费数据到hbase里
while(true) {
//第三部分
ConsumerRecords<String, String> poll = consumer.poll(Duration.ofMillis(100));
List<Put> datas = new ArrayList<>(); //这个代码务必在局部变量里,放在外面会使机器爆掉
for(ConsumerRecord<String,String> p:poll){
System.out.println(p.value());
String[] split = p.value().split(",");
Put put = new Put(Bytes.toBytes((split[0] + split[1] ).hashCode()));
put.addColumn("uf".getBytes(),"userid".getBytes(),split[0].getBytes());
put.addColumn("uf".getBytes(),"id".getBytes(),split[1].getBytes());
datas.add(put);
}
eventAttendTable.put(datas);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
三.设计模式修改代码
1.定义一个接口IParseRecord(将上述代码第三部分封装)
/*
*组装者
* 将Kafka消费的信息,通过假工转换,得到list<put>对象,用于hbase存储使用
*/
public interface IParseRecord {
public List<Put> parse( ConsumerRecords<String, String> records);
}
2.实现这个接口(业务逻辑自己定)
public class UserFriendHandler implements IParseRecord {
@Override
public List<Put> parse(ConsumerRecords<String, String> records) {
List<Put> datas=new ArrayList<>();
for(ConsumerRecord<String,String> p:records){
System.out.println(p.value());
String[] split = p.value().split(",");
Put put = new Put(Bytes.toBytes((split[0] + split[1]).hashCode()));
put.addColumn("uf".getBytes(),"userid".getBytes(),split[0].getBytes());
put.addColumn("uf".getBytes(),"id".getBytes(),split[1].getBytes());
datas.add(put);
}
return datas;
}
}
3.在定义一个接口IWriter(第二部分封装)
public interface IWriter {
public int write(ConsumerRecords<String, String> records,String tableName) throws IOException;
}
4.实现这个接口
public class HbaseWriter implements IWriter {
private Configuration conf;
private IParseRecord parseRecord;
public IParseRecord getParseRecord() {
return parseRecord;
}
public void setParseRecord(IParseRecord parseRecord) {
this.parseRecord = parseRecord;
}
public HbaseWriter(IParseRecord parseRecord ) {
this.parseRecord = parseRecord;
conf = HBaseConfiguration.create();
conf.set("hbase.rootdir","hdfs://192.168.195.20:9000/hbase");
conf.set("hbase.zookeeper.quorum","192.168.195.20");
conf.set("hbase.zookeeper.property.clientPort","2181");
}
@Override
public int write(ConsumerRecords<String, String> records, String tableName) throws IOException {
Connection conn = ConnectionFactory.createConnection(conf);
Table eventTable = conn.getTable(TableName.valueOf(tableName));
List<Put> datas = parseRecord.parse(records);
eventTable.put(datas);
return datas.size();
}
}
5.在实现一个接口IWorker,使代码更加简洁,可读性更强(第一部分封装)
public interface IWorker {
public void fillData();
}
6.实现这个接口
public class HbaseWorker implements IWorker{
private IWriter writers;
private String topic;
private Properties prop;
private String target;
public HbaseWorker( IWriter writer,String topic,String targetTable) {
this(writer,"mygroupDefault",topic,targetTable);
}
// kafka 消费端属性
public HbaseWorker( IWriter writer,String groupName,String topic,String targetTable) {
prop = new Properties();
prop.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.195.20:9092");
prop.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
prop.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
prop.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 30000);
prop.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
prop.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, 1000);
prop.put(ConsumerConfig.GROUP_ID_CONFIG, "user");
prop.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
this.topic=topic;
this.target=targetTable;
this.writers=writer;
}
@Override
public void fillData() {
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(prop);
consumer.subscribe(Collections.singleton(this.topic));
try {
while(true){
ConsumerRecords<String, String> poll = consumer.poll(Duration.ofMillis(100));
Integer rowNum=writers.write(poll,this.target);
System.out.println("行数:"+rowNum);
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
7.最后的代码改为剩下的样子为:
public class UsefriendTohb2 {
public static void main(String[] args) {
IParseRecord record=new UserFriendHandler();
IWriter writer=new HbaseWriter(record);
IWorker worker=new HbaseWorker(writer,
"eventattend",
"user_friends",
"events_db:user_friend");
worker.fillData();
}
}
8.每次当我们需要重新更改需求的时候,只需要更改UserFriendHandler里的业务逻辑和UsefriendTohb2里的对应表名等,就可以直接生成数据,这样简洁明了,代码可读性更高,修改逻辑更加方便,值得拥有