Java读kafka数据写入hbase(清澈)

最新推荐文章于 2024-08-02 18:24:49 发布

Giao.芥末77

最新推荐文章于 2024-08-02 18:24:49 发布

阅读量533

点赞数

文章标签： kafka hbase

本文链接：https://blog.csdn.net/suqun7/article/details/115603966

版权

文章目录

maven

<dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.11</artifactId>
            <version>2.0.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.0.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-client -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>1.2.0</version>
        </dependency>

注意事项

重置消费者组的游标

kafka-consumer-groups.sh --bootstrap-server single:9092 --group sq7-reset-offsets --all-topics --to-earliest --execute

做主机映射
java 连接hbase 可以加快连接速度 C:\Windows\System32\drivers\etc\hosts

代码实现

创建hbase表

hbase(main):001:0> create 'intes:event_attendees_hb','base'
hbase(main):001:0> create 'intes:users_hb','base'
hbase(main):001:0> create 'intes:train_hb','base'
hbase(main):001:0> create 'intes:user_friends_hb','base'
hbase(main):001:0> create 'intes:events_hb','base','other'

接口层读写

//读kafka的接口
public interface Read {
   
    void readKafka(Properties prop, String topic);
}

//往hbase中写的接口
public interface Write {
   
    void saveDataToHbase(ConsumerRecords<String, String> records);
}

工具类

分别创建kafka的消费者和hbase的连接connection

public class KafkaUtils {
   
    //kafka读 获得一个消费者
    public static KafkaConsumer<String, String> createConsumer(Properties prop) {
   
        return new KafkaConsumer<String, String>(prop);
    }
}

//hbase 的connection,连接池
public class HbaseUtils {
   
    public static Connection getConnection(Configuration conf) throws IOException {
   
    //使用连接池，每个线程创建一个连接，使用完后不关，因此将connection建在外面
        ExecutorService es = Executors.newFixedThreadPool(10);
        Connection connection = ConnectionFactory.createConnection(conf, es);
        return connection;
    }
}

//配置类
public class HbaseConf {
   
    public static Configuration getconf() {
   
        Configuration configuration = HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.quorum", "192.168.171.200:2181");
        return configuration;
    }
}

读kafka的实现类

public class KafkaReadImpl implements Read {
   
    private Write write;

    //构造器
    public KafkaReadImpl(Write write) {
   
        this.write = write;
    }

    @Override
    public void readKafka(Properties prop, String topic) {
   
        KafkaConsumer<String, String> consumer = KafkaUtils.createConsumer(prop);
        consumer.subscribe(Arrays.asList(topic));
        while (true) {
   
            //每100ms来一批数据，创建一个connection
            ConsumerRecords<String, String> res = consumer.poll(Duration.ofMillis(100));