Huid学习六：Structured Streaming实时写Hudi

Hub-Link

已于 2022-06-18 23:15:03 修改

阅读量935

点赞数

分类专栏：数据湖文章标签：大数据 spark

于 2022-06-18 23:12:11 首次发布

本文链接：https://blog.csdn.net/NC_NE/article/details/125352954

版权

一、整体架构

二、创建Kafka生产者，模拟生产数据

1、创建hudi_kafka topic

bin/kafka-topics.sh --zookeeper 192.168.74.100:2181 --create --replication-factor 1 --partitions 3 --topic hudi_kafka

2、模拟生产数据

/**
 * @author oyl
 * @create 2022-06-18 17:35
 * @Description 摸你生产kafka数据
 */
public class HudiProducer {
    
    public static void main(String[] args) {

        Properties props = new Properties();
        props.put("bootstrap.servers", "hadoop100:9092,hadoop101:9092,hadoop102:9092");
        props.put("acks", "-1");
        props.put("batch.size", "1048576");
        props.put("linger.ms", "5");
        props.put("compression.type", "snappy");
        props.put("buffer.memory", "33554432");
        props.put("key.serializer",
                "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer",
                "org.apache.kafka.common.serialization.StringSerializer");

        KafkaProducer<String,String> producer = new KafkaProducer<>(props);

        Random random = new Random();
        for (int i = 0; i < 1000; i++)