spark streaming任务,读kafka写入mysql

最新推荐文章于 2022-04-24 09:04:27 发布

灰二和杉菜

最新推荐文章于 2022-04-24 09:04:27 发布

阅读量2.0k

点赞数

分类专栏： Apache Spark 文章标签： spark streaming checkpoint mysql kafka

本文链接：https://blog.csdn.net/qq475781638/article/details/102785851

版权

本文介绍了如何使用Spark Streaming直接方式读取Kafka消息，配置checkpoint，并将数据写入MySQL。详细过程包括发送数据到Kafka、Spark Streaming程序的消费逻辑、依赖的MySQL连接池和插入操作，以及程序的运行步骤。

摘要由CSDN通过智能技术生成

1.背景

spark streaming direct方式读取kafka消息，设置checkpoint 并写入mysql

2.发送数据到kafka

package com.bigdata.kafka;

import java.util.Properties;
import java.util.concurrent.TimeUnit;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;

public class KafkaSend {
   

    public static void main(String[] args) {
   
        Properties props = new Properties();
        props.put("bootstrap.servers", "dn1:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        //生产者发送消息
        String topic = "ss";//topic
        Producer<String, String> procuder = new KafkaProducer<String,String>(props);
        for (int i = 1; i <= 10000; i++) {
   
            String value = "value_" + i;
            ProducerRecord<String, String> msg = new ProducerRecord<String, String>(topic, value);
            procuder.send(msg);
            try {
   
                Thread.sleep(1000);
                System.out.println(msg.toString());
            } catch (InterruptedException e) {
   
                e.printStackTrace();
            }
        }
        System.out.println("send message over.");
        procuder.close(100,TimeUnit.MILLISECONDS);
    }
}