Knative接收Kafka消息，通过KafkaSource和基于Java实现

最新推荐文章于 2024-01-13 14:36:31 发布

AaronXym

最新推荐文章于 2024-01-13 14:36:31 发布

阅读量729

点赞数

文章标签： knative KafkaSource

本文链接：https://blog.csdn.net/Crazy_xym/article/details/108261030

版权

Knative接收Kafka消息

knative可以通过POST接收cloudevents格式的HTTP请求，同时knative提供kafkaSource，kafkaChannel，kafkaBinding的方式接收普通kafka消息并转换成cloudevents发到knative service。这里提供基于Java的实现以及kafkaSource的实现。

Java

项目搭建

首先下载cloudevents-java SDK ，然后利用maven将这4个包进行打包，然后手动引入项目在这里插入图片描述

接下来pom这么写，因为向Knative Broker发送HTTP请求，所以这里使用OKHttp3

        <dependency>
            <groupId>io.cloudevents</groupId>
            <artifactId>cloudevents-kafka</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>io.cloudevents</groupId>
            <artifactId>cloudevents-json-jackson</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.8.6</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.11</version>
        </dependency>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.25</version>
            <scope>compile</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.squareup.okhttp3/okhttp -->
        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>okhttp</artifactId>
            <version>4.8.0</version>
        </dependency>

在打包方式上按下面

<build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifest>
                            <!--这里要替换成jar包main方法所在类 -->
                            <mainClass>com.aaron.consumer.MultiTopicListener</mainClass>
                        </manifest>
                        <manifestEntries>
                            <Class-Path>.</Class-Path>
                        </manifestEntries>
                    </archive>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id> <!-- this is used for inheritance merges -->
                        <phase>package</phase> <!-- 指定在打包节点执行jar包合并操作 -->
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

准备工作完成后，开始解决Java如何监听

Java监听多topic

首先得解决如何获取bootstrapServer以及topic，因为Java程序是会放到k8s中运行的，所以在yaml中进行参数编写。标签是env具体deployment.yaml如下

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-cloudevent-test-deploy
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-cloudevent-test
      release: stabel
  template:
    metadata:
      labels:
        app: kafka-cloudevent-test
        release: stabel
        env: test
    spec:
      containers:
        - name: kafka-cloudevent-test
          image: docker.io/kafka-cloudevents-test:1.0
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 80
          env:
            - name: broker
              value: http://broker-ingress.knative-eventing.svc.cluster.local/default/kafka-broker
            - name: kafka.configs
              value: "[\"localhost:9092#raw\",\"localhost:9092#test\"]"

这里为了简便topic和bootstrapServer的编写工作，将其以bootstrapServer#topic的形式组成一个字符串。上面env的作用就是为app设置环境参数，这样Java可以使用System.getenv()获取到kafka集群参数。
这里为了优化代码结构，创建了KafkaConfig和KnativeConfig两个POJO

class KafkaConfig{
    private String bootStrapServer;
    private String topic;
}

class KnativeConfig{
    public static final String BROKER = "broker";
    private String broker;
}

这里broker是指在knative中broker通过ingress接收HTTP请求，因此这个broker地址就是ingress的url，可以通过kubectl get broker查看在这里插入图片描述

KafkaUtil

public class KafkaUtil {
    private static Logger logger = LoggerFactory.getLogger(KafkaUtil.class);
    private static final String BOOTSTRAP_SERVERS_CONFIG = "bootstrap.servers";
    private static final String TOPIC = "topic";
    private static final String KAFKA_CONFIGS = "kafka.configs";
    private static Gson gson = new GsonBuilder().create();

    public static KnativeConfig getKnativeConfig(){
        Map<String,String> env = System.getenv();
        KnativeConfig config = new KnativeConfig();
        String broker = env.get(KnativeConfig.BROKER);
        if (StringUtils.isNotBlank(broker)){
            config.setBroker(broker);
            return config;
        }
        throw new IllegalArgumentException("broker can not be null");
    }

    public static List<KafkaConfig> getMultiKafkaConfig(){
        Map<String,String> env = System.getenv();
        String configsJson = env.get(KAFKA_CONFIGS);
        if (StringUtils.isNotBlank(configsJson)){
            String[] kafkaConfigs = gson.fromJson(configsJson,String[].class);
            return Arrays.stream(kafkaConfigs).map(KafkaUtil::split).collect(Collectors.toList());
        }
        throw new IllegalArgumentException("kafka.configs can not be null");
    }

    /**
     * 将bootstrapServer#topic字符串分割
     * @param strs 字符串
     * @return 0为bootstrapServer 1为topic
     */
    public static KafkaConfig split(String strs){
        String[] args = strs.split("#");
        return new KafkaConfig(args[0],args[1]);
    }

    public static KafkaConsumer<String,String> createConsumer(KafkaConfig config){
        Properties properties = new Properties();
        properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,config.getBootStrapServer());
        properties.put(ConsumerConfig.GROUP_ID_CONFIG, "sample-cloudevents-consumer");
        properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        // 这里对Kafka消息解析
        properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
        properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
        KafkaConsumer<String,String> consumer = new KafkaConsumer<>(properties);
        consumer.subscribe(Collections.singletonList(config.getTopic()));
        return consumer;
    }
}

CloudEventUtil

public class CloudEventUtil {
    private static Logger logger = LoggerFactory.getLogger(KafkaCloudEventsConverter.class);
    private static final String TYPE = "dev.knative.kafka.event";
    private static final String JSON = "application/json";
    private static final String CE_JSON = "application/cloudevents+json; charset=UTF-8";
    private static final String SUBJECT_PATTERN = "partition:%s#%s";
    private static final String SOURCE = "%s#%s";
    private static Gson gson = new GsonBuilder().create();


    public static Request buildCloudEventRequest(ConsumerRecord<String,String> record, String url){
        Request.Builder builder = new Request.Builder();
        String subject = String.format(SUBJECT_PATTERN,record.partition(),record.offset());
        String source = String.format(SOURCE,KafkaUtil.getKafkaConfig()[0],record.topic());
        MediaType jsonType = MediaType.parse("application/json");
        Headers.Builder headerBuilder = new Headers.Builder();
        RequestBody requestBody = RequestBody.create(jsonType,gson.toJson(record.value()));
        headerBuilder.add("Ce-Specversion","1.0");
        headerBuilder.add("Ce-Type",TYPE);
        headerBuilder.add("Ce-Source",source);
        headerBuilder.add("Content-Type",JSON);
        headerBuilder.add("Ce-Subject",subject);
        return builder.headers(headerBuilder.build()).post(requestBody).url(url).build();
    }


    public static void sendMsgToBroker(ConsumerRecord<String,String> record, String broker){
        OkHttpClient client = HttpUtil.getClient();
        Request brokerRequest = CloudEventUtil.buildCloudEventRequest(record,broker);
        logger.info("发送---");
        try (Response response = client.newCall(brokerRequest).execute()){
        } catch (IOException e) {
            logger.error("发送失败：{}",e.getMessage());
        }
    }
}

public class HttpUtil {
    private static OkHttpClient client;

    /**
     * 单线程操作不需要DCL
     * @return OkHttpClient
     */
    public static OkHttpClient getClient() {
        if (client == null){
            client = new OkHttpClient.Builder().build();
        }
        return client;
    }
}

编写监听代码

public class MultiTopicListener implements Runnable{
    Logger logger = LoggerFactory.getLogger(MultiTopicListener.class);
    static Gson gson = new GsonBuilder().create();

    @Override
    public void run(){
        String broker = KafkaUtil.getKnativeConfig().getBroker();
        List<KafkaConfig> kafkaConfigs = KafkaUtil.getMultiKafkaConfig();
        List<KafkaConsumer> consumers = kafkaConfigs.stream().map(KafkaUtil::createConsumer).collect(Collectors.toList());
        while (true){
            consumers.forEach(consumer -> {
                logger.info("开始监听");
                ConsumerRecords<String,String> record = consumer.poll(Duration.ofMillis(1000));
                record.forEach(r -> {
                    CloudEventUtil.sendMsgToBroker(r,broker);
                });
            });
        }
    }

    public static void main(String[] args) {
        Thread thread = new Thread(new MultiTopicListener());
        thread.start();
    }
}

打包以POD方式运行在K8S上

这里使用Docker进行镜像制作，通过dockerhub管理镜像，没有的同学自己装下。
Dockerfile

FROM openjdk:8-jdk-alpine

#容器中创建目录
RUN mkdir -p /home/user/kafka-cloudevent-test

#编译后的jar包copy到容器中创建到目录内
COPY kafka-cloudevent-test-1.0-SNAPSHOT-jar-with-dependencies.jar /home/user/kafka-cloudevent-test/kafka-cloudevent-test-1.0-SNAPSHOT-jar-with-dependencies.jar

#指定容器启动时要执行的命令
ENTRYPOINT ["java","-jar","/home/user/kafka-cloudevent-test/kafka-cloudevent-test-1.0-SNAPSHOT-jar-with-dependencies.jar"]

再把maven pkg做好的jar放到服务器上。使用下面命令制作镜像并上传到dockerhub
docker build -f Dockerfile -t kafka-cloudevents-test:1.0 .
docker push kafka-cloudevents-test
编写deployment和service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-cloudevent-test-deploy
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-cloudevent-test
      release: stabel
  template:
    metadata:
      labels:
        app: kafka-cloudevent-test
        release: stabel
        env: test
    spec:
      containers:
        - name: kafka-cloudevent-test
          image: docker.io/kafka-cloudevents-test:1.0
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 80
          env:
            - name: broker
              value: http://broker-ingress.knative-eventing.svc.cluster.local/default/kafka-broker
            - name: kafka.configs
              value: "[\"localhost:9092#raw\",\"localhost:9092#test\"]"

apiVersion: v1
kind: Service
metadata:
  name: kafka-cloudevent-test
  namespace: default
spec:
  type: NodePort
  selector:
    app: kafka-cloudevent-test
    release: stabel
  ports:
    - name: http
      port: 8080
      targetPort: 8080
      nodePort: 30080
  ref:
    apiVersion: eventing.knative.dev/v1beta1
    kind: Broker
    name: kafka-source-broker

使用kubectl apply -f deployment和kubectl apply -f service.yaml运行

到这里就结束了，可以测试一波。

KafkaSource

kafkaSource的deployment.yaml如下

apiVersion: sources.knative.dev/v1alpha1
kind: KafkaSource
metadata:
  name: kafka-source
spec:
  consumerGroup: knative-group
  bootstrapServers: ["localhost:9092","localhost:9092"]
  topics: ["raw","test"]
  sink:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: event-display

kubectl apply -f deployment执行，这个sink标签就是指EventSource向具体哪个应用发数据。这里采用了默认的event-display进行数据展示。

参考资料及命令

https://knative.dev/v0.15-docs/eventing/samples/kafka/source/
kubectl get po获取pod
kubectl get deployment获取deployment
kubectl describe pod xxxxx获取xxxxx pod的状态
kubectl delete deployment xxxxx删除xxxxx的部署，这时对应的pod也会被删除
kubectl delete -f deployment.yaml删除该deployment对应的部署。

AaronXym

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
Knative接收Kafka消息，通过KafkaSource和基于Java实现

Knative接收Kafka消息knative可以通过POST接收cloudevents格式的HTTP请求，同时knative提供kafkaSource，kafkaChannel，kafkaBinding的方式接收普通kafka消息并转换成cloudevents发到knative service。这里提供基于Java的实现以及kafkaSource的实现。Java项目搭建首先下载cloudevents-java SDK ，然后利用maven将这4个包进行打包，然后手动引入项目接下来pom这么写，
复制链接

扫一扫