Flink消费kafka数据-java

最新推荐文章于 2024-05-23 10:57:27 发布

熬夜的小王

最新推荐文章于 2024-05-23 10:57:27 发布

阅读量622

点赞数

分类专栏：大数据专栏文章标签： kafka java flink

本文链接：https://blog.csdn.net/weixin_44947939/article/details/128320151

版权

大数据专栏专栏收录该内容

10 篇文章 0 订阅

订阅专栏

Flink消费kafka数据-java

1.pom文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.wgy</groupId>
    <artifactId>Project0818</artifactId>
    <version>1.0-SNAPSHOT</version>
   <dependencies>
       <dependency>
           <groupId>org.apache.flink</groupId>
           <artifactId>flink-java</artifactId>
           <version>1.9.0</version>
       </dependency>
       <dependency>
           <groupId>org.apache.flink</groupId>
           <artifactId>flink-streaming-java_2.11</artifactId>
           <version>1.9.0</version>
       </dependency>
       <dependency>
           <groupId>org.apache.flink</groupId>
           <artifactId>flink-connector-kafka_2.11</artifactId>
           <version>1.9.0</version>
       </dependency>
       <dependency>
           <groupId>org.apache.flink</groupId>
           <artifactId>flink-connector-hive_2.11</artifactId>
           <version>1.9.0</version>
       </dependency>
   </dependencies>
</project>

2.代码

import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;

import java.util.Properties;

public class StreamWordCount
{
    public static void main(String[] args) throws Exception {
        Properties properties = new Properties();
        properties.setProperty("bootstrap.servers","hadoop102:9092");
        properties.setProperty("group.id","consumer-group");
        properties.setProperty("key.deserializer","org.apache.kafka.common.serialization.StringSerializer");
        properties.setProperty("value.deserializer","org.apache.kafka.common.serialization.StringSerializer");
        properties.put("auto.offset.rest", "latest");
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStream<String> dataStream = env.addSource(new FlinkKafkaConsumer<String>("first", new SimpleStringSchema(), properties));
        dataStream.print();
        env.execute();
    }
}

3.结果

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
1> 
1> lala
1> hehe
1> hello java