![](https://img-blog.csdnimg.cn/20201014180756923.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
Kafka
drl_blogs
这个作者很懒,什么都没留下…
展开
-
示例:Spark Streming+Kafka整合(spark-streaming-kafka-0-10_2.11)
import org.apache.kafka.common.serialization.StringDeserializerimport org.apache.log4j.{Level, Logger}import org.apache.spark.SparkConfimport org.apache.spark.streaming.kafka010.ConsumerStrategies....原创 2019-07-05 11:01:03 · 2826 阅读 · 0 评论 -
kafka_2.11-0.11.0.0 安装配置
1.上传kafka_2.11-0.11.0.0.tgz2.解压tar -zxvf kafka_2.11-0.11.0.0.tgz -C /usr/local/3.修改server.propertiesvim server.properties修改内容如#broker的全局唯一编号,不能重复broker.id=0#删除topic功能使能delete.topic.enable=t...原创 2019-06-24 10:32:18 · 738 阅读 · 0 评论 -
示例:python模拟日志生成+Flume+Kafka+Spark Streaming
生成模拟数据编写 generate_log.py#coding=UTF-8import randomimport timeurl_paths=[ "class/112.html", "class/128.html", "class/145.html", "class/130.html", "class/146.html", "cla...原创 2019-07-02 16:44:44 · 955 阅读 · 0 评论 -
示例:Log4j日志模拟数据整合 Flume+Kafka+Spark Streaming
flume_kafka.confagent1.sources = avro-sourceagent1.channels = logger-channelagent1.sinks = kafka-sink# define sourceagent1.sources.avro-source.type = avroagent1.sources.avro-source.bind = 0....原创 2019-07-01 21:22:06 · 246 阅读 · 0 评论 -
示例:Spark Streming+Kafka整合 (spark-streaming-kafka-0-8_2.11)
文章目录ReceiverDirectReceiverimport org.apache.log4j.{Level, Logger}import org.apache.spark.SparkConfimport org.apache.spark.streaming.kafka.KafkaUtilsimport org.apache.spark.streaming.{Seconds, Str...原创 2019-07-01 17:24:20 · 2239 阅读 · 0 评论 -
示例:Kafka Stream 数据清洗
接受数据后的处理逻辑import org.apache.kafka.streams.processor.Processor;import org.apache.kafka.streams.processor.ProcessorContext;public class LogProcessor implements Processor<byte[], byte[]> { p...原创 2019-06-25 16:07:44 · 1147 阅读 · 0 评论 -
示例:kafka拦截器
拦截内容在内容前加时间戳import java.util.Map;import org.apache.kafka.clients.producer.ProducerInterceptor;import org.apache.kafka.clients.producer.ProducerRecord;import org.apache.kafka.clients.producer.Re...原创 2019-06-25 14:58:48 · 293 阅读 · 0 评论