Flume
drl_blogs
这个作者很懒,什么都没留下…
展开
-
示例:Spark Streaming+Flume整合
文章目录pushpullpushimport org.apache.log4j.{Level, Logger}import org.apache.spark.SparkConfimport org.apache.spark.streaming.dstream.ReceiverInputDStreamimport org.apache.spark.streaming.flume.{Flum...原创 2019-07-01 14:49:50 · 219 阅读 · 0 评论 -
flume常用配置文件
文章目录kafka-hdfs.confflume-kafka.confflume-hdfs.confflume_pull_streaming.confflume_push_streaming.confkafka-hdfs.conf#定义agent名, source、channel、sink的名称a1.channels = c1a1.sinks = k1a1.channels.c1.ty...原创 2019-07-09 14:24:13 · 576 阅读 · 0 评论 -
示例:python模拟日志生成+Flume+Kafka+Spark Streaming
生成模拟数据编写 generate_log.py#coding=UTF-8import randomimport timeurl_paths=[ "class/112.html", "class/128.html", "class/145.html", "class/130.html", "class/146.html", "cla...原创 2019-07-02 16:44:44 · 970 阅读 · 0 评论 -
示例:Log4j日志模拟数据整合 Flume+Kafka+Spark Streaming
flume_kafka.confagent1.sources = avro-sourceagent1.channels = logger-channelagent1.sinks = kafka-sink# define sourceagent1.sources.avro-source.type = avroagent1.sources.avro-source.bind = 0....原创 2019-07-01 21:22:06 · 258 阅读 · 0 评论 -
flume-1.9.0安装
上传 apache-flume-1.9.0-bin.tar.gz解压tar -zxvf apache-flume-1.9.0-bin.tar.gz -C /usr/local/配置环境变量vim /etc/profile添加内容export FLUME_HOME=/usr/local/apache-flume-1.9.0-binexport PATH=$PATH:$FL...原创 2019-06-26 15:14:29 · 979 阅读 · 0 评论