spark streaming + flume(Python版)

最新推荐文章于 2023-07-10 18:13:46 发布

瞳小妞x

最新推荐文章于 2023-07-10 18:13:46 发布

阅读量531

点赞数

                版权声明：本文为博主原创文章，未经博主允许不得转载。                    https://blog.csdn.net/u013305783/article/details/86219360                </div>
                      <link rel="stylesheet" href="https://csdnimg.cn/release/phoenix/template/css/ck_htmledit_views-cd6c485e8b.css">
                          <div id="content_views" class="markdown_views prism-atom-one-dark">
        <!-- flowchart 箭头图标 勿删 -->
        <svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
          <path stroke-linecap="round" d="M5,0 0,2.5 5,5z" id="raphael-marker-block" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0);"></path>
        </svg>
        <h1><a name="t0"></a><a id="flume_0" target="_blank"></a>一、flume安装</h1>

（一）概述

Flume是一个分布式、可靠、和高可用的海量日志采集、聚合和传输的系统。Flume可以采集文件，socket数据包等各种形式源数据，又可以将采集到的数据输出到HDFS、hbase、hive、kafka等众多外部存储系统中，一般的采集需求，通过对flume的简单配置即可实现，Flume针对特殊场景也具备良好的自定义扩展能力，因此flume可以适用于大部分的日常数据采集场景

（二）运行机制

1、 Flume分布式系统中最核心的角色是agent，flume采集系统就是由一个个agent所连接起来形成

2、每一个agent相当于一个数据传递员，内部有三个组件：

a)	Source：采集源，用于跟数据源对接，以获取数据
b)	Sink：下沉地，采集数据的传送目的，用于往下一级agent传递数据或者往最终存储系统传递数据
c)	Channel：angent内部的数据传输通道，用于从source将数据传递到sink

 
 
 
 1
2
3

在这里插入图片描述

（三）Flume采集系统结构图

1、简单结构

单个agent采集数据

在这里插入图片描述

2、复杂结构

多级agent之间串联
在这里插入图片描述

（四）Flume的安装部署

1、去apache官网上下载安装包，并解压tar -zxvf apache-flume-1.8.0-bin，并修改conf目录下flume-env.sh，在里面配置JAVA_HOME

2、根据数据采集的需求配置采集方案，描述在配置文件中(文件名可任意自定义)
3、指定采集方案配置文件，在相应的节点上启动flume agent

二、flume push方式

1、spark streaming程序

首先是flume通过push方式将采集到的数据传递到spark程序上，这种方式基本不用。spark代码如下：

import pyspark
from pyspark.sql import SparkSession
from pyspark.streaming import StreamingContext
from pyspark.streaming.flume import FlumeUtils

if name == “main”:
spark = SparkSession
.builder
.appName(“PythonWordCount”)
.master(“local[2]”)
.getOrCreate()
sc = spark.sparkContext
ssc = StreamingContext(sc, 5)
# hostname = sys.argv[1]
# port = int(sys.argv[2])
flumeStream = FlumeUtils.createStream(ssc, “localhost”, 8888, pyspark.StorageLevel.MEMORY_AND_DISK_SER_2)
line = flumeStream.map(lambda x: x[1])
words = line.flatMap(lambda x: x.split(" "))
datas = words.map(lambda x: (x, 1))
result = datas.reduceByKey(lambda agg, obj: agg + obj)
result.pprint()
ssc.start()
ssc.awaitTermination()

注意：要指定并行度，如在本地运行设置setMaster(“local[2]”)，相当于启动两个线程，一个给receiver，一个给computer。否则会出现如下问题

2019-01-09 19:36:16 INFO  ReceiverSupervisorImpl:54 - Called receiver 0 onStart
2019-01-09 19:36:16 INFO  ReceiverSupervisorImpl:54 - Waiting for receiver to be stopped
2019-01-09 19:36:20 INFO  JobScheduler:54 - Added jobs for time 1547033780000 ms
2019-01-09 19:36:25 INFO  JobScheduler:54 - Added jobs for time 1547033785000 ms
2019-01-09 19:36:30 INFO  JobScheduler:54 - Added jobs for time 1547033790000 ms
2019-01-09 19:36:35 INFO  JobScheduler:54 - Added jobs for time 1547033795000 ms
2019-01-09 19:36:40 INFO  JobScheduler:54 - Added jobs for time 1547033800000 ms

 
 
 
 1
2
3
4
5
6
7

如果是在集群中运行，必须要求集群中可用core数大于1

2、flume conf文件

<font size=4>&emsp; &emsp;在flume的conf目录下新建flume-push.conf内容如下</font></br>
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

source

a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /home/hadoop/log/flume
a1.sources.r1.fileHeader = true

Describe the sink

a1.sinks.k1.type = avro
#这是接收方
a1.sinks.k1.hostname = 192.168.62.131
a1.sinks.k1.port = 8888

Use a channel which buffers events in memory

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

Bind the source and sink to the channel

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

需要先将spark程序运行，使用以下命令：

spark/bin/spark-submit  --driver-class-path /home/hadoop/spark/jars/*:/home/hadoop/jar/flume/* /tmp/pycharm_project_563/day5/FlumePushWordCount.py

 
 
 
 1

可能会出现以下问题

	Spark Streaming's Flume libraries not found in class path. Try one of the following.

Include the Flume library and its dependencies with in the
spark-submit command as

$ bin/spark-submit --packages org.apache.spark:spark-streaming-flume:2.4.0 …
Download the JAR of the artifact from Maven Central http://search.maven.org/,
Group Id = org.apache.spark, Artifact Id = spark-streaming-flume-assembly, Version = 2.4.0.
Then, include the jar in the spark-submit command as

$ bin/spark-submit --jars <spark-streaming-flume-assembly.jar> …
Traceback (most recent call last):
File “/tmp/pycharm_project_563/day5/FlumePushWordCount.py”, line 12, in <module>
flumeStream = FlumeUtils.createStream(ssc, “192.168.62.131”, “8888”)
File “/home/hadoop/spark/python/pyspark/streaming/flume.py”, line 67, in createStream
helper = FlumeUtils._get_helper(ssc._sc)
File “/home/hadoop/spark/python/pyspark/streaming/flume.py”, line 130, in _get_helper
return sc._jvm.org.apache.spark.streaming.flume.FlumeUtilsPythonHelper()
TypeError: ‘JavaPackage’ object is not callable

需要去maven仓库下载spark-streaming-flume-assembly.jar，然后放到上面指定的jar目录中去。

然后运行flume

bin/flume-ng agent -n a1 -c conf/ -f conf/flume-push.conf -Dflume.root.logger=WARN,console

 
 
 
 1

然后在/home/hadoop/log/flume目录下新建log文件，运行spark的日志中出现如下：

在这里插入图片描述

三、poll方式

1、spark streaming程序

这种方式是有spark主动去flume拉取数据，代码如下：

from pyspark.sql import SparkSession
from pyspark.streaming import StreamingContext
from pyspark.streaming.flume import FlumeUtils

if name == “main”:
spark = SparkSession
.builder
.appName(“PythonWordCount”)
.master(“local[2]”)
.getOrCreate()
sc = spark.sparkContext
ssc = StreamingContext(sc, 5)
addresses = [(“localhost”, 8888)]
flumeStream = FlumeUtils.createPollingStream(ssc, addresses)
line = flumeStream.map(lambda x: x[1])
words = line.flatMap(lambda x: x.split(" "))
datas = words.map(lambda x: (x, 1))
result = datas.reduceByKey(lambda agg, obj: agg + obj)

result.pprint()
ssc.start()
ssc.awaitTermination()

如果是本地模式同样需要指定并行度，如果是在集群中运行，必须要求集群中可用core数大于1

2、flume conf文件

在flume的conf目录下新建flume-poll.conf内容如下:

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# source
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /home/hadoop/log/flume
a1.sources.r1.fileHeader = true

Describe the sink

a1.sinks.k1.type = org.apache.spark.streaming.flume.sink.SparkSink
a1.sinks.k1.hostname = localhost
a1.sinks.k1.port = 8888

Use a channel which buffers events in memory

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

Bind the source and sink to the channel

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

由于是poll方式，需要的flume

bin/flume-ng agent -n a1 -c conf/ -f conf/flume-poll.conf -Dflume.root.logger=WARN,console

 
 
 
 1

启动spark程序

spark/bin/spark-submit  --driver-class-path /home/hadoop/spark/jars/*:/home/hadoop/jar/flume/* /tmp/pycharm_project_563/day5/FlumePollWordCount.py 

 
 
 
 1

同样在/home/hadoop/log/flume目录下新建log文件，将原先生成的COMPLETED文件删除，rm flume/aaa.txt.COMPLETED ，运行spark的日志中出现如下：

在这里插入图片描述

      </div>
      <link href="https://csdnimg.cn/release/phoenix/mdeditor/markdown_views-e44c3c0e64.css" rel="stylesheet">
              </div>

瞳小妞x

关注

0
点赞
踩
4

收藏

觉得还不错? 一键收藏
0
评论
spark streaming + flume(Python版)

版权声明：本文为博主原创文章，未经博主允许不得转载。 h...
复制链接

扫一扫

spark streaming + flume(Python版)

（一）概述

（二）运行机制

（三）Flume采集系统结构图

1、简单结构

2、复杂结构

（四）Flume的安装部署

二、flume push方式

1、spark streaming程序

2、flume conf文件

source

Describe the sink

Use a channel which buffers events in memory

Bind the source and sink to the channel

三、poll方式

1、spark streaming程序

2、flume conf文件

Describe the sink

Use a channel which buffers events in memory

Bind the source and sink to the channel

“相关推荐”对你有帮助么？