一、需求背景
在做某项目时，遇到一个需求是这样的：每天产生的预演数据会存放在hdfs中某个目录，文件名假设为preview20200723，这个文件在当天可能会一直有数据在追加（间断性），也可能一次性写完（持续性），需要利用现有的技术监控这个目录中数据的变化，将获取到的json数据做解析再保留到数仓中（此部分为Spark编辑部分，本文不做测试）。
二、技术选型
flume + spark streaming，后期可以再添加kafka做个缓存机制，实现高可用性。
三、实现步骤
注：本文所做的代码实现，仅仅是测试，只实现整体的思路。实际应用可以根据需要修改配置和部分代码。
1、poll方式
1.1、安装flume
下载链接http://www.apache.org/dyn/closer.lua/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
本文使用的是最新版本1.9（但其实已经有一年没有更新版本了）。
注：如果使用1.9以下版本，就需要在lib目录中添加scala-library-2.11.12.jar。
1.2、配置flume文件
flume-poll-spark.conf
a1.sources = r1
a1.sinks = k1
a1.channels = c1
#source
a1.sources.r1.channels = c1
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /root/logs
a1.sources.r1.fileHeader = true
#尝试使用端口发送信息来测试流程，不过需要另开一个命令窗口，打开44444端口
#a1.sources.r1.type = netcat
#a1.sources.r1.bind = localhost
#a1.sources.r1.port = 44444
#channel
a1.channels.c1.type =memory
a1.channels.c1.capacity = 20000
a1.channels.c1.transactionCapacity=5000
#sinks
a1.sinks.k1.channel = c1
a1.sinks.k1.type = org.apache.spark.streaming.flume.sink.SparkSink
a1.sinks.k1.hostname=localhost
a1.sinks.k1.port = 8888
a1.sinks.k1.batchSize= 2000
值得注意的是，这里的sink是根据org.apache.spark.streaming.flume.sink.SparkSink这个类来创建的，而这个类是需要导入spark-sink的包的，本文使用的是spark-streaming-flume-sink_2.11-2.0.2.jar（一开始我导的时候没有看清楚是有加sink的，所以一直很纳闷为什么会报连接不上地址的错误，请各位小伙伴导包要看仔细）。
将下载好的jar包放在flume根目录下面的lib目录中，在flume启动的时候会去寻找SparkSink类，然后flume会自行创建，并根据配置文件传入hostname和port。有兴趣的可以看看spark-streaming-flume-sink_2.11-2.0.2.jar中的SparkSink源码。
1.3、启动flume
bin/flume-ng agent -n a1 -c conf -f conf/flume-poll-spark.conf -Dflume.root.logger=INFO,console
在windows上启动，需要将-Dflume.root.logger=INFO,console去掉，并修改斜杠
1.4、导入依赖
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.steven</groupId>
<artifactId>spark-demo</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<scala.version>2.11.8</scala.version>
<hadoop.version>2.7.4</hadoop.version>
<spark.version>2.0.2</spark.version>
</properties>
<dependencies>
<!--配置scala类库-->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<!--配置spark核心依赖-->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<!--配置hadoop-->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<!--配置mysql驱动-->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.41</version>
</dependency>
<!--配置spark sql-->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-flume -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-flume_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
</dependencies>
<build>
<!--配置scala主目录-->
<sourceDirectory>src/main/scala</sourceDirectory>
<!--配置scala测试目录-->
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<!--配置scala的Maven插件-->
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<!--配置maven的shade插件-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass></mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
1.5、代码实现
package com.steven.spark.streaming
import java.net.InetSocketAddress
import org.apache.hadoop.fs.Path
import org.apache.hadoop.io.{LongWritable, Text}
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat
import org.apache.spark.serializer.KryoSerializer
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.dstream.{DStream, InputDStream, ReceiverInputDStream}
import org.apache.spark.streaming.flume.{FlumeUtils, SparkFlumeEvent}
import org.apache.spark.streaming.{Durations, StreamingContext}
import org.apache.spark.{SparkConf, SparkContext}
/**
* author:seven lin
* date:2020/6/1422:56
* description:
**/
object ListenFile {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("sparkstreamingfile").setMaster("local[2]")
val sc = new SparkContext(conf)
sc.setLogLevel("WARN")
//创建一个streamingcontext对象，并设置批次间隔时间
val ssc = new StreamingContext(sc, Durations.seconds(5))
//设置监听的地址
val address = Seq(new InetSocketAddress("192.168.25.161", 8888))
//获取flume中数据
val stream: ReceiverInputDStream[SparkFlumeEvent] = FlumeUtils.createPollingStream(ssc,address,StorageLevel.MEMORY_AND_DISK)
//从Dstream中获取flume中的数据 {"header":xxxxx "body":xxxxxx}
val lineDstream: DStream[String] = stream.map(x => new String(x.event.getBody.array()))
//打印内容
lineDstream.print()
ssc.start()
ssc.awaitTermination()
}
}
1.6、启动spark
这里只做测试，没有修改配置，就默认配置。默认的driver和executor内存大小为1G。
spark-submit --class com.steven.spark.streaming.ListenFile spark-demo-1.0-SNAPSHOT.jar
1.7、结果展示
6.txt内容
nihaoma
helloworld
spark is niubi
将6.txt直接放到
-------------------------------------------
Time: 1595083560000 ms
-------------------------------------------
nihaoma
helloworld
spark is niubi
-------------------------------------------
Time: 1595083570000 ms
-------------------------------------------
pull方式要先启动flume再启动spark，push方式则相反。
2、push方式
2.1、安装flume
同1.1
2.2、配置flume文件
flume-push-spark.conf
a1.sources = r1
a1.sinks = k1
a1.channels = c1
#source
a1.sources.r1.channels = c1
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /root/logs
a1.sources.r1.fileHeader = true
#channel
a1.channels.c1.type =memory
a1.channels.c1.capacity = 20000
a1.channels.c1.transactionCapacity=1000
#sinks
a1.sinks.k1.channel = c1
a1.sinks.k1.type = avro
a1.sinks.k1.hostname=192.168.25.161
a1.sinks.k1.port = 8888
a1.sinks.k1.batchSize= 1000
2.3、导入依赖
同1.4
2.4、代码实现
将获取数据的方法修改如下：
//接收flume的数据
val stream: ReceiverInputDStream[SparkFlumeEvent] = FlumeUtils.createStream(ssc,"192.168.25.161",8888,StorageLevel.MEMORY_AND_DISK)
2.5、启动spark
同1.6
2.6、启动flume
bin/flume-ng agent -n a1 -c conf -f conf/flume-push-spark.conf -Dflume.root.logger=INFO,console
2.7、结果展示
同1.7
总结
1、使用过程中出现过以下问题：
2020-07-18 20:12:05,701 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2020-07-18 20:12:09,702 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
原因是配置的channel容量太小，以至于我将一个较大文件直接存放的话，容量占满，而sink端原先我是没有设置批处理量的，导致消费速度跟不上。调整batchsize为1000之后解决。
2、注意poll方式的时候，flume创建的sink类型为sparksink，所以要导入相应的jar包，否则它启动之后创建不了。
3、在使用之前，查看一下端口是否有被占用。查看命令ss -lntpd | grep :8888。
扫一扫
390
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
