用 Flume 收集实时点击日志,以 Http 请求的形式 Post Json 数据,传入 Flume,通过 SparkStreaming 对数据进行处理,此处 SparkStreaming 于 Flume 对接的方式是 Push,所以启动顺序是先启动 SparkStreaming 再 启动 Flume
添加依赖到 Maven
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>groupId</groupId>
<artifactId>KanChaNiDeLian</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<spark.version>2.2.0</spark.version>
<scala.version>2.11</scala.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-flume -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-flume_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-flume-sink -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-flume-sink_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-flume-assembly -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-flume-assembly_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.google.code.gson/gson -->
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.5</version>
</dependency>
<!--解析json字符串-->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.36</version>
</dependency>
</dependencies>
<repositories>
<repository>
<id>alimaven</id>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
</repository>
</repositories>
<build>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.15.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
</plugins>
</build>
</project>
配置后台代码发送 Http 请求
在 Django 中发送 Post 请求, views.py 如下,需要注意的是,向 Flume 发送的 Json 的格式必须是 [{“headers”:{}, “body”:{}}, …]
from django.shortcuts import HttpResponse, render
import json
import requests
import random
# 预加载模型
# abc = random.randint(1, 10)
# print(abc)
def get_json(request):
# global abc
random_info = random.randint(1, 10)
# 需要注意flume传输的json的格式
j = [{
"headers": {
"userId": '233',
"age": '20',
"name": 'madala',
"itemId": '777',
"click": '1',
},
"body": "abc abc df df"
}]
# print(abc)
json_data = json.dumps(j)
# 发送请求
headers = {'Content-Type': 'application/json;charset=UTF-8'}
res = requests.post('http://localhost:5555', data=json_data, headers=headers)
print(res)
return HttpResponse(json_data, content_type='application/json')
配置 Flume
配置 Flume 的 Agent 大致步骤:
- 配置 Source
- 配置 Sink
- 配置 Chanel
- 连接 Chanel 和 Source,Chanel 和 Sink
a1.sources = r1
a1.sinks = k1
a1.channels = c1
a1.sources.r1.type = http
a1.sources.r1.bind = localhost
a1.sources.r1.port = 5555
# 注意这里是 channels,少了个 s 会启动不起来
a1.sources.r1.channels = c1
a1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
# 这个是 poll 的方式
# a1.sinks.k1.type = org.apache.spark.streaming.flume.sink.SparkSink
# a1.sinks.k1.hostname = localhost
# a1.sinks.k1.port = 1234
# a1.sinks.k1.channel = c1
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = localhost
a1.sinks.k1.port = 1234
a1.sinks.k1.channel = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
Spark-Streaming
每隔 1s 处理一次 DStream,使用 fastjson 包处理json
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.dstream.{DStream, ReceiverInputDStream}
import org.apache.spark.streaming.flume.{FlumeUtils, SparkFlumeEvent}
import com.alibaba.fastjson.JSON
object FlumePushStreaming {
def main(args: Array[String]): Unit = {
val sparkConf: SparkConf = new SparkConf()
.setMaster("local[3]")
val sc = new SparkContext(sparkConf)
sc.setLogLevel("WARN")
val ssc = new StreamingContext(sc, Seconds(1))
val stream: ReceiverInputDStream[SparkFlumeEvent] = FlumeUtils.createStream(ssc, "localhost", 1234)
// 取出 headers 部分
val dstream: DStream[String] = stream.map(x=>x.event.getHeaders.toString)
//处理 json 字符串
val dataRDD = dstream.map(json => {
// 对原来的json字符串进行修正
val first:String = json.replaceAll("=", "\":\"")
.replaceAll(",", "\",\"")
.replaceAll("[{]", "{\"")
.replaceAll("[}]", "\"}")
.replaceAll(" ", "")
println(first)
val fixedJson = first
val jsonObject = JSON.parseObject(fixedJson)
val userId = jsonObject.getOrDefault("userId", null)
val age = jsonObject.getOrDefault("age", null)
val name = jsonObject.getOrDefault("name", null)
val itemId = jsonObject.getOrDefault("itemId", null)
val click = jsonObject.getOrDefault("click", null)
println((userId, age, name, itemId, click))
(userId, age, name, itemId, click)
})
dataRDD.print()
//开启计算
ssc.start()
ssc.awaitTermination()
}
}