flume+rabbitmq+spark-streaming实时流计算

1、本文采用的程序列表
flume-ng-1.6.0-cdh5.10.1.tar.gz上篇文章链接已贴出
rabbitmq-server-3.6.5-1.noarch.rpm
spark-streaming_2.11

2、flume 对接 rabbitmq配置文件

a1.sources = r1
a1.sinks = k1
a1.channels = c1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 5140

a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 1000

a1.sinks.k1.channels = c1
a1.sinks.k1.type = com.aweber.flume.sink.rabbitmq.RabbitMQSink
a1.sinks.k1.host = xx.xx
a1.sinks.k1.port = 5672
a1.sinks.k1.virtual-host = /
a1.sinks.k1.username = admin
a1.sinks.k1.password = xxxx
a1.sinks.k1.exchange = xxxx
a1.sinks.k1.auto-properties = false
a1.sinks.k1.routing-key = xxx
a1.sinks.k1.publisher-confirms = true

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

说明：auto-properties = true 是interceptors 拦截器自定义消息头才会生效
自定义拦截器消息头主要可以配置rabbit-mq的属性比如消息过期时间expires等以下为支持的参数说明

注意发送请求：telnet xxx.xxx.xxx 5140

app-id  #应用程序的类型和版本号
content-encoding #消息的编码类型，如是否压缩
content-type    #消息体的MIME类型，如application/json
correlation-id  #一般用做关联消息的message-id，常用于消息的响应
delivery-mode   #消息的持久化类型，1为非持久化，2为持久化，性能影响巨大
expires         #消息过期时间
message-id      #消息的唯一性标识，由应用进行设置
priority        #优先级
reply-to        #构建回复消息的私有响应队列
timestamp       #消息请求时间
type            #消息类型名称，完全由应用决定如何使用该字段   如 topic
user-id         #标识已登录用户，极少使用

2.1、带有自定义消息属性配置文件,如果需要指定多个自定义属性，则添加多个 拦截器

a1.sources = r1
a1.sinks = k1
a1.channels = c1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 5140

a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = static
a1.sources.r1.interceptors.i1.key = expires
a1.sources.r1.interceptors.i1.value = 10000

a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 1000

a1.sinks.k1.channels = c1
a1.sinks.k1.type = com.aweber.flume.sink.rabbitmq.RabbitMQSink
a1.sinks.k1.host = xx.xx
a1.sinks.k1.port = 5672
a1.sinks.k1.virtual-host = /
a1.sinks.k1.username = xxxx
a1.sinks.k1.password = xxxx
a1.sinks.k1.exchange = amq.topic
a1.sinks.k1.auto-properties = true
a1.sinks.k1.routing-key = flume.event
a1.sinks.k1.publisher-confirms = true

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

2.2、flume对接rabbitmq插件文章末尾贴出

3、rabbitmq对接spark-streaming
3.1、pom文件引入插件

<dependency>
  <groupId>com.stratio.receiver</groupId>
  <artifactId>spark-rabbitmq</artifactId>
  <version>LATEST</version>
</dependency>

3.2、java对接代码

public class SparkRabbitMqTest {
    public static void main(String[] args) {
        SparkConf conf = new SparkConf().setMaster("local[4]").setAppName("app");
        JavaStreamingContext javaStreamingContext = new JavaStreamingContext(conf, Durations.seconds(5));
        Map<String, String> params = new HashMap<String, String>();

        params.put("hosts", "xxx.xxx.xx");
        params.put("queueName", "rabbitmq-queue");
        params.put("exchangeName", "amq.topic");
        params.put("routingKeys", "flume.event");
        params.put("vHost", "/");
        params.put("userName", "xxx");
        params.put("password", "xxx");

        Function<QueueingConsumer.Delivery, String> messageHandler = new Function<QueueingConsumer.Delivery, String>() {

            public String call(QueueingConsumer.Delivery message) {
                return new String(message.getBody());
            }
        };

        JavaReceiverInputDStream<String> message = RabbitMQUtils.createJavaStream(javaStreamingContext, String.class, params, messageHandler);
        message.print();
        message.flatMap(line -> Arrays.asList(line.split("@")).iterator()).mapToPair(item -> new Tuple2<>(item,1)).reduceByKey((item1,item2) -> item1 + item2).print();
        javaStreamingContext.start();
        try {
            javaStreamingContext.awaitTermination();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

    }
}

可配置参数说明：

hosts	地址	Yes (default: localhost)
virtualHost	虚拟机	Yes
queueName	队列名	Yes
exchangeName	交换机名	Yes
exchangeType	交换机类型	Yes
routingKeys	路由KEY多个可以使用逗号分隔	Yes
userName	用户名	Yes
password	密码	Yes
durable	是否持久化队列	Yes (default: true)
exclusive	exclusive	Yes (default: false)
autoDelete	是否自动参数	Yes (default: false)
ackType	basic/auto	Yes (default: basic)
fairDispatch	公平分发策略	Yes (default: false)
prefetchCount	一次拉去消息数量	Yes (default: 1)
storageLevel	消息存储类型	Yes (default: MEMORY_ONLY)
x-max-length	队列最大长度	Yes
x-message-ttl	消息过期时间针对整个队列	Yes
x-expires	消息本身有效期针对一条消息	Yes
x-max-length-bytes	消息长度 property	Yes
x-dead-letter-exchange	死信队列交换机	Yes
x-dead-letter-routing-key	死信队列路由key	Yes
x-max-priority	指定最大的优先级,值为0-255(整数)	Yes

flume对接mq插件连接
链接: https://pan.baidu.com/s/1rpgVhS9BEnotsLGDBV-WvQ 提取码: c16j