rabbitMQ整合kafka

公司使用MQTT协议进行物联网传输，数据进入rabbitMQ，接入spark的途径采用rabbitMQ->flume->kafka->spark
flumesource使用自定义source，具体代码当然是去git找。

##github地址 https://github.com/gmr/rabbitmq-flume-plugin

找到后打成jar包，加入到flume的jar文件下面，因为是CDH管理的集群，所以文件扔到jars下面，然后软连接到flume的jar下面
channel使用kafkachannel，具体可以去flume上面找。
然后就可以愉快的用spark接收数据然后盘它啦。

a1.channels = ch-1
a1.sources = src-1
a1.sinks = k1

a1.sources.src-1.channels = ch-1
a1.sources.src-1.type = com.aweber.flume.source.rabbitmq.RabbitMQSource
a1.sources.src-1.host = *****
a1.sources.src-1.port = 5672
a1.sources.src-1.virtual-host = /
a1.sources.src-1.username = admin
a1.sources.src-1.password = *******
a1.sources.src-1.queue = ********
a1.sources.src-1.prefetchCount = 10
a1.sources.src-1.interceptors = i1
a1.sources.src-1.interceptors.i1.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
a1.sources.src-1.interceptors.i1.preserveExisting = true

a1.channels.ch-1.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.ch-1.kafka.bootstrap.servers = *******:9092,*******:9092,*******:9092
a1.channels.ch-1.kafka.topic = all
a1.channels.ch-1.kafka.consumer.group.id = xx1

#rabbit-flume-kafka.properties的内容
#queue = rk_queue_test 队列名可定义 在生产者端定义
#virtual-host = TEST 虚拟主机可定义 在生产者端定义

##运行flume代码

nohup /opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/bin/flume-ng agent --name a1 --conf conf --conf-file /opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/conf/rabbit-flume-kafka.properties
-Dflume.root.logger=INFO,console &

ps -aux |grep 1906

##建立软连接

 ln -s /opt/cloudera/parcels/CDH/jars/rabbitmq-flume-plugin-standalone-1.1.0.jar /opt/cloudera/parcels/CDH/lib/flume-ng/lib/rabbitmq-flume-plugin-standalone-1.1.0.jar 

 ln -s /opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/mysql-connector-java-5.1.40.jar /opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop-yarn/lib/mysql-connector-java-5.1.40.jar

##kafka运行指令 topic在flume中设置 因为使用了kafkaSink

 /opt/cloudera/parcels/KAFKA-3.0.0-1.3.0.0.p0.40/bin/kafka-console-consumer --bootstrap-server *******:9092,*******:9092,*******:9092 --topic test80 --from-beginning

 /opt/cloudera/parcels/KAFKA-3.0.0-1.3.0.0.p0.40/bin/kafka-console-consumer --bootstrap-server *******:9092,*******:9092,*******:9092 --topic testview --from-beginning

 /opt/cloudera/parcels/KAFKA-3.0.0-1.3.0.0.p0.40/bin/kafka-console-producer  --broker-list *******:9092,*******:9092,*******:9092 --topic test

 ./kafka-consumer-groups --describe --group f1 --zookeeper ******:2181

./kafka-consumer-groups  --new-consumer --bootstrap-server *******:9092,*******:9092,*******:9092 --list

./spark-submit \
--class Report.kafka2view \
--master local[2] \
/opt/moudle/ViewTest-1.0-SNAPSHOT.jar

./spark2-submit --master local --class ****.Report.kafka2view --executor-memory 1G --total-executor-cores 2 /usr/local/src/streamingView-1.0.0-jar-with-dependencies.jar 

spark-submit --class **************
–master yarn
–deploy-mode cluster
–driver-memory 2g
–executor-memory 5g
–num-executors 15
–executor-cores 3
–conf “spark.streaming.backpressure.enabled=true”
–conf spark.default.parallelism=200
–queue online
–conf spark.yarn.am.nodeLabelExpression=label_online
–conf “spark.streaming.kafka.maxRatePerPartition=5000”
–conf spark.scheduler.listenerbus.eventqueue.size=100000
hdfs://********/****.jar 2 150 hh1 11