Spark Streaming实时流项目实战 笔记十

Spark Streaming整合Kafka实战

实战一：Receiver-based

1）先启动zookeeper
2）启动kafka

[hadoop@hadoop000 bin]$ ./kafka-server-start.sh -daemon /home/hadoop/app/kafka_2.11-0.9.0.0/config/server
server-1.properties   server-3.properties   server.properties~
server-2.properties   server-3.properties~  
server-2.properties~  server.properties     
[hadoop@hadoop000 bin]$ ./kafka-server-start.sh -daemon /home/hadoop/app/kafka_2.11-0.9.0.0/config/server.properties
[hadoop@hadoop000 bin]$ jps

3）创建topic

[hadoop@hadoop000 bin]$ ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic  kafka_streaming_topic
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "kafka_streaming_topic".


[hadoop@hadoop000 bin]$ ./kafka-topics.sh --list --zookeeper localhost:2181
hello_topic
kafka_streaming_topic
my-replicated-topic

4）通过控制台测试本控制台是否能够正常的生产和消费信息

./kafka-console-producer.sh --broker-list localhost:9092 --topic kafka_streaming_topic

./kafka-console-consumer.sh --zookeeper localhost:2181 --topic kafka_streaming_topic

spark-submit
–class com.imooc.spark.KafkaReceiverWordCount
–master local[2]
–name KafkaReceiverWordCount
–packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.2.0
/home/hadoop/lib/sparktrain-1.0.jar hadoop000:2181 test kafka_streaming_topic 1

实战二：Direct-Approach

Kafka: Spark Streaming 2.2.0 is compatible with Kafka broker versions 0.8.2.1 or higher. See the Kafka Integration Guide for more details.（版本选择）