CDH 5.16.1 使用 flume、kafka、sparkstreaming做实时

最新推荐文章于 2021-06-24 13:42:28 发布

大壮vip

最新推荐文章于 2021-06-24 13:42:28 发布

阅读量842

点赞数

本文链接：https://blog.csdn.net/qq_33792843/article/details/90371114

版权

大数据同时被 3 个专栏收录

136 篇文章 0 订阅

订阅专栏

Spark

20 篇文章 0 订阅

订阅专栏

kafka

10 篇文章 0 订阅

订阅专栏

本文的主要目的就是为了实现以下需求：

通过flume收集日志；
将收集到的日志分发给kafka；
通过sparksteaming对kafka获取的日志进行处理；
然后将处理的结果存储到hdfs的指定目录下。

第一步，我们创建flume配置文件，直接在cm上面改就行了。

a1.sources = r1

a1.channels = c1

a1.sinks =s1
 
#sources端配置

a1.sources.r1.type=exec

a1.sources.r1.command=tail -F /usr/local/soft/flume/flume_dir/kafka.log

a1.sources.r1.channels=c1
 
#channels端配置

a1.channels.c1.type=memory

a1.channels.c1.capacity=10000

a1.channels.c1.transactionCapacity=100
 
#设置Kafka接收器

a1.sinks.s1.type= org.apache.flume.sink.kafka.KafkaSink
 
#设置Kafka的broker地址和端口

a1.sinks.s1.brokerList=node1:9092,node2:9092,node3:9092
 
#设置Kafka的Topic

a1.sinks.s1.topic=realtime
 
#设置序列化方式

a1.sinks.s1.serializer.class=kafka.serializer.StringEncoder

a1.sinks.s1.channel=c1

注意，关于配置文件中注意3点：

1.配置文件：

　　a. a1.sources.r1.command=tail -F /usr/local/soft/flume/flume_dir//kafka.log

　　b. a1.sinks.s1.brokerList= n1:9092,n2:9092,n3:9092

　　c . a1.sinks.s1.topic=realtime

2.很明显，由配置文件可以了解到:

　　a.我们需要在/usr/local/soft/flume/flume_dir/下建一个kafka.log的文件，且向文件中输出内容（下面会说到）;

　　b.flume连接到kafka的地址是 manager:9092,namenode:9092,datanode:9092，注意不要配置出错了；

　　c.flume会将采集后的内容输出到Kafka topic 为realtime上，所以我们启动zk,kafka后需要打开一个终端消费topic realtime的内容。这样就可以看到flume与kafka之间玩起来了~~

下一步，我们测试。

编写测试脚本kafka_output.sh
在/usr/local/soft/flume/flume_dir/下建立空文件kafka.log。在root用户目录下新建脚本kafka_output.sh(一定要给予可执行权限)，用来向kafka.log输入内容，脚本内容如下:
for((i=0;i<=1000;i++));

do echo "kafka_test-"+$i>>/usr/local/soft/flume/flume_dir/kafka.log;

done

此时要确保，cm上面的zookeeper、flume、kafka已经正确启动了。

请注意，cdh5.16的kafka安装目录在

/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin

kafka-topics --create --zookeeper node1:2181,node2:2181,node3:2181 --replication-factor 3 --partitions 1 --topic realti
me

看一下控制台日志

[root@node1 bin]# kafka-topics  --create --zookeeper node1:2181,node2:2181,node3:2181 --replication-factor 3 --partitions 1  --topic  rea
ltime-bash: kafka-topics : command not found
[root@node1 bin]# kafka-topics --create --zookeeper node1:2181,node2:2181,node3:2181 --replication-factor 3 --partitions 1 --topic realti
me19/05/20 13:33:03 INFO utils.Log4jControllerRegistration$: Registered kafka:type=kafka.Log4jController MBean
19/05/20 13:33:06 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Initializing a new session to node1:2181,node2:2181,node3:2181.
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.14.2--1, built on 03/27/2018 20:39 GMT
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:host.name=node1
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_212
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:java.class.path=.:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x8
6_64/lib/dt.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/lib/tools.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/activation-1.1.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/activation-1.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/aopalliance-1.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/apacheds-i18n-2.0.0-M15.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/apacheds-jdbm1-2.0.0-M2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/api-asn1-api-1.0.0-M20.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/api-util-1.0.0-M20.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/asm-3.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/avro-1.7.6-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/caffeine-2.7.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/cglib-2.2.1-v20090111.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/checker-qual-2.6.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-beanutils-1.8.3.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-beanutils-core-1.8.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-cli-1.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-codec-1.9.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-collections-3.2.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-compress-1.4.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-configuration-1.6.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-digester-1.8.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-el-1.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-httpclient-3.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-io-2.4.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-lang-2.6.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-lang3-3.5.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-logging-1.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-math3-3.1.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-net-3.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/commons-pool2-2.4.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/compileScala.mapping:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/connect-api-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/connect-basic-auth-extension-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/connect-file-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/connect-json-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/connect-runtime-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/connect-transforms-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/error_prone_annotations-2.3.3.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/gson-2.2.4.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/guava-20.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/guice-3.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/guice-servlet-3.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hadoop-annotations-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hadoop-archives-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hadoop-auth-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hadoop-common-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hadoop-mapreduce-client-common-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hadoop-mapreduce-client-core-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hadoop-yarn-api-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hadoop-yarn-client-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hadoop-yarn-common-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hadoop-yarn-server-common-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hadoop-yarn-server-nodemanager-2.6.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hive-hcatalog-core-1.1.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hive-hcatalog-server-extensions-1.1.0-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hk2-api-2.5.0-b42.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hk2-locator-2.5.0-b42.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/hk2-utils-2.5.0-b42.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/htrace-core4-4.0.1-incubating.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/httpclient-4.4.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/httpcore-4.4.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jackson-annotations-2.9.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jackson-annotations-2.9.8.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jackson-core-2.9.8.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jackson-core-asl-1.9.13-cloudera.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jackson-databind-2.9.8.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jackson-jaxrs-1.8.8.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jackson-jaxrs-base-2.9.8.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jackson-jaxrs-json-provider-2.9.8.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jackson-mapper-asl-1.9.13-cloudera.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jackson-module-jaxb-annotations-2.9.8.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jackson-xc-1.8.8.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/javax.annotation-api-1.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/javax.inject-1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/java-xmlbuilder-0.4.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jaxb-api-2.2.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jersey-client-2.27.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jersey-common-2.27.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jersey-container-servlet-2.27.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jersey-guice-1.9.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jersey-hk2-2.27.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jersey-server-2.27.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jets3t-0.9.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jettison-1.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jetty-6.1.26.cloudera.4.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jetty-client-9.4.12.v20180830.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jetty-continuation-9.4.12.v20180830.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jetty-http-9.4.12.v20180830.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jetty-io-9.4.12.v20180830.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jetty-security-9.4.12.v20180830.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jetty-server-9.4.12.v20180830.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jetty-servlet-9.4.12.v20180830.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jetty-servlets-9.4.12.v20180830.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jetty-util-6.1.26.cloudera.4.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jetty-util-9.4.12.v20180830.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jsch-0.1.42.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jsp-api-2.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/jsr305-3.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/kafka_2.11-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/kafka_2.11-2.1.0-kafka-4.0.0-sources.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/kafka-clients-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/kafka-log4j-appender-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/kafka-streams-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/kafka-streams-examples-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/kafka-streams-scala_2.11-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/kafka-streams-test-utils-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/kafka-tools-2.1.0-kafka-4.0.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/leveldbjni-all-1.8.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/libthrift-0.9.3.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/log4j-1.2.17.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/lz4-java-1.5.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/maven-artifact-3.5.4.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/metrics-servlet-2.2.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/netty-3.10.5.Final.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/paranamer-2.3.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/plexus-utils-3.1.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/protobuf-java-2.5.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/reflections-0.9.11.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/rocksdbjni-5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/scala-library-2.11.12.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/scala-logging_2.11-3.9.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/scala-reflect-2.11.12.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-binding-hive-conf-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-binding-hive-follower-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-binding-kafka-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-core-common-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-core-model-db-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-core-model-indexer-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-core-model-kafka-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-hdfs-common-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-policy-common-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-policy-indexer-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-policy-kafka-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-provider-common-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-provider-db-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/sentry-provider-file-1.5.1-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/servlet-api-2.5.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/shiro-core-1.2.3.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/slf4j-api-1.7.25.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/slf4j-log4j12-1.7.5.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/snappy-java-1.1.7.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/stax-api-1.0-2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/xmlenc-0.52.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/xz-1.0.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/zkclient-0.10.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/zookeeper-3.4.5-cdh5.14.2.jar:/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin/../lib/kafka/bin/../libs/zstd-jni-1.3.5-4.jar:/etc/kafka/conf/sentry-conf19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr
/lib19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-957.el7.x86_64
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:user.name=root
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/cloudera/parcels/KAFKA-4.0.0-1.4.0.0.p0.1/bin
19/05/20 13:33:06 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=node1:2181,node2:2181,node3:2181 sessionTimeout=3
0000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@55183b2019/05/20 13:33:06 INFO zookeeper.ClientCnxn: Opening socket connection to server node2/172.16.1.118:2181. Will not attempt to authenticat
e using SASL (unknown error)19/05/20 13:33:07 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Waiting until connected.
19/05/20 13:33:07 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /172.16.1.117:58702, server: node
2/172.16.1.118:218119/05/20 13:33:07 INFO zookeeper.ClientCnxn: Session establishment complete on server node2/172.16.1.118:2181, sessionid = 0x36ac54ec66d1
500, negotiated timeout = 3000019/05/20 13:33:07 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Connected.
19/05/20 13:33:11 INFO zk.AdminZkClient: Topic creation Map(realtime-0 -> ArrayBuffer(90, 89, 91))
Created topic "realtime".
19/05/20 13:33:11 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Closing.
19/05/20 13:33:11 INFO zookeeper.ZooKeeper: Session: 0x36ac54ec66d1500 closed
19/05/20 13:33:11 INFO zookeeper.ClientCnxn: EventThread shut down
19/05/20 13:33:11 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Closed.
[root@node1 bin]#

Created topic "realtime".

看来已经成功了咯。

打开新的终端，在kafka安装目录下执行如下命令，生成对topic realtime 的消费：

kafka-console-consumer --zookeeper node1:2181,node2:2181,node3:2181 --from-beginning --topic realtime

大壮vip

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
CDH 5.16.1 使用 flume、kafka、sparkstreaming做实时

本文的主要目的就是为了实现以下需求：通过flume收集日志；将收集到的日志分发给kafka；通过sparksteaming对kafka获取的日志进行处理；然后将处理的结果存储到hdfs的指定目录下。第一步，我们创建flume配置文件，直接在cm上面改就行了。a1.sources = r1a1.channels = c1...
复制链接

扫一扫