该测试基于2.9.1
使用命令:
1.[hadoop@hadoop-hive2 ~]$ nc -l 44444
2.bin/pulsar-admin sink localrun --archive /data/apache-pulsar-2.9.1/connectors/pulsar-io-flume-2.9.1.nar --classname org.apache.pulsar.io.flume.sink.StringSink --tenant public --namespace default --name flume --inputs flume_test1 --sink-config-file /data/apache-pulsar-2.9.1/user_conf/flume-io-source.yaml --parallelism 1
注意:
yaml中文件source.conf配置文件存放路径应该是写死的
/data/apache-pulsar-2.9.1/source.conf
测试 命令:
bin/pulsar-client produce flume_test --messages "qxy"
bin/pulsar-client consume flume_test -s "first-subscription"
确定要使用的依赖包:
-rw-rw-r-- 1 hadoop hadoop 155082 Dec 27 11:06 flume-ng-sdk-1.9.0.jar
-rw-rw-r-- 1 hadoop hadoop 40628 Dec 27 11:18 flume-ng-node-1.9.0.jar
-rw-rw-r-- 1 hadoop hadoop 29070877 Dec 27 13:40 pulsar-io-flume-2.9.1.jar
-rw-rw-r-- 1 hadoop hadoop 427654 Dec 27 13:48 flume-ng-core-1.9.0.jar
-rw-rw-r-- 1 hadoop hadoop 67450 Dec 27 13:57 flume-ng-configuration-1.9.0.jar
-rw-rw-r-- 1 hadoop hadoop 1292696 Dec 27 14:12 netty-3.10.6.Final.jar
-rw-rw-r-- 1 hadoop hadoop 185582 Dec 28 11:22 avro-ipc-1.10.2.jar
-rw-rw-r-- 1 hadoop hadoop 187840 Dec 28 11:24 avro-ipc-1.7.4.jar
-rw-rw-r-- 1 hadoop hadoop 303139 Dec 28 13:36 avro-1.7.4.jar
-rw-rw-r-- 1 hadoop hadoop 1350857 Dec 28 13:45 jackson-databind-2.9.7.jar
-rw-rw-r-- 1 hadoop hadoop 773019 Dec 28 13:45 jackson-mapper-asl-1.9.3.jar
-rw-rw-r-- 1 hadoop hadoop 324036 Dec 28 13:45 jackson-core-2.9.7.jar
-rw-rw-r-- 1 hadoop hadoop 228268 Dec 28 13:45 jackson-core-asl-1.9.3.jar
-rw-rw-r-- 1 hadoop hadoop 66981 Dec 28 13:45 jackson-annotations-2.9.7.jar
需要删除的依赖包:
avro-ipc-1.10.2.jar
否则会报相应的抽象方法错误
输出:
[hadoop@hadoop-hive2 ~]$ nc -l 44444
0ªۢÔt~ÿѲ5
eªۢÔt~ÿѲ5
eappendBatch
qxy111
个人心得:
原以为,这个pulsar io flume 要与flume配合使用,后来发现不用,只是需要将flume 相应的包引入到lib中即可。