Druid Kafka indexing service (Druid 导入Kafka 数据)

最新推荐文章于 2024-06-01 14:59:20 发布

weixin_34272308

最新推荐文章于 2024-06-01 14:59:20 发布

阅读量248

点赞数

文章标签：大数据 json php

原文链接：https://my.oschina.net/u/3247419/blog/1162864

版权

2019独角兽企业重金招聘Python工程师标准>>>

1、默认已经安装Druid,如未安装请参见上篇文章<Druid入门>

2、安装Kafka环境。（单机测试）（Druid与Kafka在同一台机器上）

2.1 下载kafka

    curl -O http://www.us.apache.org/dist/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
    tar -xzf kafka_2.11-0.9.0.0.tgz
    cd kafka_2.11-0.9.0.0

2.2 启动（由于kafka与zookeeper在一台机器上未修改配置）

nohup ./bin/kafka-server-start.sh config/server.properties &

2.3 创建topic

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic wikiticker

3.Enable Druid Kafka ingestion（启用Druid Kafka摄入）(自己看着翻译吧 ๑乛◡乛๑)

imply 默认提供了一个 demo wikiticker-kafka-supervisor.json,是一个kafka的配置
curl -XPOST -H'Content-Type: application/json' -d @quickstart/wikiticker-kafka-supervisor.json http://localhost:8090/druid/indexer/v1/supervisor

返回值：{"id":"wikiticker-kafka"}

4、发送数据

./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikiticker

发送以下内容：
{"isRobot":false,"channel":"#en.wikipedia","timestamp":"2016-06-27T20:41:29.625Z","flags":"","isUnpatrolled":false,"page":"Indonesian future capital proposal","diffUrl":"https://en.wikipedia.org/w/index.php?diff=727266238&oldid=727256901","added":1467,"comment":"/* New capital city */ add Balikpapan and Palembang plus ref","commentLength":60,"isNew":false,"isMinor":false,"delta":1467,"isAnonymous":false,"user":"Gunkarta","deltaBucket":1400.0,"deleted":0,"namespace":"Main"}