Druid单机的安装和测试

最新推荐文章于 2024-08-07 09:32:25 发布

eric_lee

最新推荐文章于 2024-08-07 09:32:25 发布

阅读量984

点赞数

分类专栏：大数据开发大数据运维文章标签： druid

大数据开发同时被 2 个专栏收录

113 篇文章 0 订阅

订阅专栏

大数据运维

51 篇文章 1 订阅

订阅专栏

1、环境信息
2、安装zookeeper
3、安装druid
4、启动druid
5、验证druid
6、遇到的吭

1、环境信息
Centos6.5
32GB 8C
Zookeeper 3.4.5
Druid 0.9.2

2、安装zookeeper

tar -xzf zookeeper-3.4.6.tar.gz
cd zookeeper-3.4.6
cp conf/zoo_sample.cfg conf/zoo.cfg
./bin/zkServer.sh start

3、安装 druid
3、1解压druid
3、2 执行 bin/init 初始化化环境

4、启动druid
4、1启动5个终端执行下面5个命名，或者启动一个将命令执行丢入后台

java `cat conf-quickstart/druid/historical/jvm.config | xargs` -cp "conf-quickstart/druid/_common:conf-quickstart/druid/historical:lib/*" io.druid.cli.Main server historical
java `cat conf-quickstart/druid/broker/jvm.config | xargs` -cp "conf-quickstart/druid/_common:conf-quickstart/druid/broker:lib/*" io.druid.cli.Main server broker
java `cat conf-quickstart/druid/coordinator/jvm.config | xargs` -cp "conf-quickstart/druid/_common:conf-quickstart/druid/coordinator:lib/*" io.druid.cli.Main server coordinator
java `cat conf-quickstart/druid/overlord/jvm.config | xargs` -cp "conf-quickstart/druid/_common:conf-quickstart/druid/overlord:lib/*" io.druid.cli.Main server overlord
java `cat conf-quickstart/druid/middleManager/jvm.config | xargs` -cp "conf-quickstart/druid/_common:conf-quickstart/druid/middleManager:lib/*" io.druid.cli.Main server middleManager

4、2验证是否正常
执行完成以后可以查看服务是否启动
http://10.20.23.82:8081/#/indexing-service
这里写图片描述
 http://10.20.23.82:8090/console.html

5、验证druid
5、1 修改 wikiticker-2015-09-12-sampled.json 文件为本地local模式的

{
  "type" : "index",
  "spec" : {
    "ioConfig" : {
      "type" : "index",
      "firehose" : {
        "type" : "local",
        "baseDir" : "/home/hadoop/druid-0.9.2/quickstart",
        "filter" : "wikiticker-2015-09-12-sampled.json"
      }
    },
    "dataSchema" : {
      "dataSource" : "wikiticker",
      "granularitySpec" : {
        "type" : "uniform",
        "segmentGranularity" : "day",
        "queryGranularity" : "none",
        "intervals" : ["2015-09-12/2015-09-13"]
      },
      "parser" : {
        "type" : "string",
        "parseSpec" : {
          "format" : "json",
          "dimensionsSpec" : { "dimensions" : [ "channel", "cityName", "comment", "countryIsoCode", "countryName", "isAnonymous", "isMinor", "isNew", "isRobot", "isUnpatrolled", "metroCode", "namespace", "page", "regionIsoCode", "regionName", "user" ] },
          "timestampSpec" : { "format" : "auto", "column" : "time" } }
      },
      "metricsSpec" : [
        {
          "name" : "count",
          "type" : "count"
        },
        {
          "name" : "added",
          "type" : "longSum",
          "fieldName" : "added"
        },
        {
          "name" : "deleted",
          "type" : "longSum",
          "fieldName" : "deleted"
        },
        {
          "name" : "delta",
          "type" : "longSum",
          "fieldName" : "delta"
        },
        {
          "name" : "user_unique",
          "type" : "hyperUnique",
          "fieldName" : "user"
        }
      ]
    },
    "tuningConfig" : {
      "type" : "index",
      "partitionsSpec" : {
        "type" : "hashed",
        "targetPartitionSize" : 5000000
      },
      "jobProperties" : {}
    }
  }
}

5、2 执行 curl -X ‘POST’ -H ‘Content-Type:application/json’ -d @quickstart/wikiticker-index.json localhost:8090/druid/indexer/v1/task 命令来overlord数据

[hadoop@SZB-L0038787 druid-0.9.2]$ curl -X 'POST' -H 'Content-Type:application/json' -d @quickstart/wikiticker-index.json localhost:8090/druid/indexer/v1/task
{"task":"index_wikiticker_2017-05-22T02:06:52.225Z"}

5、3 查看运行的情况，可以在console上面查看是否运行成功
这里写图片描述

5、4查看数据
运行如下命令

curl -L -H'Content-Type: application/json' -XPOST --data-binary @quickstart/wikiticker-top-pages.json http://localhost:8082/druid/v2/?pretty

[hadoop@SZB-L0038787 druid-0.9.2]$ curl -L -H'Content-Type: application/json' -XPOST --data-binary @quickstart/wikiticker-top-pages.json http://localhost:8082/druid/v2/?pretty
[ {
  "timestamp" : "2015-09-12T00:46:58.771Z",
  "result" : [ {
    "edits" : 33,
    "page" : "Wikipedia:Vandalismusmeldung"
  }, {
    "edits" : 28,
    "page" : "User:Cyde/List of candidates for speedy deletion/Subpage"
  }, {
    "edits" : 27,
    "page" : "Jeremy Corbyn"
  }, {
    "edits" : 21,
    "page" : "Wikipedia:Administrators' noticeboard/Incidents"
  }, {
    "edits" : 20,
    "page" : "Flavia Pennetta"
  }, {
    "edits" : 18,
    "page" : "Total Drama Presents: The Ridonculous Race"
  }, {
    "edits" : 18,
    "page" : "User talk:Dudeperson176123"
  }, {
    "edits" : 18,
    "page" : "Wikipédia:Le Bistro/12 septembre 2015"
  }, {
    "edits" : 17,
    "page" : "Wikipedia:In the news/Candidates"
  }, {
    "edits" : 17,
    "page" : "Wikipedia:Requests for page protection"
  }, {
    "edits" : 16,
    "page" : "Utente:Giulio Mainardi/Sandbox"
  }, {
    "edits" : 16,
    "page" : "Wikipedia:Administrator intervention against vandalism"
  }, {
    "edits" : 15,
    "page" : "Anthony Martial"
  }, {
    "edits" : 13,
    "page" : "Template talk:Connected contributor"
  }, {
    "edits" : 12,
    "page" : "Chronologie de la Lorraine"
  }, {
    "edits" : 12,
    "page" : "Wikipedia:Files for deletion/2015 September 12"
  }, {
    "edits" : 12,
    "page" : "Гомосексуальный образ жизни"
  }, {
    "edits" : 11,
    "page" : "Constructive vote of no confidence"
  }, {
    "edits" : 11,
    "page" : "Homo naledi"
  }, {
    "edits" : 11,
    "page" : "Kim Davis (county clerk)"
  }, {
    "edits" : 11,
    "page" : "Vorlage:Revert-Statistik"
  }, {
    "edits" : 11,
    "page" : "Конституция Японской империи"
  }, {
    "edits" : 10,
    "page" : "The Naked Brothers Band (TV series)"
  }, {
    "edits" : 10,
    "page" : "User talk:Buster40004"
  }, {
    "edits" : 10,
    "page" : "User:Valmir144/sandbox"
  } ]
} ]

6、遇到的坑
6、1 大小写的区别，路径的配置baseDir只能这么写

2017-05-22T01:33:31,841 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_wikiticker_2017-05-22T01:33:26.102Z, type=index, dataSource=wikiticker}]
com.metamx.common.IAE: baseDir is null
    at io.druid.segment.realtime.firehose.LocalFirehoseFactory.connect(LocalFirehoseFactory.java:89) ~[druid-server-0.9.2.jar:0.9.2]
    at io.druid.segment.realtime.firehose.LocalFirehoseFactory.connect(LocalFirehoseFactory.java:46) ~[druid-server-0.9.2.jar:0.9.2]
    at io.druid.indexing.common.task.IndexTask.getDataIntervals(IndexTask.java:255) ~[druid-indexing-service-0.9.2.jar:0.9.2]
    at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:201) ~[druid-indexing-service-0.9.2.jar:0.9.2]
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2.jar:0.9.2]
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2.jar:0.9.2]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
2017-05-22T01:33:31,852 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_wikiticker_2017-05-22T01:33:26.102Z] status changed to [FAILED].
2017-05-22T01:33:31,858 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_wikiticker_2017-05-22T01:33:26.102Z",
  "status" : "FAILED",
  "duration" : 326
}

6、2 源文件的路径，在druid中配置路径baseDir 这个建议填写绝对路径，filter这个只填写文件名称。如果你需要填写信息的话filter这个参数带的路径是没有实际意义的

2017-05-22T01:34:40,650 INFO [task-runner-0-priority-0] io.druid.segment.realtime.firehose.LocalFirehoseFactory - Searching for all [quickstart/wikiticker-2015-09-12-sampled.json] in and beneath [/home/hadoop/druid-0.9.2/.]
2017-05-22T01:34:40,673 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_wikiticker_2017-05-22T01:34:36.301Z, type=index, dataSource=wikiticker}]
com.metamx.common.ISE: Found no files to ingest! Check your schema.
    at io.druid.segment.realtime.firehose.LocalFirehoseFactory.connect(LocalFirehoseFactory.java:100) ~[druid-server-0.9.2.jar:0.9.2]
    at io.druid.segment.realtime.firehose.LocalFirehoseFactory.connect(LocalFirehoseFactory.java:46) ~[druid-server-0.9.2.jar:0.9.2]
    at io.druid.indexing.common.task.IndexTask.getDataIntervals(IndexTask.java:255) ~[druid-indexing-service-0.9.2.jar:0.9.2]
    at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:201) ~[druid-indexing-service-0.9.2.jar:0.9.2]
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2.jar:0.9.2]
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2.jar:0.9.2]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
2017-05-22T01:34:40,678 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_wikiticker_2017-05-22T01:34:36.301Z] status changed to [FAILED].
2017-05-22T01:34:40,680 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_wikiticker_2017-05-22T01:34:36.301Z",
  "status" : "FAILED",
  "duration" : 232
}

6、3 路径错误，在druid中有一个环境路径需要提前手工创建
environment:java.io.tmpdir=var/tmp

2017-05-22T02:05:47,698 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_wikiticker_2017-05-22T02:05:40.252Z, type=index, dataSource=wikiticker}]
java.lang.IllegalStateException: Failed to create directory within 10000 attempts (tried 1495418747661-0 to 1495418747661-9999)
    at com.google.common.io.Files.createTempDir(Files.java:600) ~[guava-16.0.1.jar:?]
    at io.druid.segment.indexing.RealtimeTuningConfig.createNewBasePersistDirectory(RealtimeTuningConfig.java:56) ~[druid-server-0.9.2.jar:0.9.2]
    at io.druid.segment.indexing.RealtimeTuningConfig.<init>(RealtimeTuningConfig.java:118) ~[druid-server-0.9.2.jar:0.9.2]
    at io.druid.indexing.common.task.IndexTask.convertTuningConfig(IndexTask.java:147) ~[druid-indexing-service-0.9.2.jar:0.9.2]
    at io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:397) ~[druid-indexing-service-0.9.2.jar:0.9.2]
    at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:233) ~[druid-indexing-service-0.9.2.jar:0.9.2]
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2.jar:0.9.2]
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2.jar:0.9.2]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
2017-05-22T02:05:47,705 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_wikiticker_2017-05-22T02:05:40.252Z] status changed to [FAILED].
2017-05-22T02:05:47,711 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_wikiticker_2017-05-22T02:05:40.252Z",
  "status" : "FAILED",
  "duration" : 2776
}