Druid:Druid.io 部署&使用文档
时间 2016-12-23
原文 http://blog.csdn.net/silentwolfyh/article/details/53843335
参考:
http://lxw1234.com/archives/2015/11/554.htm 海量数据实时OLAP分析系统-Druid.io安装配置和体验
http://druid.io/docs/0.9.2/design/design.html Druid官网搭建
Druid.io 部署&使用文档
1.集群模式下部署
Prerequisites : Java 7 or higher & Zookeeper & mysql
下载Druid.io :
curl -O http://static.druid.io/artifacts/releases/druid-0.9.1.1-bin.tar.gz tar -xzf druid-0.9.1.1-bin.tar.gz cd druid-0.9.1.1
文件夹目录结构 :
- LICENSE - the license files.
- bin/ - scripts related to the single-machine quickstart.
- conf/* - template configurations for a clustered setup.
- conf-quickstart/* - configurations for the single-machine quickstart.
- extensions/* - all Druid extensions.
- hadoop-dependencies/* - Druid Hadoop dependencies.
- lib/* - all included software packages for core Druid.
- quickstart/* - files related to the single-machine quickstart.
所有配置文件均在 conf/* 目录下.
配饰HDFS为Druid.io的deep storage & 配置zk & 配置mysql
修改 conf/druid/_common/common.runtime.properties 文件.
# # Extensions # # This is not the full list of Druid extensions, but common ones that people often use. You may need to change this list # based on your particular setup. #使用 "mysql-metadata-storage" 作为metadata的存储 #使用 "druid-hdfs-storage" 作为 deep storage #使用 "druid-parquet-extensions" 向druid中插入parquet数据 druid.extensions.loadList=["druid-kafka-eight", "druid-histogram", "druid-datasketches", "mysql-metadata-storage", "druid-hdfs-storage", "druid-avro-extensions", "druid-parquet-extensions"] # If you have a different version of Hadoop, place your Hadoop client jar files in your hadoop-dependencies directory # and uncomment the line below to point to your directory. #druid.extensions.hadoopDependenciesDir=/my/dir/hadoop-dependencies # # Logging # # Log all runtime properties on startup. Disable to avoid logging properties on startup: druid.startup.logging.logProperties=true # # Zookeeper # druid.zk.service.host=tagtic-slave01:2181,tagtic-slave02:2181,tagtic-slave03:2181 druid.zk.paths.base=/druid # # Metadata storage # # For MySQL: druid.metadata.storage.type=mysql druid.metadata.storage.connector.connectURI=jdbc:mysql://tagtic-master:3306/druid druid.metadata.storage.connector.user=root druid.metadata.storage.connector.password=123456 # # Deep storage # # For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp): druid.storage.type=hdfs druid.storage.storageDirectory=/druid/segments # # Indexing service logs # # For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp): druid.indexer.logs.type=hdfs druid.indexer.logs.directory=/druid/indexing-logs # # Service discovery # druid.selectors.indexing.serviceName=druid/overlord druid.selectors.coordinator.serviceName=druid/coordinator # # Monitoring # druid.monitoring.monitors=["com.metamx.metrics.JvmMonitor"] druid.emitter=logging druid.emitter.logging.logLevel=info
将 Hadoop 的配置文件(core-site.xml, hdfs-site.xml, yarn-site.xml, mapred-site.xml) cp 到 conf/druid/_common 目录下
修改 conf/druid/middleManager/runtime.properties 文件.
druid.service=druid/middleManager druid.port=18091 # Number of tasks per middleManager druid.worker.capacity=3 # Task launch parameters # **CDH版本添加 -Dhadoop.mapreduce.job.classloader=true 来解决hadoop indexer导入时jar包冲突问题** druid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dhadoop.mapreduce.job.classloader=true druid.indexer.task.baseTaskDir=var/druid/task # HTTP server threads druid.server.http.numThreads=25 # Processing threads and buffers druid.processing.buffer.sizeBytes=536870912 druid.processing.numThreads=2 # Hadoop indexing druid.indexer.task.hadoopWorkingPath=/tmp/druid-indexing druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.6.0"]
Hadoop集群版本必须和Druid.io中版本同一,可以通过pull-deps下载相同hadoop-dependencies版本,e.g. :
java -classpath "lib/*" io.druid.cli.Main tools pull-deps --defaultVersion 0.9.1.1 -c io.druid.extensions:mysql-metadata-storage:0.9.1.1 -c druid-hdfs-storage -h org.apache.hadoop:hadoop-client:2.6.0
项目中Druid.io配置端口号:
- druid.service=druid/coordinator druid.port=18081
- druid.service=druid/broker druid.port=18082
- druid.service=druid/historical druid.port=18083
- druid.service=druid/overlord druid.port=18090
- druid.service=druid/middleManager druid.port=18091
2.启动Druid.io
java `cat conf/druid/coordinator/jvm.config | xargs` -cp conf/druid/_common:conf/druid/coordinator:lib/* io.druid.cli.Main server coordinator &>> logs/coordinator.log & java `cat conf/druid/overlord/jvm.config | xargs` -cp conf/druid/_common:conf/druid/overlord:lib/* io.druid.cli.Main server overlord &>> logs/overlord.log & java `cat conf/druid/historical/jvm.config | xargs` -cp conf/druid/_common:conf/druid/historical:lib/* io.druid.cli.Main server historical &>> logs/historical.log & java `cat conf/druid/middleManager/jvm.config | xargs` -cp conf/druid/_common:conf/druid/middleManager:lib/* io.druid.cli.Main server middleManager &>> logs/middleManager.log & java `cat conf/druid/broker/jvm.config | xargs` -cp conf/druid/_common:conf/druid/broker:lib/* io.druid.cli.Main server broker &>> logs/broker.log &
3.从HDFS导入数据到Druid.io
批量导入Batch Data Ingestion
导入Parquet文件
Druid作业查看 Coordinator : http://tagtic-master:18090/console.html
Druid集群查看Cluster : http://tagtic-master:18081/#/
解决传入数据时区问题 Hadoop Configuration,在conf/druid/_common/mapred-site.xml中添加
<property> <name>mapreduce.map.java.opts</name> <value>-server -Xmx1536m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</value> </property> <property> <name>mapreduce.reduce.java.opts</name> <value>-server -Xmx2560m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</value> </property>
相关文章
- 1. Redis的部署使用文档
- 2. TFS 使用文档 - 部署篇
- 3. TFS使用文档-部署篇
- 4. openfire部署文档(备用)
- 5. Redis部署文档
- 6. Nagios部署文档
- 7. TProfiler部署文档
- 8. docker部署文档
- 9. Tomcat部署文档
- 10. SAMBA部署文档
- 更多相关文章...
相关标签/搜索