Presto常用配置详细

Presto常用配置详细

下载presto-server安装包

http://repo2.maven.org/maven2/com/facebook/presto/presto-server/

presto更新很快，这里是 presto-server-0.179.tar.gz

1、解压

tar zxfv presto-server-0.179.tar.gz

2、presto-server路径下，创建etc目录

mkdir etc

3、config.properties 配置文件，master与slave是不同的

master配置

coordinator=true
discovery-server.enabled=true
node-scheduler.include-coordinator=true
http-server.http.port=8888
query.max-memory=500GB
query.max-memory-per-node=76GB
discovery.uri=http://hadoopslave10:8888

failure-detector.threshold=0.10

exchange.http-client.request-timeout=30m

http-server.threads.max=1500
task.max-worker-threads=64

exchange.min-error-duration=30m
exchange.max-error-duration=150m

query.max-length=100000000

slave配置

coordinator=false
http-server.http.port=8889
query.max-memory=500GB
query.max-memory-per-node=76GB
discovery.uri=http://hadoopslave10:8888

exchange.http-client.request-timeout=30m

http-server.threads.max=1500
task.max-worker-threads=64

exchange.min-error-duration=30m
exchange.max-error-duration=150m

query.max-length=100000000                  

区别在于 ，master设置

coordinator=true
discovery-server.enabled=true
node-scheduler.include-coordinator=true

配置资源队列：

修改Presto/etc/config.properties，添加以下配置
query.queue-config-file=/home/presto/presto-server/etc/queue_config.json

{
  "queues": {
    "user.${USER}": {
      "maxConcurrent": 5,
      "maxQueued": 20
    },
    "test": {
      "maxConcurrent": 2,
      "maxQueued": 3
    },
    "admin": {
      "maxConcurrent": 100,
      "maxQueued": 100
    },
    "global": {
      "maxConcurrent": 100,
      "maxQueued": 1000
    }
  },
  "rules": [
    { 
      "user": "bob",   #bob为管理员，最多可以同时进行100个查询，有100个在等待队列中
      "queues": ["admin"]
    },
    { 
      "user": "guest",#guest用户使用test的策略，最多可以同时进行2个查询，有3个在等待队列中
      "queues": ["test"]
    },
    {
      "queues": [
        "user.${USER}",
        "global"
      ] 
    }   
  ]   
} 

2、jvm.config 配置文件

-server
-Xmx128G
-XX:+UseConcMarkSweepGC
-XX:+ExplicitGCInvokesConcurrent
-XX:+CMSClassUnloadingEnabled
-XX:+AggressiveOpts
-XX:+HeapDumpOnOutOfMemoryError
-XX:OnOutOfMemoryError=kill -9 %p
-XX:ReservedCodeCacheSize=150M

3、log.properties配置文件

com.facebook.presto=DEBUG

4、node.properties文件

node.environment=datamart
node.id=hadoopmaster1
node.data-dir=/home/presto/presto-server-0.179/data                                                

注意node.id 每一个节点不相同的，通常与hostname一样

5、etc目录下，创建catalog目录

mkdir catalog

6、hive.properties配置文件

connector.name=hive-hadoop2
hive.metastore.uri=thrift://hadoopmaster1:9083
hive.config.resources=/etc/hadoop/conf/core-site.xml, /etc/hadoop/conf/hdfs-site.xml
hive.allow-drop-table=true
hive.storage-format=PARQUET
hive.metastore-cache-ttl=0s
hive.metastore-refresh-interval=1s
hive.max-partitions-per-writers=1000

7、集群重启脚本，每个节点配置免密码

# restart slaves 
ssh hadoopmaster1 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopmaster2 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave1 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave2 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave3 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave4 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave5 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave6 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave7 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave8 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave9 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave11 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave12 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave13 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave14 "/home/presto/presto-server-0.179/bin/launcher restart" &
ssh hadoopslave15 "/home/presto/presto-server-0.179/bin/launcher restart" &

#restart master
ssh hadoopslave10 "/home/presto/presto-server-0.179/bin/launcher restart" &