presto 集群架构

3台服务器，10.8.1.8、10.8.1.7、10.8.1.6

每台服务器，部署3个presto实例，分别为presto-server-0.248、presto-server-0.248.01、presto-server-0.248.02

1. 文件、目录操作

创建目录etc，并创建子目录catalog

etc
├── catalog
│   └── hive.properties
├── config.properties
├── jvm.config
├── jvm.config.bak
├── log.properties
├── node.properties
└── queue_config.json

创建data，构建软连接etc、plugin，创建子目录var/log、var/run

data
├── etc -> /home/user/presto-server-0.248/etc
├── plugin -> /home/user/presto-server-0.248/plugin
└── var
    ├── log
    │   ├── http-request.log
    │   ├── launcher.log
    │   └── server.log
    └── run
        └── launcher.pid

2. etc 配置文件

2.1 协调节点coordinator

分别是3台服务器presto-server-0.248实例

并分离出discovery，均指定discovery.uri=http://10.88.1.88:9990

其config.properties内容如下：

coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=9990
query.max-memory=50GB
query.max-memory-per-node=8GB
query.max-total-memory-per-node=16GB
discovery-server.enabled=true
discovery.uri=http://10.88.1.88:9990
task.max-worker-threads=32
task.concurrency=32
node-scheduler.max-splits-per-node=100
node-scheduler.max-pending-splits-per-task=80
node-scheduler.network-topology=flat
exchange.http-client.max-connections=1000
exchange.http-client.max-connections-per-server=1000
exchange.http-client.connect-timeout=1m
exchange.http-client.read-timeout=1m
query.client.timeout=5m
query.min-expire-age=30m

2.2 工作节点worker

其config.properties内容如下：

coordinator=false
http-server.http.port=9991
query.max-memory=50GB
query.max-memory-per-node=8GB
query.max-total-memory-per-node=16GB
discovery.uri=http://10.88.1.88:9990
task.max-worker-threads=32
task.concurrency=32
node-scheduler.max-splits-per-node=100
node-scheduler.max-pending-splits-per-task=80
node-scheduler.network-topology=flat
exchange.http-client.max-connections=1000
exchange.http-client.max-connections-per-server=1000
exchange.http-client.connect-timeout=1m
exchange.http-client.read-timeout=1m
query.client.timeout=5m
query.min-expire-age=30m

2.3 node.properties

node.environment=production
node.id=ffffffff-ffff-ffff-ffff-fffffffffffe
node.data-dir=/home/user/presto-server-0.248/data

只要node.id唯一，且data-dir写对实例对应的路径即可。

coordinator和worker，相同

2.4 jvm.config

jvm优化和配置，coordinator和worker相同。

-server
-Xmx64G
-XX:-UseBiasedLocking
-XX:+UseG1GC                               --垃圾回收器
-XX:MaxGCPauseMillis=200
-XX:G1HeapRegionSize=32M
-XX:G1ReservePercent=25
-XX:InitiatingHeapOccupancyPercent=15
-XX:+ExplicitGCInvokesConcurrent
-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-XX:ReservedCodeCacheSize=512M
-XX:PerMethodRecompilationCutoff=10000
-XX:PerBytecodeRecompilationCutoff=10000
-Djdk.attach.allowAttachSelf=true
-Djdk.nio.maxCachedBufferSize=2000000

2.5 日志配置log.properties

com.facebook.presto=INFO

2.6 配置数据源

比如hive，在catalog目录下创建文件hive.properties

connector.name=hive-hadoop2      --对应hive版本
hive.metastore.uri=thrift://10.88.1.86:9083,thrift://10.88.1.87:9083,thrift://10.88.1.88:9083 --多metastore进程实例，用于并发
hive.config.resources=/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml
hive.parquet.use-column-names=true
hive.orc.use-column-names=true
hive.allow-drop-table=true
hive.allow-rename-table=true
#hive.allow-register-partition-procedure=true
#hive.create-empty-bucket-files=false
hive.non-managed-table-writes-enabled=true
hive.max-partitions-per-writers=100

3. 启动和停止服务

bin/launcher start

bin/launcher stop

4.客户端

./presto-cli --server 10.8.1.8:9990 --catalog hive --schema iot

5. hue集成

[[interpreters]]
    [[[presto]]]
     name=Presto SQL
     interface=sqlalchemy
     options='{"url": "presto://10.8.1.8:9990/hive/iot"}'