3台服务器,10.8.1.8、10.8.1.7、10.8.1.6
每台服务器,部署3个presto实例,分别为presto-server-0.248、presto-server-0.248.01、presto-server-0.248.02
1. 文件、目录操作
创建目录etc,并创建子目录catalog
etc
├── catalog
│ └── hive.properties
├── config.properties
├── jvm.config
├── jvm.config.bak
├── log.properties
├── node.properties
└── queue_config.json
创建data,构建软连接etc、plugin,创建子目录var/log、var/run
data
├── etc -> /home/user/presto-server-0.248/etc
├── plugin -> /home/user/presto-server-0.248/plugin
└── var
├── log
│ ├── http-request.log
│ ├── launcher.log
│ └── server.log
└── run
└── launcher.pid
2. etc 配置文件
2.1 协调节点coordinator
分别是3台服务器presto-server-0.248实例
并分离出discovery,均指定discovery.uri=http://10.88.1.88:9990
其config.properties内容如下:
coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=9990
query.max-memory=50GB
query.max-memory-per-node=8GB
query.max-total-memory-per-node=16GB
discovery-server.enabled=true
discovery.uri=http://10.88.1.88:9990
task.max-worker-threads=32
task.concurrency=32
node-scheduler.max-splits-per-node=100
node-scheduler.max-pending-splits-per-task=80
node-scheduler.network-topology=flat
exchange.http-client.max-connections=1000
exchange.http-client.max-connections-per-server=1000
exchange.http-client.connect-timeout=1m
exchange.http-client.read-timeout=1m
query.client.timeout=5m
query.min-expire-age=30m
2.2 工作节点worker
其config.properties内容如下:
coordinator=false
http-server.http.port=9991
query.max-memory=50GB
query.max-memory-per-node=8GB
query.max-total-memory-per-node=16GB
discovery.uri=http://10.88.1.88:9990
task.max-worker-threads=32
task.concurrency=32
node-scheduler.max-splits-per-node=100
node-scheduler.max-pending-splits-per-task=80
node-scheduler.network-topology=flat
exchange.http-client.max-connections=1000
exchange.http-client.max-connections-per-server=1000
exchange.http-client.connect-timeout=1m
exchange.http-client.read-timeout=1m
query.client.timeout=5m
query.min-expire-age=30m
2.3 node.properties
node.environment=production
node.id=ffffffff-ffff-ffff-ffff-fffffffffffe
node.data-dir=/home/user/presto-server-0.248/data
只要node.id唯一,且data-dir写对实例对应的路径即可。
coordinator和worker,相同
2.4 jvm.config
jvm优化和配置,coordinator和worker相同。
-server
-Xmx64G
-XX:-UseBiasedLocking
-XX:+UseG1GC --垃圾回收器
-XX:MaxGCPauseMillis=200
-XX:G1HeapRegionSize=32M
-XX:G1ReservePercent=25
-XX:InitiatingHeapOccupancyPercent=15
-XX:+ExplicitGCInvokesConcurrent
-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-XX:ReservedCodeCacheSize=512M
-XX:PerMethodRecompilationCutoff=10000
-XX:PerBytecodeRecompilationCutoff=10000
-Djdk.attach.allowAttachSelf=true
-Djdk.nio.maxCachedBufferSize=2000000
2.5 日志配置log.properties
com.facebook.presto=INFO
2.6 配置数据源
比如hive,在catalog目录下创建文件hive.properties
connector.name=hive-hadoop2 --对应hive版本
hive.metastore.uri=thrift://10.88.1.86:9083,thrift://10.88.1.87:9083,thrift://10.88.1.88:9083 --多metastore进程实例,用于并发
hive.config.resources=/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml
hive.parquet.use-column-names=true
hive.orc.use-column-names=true
hive.allow-drop-table=true
hive.allow-rename-table=true
#hive.allow-register-partition-procedure=true
#hive.create-empty-bucket-files=false
hive.non-managed-table-writes-enabled=true
hive.max-partitions-per-writers=100
3. 启动和停止服务
bin/launcher start
bin/launcher stop
4.客户端
./presto-cli --server 10.8.1.8:9990 --catalog hive --schema iot
5. hue集成
[[interpreters]]
[[[presto]]]
name=Presto SQL
interface=sqlalchemy
options='{"url": "presto://10.8.1.8:9990/hive/iot"}'