presto安装及初步使用

最新推荐文章于 2024-09-29 08:31:25 发布

xiaolin_xinji

最新推荐文章于 2024-09-29 08:31:25 发布

阅读量6.1k

点赞数 3

分类专栏： PRESTO

本文链接：https://blog.csdn.net/weixin_44131414/article/details/102629063

版权

PRESTO 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

1.下载安装包

#下载页面 http://prestodb.github.io/download.html
服务端:
presto-server-0.227.tar
客户端:
presto-cli-0.227-executable

2.解压安装包并且创建对应目录和文件

解压

[hadoop@hadoop001 software]$ tar -zxvf  presto-server-0.227.tar -C ~/app/
#在解压的路径下 创建 etc 目录
[hadoop@hadoop001 presto-server-0.227]$ pwd 
/home/hadoop/app/presto-server-0.227
[hadoop@hadoop001 presto-server-0.227]$ ll
total 208
drwxr-xr-x  3 hadoop hadoop     93 Oct 18 09:54 bin
drwxrwxr-x  3 hadoop hadoop    104 Oct 18 10:49 etc
drwxrwxr-x  2 hadoop hadoop   8192 Oct 18 09:49 lib
-rw-r--r--  1 hadoop hadoop 191539 Oct 17 08:06 NOTICE
drwxrwxr-x 30 hadoop hadoop   4096 Oct 18 09:49 plugin
-rw-r--r--  1 hadoop hadoop    126 Oct 17 08:06 README.txt

3. 在etc目录分别创建4个配置文件;创建/catalog/ 目录,创建外部数据源配置文件

https://prestodb.io/docs/current/installation/deployment.html

drwxrwxr-x 2 hadoop hadoop  51 Oct 18 14:05 catalog
-rw-rw-r-- 1 hadoop hadoop 241 Oct 18 10:45 config.properties
-rw-rw-r-- 1 hadoop hadoop 170 Oct 18 10:00 jvm.config
-rw-rw-r-- 1 hadoop hadoop  25 Oct 18 10:08 log.properties
-rw-rw-r-- 1 hadoop hadoop 116 Oct 18 10:49 node.properties

node.properties

# 集群名称, 所有在同一个集群中的Presto节点必须拥有相同的集群名称
	node.environment=production
# 每个Presto节点的唯一标示。每个节点的node.id都必须是唯一的
	node.id=ffffffff-ffff-ffff-ffff-ffffffffffff
#该路径要自己创建 随意的目录
	node.data-dir=/var/presto/data

jvm.config 根据机器的内存配置适合的内存大小

-server
-Xmx16G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError

config.properties a single machine(单机)配置coordinator worker
适当修改内存大小

# 指定是否运维Presto实例作为一个coordinator(接收来自客户端的查询情切管理每个查询的执行过程)
	coordinator=true
# 是否允许在coordinator服务中进行调度工作,
node-scheduler.include-coordinator=true
http-server.http.port=8080
query.max-memory=1GB
query.max-memory-per-node=512MB
query.max-total-memory-per-node=512MB
discovery-server.enabled=true
discovery.uri=http://hadoop001:8080
#  如果节点是worker 则是以下配置
	# coordinator=false
	# http-server.http.port=8080
	# query.max-memory=50GB
	# query.max-memory-per-node=1GB
	# query.max-total-memory-per-node=2GB
	# discovery.uri=http://example.net:8080

log.properties 配置日志级别

com.facebook.presto=INFO

4. catalog目录下的配置,列举mysql hive 配置

[hadoop@hadoop001 catalog]$ ll
total 8
# hive
-rw-rw-r-- 1 hadoop hadoop 173 Oct 18 14:05 hive.properties
# mysql
-rw-rw-r-- 1 hadoop hadoop 112 Oct 18 10:36 mysql.properties

hive.properties的配置

connector.name=hive-hadoop2
#需要启动hive metastore服务
hive.metastore.uri=thrift://hadoop001:9083
# hadoop集群的配置文件目录
hive.config.resources=${HADOOP_HOME}/etc/hadoop/core-site.xml,${HADOOP_HOME}/etc/hadoop/hdfs-site.xml

mysql.properties的配置

connector.name=mysql
connection-url=jdbc:mysql://hadoop001:3306
connection-user=root
connection-password=123456

5. 将客户端放到 /home/hadoop/app/presto-server-0.227/bin

安装文件的bin目录并修改名称为 presto ,赋予执行权限 chmod +x

[hadoop@hadoop001 bin]$ ll
total 14276
-rwxr-xr-x 1 hadoop hadoop     1450 Dec 31  2018 launcher
-rw-r--r-- 1 hadoop hadoop       78 Oct 17 08:15 launcher.properties
-rwxr-xr-x 1 hadoop hadoop    14173 Dec 31  2018 launcher.py
-rwxr-xr-x 1 hadoop hadoop 14592223 Oct 18 09:31 presto
drwxr-xr-x 4 hadoop hadoop       45 Oct 17 08:15 procname

6. 测试

--启动服务
bin/launcher run
--启动客户端
./presto --server hadoop001:8080 --catalog myslq

在这里插入图片描述

#查询hive的数据库
presto> SHOW SCHEMAS FROM hive;
       Schema       
--------------------
 default            
 flumedb            
 information_schema 
 workdb             
(4 rows)
# 查询数据库workdb的表
presto> SHOW tables FROM hive.workdb;
   Table    
------------
 sql_work01 
 sql_work02 
(2 rows)
#mysql的数据库查询
presto> show schemas from mysql;
       Schema       
--------------------
 hivedb             
 information_schema 
 performance_schema 
 sparkdb            
 test               
(5 rows)
# 数据库sparkdb表查询 
presto> show tables from mysql.sparkdb;
  Table   
----------
 dept     
 sys_ad   
 sys_area 
(3 rows)
# 联合查询 hive 与mysql的数据
presto> select * from mysql.sparkdb.dept d join hive.default.emp e on d.deno = e.deno;
 deno |   dename   | address | emno |  name  |   duty    | other |   salary   |   jrsj   | deno 
------+------------+---------+------+--------+-----------+-------+------------+----------+------
 10   | ACCOUNTING | NEWYORK | 7934 | MILLER | CLERK     | 7782  | 1982-1-23  | 1300.00  | 10   
 10   | ACCOUNTING | NEWYORK | 7839 | KING   | PRESIDENT |       | 1981-11-17 | 5000.00  | 10   
 10   | ACCOUNTING | NEWYORK | 7782 | CLARK  | MANAGER   | 7839  | 1981-6-9   | 2450.00  | 10   
 20   | RESEARCH   | DALLAS  | 8888 | HIVE   | PROGRAM   | 7839  | 1988-1-23  | 10300.00 | 20   
 20   | RESEARCH   | DALLAS  | 7902 | FORD   | ANALYST   | 7566  | 1981-12-3  | 3000.00  | 20   
 20   | RESEARCH   | DALLAS  | 7876 | ADAMS  | CLERK     | 7788  | 1987-5-23  | 1100.00  | 20   
 20   | RESEARCH   | DALLAS  | 7788 | SCOTT  | ANALYST   | 7566  | 1987-4-19  | 3000.00  | 20   
 20   | RESEARCH   | DALLAS  | 7566 | JONES  | MANAGER   | 7839  | 1981-4-2   | 2975.00  | 20   
 20   | RESEARCH   | DALLAS  | 7369 | SMITH  | CLERK     | 7902  | 1980-12-17 | 800.00   | 20   
 30   | SALES      | CHICAGO | 7900 | JAMES  | CLERK     | 7698  | 1981-12-3  | 950.00   | 30   
 30   | SALES      | CHICAGO | 7844 | TURNER | SALESMAN  | 7698  | 1981-9-8   | 1500.00  | 30   
 30   | SALES      | CHICAGO | 7698 | BLAKE  | MANAGER   | 7839  | 1981-5-1   | 2850.00  | 30   
 30   | SALES      | CHICAGO | 7654 | MARTIN | SALESMAN  | 7698  | 1981-9-28  | 1250.00  | 30   
 30   | SALES      | CHICAGO | 7521 | WARD   | SALESMAN  | 7698  | 1981-2-22  | 1250.00  | 30   
 30   | SALES      | CHICAGO | 7499 | ALLEN  | SALESMAN  | 7698  | 1981-2-20  | 1600.00  | 30   
(15 rows)

Query 20191018_083410_00039_s3sdr, FINISHED, 1 node
Splits: 66 total, 66 done (100.00%)