presto 使用 部署_presto安装及初步使用

1.下载安装包

#下载页面 http://prestodb.github.io/download.html

服务端:

presto-server-0.227.tar

客户端:

presto-cli-0.227-executable

2.解压安装包并且创建对应目录和文件

解压

[[email protected] software]$ tar -zxvf presto-server-0.227.tar -C ~/app/

#在解压的路径下 创建 etc 目录

[[email protected] presto-server-0.227]$ pwd

/home/hadoop/app/presto-server-0.227

[[email protected] presto-server-0.227]$ ll

total 208

drwxr-xr-x 3 hadoop hadoop 93 Oct 18 09:54 bin

drwxrwxr-x 3 hadoop hadoop 104 Oct 18 10:49 etc

drwxrwxr-x 2 hadoop hadoop 8192 Oct 18 09:49 lib

-rw-r--r-- 1 hadoop hadoop 191539 Oct 17 08:06 NOTICE

drwxrwxr-x 30 hadoop hadoop 4096 Oct 18 09:49 plugin

-rw-r--r-- 1 hadoop hadoop 126 Oct 17 08:06 README.txt

3. 在etc目录分别创建4个配置文件;创建/catalog/ 目录,创建外部数据源配置文件

drwxrwxr-x 2 hadoop hadoop 51 Oct 18 14:05 catalog

-rw-rw-r-- 1 hadoop hadoop 241 Oct 18 10:45 config.properties

-rw-rw-r-- 1 hadoop hadoop 170 Oct 18 10:00 jvm.config

-rw-rw-r-- 1 hadoop hadoop 25 Oct 18 10:08 log.properties

-rw-rw-r-- 1 hadoop hadoop 116 Oct 18 10:49 node.properties

node.properties

node.environment=production

node.id=ffffffff-ffff-ffff-ffff-ffffffffffff

#该路径要自己创建 随意的目录

node.data-dir=/var/presto/data

jvm.config 根据机器的内存 配置适合的内存大小

-server

-Xmx16G

-XX:+UseG1GC

-XX:G1HeapRegionSize=32M

-XX:+UseGCOverheadLimit

-XX:+ExplicitGCInvokesConcurrent

-XX:+HeapDumpOnOutOfMemoryError

-XX:+ExitOnOutOfMemoryError

config.properties a single machine(单机)配置coordinator worker

适当修改内存大小

coordinator=true

node-scheduler.include-coordinator=true

http-server.http.port=8080

query.max-memory=1GB

query.max-memory-per-node=512MB

query.max-total-memory-per-node=512MB

discovery-server.enabled=true

discovery.uri=http://hadoop001:8080

log.properties 配置日志级别

com.facebook.presto=INFO

4. catalog目录下的配置,列举mysql hive 配置

[[email protected] catalog]$ ll

total 8

# hive

-rw-rw-r-- 1 hadoop hadoop 173 Oct 18 14:05 hive.properties

# mysql

-rw-rw-r-- 1 hadoop hadoop 112 Oct 18 10:36 mysql.properties

hive.properties的配置

connector.name=hive-hadoop2

#需要启动hive metastore服务

hive.metastore.uri=thrift://hadoop001:9083

# hadoop集群的配置文件目录

hive.config.resources=${HADOOP_HOME}/etc/hadoop/core-site.xml,${HADOOP_HOME}/etc/hadoop/hdfs-site.xml

mysql.properties的配置

connector.name=mysql

connection-url=jdbc:mysql://hadoop001:3306

connection-user=root

connection-password=123456

5. 将客户端放到 /home/hadoop/app/presto-server-0.227/bin

安装文件的bin目录 并修改名称 为 presto ,赋予执行权限 chmod +x

[[email protected] bin]$ ll

total 14276

-rwxr-xr-x 1 hadoop hadoop 1450 Dec 31 2018 launcher

-rw-r--r-- 1 hadoop hadoop 78 Oct 17 08:15 launcher.properties

-rwxr-xr-x 1 hadoop hadoop 14173 Dec 31 2018 launcher.py

-rwxr-xr-x 1 hadoop hadoop 14592223 Oct 18 09:31 presto

drwxr-xr-x 4 hadoop hadoop 45 Oct 17 08:15 procname

6. 测试

--启动服务

bin/launcher run

--启动客户端

./presto --server hadoop001:8080 --catalog myslq

#查询hive的数据库

presto> SHOW SCHEMAS FROM hive;

Schema

--------------------

default

flumedb

information_schema

workdb

(4 rows)

# 查询数据库workdb的表

presto> SHOW tables FROM hive.workdb;

Table

------------

sql_work01

sql_work02

(2 rows)

#mysql的数据库查询

presto> show schemas from mysql;

Schema

--------------------

hivedb

information_schema

performance_schema

sparkdb

test

(5 rows)

# 数据库sparkdb表查询

presto> show tables from mysql.sparkdb;

Table

----------

dept

sys_ad

sys_area

(3 rows)

# 联合查询 hive 与mysql的数据

presto> select * from mysql.sparkdb.dept d join hive.default.emp e on d.deno = e.deno;

deno | dename | address | emno | name | duty | other | salary | jrsj | deno

------+------------+---------+------+--------+-----------+-------+------------+----------+------

10 | ACCOUNTING | NEWYORK | 7934 | MILLER | CLERK | 7782 | 1982-1-23 | 1300.00 | 10

10 | ACCOUNTING | NEWYORK | 7839 | KING | PRESIDENT | | 1981-11-17 | 5000.00 | 10

10 | ACCOUNTING | NEWYORK | 7782 | CLARK | MANAGER | 7839 | 1981-6-9 | 2450.00 | 10

20 | RESEARCH | DALLAS | 8888 | HIVE | PROGRAM | 7839 | 1988-1-23 | 10300.00 | 20

20 | RESEARCH | DALLAS | 7902 | FORD | ANALYST | 7566 | 1981-12-3 | 3000.00 | 20

20 | RESEARCH | DALLAS | 7876 | ADAMS | CLERK | 7788 | 1987-5-23 | 1100.00 | 20

20 | RESEARCH | DALLAS | 7788 | SCOTT | ANALYST | 7566 | 1987-4-19 | 3000.00 | 20

20 | RESEARCH | DALLAS | 7566 | JONES | MANAGER | 7839 | 1981-4-2 | 2975.00 | 20

20 | RESEARCH | DALLAS | 7369 | SMITH | CLERK | 7902 | 1980-12-17 | 800.00 | 20

30 | SALES | CHICAGO | 7900 | JAMES | CLERK | 7698 | 1981-12-3 | 950.00 | 30

30 | SALES | CHICAGO | 7844 | TURNER | SALESMAN | 7698 | 1981-9-8 | 1500.00 | 30

30 | SALES | CHICAGO | 7698 | BLAKE | MANAGER | 7839 | 1981-5-1 | 2850.00 | 30

30 | SALES | CHICAGO | 7654 | MARTIN | SALESMAN | 7698 | 1981-9-28 | 1250.00 | 30

30 | SALES | CHICAGO | 7521 | WARD | SALESMAN | 7698 | 1981-2-22 | 1250.00 | 30

30 | SALES | CHICAGO | 7499 | ALLEN | SALESMAN | 7698 | 1981-2-20 | 1600.00 | 30

(15 rows)

Query 20191018_083410_00039_s3sdr, FINISHED, 1 node

Splits: 66 total, 66 done (100.00%)