实时OLAP（六）Facebook Presto connector Apache Pinot 打造实时OLAP系统

感谢uber开源了presto pinot connector 

Download and build Pinot:

git clone https://github.com/apache/incubator-pinot.git
cd incubator-pinot/
git checkout tags/release-0.1.0
mvn install package -DskipTests -Pbin-dist

Start Pinot:

cd pinot-distribution/target/apache-pinot-incubating-0.1.0-bin/apache-pinot-incubating-0.1.0-bin
bin/start-controller.sh > ~/tmp/pinot_controller_cmd.log &
bin/start-broker.sh > ~/tmp/pinot_broker_cmd.log &
bin/start-server.sh > ~/tmp/pinot_server_cmd.log &

Prepare Pinot Data
Create some Pinot segments:

mkdir airlineStatsData
cp sample_data/airlineStats_data.avro airlineStatsData/.

bin/pinot-admin.sh CreateSegment -schemaFile sample_data/airlineStats_schema.json -dataDir airlineStatsData -tableName airlineStats -segmentName airlineStatsSeg_0 -outDir airlineStatsData/airlineStatsSeg_0
bin/pinot-admin.sh CreateSegment -schemaFile sample_data/airlineStats_schema.json -dataDir airlineStatsData -tableName airlineStats -segmentName airlineStatsSeg_1 -outDir airlineStatsData/airlineStatsSeg_1
bin/pinot-admin.sh CreateSegment -schemaFile sample_data/airlineStats_schema.json -dataDir airlineStatsData -tableName airlineStats -segmentName airlineStatsSeg_2 -outDir airlineStatsData/airlineStatsSeg_2

Create Table:

bin/pinot-admin.sh AddSchema  -schemaFile sample_data/airlineStats_schema.json -exec
bin/pinot-admin.sh AddTable -filePath sample_data/airlineStats_offline_table_config.json -controllerPort 9000 -exec

Upload Segments:

bin/pinot-admin.sh UploadSegment -controllerPort 9000 -segmentDir airlineStatsData/airlineStatsSeg_0
bin/pinot-admin.sh UploadSegment -controllerPort 9000 -segmentDir airlineStatsData/airlineStatsSeg_1
bin/pinot-admin.sh UploadSegment -controllerPort 9000 -segmentDir airlineStatsData/airlineStatsSeg_2

Query:

http://localhost:9000/query#


Build Presto:

git clone git@github.com:prestodb/presto.git
mvn clean install -DskipTests -T 4

Running Presto in IDE
Use the following options to create a run configuration:

Main Class: com.facebook.presto.server.PrestoServer
VM Options: -ea -XX:+UseG1GC -XX:G1HeapRegionSize=32M -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -Xmx2G -Dconfig=etc/config.properties -Dlog.levels-file=etc/log.properties
Working directory: $MODULE_DIR$
Use classpath of module: presto-main
The working directory should be the presto-main subdirectory. In IntelliJ, using $MODULE_DIR$ accomplishes this automatically.

Running Presto CLI:
presto-cli/target/presto-cli-*-executable.jar  --catalog pinot
presto> use airlinestats;
Query Pinot table through Presto

presto:airlinestats> select count(AirlineID) from airlinestats;
 _col0
-------
 29238
(1 row)

Query 20190828_224004_00003_3wqm2, FINISHED, 1 node
Splits: 20 total, 20 done (100.00%)
0:04 [29.2K rows, 114KB] [8.29K rows/s, 32.4KB/s]

OLAP System Union Query: