本实验参考此文档。
这个实验可以了解以下知识点:
- 如何从对象存储倒入数据库到MySQL
- 如何使用Autopilot中的Auto Parallel Load特性加载数据到HeatWave
在这里提供了很多MySQL的示例数据库。我们需要的airport DB从这里下载,约640MB。
下载并解压:
$ wget https://downloads.mysql.com/docs/airport-db.zip
$ ls -l airport-db.zip
-rw-rw-r--. 1 opc opc 655695008 Sep 10 17:26 airport-db.zip
$ unzip airport-db.zip
$ ls -l airport-db
total 641612
-rwxr-xr-x. 1 opc opc 1589 Sep 10 17:25 airportdb@airline@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@airline@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 627 Sep 10 17:25 airportdb@airline.json
-rwxr-xr-x. 1 opc opc 1259 Sep 10 17:25 airportdb@airline.sql
-rwxr-xr-x. 1 opc opc 24022 Sep 10 17:25 airportdb@airplane@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@airplane@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 628 Sep 10 17:25 airportdb@airplane.json
-rwxr-xr-x. 1 opc opc 1105 Sep 10 17:25 airportdb@airplane.sql
-rwxr-xr-x. 1 opc opc 181412 Sep 10 17:25 airportdb@airplane_type@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@airplane_type@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 605 Sep 10 17:25 airportdb@airplane_type.json
-rwxr-xr-x. 1 opc opc 1115 Sep 10 17:25 airportdb@airplane_type.sql
-rwxr-xr-x. 1 opc opc 142410 Sep 10 17:25 airportdb@airport@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@airport@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 408522 Sep 10 17:25 airportdb@airport_geo@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@airport_geo@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 774 Sep 10 17:25 airportdb@airport_geo.json
-rwxr-xr-x. 1 opc opc 1366 Sep 10 17:25 airportdb@airport_geo.sql
-rwxr-xr-x. 1 opc opc 612 Sep 10 17:25 airportdb@airport.json
-rwxr-xr-x. 1 opc opc 582 Sep 10 17:25 airportdb@airport_reachable.json
-rwxr-xr-x. 1 opc opc 1010 Sep 10 17:25 airportdb@airport_reachable.sql
-rwxr-xr-x. 1 opc opc 9 Sep 10 17:25 airportdb@airport_reachable.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@airport_reachable.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 1209 Sep 10 17:25 airportdb@airport.sql
-rwxr-xr-x. 1 opc opc 23990482 Sep 10 17:25 airportdb@booking@0.tsv.zst
-rwxr-xr-x. 1 opc opc 512 Sep 10 17:25 airportdb@booking@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24787816 Sep 10 17:25 airportdb@booking@10.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@10.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24605829 Sep 10 17:25 airportdb@booking@11.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@11.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24662300 Sep 10 17:25 airportdb@booking@12.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@12.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24598565 Sep 10 17:25 airportdb@booking@13.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@13.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24474339 Sep 10 17:25 airportdb@booking@14.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@14.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24659075 Sep 10 17:25 airportdb@booking@15.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@15.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24460743 Sep 10 17:25 airportdb@booking@16.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@16.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24872813 Sep 10 17:25 airportdb@booking@17.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@17.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24730510 Sep 10 17:25 airportdb@booking@18.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@18.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24664897 Sep 10 17:25 airportdb@booking@19.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@19.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 23821261 Sep 10 17:25 airportdb@booking@1.tsv.zst
-rwxr-xr-x. 1 opc opc 520 Sep 10 17:25 airportdb@booking@1.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24477378 Sep 10 17:25 airportdb@booking@20.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@20.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24603821 Sep 10 17:25 airportdb@booking@21.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@21.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24510521 Sep 10 17:25 airportdb@booking@22.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@22.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24492613 Sep 10 17:25 airportdb@booking@23.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@23.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 7089904 Sep 10 17:25 airportdb@booking@@24.tsv.zst
-rwxr-xr-x. 1 opc opc 160 Sep 10 17:25 airportdb@booking@@24.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 29859055 Sep 10 17:25 airportdb@booking@2.tsv.zst
-rwxr-xr-x. 1 opc opc 648 Sep 10 17:25 airportdb@booking@2.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 22284735 Sep 10 17:25 airportdb@booking@3.tsv.zst
-rwxr-xr-x. 1 opc opc 488 Sep 10 17:25 airportdb@booking@3.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 28336025 Sep 10 17:25 airportdb@booking@4.tsv.zst
-rwxr-xr-x. 1 opc opc 616 Sep 10 17:25 airportdb@booking@4.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24829241 Sep 10 17:25 airportdb@booking@5.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@5.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24574052 Sep 10 17:25 airportdb@booking@6.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@6.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24435530 Sep 10 17:25 airportdb@booking@7.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@7.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24483962 Sep 10 17:25 airportdb@booking@8.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@8.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24687308 Sep 10 17:25 airportdb@booking@9.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@9.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 646 Sep 10 17:25 airportdb@booking.json
-rwxr-xr-x. 1 opc opc 1371 Sep 10 17:25 airportdb@booking.sql
-rwxr-xr-x. 1 opc opc 89932 Sep 10 17:25 airportdb@employee@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@employee@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 883 Sep 10 17:25 airportdb@employee.json
-rwxr-xr-x. 1 opc opc 2016 Sep 10 17:25 airportdb@employee.sql
-rwxr-xr-x. 1 opc opc 10133356 Sep 10 17:25 airportdb@flight@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 256 Sep 10 17:25 airportdb@flight@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 712 Sep 10 17:25 airportdb@flight.json
-rwxr-xr-x. 1 opc opc 1035 Sep 10 17:25 airportdb@flight_log.json
-rwxr-xr-x. 1 opc opc 1827 Sep 10 17:25 airportdb@flight_log.sql
-rwxr-xr-x. 1 opc opc 9 Sep 10 17:25 airportdb@flight_log.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@flight_log.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 172016 Sep 10 17:25 airportdb@flightschedule@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@flightschedule@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 829 Sep 10 17:25 airportdb@flightschedule.json
-rwxr-xr-x. 1 opc opc 1686 Sep 10 17:25 airportdb@flightschedule.sql
-rwxr-xr-x. 1 opc opc 1821 Sep 10 17:25 airportdb@flight.sql
-rwxr-xr-x. 1 opc opc 1199 Sep 10 17:25 airportdb.json
-rwxr-xr-x. 1 opc opc 494930 Sep 10 17:25 airportdb@passenger@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@passenger@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 1672340 Sep 10 17:25 airportdb@passengerdetails@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 32 Sep 10 17:25 airportdb@passengerdetails@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 748 Sep 10 17:25 airportdb@passengerdetails.json
-rwxr-xr-x. 1 opc opc 1569 Sep 10 17:25 airportdb@passengerdetails.sql
-rwxr-xr-x. 1 opc opc 633 Sep 10 17:25 airportdb@passenger.json
-rwxr-xr-x. 1 opc opc 1183 Sep 10 17:25 airportdb@passenger.sql
-rwxr-xr-x. 1 opc opc 583 Sep 10 17:25 airportdb.sql
-rwxr-xr-x. 1 opc opc 35538282 Sep 10 17:25 airportdb@weatherdata@0.tsv.zst
-rwxr-xr-x. 1 opc opc 1752 Sep 10 17:25 airportdb@weatherdata@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 4733492 Sep 10 17:25 airportdb@weatherdata@@1.tsv.zst
-rwxr-xr-x. 1 opc opc 240 Sep 10 17:25 airportdb@weatherdata@@1.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 738 Sep 10 17:25 airportdb@weatherdata.json
-rwxr-xr-x. 1 opc opc 1310 Sep 10 17:25 airportdb@weatherdata.sql
-rwxr-xr-x. 1 opc opc 2586 Sep 10 17:25 @.done.json
-rwxr-xr-x. 1 opc opc 859 Sep 10 17:25 @.json
-rwxr-xr-x. 1 opc opc 46349 Sep 10 17:25 @.manifest.json
-rwxr-xr-x. 1 opc opc 240 Sep 10 17:25 @.post.sql
-rwxr-xr-x. 1 opc opc 4884 Sep 10 17:25 README.txt
-rwxr-xr-x. 1 opc opc 240 Sep 10 17:25 @.sql
安装OCI CLI,OCI Linux 7中居然已经安装好了,还是最新版本。
使用OCI CLI将目录中的数据库文件批量上传到对象存储:
$ oci os object bulk-upload -ns ocichina001 -bn airport --src-dir ~/airport-db
然后将数据库从对象存储导入,并不需要事先建库,会自动建库:
$ mysqlsh root@10.0.1.233
MySQL 10.0.1.233:33060+ ssl JS > util.loadDump("", {threads: 16, loadIndexes: "false", osBucketName: "airport", osNamespace: "ocichina001", ignoreVersion: true})
Loading DDL and Data from OCI ObjectStorage bucket=airport, prefix='' using 16 threads.
Opening dump...
Target is MySQL 8.0.26-u1-cloud (MySQL Database Service). Dump was produced from MySQL 8.0.26-cloud
Fetching dump data from remote location...
Fetching 14 table metadata files for schema `airportdb`...
Checking for pre-existing objects...
Executing common preamble SQL
Executing DDL script for schema `airportdb`
[Worker005] Executing DDL script for `airportdb`.`flight`
[Worker003] Executing DDL script for `airportdb`.`employee`
[Worker011] Executing DDL script for `airportdb`.`weatherdata`
[Worker012] Executing DDL script for `airportdb`.`booking`
[Worker014] Executing DDL script for `airportdb`.`flight_log`
[Worker006] Executing DDL script for `airportdb`.`passengerdetails`
[Worker010] Executing DDL script for `airportdb`.`flightschedule`
[Worker002] Executing DDL script for `airportdb`.`airport_reachable`
[Worker013] Executing DDL script for `airportdb`.`airplane_type` (indexes removed for deferred creation)
[Worker009] Executing DDL script for `airportdb`.`airplane`
[Worker015] Executing DDL script for `airportdb`.`airline`
[Worker008] Executing DDL script for `airportdb`.`passenger`
[Worker007] Executing DDL script for `airportdb`.`airport`
[Worker000] Executing DDL script for `airportdb`.`airport_geo`
NOTE: [Worker007] Error processing table `airportdb`.`airport`, will retry after delay: MySQL Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
[Worker013] airportdb@airplane_type@@0.tsv.zst: Records: 342 Deleted: 0 Skipped: 0 Warnings: 0
[Worker008] airportdb@airline@@0.tsv.zst: Records: 113 Deleted: 0 Skipped: 0 Warnings: 0
[Worker000] airportdb@employee@@0.tsv.zst: Records: 1000 Deleted: 0 Skipped: 0 Warnings: 0
[Worker014] airportdb@airplane@@0.tsv.zst: Records: 5583 Deleted: 0 Skipped: 0 Warnings: 0
[Worker012] airportdb@airport@@0.tsv.zst: Records: 9854 Deleted: 0 Skipped: 0 Warnings: 0
[Worker002] airportdb@flightschedule@@0.tsv.zst: Records: 9881 Deleted: 0 Skipped: 0 Warnings: 0
[Worker006] airportdb@passenger@@0.tsv.zst: Records: 36095 Deleted: 0 Skipped: 0 Warnings: 0
[Worker003] airportdb@airport_geo@@0.tsv.zst: Records: 9854 Deleted: 0 Skipped: 0 Warnings: 0
[Worker005] airportdb@passengerdetails@@0.tsv.zst: Records: 36095 Deleted: 0 Skipped: 0 Warnings: 0
[Worker011] airportdb@flight@@0.tsv.zst: Records: 462553 Deleted: 0 Skipped: 0 Warnings: 0
[Worker004] airportdb@weatherdata@0.tsv.zst: Records: 4082688 Deleted: 0 Skipped: 0 Warnings: 0
[Worker004] airportdb@weatherdata@@1.tsv.zst: Records: 543744 Deleted: 0 Skipped: 0 Warnings: 0
[Worker010] airportdb@booking@3.tsv.zst: Records: 2024926 Deleted: 0 Skipped: 0 Warnings: 0
[Worker001] airportdb@booking@0.tsv.zst: Records: 2160442 Deleted: 0 Skipped: 0 Warnings: 0
[Worker009] airportdb@booking@1.tsv.zst: Records: 2159969 Deleted: 0 Skipped: 0 Warnings: 0
[Worker008] airportdb@booking@6.tsv.zst: Records: 2214973 Deleted: 0 Skipped: 0 Warnings: 0
[Worker002] airportdb@booking@10.tsv.zst: Records: 2214991 Deleted: 0 Skipped: 0 Warnings: 0
[Worker003] airportdb@booking@12.tsv.zst: Records: 2214478 Deleted: 0 Skipped: 0 Warnings: 0
[Worker013] airportdb@booking@5.tsv.zst: Records: 2215073 Deleted: 0 Skipped: 0 Warnings: 0
[Worker012] airportdb@booking@9.tsv.zst: Records: 2214959 Deleted: 0 Skipped: 0 Warnings: 0
[Worker000] airportdb@booking@7.tsv.zst: Records: 2214707 Deleted: 0 Skipped: 0 Warnings: 0
[Worker014] airportdb@booking@8.tsv.zst: Records: 2214781 Deleted: 0 Skipped: 0 Warnings: 0
[Worker005] airportdb@booking@13.tsv.zst: Records: 2215250 Deleted: 0 Skipped: 0 Warnings: 0
[Worker006] airportdb@booking@11.tsv.zst: Records: 2215122 Deleted: 0 Skipped: 0 Warnings: 0
[Worker007] airportdb@booking@4.tsv.zst: Records: 2531382 Deleted: 0 Skipped: 0 Warnings: 0
[Worker015] airportdb@booking@2.tsv.zst: Records: 2699650 Deleted: 0 Skipped: 0 Warnings: 0
[Worker011] airportdb@booking@14.tsv.zst: Records: 2215310 Deleted: 0 Skipped: 0 Warnings: 0
[Worker000] airportdb@booking@@24.tsv.zst: Records: 644485 Deleted: 0 Skipped: 0 Warnings: 0
[Worker004] airportdb@booking@15.tsv.zst: Records: 2215229 Deleted: 0 Skipped: 0 Warnings: 0
[Worker010] airportdb@booking@16.tsv.zst: Records: 2215038 Deleted: 0 Skipped: 0 Warnings: 0
[Worker001] airportdb@booking@17.tsv.zst: Records: 2214973 Deleted: 0 Skipped: 0 Warnings: 0
[Worker009] airportdb@booking@18.tsv.zst: Records: 2215177 Deleted: 0 Skipped: 0 Warnings: 0
[Worker002] airportdb@booking@20.tsv.zst: Records: 2214883 Deleted: 0 Skipped: 0 Warnings: 0
[Worker012] airportdb@booking@23.tsv.zst: Records: 2215175 Deleted: 0 Skipped: 0 Warnings: 0
[Worker008] airportdb@booking@19.tsv.zst: Records: 2214884 Deleted: 0 Skipped: 0 Warnings: 0
[Worker003] airportdb@booking@21.tsv.zst: Records: 2214070 Deleted: 0 Skipped: 0 Warnings: 0
[Worker013] airportdb@booking@22.tsv.zst: Records: 2214692 Deleted: 0 Skipped: 0 Warnings: 0
Executing common postamble SQL
37 chunks (59.50M rows, 2.03 GB) for 12 tables in 1 schemas were loaded in 2 min 6 sec (avg throughput 16.15 MB/s)
0 warnings were reported during the load.
验证数据库已创建,表已导入:
MySQL 10.0.1.233:33060+ ssl JS > \sql
Switching to SQL mode... Commands end with ;
MySQL 10.0.1.233:33060+ ssl SQL > use airportdb;
Default schema set to `airportdb`.
Fetching table and column names from `airportdb` for auto-completion... Press ^C to stop.
MySQL 10.0.1.233:33060+ ssl airportdb SQL > select count(*) from flight;
+----------+
| count(*) |
+----------+
| 462553 |
+----------+
1 row in set (0.0134 sec)
添加HeatWave Cluster,默认的2个节点即可。
加载数据到HeatWave,这是Autopilot的特性:
MySQL 10.0.1.233:33060+ ssl airportdb SQL > CALL sys.heatwave_load(JSON_ARRAY('airportdb'), NULL);
+------------------------------------------+
| INITIALIZING HEATWAVE AUTO LOAD |
+------------------------------------------+
| Version: 1.15 |
| |
| Load Mode: normal |
| Load Policy: disable_unsupported_columns |
| Output Mode: normal |
| |
+------------------------------------------+
6 rows in set (5.2867 sec)
+-----------------------------------------------------------------------------------------------+
| OFFLOAD ANALYSIS |
+-----------------------------------------------------------------------------------------------+
| Verifying input schemas: 1 |
| User excluded items: 0 |
| |
| SCHEMA OFFLOADABLE OFFLOADABLE SUMMARY OF |
| NAME TABLES COLUMNS ISSUES |
| ------ ----------- ----------- ---------- |
| `airportdb` 14 104 2 table(s) are partially loadable |
| |
| Total offloadable schemas: 1 |
| |
+-----------------------------------------------------------------------------------------------+
10 rows in set (5.2867 sec)
+-----------------------------------------------------------------------------------------------------------------------------+
| CAPACITY ESTIMATION |
+-----------------------------------------------------------------------------------------------------------------------------+
| Default load pool for tables: TRANSACTIONAL |
| Default encoding for string columns: VARLEN (unless specified in the schema) |
| Estimating memory footprint for 1 schema(s) |
| |
| TOTAL ESTIMATED ESTIMATED TOTAL DICTIONARY VARLEN ESTIMATED |
| SCHEMA OFFLOADABLE HEATWAVE NODE MYSQL NODE STRING ENCODED ENCODED LOAD |
| NAME TABLES FOOTPRINT FOOTPRINT COLUMNS COLUMNS COLUMNS TIME |
| ------ ----------- --------- --------- ------- ---------- ------- --------- |
| `airportdb` 14 4.16 GiB 56.00 MiB 35 0 35 17.00 s |
| |
| Sufficient MySQL host memory available to load all tables. |
| Sufficient HeatWave cluster memory available to load all tables. |
| |
+-----------------------------------------------------------------------------------------------------------------------------+
13 rows in set (5.2867 sec)
+---------------------------------------------------------------------------------------------------------------------------------------+
| EXECUTING LOAD |
+---------------------------------------------------------------------------------------------------------------------------------------+
| HeatWave Load script generated |
| Retrieve load script containing 46 generated DDL command(s) using the query below: |
| SELECT log->>"$.sql" AS "Load Script" FROM sys.heatwave_load_report WHERE type = "sql" ORDER BY id; |
| |
| Adjusting load parallelism dynamically per table |
| Using current parallelism of 32 thread(s) as maximum |
| |
| Using SQL_MODE: ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION |
| |
| Proceeding to load 14 tables into HeatWave |
| |
| Applying changes will take approximately 16.97 s |
| |
+---------------------------------------------------------------------------------------------------------------------------------------+
13 rows in set (5.2867 sec)
+----------------------------------------+
| LOADING TABLE |
+----------------------------------------+
| TABLE (1 of 14): `airportdb`.`airline` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 4 |
| Table loaded using 1 thread(s) |
| |
+----------------------------------------+
7 rows in set (5.2867 sec)
+-----------------------------------------+
| LOADING TABLE |
+-----------------------------------------+
| TABLE (2 of 14): `airportdb`.`airplane` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 4 |
| Table loaded using 1 thread(s) |
| |
+-----------------------------------------+
7 rows in set (5.2867 sec)
+----------------------------------------------+
| LOADING TABLE |
+----------------------------------------------+
| TABLE (3 of 14): `airportdb`.`airplane_type` |
| Commands executed successfully: 5 of 5 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 2 |
| Table loaded using 1 thread(s) |
| |
+----------------------------------------------+
7 rows in set (5.2867 sec)
+----------------------------------------+
| LOADING TABLE |
+----------------------------------------+
| TABLE (4 of 14): `airportdb`.`airport` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 4 |
| Table loaded using 1 thread(s) |
| |
+----------------------------------------+
7 rows in set (5.2867 sec)
+--------------------------------------------+
| LOADING TABLE |
+--------------------------------------------+
| TABLE (5 of 14): `airportdb`.`airport_geo` |
| Commands executed successfully: 5 of 5 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 6 |
| Table loaded using 1 thread(s) |
| |
+--------------------------------------------+
7 rows in set (5.2867 sec)
+--------------------------------------------------+
| LOADING TABLE |
+--------------------------------------------------+
| TABLE (6 of 14): `airportdb`.`airport_reachable` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 2 |
| Table loaded using 1 thread(s) |
| |
+--------------------------------------------------+
7 rows in set (5.2867 sec)
+----------------------------------------+
| LOADING TABLE |
+----------------------------------------+
| TABLE (7 of 14): `airportdb`.`booking` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 5 |
| Table loaded using 32 thread(s) |
| |
+----------------------------------------+
7 rows in set (5.2867 sec)
+-----------------------------------------+
| LOADING TABLE |
+-----------------------------------------+
| TABLE (8 of 14): `airportdb`.`employee` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 15 |
| Table loaded using 1 thread(s) |
| |
+-----------------------------------------+
7 rows in set (5.2867 sec)
+----------------------------------------+
| LOADING TABLE |
+----------------------------------------+
| TABLE (9 of 14): `airportdb`.`flight` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 8 |
| Table loaded using 9 thread(s) |
| |
+----------------------------------------+
7 rows in set (5.2867 sec)
+--------------------------------------------+
| LOADING TABLE |
+--------------------------------------------+
| TABLE (10 of 14): `airportdb`.`flight_log` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 19 |
| Table loaded using 1 thread(s) |
| |
+--------------------------------------------+
7 rows in set (5.2867 sec)
+------------------------------------------------+
| LOADING TABLE |
+------------------------------------------------+
| TABLE (11 of 14): `airportdb`.`flightschedule` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 13 |
| Table loaded using 1 thread(s) |
| |
+------------------------------------------------+
7 rows in set (5.2867 sec)
+-------------------------------------------+
| LOADING TABLE |
+-------------------------------------------+
| TABLE (12 of 14): `airportdb`.`passenger` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 4 |
| Table loaded using 1 thread(s) |
| |
+-------------------------------------------+
7 rows in set (5.2867 sec)
+--------------------------------------------------+
| LOADING TABLE |
+--------------------------------------------------+
| TABLE (13 of 14): `airportdb`.`passengerdetails` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 9 |
| Table loaded using 1 thread(s) |
| |
+--------------------------------------------------+
7 rows in set (5.2867 sec)
+---------------------------------------------+
| LOADING TABLE |
+---------------------------------------------+
| TABLE (14 of 14): `airportdb`.`weatherdata` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 9 |
| Table loaded using 18 thread(s) |
| |
+---------------------------------------------+
7 rows in set (5.2867 sec)
+-------------------------------------------------------------------------------+
| LOAD SUMMARY |
+-------------------------------------------------------------------------------+
| |
| SCHEMA TABLES TABLES COLUMNS LOAD |
| NAME LOADED FAILED LOADED DURATION |
| ------ ------ ------ ------- -------- |
| `airportdb` 14 0 104 5.16 s |
| |
+-------------------------------------------------------------------------------+
6 rows in set (5.2867 sec)
Query OK, 0 rows affected (5.2867 sec)
确认以下查询可以利用HeatWave能力(engine RAPID):
MySQL 10.0.1.233:33060+ ssl airportdb SQL > EXPLAIN SELECT booking.price, count(*) FROM booking WHERE booking.price > 500 GROUP BY booking.price ORDER BY booking.price LIMIT 10 \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: booking
partitions: NULL
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 51306483
filtered: 33.32999801635742
Extra: Using where; Using temporary; Using filesort; Using secondary engine RAPID
1 row in set, 1 warning (0.0054 sec)
Note (code 1003): /* select#1 */ select `airportdb`.`booking`.`price` AS `price`,count(0) AS `count(*)` from `airportdb`.`booking` where (`airportdb`.`booking`.`price` > 500.00) group by `airportdb`.`booking`.`price` order by `airportdb`.`booking`.`price` limit 10
执行查询,使用和禁用HeatWave的查询时间为0.1596和9.0713秒,性能提升为56.8倍。
MySQL 10.0.1.233:33060+ ssl airportdb SQL > SELECT booking.price, count(*) FROM booking WHERE booking.price > 500 GROUP BY booking.price ORDER BY booking.price LIMIT 10;
+--------+----------+
| price | count(*) |
+--------+----------+
| 500.01 | 860 |
| 500.02 | 1207 |
| 500.03 | 1135 |
| 500.04 | 1010 |
| 500.05 | 1016 |
| 500.06 | 1039 |
| 500.07 | 1002 |
| 500.08 | 1095 |
| 500.09 | 1117 |
| 500.10 | 1106 |
+--------+----------+
10 rows in set (0.1596 sec)
MySQL 10.0.1.233:33060+ ssl airportdb SQL > SET SESSION use_secondary_engine=OFF;
Query OK, 0 rows affected (0.0004 sec)
MySQL 10.0.1.233:33060+ ssl airportdb SQL > SELECT booking.price, count(*) FROM booking WHERE booking.price > 500 GROUP BY booking.price ORDER BY booking.price LIMIT 10;
+--------+----------+
| price | count(*) |
+--------+----------+
| 500.01 | 860 |
| 500.02 | 1207 |
| 500.03 | 1135 |
| 500.04 | 1010 |
| 500.05 | 1016 |
| 500.06 | 1039 |
| 500.07 | 1002 |
| 500.08 | 1095 |
| 500.09 | 1117 |
| 500.10 | 1106 |
+--------+----------+
10 rows in set (9.0713 sec)
运行以下三个可选的查询,未使用HeatWave的查询时间为(9.0642,12.7548,23.3557 ),使用HeatWave的查询时间为(0.0800,2.2155,0.2260)。性能提升分别为(113.3,5.76,103.3)倍。
MySQL 10.0.1.233:33060+ ssl airportdb SQL > SELECT
-> booking.price,
-> count(*)
-> FROM
-> booking
-> WHERE
-> booking.price > 500
-> GROUP BY
-> booking.price
-> ORDER BY
-> booking.price
-> LIMIT
-> 10;
+--------+----------+
| price | count(*) |
+--------+----------+
| 500.01 | 860 |
| 500.02 | 1207 |
| 500.03 | 1135 |
| 500.04 | 1010 |
| 500.05 | 1016 |
| 500.06 | 1039 |
| 500.07 | 1002 |
| 500.08 | 1095 |
| 500.09 | 1117 |
| 500.10 | 1106 |
+--------+----------+
10 rows in set (9.0642 sec)
MySQL 10.0.1.233:33060+ ssl airportdb SQL > SELECT
-> airline.airlinename,
-> AVG(datediff(departure,birthdate)/365.25) as avg_age,
-> count(*) as nb_people
-> FROM
-> booking, flight, airline, passengerdetails
-> WHERE
-> booking.flight_id=flight.flight_id AND
-> airline.airline_id=flight.airline_id AND
-> booking.passenger_id=passengerdetails.passenger_id AND
-> country IN ("SWITZERLAND", "FRANCE", "ITALY")
-> GROUP BY
-> airline.airlinename
-> ORDER BY
-> airline.airlinename, avg_age
-> LIMIT 10;
+----------------------+-------------+-----------+
| airlinename | avg_age | nb_people |
+----------------------+-------------+-----------+
| Afghanistan Airlines | 45.92612137 | 20570 |
| Albania Airlines | 46.02485633 | 21804 |
| American Samoa Airli | 46.10098466 | 15249 |
| Angola Airlines | 45.97214161 | 18539 |
| Argentina Airlines | 46.03115082 | 21298 |
| Australia Airlines | 46.07925759 | 19986 |
| Azerbaijan Airlines | 46.01659919 | 16011 |
| Bahamas Airlines | 46.22542566 | 22564 |
| Belarus Airlines | 46.10818585 | 18226 |
| Bhutan Airlines | 46.13252703 | 22873 |
+----------------------+-------------+-----------+
10 rows in set (12.7548 sec)
MySQL 10.0.1.233:33060+ ssl airportdb SQL > SELECT
count(*) as nb_tickets
FROM
booking, flight, airline, airport_geo
WHERE
booking.flight_id=flight.flight_id AND
airline.airline_id=flight.airline_id AND
flight.from=airport_geo.airport_id AND
airport_geo.country = "UNITED STATES"
GROUP BY
airline.airlinename
ORDER BY
nb_tickets desc, airline.airlinename
LIMIT 10; airline.airlinename,
-> SUM(booking.price) as price_tickets,
-> count(*) as nb_tickets
-> FROM
-> booking, flight, airline, airport_geo
-> WHERE
-> booking.flight_id=flight.flight_id AND
-> airline.airline_id=flight.airline_id AND
-> flight.from=airport_geo.airport_id AND
-> airport_geo.country = "UNITED STATES"
-> GROUP BY
-> airline.airlinename
-> ORDER BY
-> nb_tickets desc, airline.airlinename
-> LIMIT 10;
+----------------------+---------------+------------+
| airlinename | price_tickets | nb_tickets |
+----------------------+---------------+------------+
| Falkland Is Airlines | 54329614.83 | 216237 |
| Micronesia Airlines | 53875825.72 | 214275 |
| Brazil Airlines | 50039222.97 | 199451 |
| Cyprus Airlines | 47466718.63 | 189493 |
| Yugoslavia Airlines | 46905122.48 | 186865 |
| Italy Airlines | 45731672.06 | 182152 |
| Peru Airlines | 45742817.89 | 182111 |
| Luxembourg Airlines | 44537553.63 | 177952 |
| Chad Airlines | 43162776.69 | 172159 |
| Australia Airlines | 41201251.28 | 164501 |
+----------------------+---------------+------------+
10 rows in set (23.3557 sec)
MySQL 10.0.1.233:33060+ ssl airportdb SQL > SET SESSION use_secondary_engine=ON ;
Query OK, 0 rows affected (0.0003 sec)
MySQL 10.0.1.233:33060+ ssl airportdb SQL > SELECT
-> booking.price,
-> count(*)
-> FROM
-> booking
-> WHERE
-> booking.price > 500
-> GROUP BY
-> booking.price
-> ORDER BY
-> booking.price
-> LIMIT
-> 10;
+--------+----------+
| price | count(*) |
+--------+----------+
| 500.01 | 860 |
| 500.02 | 1207 |
| 500.03 | 1135 |
| 500.04 | 1010 |
| 500.05 | 1016 |
| 500.06 | 1039 |
| 500.07 | 1002 |
| 500.08 | 1095 |
| 500.09 | 1117 |
| 500.10 | 1106 |
+--------+----------+
10 rows in set (0.0800 sec)
MySQL 10.0.1.233:33060+ ssl airportdb SQL > SELECT
-> airline.airlinename,
-> AVG(datediff(departure,birthdate)/365.25) as avg_age,
-> count(*) as nb_people
-> FROM
-> booking, flight, airline, passengerdetails
-> WHERE
-> booking.flight_id=flight.flight_id AND
-> airline.airline_id=flight.airline_id AND
-> booking.passenger_id=passengerdetails.passenger_id AND
-> country IN ("SWITZERLAND", "FRANCE", "ITALY")
-> GROUP BY
-> airline.airlinename
ORDER BY
airline.airlinename, avg_age
LIMIT 10; ORDER BY
-> airline.airlinename, avg_age
-> LIMIT 10;
+----------------------+-------------+-----------+
| airlinename | avg_age | nb_people |
+----------------------+-------------+-----------+
| Afghanistan Airlines | 45.92607116 | 20570 |
| Albania Airlines | 46.02480595 | 21804 |
| American Samoa Airli | 46.10093492 | 15249 |
| Angola Airlines | 45.97209191 | 18539 |
| Argentina Airlines | 46.03110048 | 21298 |
| Australia Airlines | 46.07920841 | 19986 |
| Azerbaijan Airlines | 46.01654847 | 16011 |
| Bahamas Airlines | 46.22537506 | 22564 |
| Belarus Airlines | 46.10813567 | 18226 |
| Bhutan Airlines | 46.13247687 | 22873 |
+----------------------+-------------+-----------+
10 rows in set (2.2155 sec)
MySQL 10.0.1.233:33060+ ssl airportdb SQL > SELECT
-> airline.airlinename,
-> SUM(booking.price) as price_tickets,
-> count(*) as nb_tickets
-> FROM
-> booking, flight, airline, airport_geo
-> WHERE
-> booking.flight_id=flight.flight_id AND
-> airline.airline_id=flight.airline_id AND
-> flight.from=airport_geo.airport_id AND
-> airport_geo.country = "UNITED STATES"
-> GROUP BY
-> airline.airlinename
-> ORDER BY
-> nb_tickets desc, airline.airlinename
-> LIMIT 10;
+----------------------+---------------+------------+
| airlinename | price_tickets | nb_tickets |
+----------------------+---------------+------------+
| Falkland Is Airlines | 54329614.83 | 216237 |
| Micronesia Airlines | 53875825.72 | 214275 |
| Brazil Airlines | 50039222.97 | 199451 |
| Cyprus Airlines | 47466718.63 | 189493 |
| Yugoslavia Airlines | 46905122.48 | 186865 |
| Italy Airlines | 45731672.06 | 182152 |
| Peru Airlines | 45742817.89 | 182111 |
| Luxembourg Airlines | 44537553.63 | 177952 |
| Chad Airlines | 43162776.69 | 172159 |
| Australia Airlines | 41201251.28 | 164501 |
+----------------------+---------------+------------+
10 rows in set (0.2260 sec)