spark start-thriftserver.sh & Kubernetes

启动命令

sh sbin/start-thriftserver.sh
–master k8s://https://192.168.99.108:8443
–name spark-thriftserver
–conf spark.executor.instances=1
–conf spark.kubernetes.container.image=zhixingheyitian/spark:spark2.4.1

如果默认端口10000被占，
则使用如下 环境变量 测试

export HIVE_SERVER2_THRIFT_PORT=10013

可参考官方文档： https://spark.apache.org/docs/3.0.0-preview2/sql-distributed-sql-engine.html

web UI

换一台机器执行query

这里以TPC-DS 为例

./sql/select.sql

# cat ./sql/select.sql
SELECT * FROM tpcds_1g_parquet.store_sales WHERE ss_customer_sk BETWEEN 10 AND 80 limit 10;

执行 ./bin/beeline

# ./bin/beeline -u jdbc:hive2://10.239.47.141:10000 -f ./sql/select.sql
Connecting to jdbc:hive2://10.239.47.141:10000
log4j:WARN No appenders could be found for logger (org.apache.hive.jdbc.Utils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Connected to: Spark SQL (version 2.4.1)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://10.239.47.141:10000> SELECT * FROM tpcds_1g_parquet.store_sales WHERE ss_customer_sk BETWEEN 10 AND 80 limit 10;
+------------------+------------------+-------------+-----------------+--------------+--------------+-------------+--------------+--------------+-------------------+--------------+--------------------+----------------+-----------------+----------------------+---------------------+------------------------+--------------------+-------------+----------------+--------------+----------------------+----------------+--+
| ss_sold_date_sk  | ss_sold_time_sk  | ss_item_sk  | ss_customer_sk  | ss_cdemo_sk  | ss_hdemo_sk  | ss_addr_sk  | ss_store_sk  | ss_promo_sk  | ss_ticket_number  | ss_quantity  | ss_wholesale_cost  | ss_list_price  | ss_sales_price  | ss_ext_discount_amt  | ss_ext_sales_price  | ss_ext_wholesale_cost  | ss_ext_list_price  | ss_ext_tax  | ss_coupon_amt  | ss_net_paid  | ss_net_paid_inc_tax  | ss_net_profit  |
+------------------+------------------+-------------+-----------------+--------------+--------------+-------------+--------------+--------------+-------------------+--------------+--------------------+----------------+-----------------+----------------------+---------------------+------------------------+--------------------+-------------+----------------+--------------+----------------------+----------------+--+
| 2451797          | 42040            | 17531       | 50              | 1378426      | 5249         | 2025        | 8            | 80           | 307               | 78           | 75.96              | 100.26         | 70.18           | 0.0                  | 5474.04             | 5924.88                | 7820.28            | 383.18      | 0.0            | 5474.04      | 5857.22              | -450.84        |
| 2451797          | 42040            | 3560        | 50              | 1378426      | 5249         | 2025        | 8            | 133          | 307               | 15           | 34.19              | 49.57          | 47.58           | 0.0                  | 713.7               | 512.85                 | 743.55             | 28.54       | 0.0            | 713.7        | 742.24               | 200.85         |
| 2451797          | 42040            | 14641       | 50              | 1378426      | 5249         | 2025        | 8            | 108          | 307               | 60           | 56.61              | 67.93          | 44.83           | 0.0                  | 2689.8              | 3396.6                 | 4075.8             | 242.08      | 0.0            | 2689.8       | 2931.88              | -706.8         |
| 2451797          | 42040            | 6146        | 50              | 1378426      | 5249         | 2025        | 8            | 162          | 307               | 86           | 88.72              | 135.74         | 65.15           | 2689.39              | 5602.9              | 7629.92                | 11673.64           | 145.67      | 2689.39        | 2913.51      | 3059.18              | -4716.41       |
| 2451797          | 42040            | 12272       | 50              | 1378426      | 5249         | 2025        | 8            | 185          | 307               | 56           | 69.44              | 138.18         | 69.09           | 2282.73              | 3869.04             | 3888.64                | 7738.08            | 15.86       | 2282.73        | 1586.31      | 1602.17              | -2302.33       |
| 2451797          | 42040            | 9113        | 50              | 1378426      | 5249         | 2025        | 8            | 262          | 307               | 60           | 76.06              | 149.07         | 137.14          | 0.0                  | 8228.4              | 4563.6                 | 8944.2             | 411.42      | 0.0            | 8228.4       | 8639.82              | 3664.8         |
| 2451797          | 42040            | 3650        | 50              | 1378426      | 5249         | 2025        | 8            | 83           | 307               | 52           | 80.38              | 114.13         | 97.01           | 4237.39              | 5044.52             | 4179.76                | 5934.76            | 32.28       | 4237.39        | 807.13       | 839.41               | -3372.63       |
| 2451797          | 42040            | 8855        | 50              | 1378426      | 5249         | 2025        | 8            | 224          | 307               | 73           | 21.54              | 22.18          | 6.87            | 0.0                  | 501.51              | 1572.42                | 1619.14            | 20.06       | 0.0            | 501.51       | 521.57               | -1070.91       |
| 2451797          | 42040            | 4184        | 50              | 1378426      | 5249         | 2025        | 8            | 101          | 307               | 88           | 79.63              | 97.14          | 46.62           | 0.0                  | 4102.56             | 7007.44                | 8548.32            | 287.17      | 0.0            | 4102.56      | 4389.73              | -2904.88       |
| 2451797          | 42040            | 7825        | 50              | 1378426      | 5249         | 2025        | 8            | 137          | 307               | 30           | 24.11              | 25.55          | 23.5            | 0.0                  | 705.0               | 723.3                  | 766.5              | 35.25       | 0.0            | 705.0        | 740.25               | -18.3          |
+------------------+------------------+-------------+-----------------+--------------+--------------+-------------+--------------+--------------+-------------------+--------------+--------------------+----------------+-----------------+----------------------+---------------------+------------------------+--------------------+-------------+----------------+--------------+----------------------+----------------+--+
10 rows selected (31.454 seconds)
0: jdbc:hive2://10.239.47.141:10000> 
0: jdbc:hive2://10.239.47.141:10000> 
0: jdbc:hive2://10.239.47.141:10000> 
0: jdbc:hive2://10.239.47.141:10000> 
0: jdbc:hive2://10.239.47.141:10000> 
Closing: 0: jdbc:hive2://10.239.47.141:10000

执行查询后的UI 状况

jobs

executors