spark-sql(二)---ThriftServer模式

ThriftServer

ThriftServer和和hiveserver2类似，提供了jdbc/odbc的接口，spark-sql以这种模式启动后可以以服务的形式提供SQL查询。
客户端可以用jdbc直接访问spark-sql，用sql进行交互。

启动

安装配置接前面链接
http://blog.csdn.net/dante_003/article/details/78652534
配置文件用的上个连接 conf/hive-site.xml

sbin/start-thriftserver.sh --master yarn \
--driver-cores 2 --driver-memory 2g \
--executor-cores 13 --executor-memory 16g \
--num-executors 3

beeline

注意：一定要用spark文件夹中的beeline，不用使用hive文件夹中的beeline

[root@ht05 bin]# bin/beeline -u jdbc:hive2://localhost:10000
Connecting to jdbc:hive2://localhost:10000
17/11/28 17:43:40 INFO jdbc.Utils: Supplied authorities: localhost:10000
17/11/28 17:43:40 INFO jdbc.Utils: Resolved authority: localhost:10000
17/11/28 17:43:40 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://localhost:10000
Connected to: Spark SQL (version 2.2.0)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1.spark2 by Apache Hive
0: jdbc:hive2://localhost:10000> show tables;
+-----------+---------------+--------------+--+
| database  |   tableName   | isTemporary  |
+-----------+---------------+--------------+--+
| default   | test_orc      | false        |
| default   | test_orc_t    | false        |
| default   | test_phoenix  | false        |
| default   | test_txt      | false        |
| default   | test_txt1     | false        |
+-----------+---------------+--------------+--+
5 rows selected (0.587 seconds)
0: jdbc:hive2://localhost:10000> select count(*) from test_txt1;
+--------------+--+
|   count(1)   |
+--------------+--+
| 12818207296  |
+--------------+--+
1 row selected (262.646 seconds)

java jdbc连接

取出jdbc驱动，位置在spark/jars/hive-jdbc-1.2.1.spark2.jar
TestSparksql.java

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;

public class TestSparksql {
    private static String driverName="org.apache.hive.jdbc.HiveDriver";

    public static void main(String[] args) throws ClassNotFoundException, SQLException {
        Class.forName(driverName);
        Connection con=DriverManager.getConnection("jdbc:hive2://192.168.2.5:10000/default", "", "");
        Statement statement = con.createStatement();

        String sql="select * from test_txt1 limit 10";
        ResultSet executeQuery = statement.executeQuery(sql);
        while(executeQuery.next()){
            String string = executeQuery.getString(1);
            System.out.println(string);
        }
        statement.close();
        con.close();
    }
}