Spark 连接hive local

采用读取recources文件夹下的配制文件方式hive-site.xml 
一.通过hiveserver2 服务连接     $HIVE_HOME/bin/hiveserver2
object NewSparkHiveDemo {

    def main(args: Array[String]): Unit = {

        System.setProperty("HADOOP_USER_NAME", "fengjiangyue")

        val spark: SparkSession = SparkSession.builder()
          .appName("spark hive Example")
          .master("local")
//          .config("hive.metastore.uris", "thrift://localhost:9083")
          .config("spark.sql.warehouse.dir", "hdfs://localhost:9000/user/hive/warehouse/")
          .enableHiveSupport()
          .getOrCreate()

        import spark.implicits._
        import spark.sql

        sql("show tables").show()

//        sql("create table if not exists src(key INT, value STRING)")
//        val data = Array((1, "val1"),(2, "val2"),(3,"val3"))
//        var df = spark.createDataFrame(data).toDF("key","value")
//        df.createOrReplaceTempView("temp_src")
//        sql("insert into src select key,value from temp_src")
//        sql("select * from src").show()

    }
}

hive-site.xml如下

<configuration>
   <property>
     <name>javax.jdo.option.ConnectionURL</name>
     <value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true</value>
     <description>JDBC connect string for a JDBC metastore</description>
   </property>
   <property>
     <name>javax.jdo.option.ConnectionDriverName</name>
     <value>com.mysql.jdbc.Driver</value>
     <description>Driver class name for a JDBC metastore</description>
   </property>
   <property>
     <name>javax.jdo.option.ConnectionUserName</name>
     <value>root</value>
     <description>username to use against metastore database</description>
   </property>
   <property>
     <name>javax.jdo.option.ConnectionPassword</name>
     <value>root</value>
     <description>password to use against metastore database</description>
   </property>
</configuration>

21/12/25 08:18:08 INFO HiveConf: Found configuration file file:/Users/fengjiangyue/IdeaProjects/sparklearn/sparkdoit/target/classes/hive-site.xml
21/12/25 08:18:09 INFO SessionState: Created HDFS directory: /tmp/hive/fengjiangyue/0266e12f-79bc-4c58-bad7-be9bbeebb47b
21/12/25 08:18:09 INFO SessionState: Created local directory: /var/folders/f8/y6vdl3zs5_10j0tyq235py540000gn/T/fengjiangyue/0266e12f-79bc-4c58-bad7-be9bbeebb47b
21/12/25 08:18:09 INFO SessionState: Created HDFS directory: /tmp/hive/fengjiangyue/0266e12f-79bc-4c58-bad7-be9bbeebb47b/_tmp_space.db
21/12/25 08:18:09 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is hdfs://localhost:9000/user/hive/warehouse/
21/12/25 08:18:10 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
21/12/25 08:18:10 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
21/12/25 08:18:10 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
21/12/25 08:18:10 INFO ObjectStore: ObjectStore, initialize called
21/12/25 08:18:10 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
21/12/25 08:18:10 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
21/12/25 08:18:11 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
21/12/25 08:18:12 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL
21/12/25 08:18:12 INFO ObjectStore: Initialized ObjectStore
21/12/25 08:18:12 INFO HiveMetaStore: Added admin role in metastore
21/12/25 08:18:12 INFO HiveMetaStore: Added public role in metastore
21/12/25 08:18:12 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/12/25 08:18:12 INFO HiveMetaStore: 0: get_all_functions
21/12/25 08:18:12 INFO audit: ugi=fengjiangyue    ip=unknown-ip-addr    cmd=get_all_functions   21/12/25 08:18:12 INFO HiveMetaStore: 0: get_database: default

二.通过metastore 服务的thrift协义    开启$HIVE_HOME/bin/hive --service hiveserver2

object NewSparkHiveDemo {
    def main(args: Array[String]): Unit = {
        System.setProperty("HADOOP_USER_NAME", "fengjiangyue")
        val spark: SparkSession = SparkSession.builder()
          .appName("spark hive Example")
          .master("local")
          .config("hive.metastore.uris", "thrift://localhost:9083")
          .config("spark.sql.warehouse.dir", "hdfs://localhost:9000/user/hive/warehouse/")
          .enableHiveSupport()
          .getOrCreate()

        import spark.implicits._
        import spark.sql

        sql("show tables").show()
    }
}

hive-site.xml配制如下：
<configuration>
    <property>
       <name>hive.metastore.uris</name>
       <value>thrift://localhost:9083</value>
   </property>
</configuration>

21/12/25 08:15:27 INFO HiveUtils: Initializing HiveMetastoreConnection version 2.3.7 using Spark classes.
21/12/25 08:15:27 INFO HiveConf: Found configuration file file:/Users/fengjiangyue/IdeaProjects/sparklearn/sparkdoit/target/classes/hive-site.xml
21/12/25 08:15:28 INFO SessionState: Created HDFS directory: /tmp/hive/fengjiangyue/8abeee58-2369-4a59-aa80-f341c182a0e8
21/12/25 08:15:28 INFO SessionState: Created local directory: /var/folders/f8/y6vdl3zs5_10j0tyq235py540000gn/T/fengjiangyue/8abeee58-2369-4a59-aa80-f341c182a0e8
21/12/25 08:15:28 INFO SessionState: Created HDFS directory: /tmp/hive/fengjiangyue/8abeee58-2369-4a59-aa80-f341c182a0e8/_tmp_space.db
21/12/25 08:15:28 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is hdfs://localhost:9000/user/hive/warehouse/
21/12/25 08:15:28 INFO metastore: Trying to connect to metastore with URI thrift://localhost:9083
21/12/25 08:15:28 WARN metastore: Failed to connect to the MetaStore Server...
21/12/25 08:15:28 INFO metastore: Waiting 1 seconds before next connection attempt.
21/12/25 08:15:29 INFO metastore: Trying to connect to metastore with URI thrift://localhost:9083
21/12/25 08:15:29 WARN metastore: Failed to connect to the MetaStore Server...
21/12/25 08:15:29 INFO metastore: Waiting 1 seconds before next connection attempt.
21/12/25 08:15:30 INFO metastore: Trying to connect to metastore with URI thrift://localhost:9083
21/12/25 08:15:30 WARN metastore: Failed to connect to the MetaStore Server...
21/12/25 08:15:30 INFO metastore: Waiting 1 seconds before next connection attempt.
21/12/25 08:15:31 WARN Hive: Failed to register all functions.

Spark SQL入门到实战之（7）spark连接hive（spark-shell和eclipse两种方式） - 大码王 - 博客园

hiveserver2和metastore service的区别和联系_遥望......-CSDN博客_hiveserver2和metastore

Hive中MetaServer与HiveServer2的应用 - 大葱拌豆腐 - 博客园