Spark操作Hive所需依赖（Java版本）

最新推荐文章于 2025-02-14 18:21:16 发布

hongmofang10

最新推荐文章于 2025-02-14 18:21:16 发布

阅读量2.1k

点赞数

分类专栏： spark Java开发

本文链接：https://blog.csdn.net/hongmofang10/article/details/108885701

版权

Java开发同时被 2 个专栏收录

16 篇文章

订阅专栏

spark

13 篇文章

订阅专栏

该博客展示了如何在Java中配置和使用Spark与Hive进行数据操作。通过Maven构建文件，详细列出了依赖项，包括Spark、Hive、ThriftServer和JDBC驱动。测试代码演示了如何创建SparkSession、连接Hive并执行基本的数据库和表操作。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.sun</groupId>
	<artifactId>spark-hive</artifactId>
	<version>0.0.1-SNAPSHOT</version>

	<properties>
		<spark.version>2.3.1</spark.version>
		<scala.version>2.11</scala.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
			<version>1.18.12</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.spark</groupId>
			<artifactId>spark-sql_${scala.version}</artifactId>
			<version>${spark.version}</version>
		</dependency>

		<dependency>
			<groupId>org.apache.spark</groupId>
			<artifactId>spark-hive_${scala.version}</artifactId>
			<version>${spark.version}</version>
		</dependency>

		<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive-thriftserver -->
		<dependency>
			<groupId>org.apache.spark</groupId>
			<artifactId>spark-hive-thriftserver_2.11</artifactId>
			<version>2.3.1</version>
			<scope>provided</scope>
		</dependency>

		<!-- https://mvnrepository.com/artifact/org.apache.hive/hive-jdbc -->
		<dependency>
			<groupId>org.apache.hive</groupId>
			<artifactId>hive-jdbc</artifactId>
			<version>2.3.1</version>
		</dependency>
		<!-- https://mvnrepository.com/artifact/mysql/mysql-connector-java -->
		<dependency>
			<groupId>mysql</groupId>
			<artifactId>mysql-connector-java</artifactId>
			<version>5.1.26</version>
		</dependency>
	</dependencies>
	<build>
		<pluginManagement><!-- lock down plugins versions to avoid using Maven 
				defaults (may be moved to parent pom) -->
			<plugins>
				<!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
				<plugin>
					<groupId>org.apache.maven.plugins</groupId>
					<artifactId>maven-assembly-plugin</artifactId>
					<configuration>
						<archive>
							<manifest>
								<mainClass>comspark.Main</mainClass>
							</manifest>
						</archive>
						<descriptorRefs>
							<descriptorRef>
								jar-with-dependencies
							</descriptorRef>
						</descriptorRefs>
					</configuration>
				</plugin>
				<plugin>
					<artifactId>maven-clean-plugin</artifactId>
					<version>3.1.0</version>
				</plugin>
				<!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
				<plugin>
					<artifactId>maven-resources-plugin</artifactId>
					<version>3.0.2</version>
				</plugin>
				<plugin>
					<artifactId>maven-compiler-plugin</artifactId>
					<version>3.8.0</version>
				</plugin>
				<plugin>
					<artifactId>maven-surefire-plugin</artifactId>
					<version>2.22.1</version>
				</plugin>
				<plugin>
					<artifactId>maven-jar-plugin</artifactId>
					<version>3.0.2</version>
				</plugin>
				<plugin>
					<artifactId>maven-install-plugin</artifactId>
					<version>2.5.2</version>
				</plugin>
				<plugin>
					<artifactId>maven-deploy-plugin</artifactId>
					<version>2.8.2</version>
				</plugin>
				<!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
				<plugin>
					<artifactId>maven-site-plugin</artifactId>
					<version>3.7.1</version>
				</plugin>
				<plugin>
					<artifactId>maven-project-info-reports-plugin</artifactId>
					<version>3.0.0</version>
				</plugin>
			</plugins>
		</pluginManagement>
	</build>
</project>

测试代码如下：

	SparkSession spark = SparkSession
                .builder()
                .appName("Java Spark Hive Example")
                .config("spark.sql.warehouse.dir","hdfs://****/user/hive/warehouse")
                .enableHiveSupport()
                .getOrCreate();

        spark.sql("show databases").show();
        spark.sql("use test").show();
        spark.sql("select * from hive01").show();