| NO | 步骤 |
| 1 | 安装配置jdk1.8 |
| 2 | 安装配置scala2.11.18 |
| 3 | 写操作hive的demo |
| 4 | 导入pom.xml中依赖jar |
| 5 | 下载hadoop的binary包，我的版本是2.7.3 |
| 6 | 下载winutils.exe将其放到HADOOP_HOME/bin/目录下</td></tr><tr><td>7</td><td>在启动类的运行参数中设置环境变量，HADOOP_HOME=D:\software1\hadoop-2.7.3，=后面是hadoop的安装目录</td></tr><tr><td>8</td><td>将hive-site.xml配置文件放到工程的resource目录中</td></tr><tr><td>9</td><td>如果有权限问题，可以关闭namenode权限检查</td></tr></tbody></table><p>pom.xml的依赖关系如下：</p><pre οnclick="hljs.copyCode(event)"><code class="language-html hljs xml"><ol class="hljs-ln" style="width:918px"><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="1"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"><span class="php"><span class="hljs-meta"><?</span>xml version=<span class="hljs-string">"1.0"</span> encoding=<span class="hljs-string">"UTF-8"</span><span class="hljs-meta">?></span></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="2"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"><span class="hljs-tag"><<span class="hljs-name">project</span> <span class="hljs-attr">xmlns</span>=<span class="hljs-string">"http://maven.apache.org/POM/4.0.0"</span></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="3"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-attr">xmlns:xsi</span>=<span class="hljs-string">"http://www.w3.org/2001/XMLSchema-instance"</span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="4"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-attr">xsi:schemaLocation</span>=<span class="hljs-string">"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"</span>></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="5"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">modelVersion</span>></span>4.0.0<span class="hljs-tag"></<span class="hljs-name">modelVersion</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="6"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> </div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="7"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">groupId</span>></span>com.lwq<span class="hljs-tag"></<span class="hljs-name">groupId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="8"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>sparkcase2<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="9"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">version</span>></span>1.0-SNAPSHOT<span class="hljs-tag"></<span class="hljs-name">version</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="10"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> </div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="11"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">properties</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="12"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">java.version</span>></span>1.8<span class="hljs-tag"></<span class="hljs-name">java.version</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="13"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">spark.version</span>></span>2.1.1<span class="hljs-tag"></<span class="hljs-name">spark.version</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="14"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"></<span class="hljs-name">properties</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="15"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> </div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="16"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">dependencies</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="17"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">dependency</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="18"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">groupId</span>></span>org.apache.spark<span class="hljs-tag"></<span class="hljs-name">groupId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="19"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>spark-core_2.11<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="20"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">version</span>></span> HADOOP_HOME/bin/目录下</td></tr><tr><td>7</td><td>在启动类的运行参数中设置环境变量，HADOOP_HOME=D:\software1\hadoop-2.7.3，=后面是hadoop的安装目录</td></tr><tr><td>8</td><td>将hive-site.xml配置文件放到工程的resource目录中</td></tr><tr><td>9</td><td>如果有权限问题，可以关闭namenode权限检查</td></tr></tbody></table><p>pom.xml的依赖关系如下：</p><pre οnclick="hljs.copyCode(event)"><code class="language-html hljs xml"><ol class="hljs-ln" style="width:918px"><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="1"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"><span class="php"><span class="hljs-meta"><?</span>xml version=<span class="hljs-string">"1.0"</span> encoding=<span class="hljs-string">"UTF-8"</span><span class="hljs-meta">?></span></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="2"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"><span class="hljs-tag"><<span class="hljs-name">project</span> <span class="hljs-attr">xmlns</span>=<span class="hljs-string">"http://maven.apache.org/POM/4.0.0"</span></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="3"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-attr">xmlns:xsi</span>=<span class="hljs-string">"http://www.w3.org/2001/XMLSchema-instance"</span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="4"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-attr">xsi:schemaLocation</span>=<span class="hljs-string">"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"</span>></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="5"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">modelVersion</span>></span>4.0.0<span class="hljs-tag"></<span class="hljs-name">modelVersion</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="6"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> </div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="7"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">groupId</span>></span>com.lwq<span class="hljs-tag"></<span class="hljs-name">groupId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="8"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>sparkcase2<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="9"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">version</span>></span>1.0-SNAPSHOT<span class="hljs-tag"></<span class="hljs-name">version</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="10"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> </div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="11"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">properties</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="12"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">java.version</span>></span>1.8<span class="hljs-tag"></<span class="hljs-name">java.version</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="13"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">spark.version</span>></span>2.1.1<span class="hljs-tag"></<span class="hljs-name">spark.version</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="14"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"></<span class="hljs-name">properties</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="15"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> </div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="16"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">dependencies</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="17"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">dependency</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="18"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">groupId</span>></span>org.apache.spark<span class="hljs-tag"></<span class="hljs-name">groupId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="19"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>spark-core_2.11<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span></div></div></li><li><div class="hljs-ln-numbers"><div class="hljs-ln-line hljs-ln-n" data-line-number="20"></div></div><div class="hljs-ln-code"><div class="hljs-ln-line"> <span class="hljs-tag"><<span class="hljs-name">version</span>></span> {spark.version}</version> |
-
</dependency>
-
-
<dependency>
-
<groupId>mysql </groupId>
-
<artifactId>mysql-connector-java </artifactId>
-
<version>5.1.38 </version>
-
</dependency>
-
-
<dependency>
-
<groupId>org.apache.spark </groupId>
-
<artifactId>spark-hive_2.11 </artifactId>
-
<version>${spark.version} </version>
-
</dependency>
-
</dependencies>
-
</project>
- hive-site.xml的配置如下:
-
<?xml version="1.0"?>
-
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-
<configuration>
-
# hive元数据服务的地址
-
<property>
-
<name>hive.metastore.uris </name>
-
<value>thrift://master:9083 </value>
-
</property>
-
<property>
-
<name>hive.server2.thrift.port </name>
-
<value>10000 </value>
-
</property>
-
<property> #元数据地址
-
<name>javax.jdo.option.ConnectionURL </name>
-
<value>jdbc:mysql://master/metastore?createDatabaseIfNotExist=true </value>
-
</property>
-
<property>
-
<name>javax.jdo.option.ConnectionDriverName </name>
-
<value>com.mysql.jdbc.Driver </value>
-
</property>
-
<property> #连接元数据数据库账号
-
<name>javax.jdo.option.ConnectionUserName </name>
-
<value>hadoop </value>
-
</property>
-
<property> #连接元数据数据库密码
-
<name>javax.jdo.option.ConnectionPassword </name>
-
<value>hadoop </value>
-
</property>
-
<property> #hive数据存储在hdfs上的目录
-
<name>hive.metastore.warehouse.dir </name>
-
<value>/warehouse </value>
-
</property>
-
<property>
-
<name>fs.defaultFS </name>
-
<value>hdfs://master:9000 </value>
-
</property>
-
<property> #元数据schema验证
-
<name>hive.metastore.schema.verification </name>
-
<value>false </value>
-
</property>
-
<property>
-
<name>datanucleus.autoCreateSchema </name>
-
<value>true </value>
-
</property>
-
<property>
-
<name>datanucleus.autoStartMechanism </name>
-
<value>SchemaTable </value>
-
</property>
-
<property>
-
<name>datanucleus.schema.autoCreateTables </name>
-
<value>true </value>
-
</property>
-
<property>
-
<name>beeline.hs2.connection.user </name>
-
<value>bigdata </value>
-
</property>
-
<property>
-
<name>beeline.hs2.connection.password </name>
-
<value>root </value>
-
</property>
-
</configuration>
演示代码:
-
package com.lwq.spark
-
-
import org.apache.spark.sql.SparkSession
-
-
object HiveCaseJob {
-
def main(args: Array[String]): Unit = {
-
val sparkSession = SparkSession.builder().appName("HiveCaseJob").master("local[*]").enableHiveSupport().getOrCreate()
-
-
sparkSession.sql("drop table if exists users")
-
sparkSession.sql("show tables").show()
-
sparkSession.sql("create table if not exists users(id int,name string) row format delimited fields terminated by ' ' stored as textfile")
-
sparkSession.sql("show tables").show()
-
sparkSession.sql("select * from users").show()
-
sparkSession.sql("load data local inpath 'src/main/resources/a.txt' overwrite into table users")
-
sparkSession.sql("select * from users").show()
-
-
}
-
}
配置运行参数:
运行过程中可能的报错：
这是HADOOP_HOME变量和winutils.exe的原因，查看变量是否配好，winutils是否放在正确目录。
因为是在windows环境，在运行的时候会模拟linux，需要winutils.exe，winutils.exe的路径代码中默认是从hadoop的安装路径的bin目录下。所以还需要上面的HADOOP_HOME变量。
还有一个错误就是提示权限不足的，因为在windows下是通过windows的用户来访问的，可以关闭hdfs的权限检查
如下：
-
3462
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
