一、目的
在开发过程中,不可避免需要在集群外部调试程序,以便快速开发,测试。
二、环境
1.IDEA
2.CDH 5.7.0(Hive、HADOOP)
三、步骤
1.创建Maven项目
2.加载对应CDH需求依赖
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>BG</groupId>
<artifactId>Hadoop</artifactId>
<version>1.0-SNAPSHOT</version>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<hadoop.version>2.6.0-cdh5.7.0</hadoop.version>
<hbase.version>1.2.0-cdh5.7.0</hbase.version>
<hive.version>1.1.0-cdh5.7.0</hive.version>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-metastore</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
3.加载hdfs-site.xml、core-site.xml和log4j.properties到resources目录下
4.将hadoop-2.6.0-cdh5.7.0的tar包直接解压缩到本地,并且配置Hadoop环境变量
5.添加winutils.exe到hadoop-2.6.0-cdh5.7.0\bin目录下,并且配置winutils.exe到系统Path中
6.开启hdfs和hive metastore以及hive server2
bin/hive --service metastore &
bin/hive --service hiveserver2 &
四、链接Hive测试代码
1.参考
https://blog.csdn.net/u010886217/article/details/83926355