hive udf 使用示例

最新推荐文章于 2024-04-21 11:38:08 发布

iteye_10738

最新推荐文章于 2024-04-21 11:38:08 发布

阅读量311

点赞数

分类专栏： hive 文章标签： hive udf

本文链接：https://blog.csdn.net/iteye_10738/article/details/82676572

版权

hive 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

在使用过程中，在执行sql的过程中需要将partition按照一定的规则动态分区：

import org.apache.hadoop.hive.ql.exec.UDF;
public class SharedRouter extends UDF {
	private static SolrCloudPainRouter cloudPainRouter;

	public String evaluate(final String shardValue
, final String collection, final String runtime) {
		return getRouter(collection, runtime).getShardIndex(shardValue);
	}

	private SolrCloudPainRouter getRouter(String collection, String runtime) {

		// 构建cloudPainRouter
		return cloudPainRouter;

	}
}

创建临时函数：

create temporary function tisshare as 'com.dfire.tis.dump.hive.SharedRouter';

在hive中调用udf测试：

select tisshare('hello','search4totalpay','daily');

函数说明，此函数一共三个参数，说明如下：

参数1：某列属性
参数2：对应分区tis中的某个索引名称
参数3：运行环境，日常使用‘daily’,线上生产环境'online'

这个udf执行的时候需要依赖一些jar包，所以在启动hive时，需要将jar添加到classpath中去：

hive 启动

hive --auxpath /root/standalone/commons-io-2.3.jar,/root/standalone/commons-logging-1.2.jar,/root/standalone/fastjson-1.1.41.jar,/root/standalone/global-diamond-config-1.0.3-SNAPSHOT.jar,/root/standalone/hive-exec-1.1.0.jar,/root/standalone/noggit-0.6.jar,/root/standalone/slf4j-api-1.7.5.jar,/root/standalone/solr-solrj-5.3.0.jar,/root/standalone/spring-2.5.6.jar,/root/standalone/terminator-manage-pojo-1.0.3-SNAPSHOT.jar,/root/standalone/tis-client-standalone-1.0.3-SNAPSHOT.jar,/root/standalone/tis-solrj-client-1.0.3-SNAPSHOT.jar,/root/standalone/zookeeper-3.4.5.jar

在hive脚本中使用例子

ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/commons-io-2.3.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/commons-logging-1.2.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/fastjson-1.1.41.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/global-diamond-config.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/hive-exec-1.1.0.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/noggit-0.6.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/slf4j-api-1.7.5.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/solr-solrj-5.3.0.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/spring-2.5.6.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/terminator-manage-pojo-1.0.3-SNAPSHOT.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/tis-client-standalone-1.0.3-SNAPSHOT.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/tis-solrj-client-1.0.3-SNAPSHOT.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/zookeeper-3.4.5.jar;
CREATE TEMPORARY FUNCTION tisshare as 'com.dfire.tis.dump.hive.SharedRouter';


select tisshare('123','search4totalpay','online')