hive udf 使用示例

 在使用过程中,在执行sql的过程中需要将partition按照一定的规则动态分区:

import org.apache.hadoop.hive.ql.exec.UDF;
public class SharedRouter extends UDF {
	private static SolrCloudPainRouter cloudPainRouter;

	public String evaluate(final String shardValue
, final String collection, final String runtime) {
		return getRouter(collection, runtime).getShardIndex(shardValue);
	}

	private SolrCloudPainRouter getRouter(String collection, String runtime) {

		// 构建cloudPainRouter
		return cloudPainRouter;

	}
}

 

 

创建临时函数:

create temporary function tisshare as 'com.dfire.tis.dump.hive.SharedRouter';
 

在hive中调用udf测试:

select tisshare('hello','search4totalpay','daily');
函数说明,此函数一共三个参数,说明如下:
  • 参数1:某列属性
  • 参数2:对应分区tis中的某个索引名称
  • 参数3:运行环境,日常使用‘daily’,线上生产环境'online'

这个udf执行的时候需要依赖一些jar包,所以在启动hive时,需要将jar添加到classpath中去:

hive 启动

hive --auxpath /root/standalone/commons-io-2.3.jar,/root/standalone/commons-logging-1.2.jar,/root/standalone/fastjson-1.1.41.jar,/root/standalone/global-diamond-config-1.0.3-SNAPSHOT.jar,/root/standalone/hive-exec-1.1.0.jar,/root/standalone/noggit-0.6.jar,/root/standalone/slf4j-api-1.7.5.jar,/root/standalone/solr-solrj-5.3.0.jar,/root/standalone/spring-2.5.6.jar,/root/standalone/terminator-manage-pojo-1.0.3-SNAPSHOT.jar,/root/standalone/tis-client-standalone-1.0.3-SNAPSHOT.jar,/root/standalone/tis-solrj-client-1.0.3-SNAPSHOT.jar,/root/standalone/zookeeper-3.4.5.jar

在hive脚本中使用例子

ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/commons-io-2.3.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/commons-logging-1.2.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/fastjson-1.1.41.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/global-diamond-config.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/hive-exec-1.1.0.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/noggit-0.6.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/slf4j-api-1.7.5.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/solr-solrj-5.3.0.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/spring-2.5.6.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/terminator-manage-pojo-1.0.3-SNAPSHOT.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/tis-client-standalone-1.0.3-SNAPSHOT.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/tis-solrj-client-1.0.3-SNAPSHOT.jar;
ADD jar hdfs://cluster-cdh/zeus/hdfs-upload-dir/jar/zookeeper-3.4.5.jar;
CREATE TEMPORARY FUNCTION tisshare as 'com.dfire.tis.dump.hive.SharedRouter';


select tisshare('123','search4totalpay','online') 
 

 

 

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值