Hive 临时和永久UDF实践

最新推荐文章于 2021-10-14 20:31:15 发布

AaronLwx

最新推荐文章于 2021-10-14 20:31:15 发布

阅读量920

点赞数

分类专栏： HIve

本文链接：https://blog.csdn.net/xiaoxiongaa0/article/details/89437041

版权

HIve 专栏收录该内容

6 篇文章 1 订阅

订阅专栏

[hadoop@hadoop004 lib]$ ls
g6-hadoop-1.0.jar  udflearn-1.0.jar

hive>
    > ADD JAR /home/hadoop/lib/udflearn-1.0.jar;
Added [/home/hadoop/lib/udflearn-1.0.jar] to class path
Added resources: [/home/hadoop/lib/udflearn-1.0.jar]

hive> CREATE TEMPORARY FUNCTION sayhello1 as 'com.hivelearn.udf.HelloUDF';
OK
Time taken: 0.316 seconds

hive> select sayhell1('girl');
FAILED: SemanticException [Error 10011]: Line 1:7 Invalid function 'sayhell1'

hive> select sayhello1('girl');
OK
hello:girl
Time taken: 0.42 seconds, Fetched: 1 row(s)

我这边java的版本为1.7，但在我的idea里面java的版本为1.8，但是在hive上面这个udf还是能够运行成功

下面开另外一个终端

hive> select sayhello1('girl');
FAILED: SemanticException [Error 10011]: Line 1:7 Invalid function 'sayhello1'

使用第二种方法创建临时函数

hive>
    >
    > CREATE TEMPORARY FUNCTION sayhello1 as 'com.hivelearn.udf.HelloUDF';
FAILED: Class com.hivelearn.udf.HelloUDF not found
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask

hive> exit;

先退出原来的终端，再打开新的终端创建临时函数，不然会报错

[hadoop@hadoop004 ~]$ hive
which: no hbase in (/home/hadoop/app/hue-3.9.0-cdh5.7.0/bin:/home/hadoop/app/hive-1.1.0-cdh5.7.0/bin:/home/hadoop/app/apache-maven-3.3.9/bin:/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/bin:/usr/java/jdk1.7.0_45/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin)

Logging initialized using configuration in jar:file:/home/hadoop/app/hive-1.1.0-cdh5.7.0/lib/hive-common-1.1.0-cdh5.7.0.jar!/hive-log4j.properties
WARNING: Hive CLI is deprecated and migration to Beeline is recommended.
hive> CREATE TEMPORARY FUNCTION sayhello1 as 'com.hivelearn.udf.HelloUDF';
OK
Time taken: 0.309 seconds


hive>
    > select sayhello1('girl');
OK
hello:girl
Time taken: 0.729 seconds, Fetched: 1 row(s)
hive>

下面创建永久生效的UDF

[hadoop@hadoop004 lib]$ hdfs dfs -mkdir /lib


[hadoop@hadoop004 lib]$ hdfs dfs -ls /
Found 6 items
drwxr-xr-x   - hadoop supergroup          0 2019-04-19 12:25 /data
drwxr-xr-x   - hadoop supergroup          0 2019-04-19 11:31 /g6
drwxr-xr-x   - hadoop supergroup          0 2019-04-19 11:45 /hello
drwxr-xr-x   - hadoop supergroup          0 2019-04-21 23:59 /lib
drwx------   - hadoop supergroup          0 2019-04-19 01:04 /tmp
drwxr-xr-x   - hadoop supergroup          0 2019-04-20 00:48 /user

[hadoop@hadoop004 lib]$ hdfs dfs -put udflearn-1.0.jar /lib/

[hadoop@hadoop004 lib]$ hdfs dfs -ls /lib
Found 1 items
-rw-r--r--   1 hadoop supergroup       3690 2019-04-22 00:00 /lib/udflearn-1.0.jar

hive> create function sayhello2 as 'com.hivelearn.udf.HelloUDF' using jar 'hdfs://hadoop004:9000/lib/udflearn-1.0.jar';
converting to local hdfs://hadoop004:9000/lib/udflearn-1.0.jar
Added [/tmp/31feb7b9-65a2-4eb9-ab24-7607abceb5dd_resources/udflearn-1.0.jar] to class path
Added resources: [hdfs://hadoop004:9000/lib/udflearn-1.0.jar]
OK
Time taken: 0.158 seconds

打开新的终端

hive> select sayhello2('girl');
converting to local hdfs://hadoop004:9000/lib/udflearn-1.0.jar
Added [/tmp/625543e8-bebf-4b34-bdb5-2ac672e597e4_resources/udflearn-1.0.jar] to class path
Added resources: [hdfs://hadoop004:9000/lib/udflearn-1.0.jar]
OK
hello:girl
Time taken: 1.121 seconds, Fetched: 1 row(s)

进入MySQL数据库

mysql> select * from FUNCS;
+---------+----------------------------+-------------+-------+-----------+-----------+------------+------------+
| FUNC_ID | CLASS_NAME                 | CREATE_TIME | DB_ID | FUNC_NAME | FUNC_TYPE | OWNER_NAME | OWNER_TYPE |
+---------+----------------------------+-------------+-------+-----------+-----------+------------+------------+
|       1 | com.hivelearn.udf.HelloUDF |  1555862678 |     1 | sayhello2 |         1 | NULL       | USER       |
+---------+----------------------------+-------------+-------+-----------+-----------+------------+------------+
1 row in set (0.00 sec)