Atlas配置Hive Hook

已于 2022-12-08 01:14:29 修改
阅读量1.2k 收藏 1
点赞数
文章标签： hive 大数据 hadoop
于 2022-06-29 20:18:22 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/maohuihua123/article/details/125528795
版权

Atlas配置Hive Hook

  1. 配置hive-site.xml

    在Hive客户端$HIVE_HOME/conf/hive-site.xml中加入内容：

    <property>
    <name>hive.exec.post.hooks</name>
    <value>org.apache.atlas.hive.hook.HiveHook</value>
</property>

  2. 解压hive-hook包

    解压hook包(这里以hive-hook为例)，并拷贝内容到atlas安装目录下

    tar -zxvf apache-atlas-2.2.0-hive-hook.tar.gz -C /opt/atlas/atlas

  3. 修改Hive-env.sh

    [root@dc6-80-283 conf]# cd /opt/hive/hive/conf
[root@dc6-80-283 conf]# cp hive-env.sh.template hive-env.sh
[root@dc6-80-283 conf]# vim hive-env.sh
# 加入内容
export HIVE_AUX_JARS_PATH=/opt/atlas/atlas/hook/hive/

    然后重启hive

  4. 复制ATLAS_HOME/conf/atlas-application.properties到HIVE_HOME/conf目录下

    cp /opt/atlas/atlas/conf/atlas-application.properties /opt/hive/hive/conf/

  5. 执行同步Hive元数据脚本

    # 这里同步的是Hive中已有数据的元数据，可以通过此脚本同步过来
[root@dc6-80-283 ]# cd /opt/atlas/atlas/hook-bin/
# 执行脚本导入元数据，期间需要输入atlas的用户名和密码：admin/admin
[root@dc6-80-283 ]# import-hive.sh 

    2022-06-29T19:30:02,605 INFO [main] com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
2022-06-29T19:30:02,612 WARN [main] com.zaxxer.hikari.util.DriverDataSource - Registered driver with driverClassName=com.mysql.jdbc.Driver was not found, trying direct instantiation.
2022-06-29T19:30:03,057 INFO [main] com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed.
2022-06-29T19:30:03,090 INFO [main] com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Starting...
2022-06-29T19:30:03,091 WARN [main] com.zaxxer.hikari.util.DriverDataSource - Registered driver with driverClassName=com.mysql.jdbc.Driver was not found, trying direct instantiation.
2022-06-29T19:30:03,118 INFO [main] com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Start completed.
2022-06-29T19:30:03,212 INFO [main] org.apache.hadoop.hive.metastore.ObjectStore - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2022-06-29T19:30:03,336 INFO [main] org.apache.hadoop.hive.metastore.MetaStoreDirectSql - Using direct SQL, underlying DB is MYSQL
2022-06-29T19:30:03,339 INFO [main] org.apache.hadoop.hive.metastore.ObjectStore - Initialized ObjectStore
2022-06-29T19:30:03,519 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:03,519 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:03,520 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:03,520 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:03,520 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:03,520 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:04,637 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:04,637 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:04,638 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:04,638 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:04,638 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:04,638 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:06,176 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - Added admin role in metastore
2022-06-29T19:30:06,180 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - Added public role in metastore
2022-06-29T19:30:06,201 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - No user is added in admin role, since config is empty
2022-06-29T19:30:06,347 INFO [main] org.apache.hadoop.hive.metastore.RetryingMetaStoreClient - RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0
2022-06-29T19:30:06,374 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - 0: get_all_functions
2022-06-29T19:30:06,376 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore.audit - ugi=root	ip=unknown-ip-addr	cmd=get_all_functions	
2022-06-29T19:30:06,406 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Importing Hive metadata
2022-06-29T19:30:06,406 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - 0: get_databases: @hive#
2022-06-29T19:30:06,407 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore.audit - ugi=root	ip=unknown-ip-addr	cmd=get_databases: @hive#	
2022-06-29T19:30:06,413 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 1 databases
2022-06-29T19:30:06,413 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - 0: get_database: @hive#default
2022-06-29T19:30:06,413 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore.audit - ugi=root	ip=unknown-ip-addr	cmd=get_database: @hive#default	
2022-06-29T19:30:06,459 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2022-06-29T19:30:06,717 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Database default is already registered - id=07cf307a-613f-42d2-bf33-12fd26e3bd67. Updating it.
2022-06-29T19:30:06,784 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2022-06-29T19:30:06,793 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Updated hive_db entity: name=default@primary, guid=07cf307a-613f-42d2-bf33-12fd26e3bd67
2022-06-29T19:30:06,809 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - 0: get_tables: db=@hive#default pat=.*
2022-06-29T19:30:06,809 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore.audit - ugi=root	ip=unknown-ip-addr	cmd=get_tables: db=@hive#default pat=.*	
2022-06-29T19:30:06,813 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 1 tables to import in database default
2022-06-29T19:30:06,815 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - 0: get_table : tbl=hive.default.t_order
2022-06-29T19:30:06,815 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore.audit - ugi=root	ip=unknown-ip-addr	cmd=get_table : tbl=hive.default.t_order	
2022-06-29T19:30:06,944 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2022-06-29T19:30:06,945 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Table default.t_order is already registered with id 63fb7492-0c8d-4185-b556-8a7a3c6b8f5c. Updating entity.
2022-06-29T19:30:07,052 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2022-06-29T19:30:07,053 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Updated hive_table entity: name=default.t_order@primary, guid=63fb7492-0c8d-4185-b556-8a7a3c6b8f5c
2022-06-29T19:30:07,053 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Successfully imported 1 tables from database default
Hive Meta Data imported successfully!!!
[root@dc6-80-283 hook-bin]#

  6. 可以看到成功导入了数据表t_order

    2022-06-29T19:30:06,813 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 1 tables to import in database default
2022-06-29T19:30:06,815 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - 0: get_table : tbl=hive.default.t_order

  7. 查看Atlas Web页面

    在这里插入图片描述

沧浪之水NaN
关注
Atlas hive Hook代码分析
weixin_40455124的博客
09-05 1378
基类及配置 atlas所有hook都（应该）基于public abstract class AtlasHook ，这个类在notifications 子项目下。 这个类首先需要关注的是下面这些可以在atlas-application.properties里面定义的配置项： public static final String ATLAS_NOTIFICATION_ASYNCHRONOUS = "atlas.notification.hook.asynchronous";
apache-atlas-2.1.0-hive-hook.tar.gz
12-13
atlas hive hook 已编译版本apache-atlas-2.1.0-hive-hook.tar.gz 可用于hive中进行数据抓取 和元数据信息管理
apache-atlas-2.0.0-hive-hook.tar.gz
08-27
Apache Atlas为组织提供开放式元数据管理和治理功能，用以构建其数据资产目录，对这些资产进行分类和管理，并为数据分析师和数据治理团队，提供围绕这些数据资产的协作功能。
APACHE-ATLAS-2.1.0 - 安装HIVE HOOK用于实时接收HIVE元数据的变化（五）
最新发布
记录并分享
06-06 752
本博文以获取HIVE元数据为例，进行流程和源码的分析。请提前安装好HADOOP和HIVE的环境，用于测试。ATLAS官网：https://atlas.apache.org/#/HookHiveHOOK是一种在处理过程中拦截事件、消息或函数调用的机制，从这种意义上讲， HIVE HOOKS 提供了使用HIVE扩展和集成外部功能的能力。
atlas hive hook 编译依赖 apache-atlas-1.2.0-hive-hook.tar.gz
02-26
atlas hive hook 资源包
Atlas之HiveHook源码简介
qq_41528616的博客
04-20 580
介绍了Hive中的Hook种类HiveQL生命周期Atlas-HiveHook原理介绍Hive中的Hook种类及各种Hook在HiveQL生命周期中的体现，最后介绍Atlas中的HiveHook流程。
atlas配置hive hook
09-06
配置 Atlas Hive Hook 的目的是将 Hive 的操作与 Atlas 的元数据服务集成，确保对 Hive 表和数据的操作能够实时地反映在 Atlas 中，实现元数据的自动化管理和审计。 以下是如何配置 Atlas Hive Hook 的详细步骤： ...
Atlas配置Hive Hook详细步骤
以下是如何配置Atlas Hive Hook的详细步骤： 1. 设置Atlas配置目录 首先，你需要创建一个目录来存放Atlas的配置文件。在这个例子中，创建了`/opt/service/atlas/apache-atlas-2.2.0-SNAPSHOT/conf`目录，并将必要...
apache-atlas-2.0.0-hive-hook ,亲测有用！
01-26
3. **配置Hive**：在Hive的`hive-site.xml`配置文件中，添加关于Atlas Hook的配置，如Atlas服务器地址、服务端口以及Hook的类名。 4. **重启服务**：更新配置后，重启Apache Atlas 和 Hive 服务，使新的配置生效。 ...
Apache Atlas的Hive Hook在CDH中的配置
zhuyao3582的博客
09-10 2981
配置Atlas的hivehook 将192.168.10.205机器中的/opt/module/atlas/distro/target/apache-atlas-0.8.4-bin/apache-atlas-0.8.4/hook/hive的所有jar包拷贝到生产环境hive节点172.165.11.49中的/opt/cloudera/parcels/CDH/lib/hive/lib目录下 ...
Atlas
10-25
Atlas
编译好的Atlas服务和HIVE-hook
02-03
版本2.1.0 apache-atlas-2.1.0-server.tar.gz和apache-atlas-2.1.0-hive-hook.tar.gz
apache-atlas-hive-hook-源码分析
windydreams的专栏
11-16 1089
apache atlas hive hook源码分析
Apache Atlas 1.2.0 Hive Hook & Bridge配置手册
狂奔的蜗牛
06-30 5842
具体编译部署方法请参阅： Apache Atlas 部署手册 参考： http://atlas.apache.org/1.2.0/Configuration.html http://atlas.apache.org/1.2.0/Hook-Hive.html 1、修改hive-site.xml 方法1：在/etc/hive/conf/hive-site.xml文件中添加如下内容： hiv...
Atlas配置Hbase Hook
沧浪之水的博客
12-08 895
Atlas配置HBase Hook 解压包解压包(这里以为例)，并拷贝内容到安装目录下 创建 在的连接 举例： 复制到目录下 配置，增加以下属性 执行同步元数据脚本
Class not found: org.apache.atlas.hive.hook.HiveHook
fairynini的博客
12-31 2467
看到集群sqoop任务部分失败了，而失败原因很明显： hive.exec.post.hooks Class not found: org.apache.atlas.hive.hook.HiveHook 主要是某个节点上的sqoop任务都会发生问题，检查该节点的hive的lib下的jar包 果然发现没有将atlas的包包搬过去， 解决方法： 将/usr/local/src/atlas/apache-atlas-2.0.0/hook/hive所有jar包拷贝/opt/cloudera/parcels/CDH/
cdh的hive配置文件位置_CDH6配置 Atlas，及 Hive Hook
weixin_39789042的博客
12-22 586
因为 CDH 社区版不能使用 Navigator，所以需要自己集成一个Apache Atlas。版本说明20200818 Updated: 目前最新版是2.1.0，如果 Hive 版本是3.1一下的，可能存在不兼容问题。在编译前修改下源码，参考这篇。Atlas: 2.0 DownloadCDH: 6.3.1 (Parcels)其它Atlas 依赖 Solr(或 ES)、HBase和 Kafka 来...
Atlas 2.1.0 实践（3）—— Atlas集成HIve
大数据流动
01-25 1779
Atlas集成Hive在安装好Atlas以后，如果想要使用起来，还要让Atlas与其他组件建立联系。其中最常用的就是Hive。通过Atlas的架构，只要配置好Hive Hook ，那么每...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值