CDH6.3.2集成Apache Atlas2.1.0

CarsonBigData

已于 2023-04-20 11:18:03 修改

阅读量1.1k

点赞数 1

分类专栏：大数据文章标签： apache log4j java

于 2022-12-22 08:54:21 首次发布

本文链接：https://blog.csdn.net/carson073/article/details/128397108

版权

大数据专栏收录该内容

38 篇文章 22 订阅

订阅专栏

1 环境准备

1.1 CDH6.3.2 环境搭建

参考文档如下
Cloudera Manager安装CDH6教程-（一）虚拟环境安装配置

 Cloudera Manager安装CDH6教程-（二）搭建Cloudera和CDH6

CM和CDH在安装的时候遇到的问题

CDH6.3.2 各组件版本

1.2 apache atlas 2.1.0 环境准备

Atlas官网下载地址

下载完解压,然后用idea打开,修改pom.xml里的hadoop版本配置
找到 /distro/src/conf/atlas-log4j.xml, 把这段注释放开

    <appender name="perf_appender" class="org.apache.log4j.DailyRollingFileAppender">
        <param name="file" value="${atlas.log.dir}/atlas_perf.log" />
        <param name="datePattern" value="'.'yyyy-MM-dd" />
        <param name="append" value="true" />
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d|%t|%m%n" />
        </layout>
    </appender>
 
    <logger name="org.apache.atlas.perf" additivity="false">
        <level value="debug" />
        <appender-ref ref="perf_appender" />
    </logger>

编译打包

# 内嵌方式打包
mvn clean -DskipTests package -Pdist,embedded-hbase-solr

# 使用外部的hbase和solr
mvn clean -DskipTests package -Pdist

这里我用第二种外部方式打包

将 apache-atlas-2.1.0-bin.tar.gz上传到CDH集群的服务器上,然后解压,我上传的路径是/usr/local/src
修改conf\atlas-application.properties文件

#集成修改hbase配置
atlas.graph.storage.hostname=cdh01.cm:2181,cdh02.cm:2181,cdh03.cm:2181
#集成修改solr配置
atlas.graph.index.search.solr.zookeeper-url=cdh01.cm:2181,cdh02.cm:2181,cdh03.cm:2181/solr
#集成修改kafka配置
atlas.notification.embedded=false		    #false外置的kafka
atlas.kafka.zookeeper.connect=cdh01.cm:2181,cdh02.cm:2181,cdh03.cm:2181
atlas.kafka.bootstrap.servers=cdh01.cm:9092,cdh02.cm:9092,cdh03.cm:9092
atlas.kafka.zookeeper.session.timeout.ms=60000
atlas.kafka.zookeeper.connection.timeout.ms=30000
atlas.kafka.enable.auto.commit=true
#集成修改其他配置
atlas.rest.address=http://cdh01.cm:21000	#访问地址端口，此值修改不生效，默认本地21000端口，此端口和impala冲突
atlas.server.run.setup.on.start=false		   #如果启用并设置为true，则在服务器启动时将运行安装步骤
atlas.audit.hbase.zookeeper.quorum=cdh01.cm:2181,cdh02.cm:2181,cdh03.cm:2181
#集成添加hive钩子配置(文件最下面即可)
#在hive中做任何操作，都会被钩子所感应到，并生成相应的事件发往atlas所订阅的kafka-topic，再由atlas进行元数据生成和存储管理
######### Hive Hook Configs #######
atlas.hook.hive.synchronous=false
atlas.hook.hive.numRetries=3
atlas.hook.hive.queueSize=10000
atlas.cluster.name=primary
    
#配置用户名密码(选做)
    #开启或关闭三种验证方法
atlas.authentication.method.kerberos=true|false
atlas.authentication.method.ldap=true|false
atlas.authentication.method.file=true
   #vim users-credentials.properties(修改该文件)
   #>>>源文件
   #username=group::sha256-password
   admin=ADMIN::8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
   rangertagsync=RANGER_TAG_SYNC::e3f67240f5117d1753c940dae9eea772d36ed5fe9bd9c94a300e40413f1afb9d
   #<<<   
   #>>>修改成用户名bigdata123，密码bigdata123
   #username=group::sha256-password
   bigdata123=ADMIN::aa0336d976ba6db36f33f75a20f68dd9035b1e0e2315c331c95c2dc19b2aac13
   rangertagsync=RANGER_TAG_SYNC::e3f67240f5117d1753c940dae9eea772d36ed5fe9bd9c94a300e40413f1afb9d
   #<<<    
   #计算sha256：echo -n "bigdata123"|sha256sum

修改conf\atlas-env.sh文件

#集成添加hbase配置->下面的目录为atlas下的hbase配置目录，需要后面加入集群hbase配置
export HBASE_CONF_DIR=/usr/local/src/atlas/apache-atlas-2.0.0/conf/hbase/conf
#export MANAGE_LOCAL_HBASE=false  (false外置的zk和hbase)
#export MANAGE_LOCAL_SOLR=false  （false外置的solr）
    
#修改内存指标（根据线上机器配置）
export ATLAS_SERVER_OPTS="-server -XX:SoftRefLRUPolicyMSPerMB=0
-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=dumps/atlas_server.hprof
-Xloggc:logs/gc-worker.log -verbose:gc
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC
-XX:+PrintGCTimeStamps"

#优化 JDK1.8(以下需要16G内存)
export ATLAS_SERVER_HEAP="-Xms15360m -Xmx15360m
-XX:MaxNewSize=5120m -XX:MetaspaceSize=100M
-XX:MaxMetaspaceSize=512m"

修改conf\atlas-log4j.xml

#去掉如下代码的注释(开启如下代码)
    <appender name="perf_appender" class="org.apache.log4j.DailyRollingFileAppender">
        <param name="file" value="${atlas.log.dir}/atlas_perf.log" />
        <param name="datePattern" value="'.'yyyy-MM-dd" />
        <param name="append" value="true" />
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d|%t|%m%n" />
        </layout>
    </appender>

    <logger name="org.apache.atlas.perf" additivity="false">
        <level value="debug" />
        <appender-ref ref="perf_appender" />
    </logger>

2 集成组件

2.1 和Solr集成

2.2 和Hbase集成

2.3 和Kafka集成

2.4 和Hive集成

3 踩坑记录

编译时报错

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) on project apache-atlas: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]

解决办法:
注释掉这两个插件, 重新打包

<!--                <plugin>-->
<!--                    <inherited>true</inherited>-->
<!--                    <groupId>org.apache.maven.plugins</groupId>-->
<!--                    <artifactId>maven-enforcer-plugin</artifactId>-->
<!--                    <version>3.0.0-M1</version>-->
<!--                </plugin>-->




<!--            <plugin>-->
<!--                <groupId>org.apache.maven.plugins</groupId>-->
<!--                <artifactId>maven-enforcer-plugin</artifactId>-->
<!--                <executions>-->
<!--                    <execution>-->
<!--                        <id>enforce-versions</id>-->
<!--                        <goals>-->
<!--                            <goal>enforce</goal>-->
<!--                        </goals>-->
<!--                        <configuration>-->
<!--                            <rules>-->
<!--                                <requireMavenVersion>-->
<!--                                    <version>[3.5.0,)</version>-->
<!--                                    <message>** MAVEN VERSION ERROR ** Maven 3.5.0 or above is required. See https://maven.apache.org/install.html </message>-->
<!--                                </requireMavenVersion>-->
<!--                                <requireJavaVersion>-->
<!--                                    <level>ERROR</level>-->
<!--                                    <version>[1.8.0-151,)</version>-->
<!--                                    <message>** JAVA VERSION ERROR ** Java 8 (Update 151) or above is required.</message>-->
<!--                                </requireJavaVersion>-->
<!--                                <requireJavaVersion>-->
<!--                                    <level>WARN</level>-->
<!--                                    <version>(,1.9]</version>-->
<!--                                    <message>** JAVA VERSION WARNING ** Java 9 and above has not been tested with Atlas.</message>-->
<!--                                </requireJavaVersion>-->
<!--                            </rules>-->
<!--                        </configuration>-->
<!--                    </execution>-->
<!--                </executions>-->
<!--            </plugin>-->

参考博客:
Apache atlas集成CDH管理元数据
 CDH6.3 从0-1搭建部署Apache Atlas2.0 windows hive 元数据治理
 记录 CDH6.3安装配置Apache Atlas2.0

CDH6 安装 Apache atlas