CDH6.3.2 集成配置 ATLAS-2.1.0

最新推荐文章于 2023-10-16 17:27:30 发布

宙宇元

最新推荐文章于 2023-10-16 17:27:30 发布

阅读量2.6k

点赞数 1

文章标签： hadoop apache kafka

本文链接：https://blog.csdn.net/qinxw1687/article/details/122998315

版权

APACHE ATLAS安装部署，数据治理。

摘要由CSDN通过智能技术生成

去官网下载ATLAS源码包：http://atlas.apache.org/2.1.0/index.html#/Downloads

一、Atlas源码编译

1.修改pom文件

因与CDH6.3.2集成，在repositories中新增以下部分：

<repository>

     <id>cloudera</id>

     <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>

     <releases>

         <enabled>true</enabled>

     </releases>

     <snapshots>

         <enabled>false</enabled>

     </snapshots>

</repository>

修改CHD对应的版本，注意连接符号。直接复制cdh的话是个+号

        <lucene-solr.version>7.4.0-cdh6.3.2</lucene-solr.version>

        <hadoop.version>3.0.0-cdh6.3.2</hadoop.version>

        <hbase.version>2.1.0-cdh6.3.2</hbase.version>

        <solr.version>7.4.0-cdh6.3.2</solr.version>

        <hive.version>2.1.1-cdh6.3.2</hive.version>

        <kafka.version>2.2.1-cdh6.3.2</kafka.version>

        <kafka.scala.binary.version>2.11</kafka.scala.binary.version>

        <calcite.version>1.16.0</calcite.version>

        <zookeeper.version>3.4.5-cdh6.3.2</zookeeper.version>

        <falcon.version>0.8</falcon.version>

        <sqoop.version>1.4.7-cdh6.3.2</sqoop.version>

2.兼容Hive2.1.1版本，修改Atlas源代码

默认是3.1 不修改的话会报错

所需修改的项目位置：atlas-release-2.1.0-rc3/addons/hive-bridge

①.src/main/java/org/apache/atlas/hive/bridge//HiveMetaStoreBridge.java 577行

String catalogName = hiveDB.getCatalogName() != null ? hiveDB.getCatalogName().toLowerCase() : null;

改为

String catalogName = null;

②.src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 81行

this.metastoreHandler = (listenerEvent != null) ? metastoreEvent.getIHMSHandler() : null;

this.metastoreHandler = null;

3.编译

注意java 版本需要和生产环境的版本一致不然会报错

mvn clean -DskipTests package -Pdist

完成之后文件在/home/software/atlas/distro/target，会编译生成很多压缩包

二、Atlas安装

1.解压

将apache-atlas-2.1.0-bin.tar.gz解压至安装目录，不要用官方文档说的server包那个包没有各种hook文件

2.修改配置文件atlas-env.sh

export HBASE_CONF_DIR=/etc/hbase/conf

export ATLAS_SERVER_HEAP="-Xms15360m -Xmx15360m -XX:MaxNewSize=5120m -XX:MetaspaceSize=100M -XX:MaxMetaspaceSize=512m"

export ATLAS_SERVER_OPTS="-server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dumps/atlas_server.hprof -Xloggc:logs/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps"

export MANAGE_LOCAL_HBASE=false

export MANAGE_LOCAL_SOLR=false

export MANAGE_EMBEDDED_CASSANDRA=false

export MANAGE_LOCAL_ELASTICSEARCH=false

3.修改配置文件atlas-application.properties

这需要重点注意，habase、kafka、solr、zookeeper等配置需要修改

#

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements.  See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership.  The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except in compliance

# with the License.  You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

#

#########  Graph Database Configs  #########

# Graph Database

#Configures the graph