Apache Atlas部署

最新推荐文章于 2024-08-07 09:11:05 发布

脚气水蟑螂药

最新推荐文章于 2024-08-07 09:11:05 发布

阅读量956

点赞数 1

分类专栏： Apache Atlas

本文链接：https://blog.csdn.net/fan_yi_bo/article/details/106750004

版权

Apache Atlas 专栏收录该内容

5 篇文章 3 订阅

订阅专栏

1.Atlas内嵌HBase和Solr的方式部署

1.1 Atlas编译

先从官网下载源码，然后解压进行编译

[root@atlas ~]# tar -zxf apache-atlas-2.0.0-sources.tar.gz -C /opt/software
[root@atlas ~]# cd /opt/software/apache-atlas-sources-2.0.0
[root@atlas apache-atlas-sources-2.0.0]# export MAVEN_OPTS="-Xms2g -Xmx2g"
[root@atlas apache-atlas-sources-2.0.0]# mvn clean package -DskipTests -Pdist,embedded-hbase-solr

经历漫长的等待后你会看到编译完成的提示，编译前最好修改下maven的仓库，这样能节省点时间，要是能翻墙，那就无所谓了，编译过程中失败了不要慌，看看什么原因解决就行了，我在部署过程中也踩了很多坑，大家可以参考下Apache Atlas部署FAQ

1.2 解压

编译好的文件在distro/target/目录下

[root@atlas apache-atlas-sources-2.0.0]# cd distro/target
[root@atlas target]# ll
total 2048464
drwxr-xr-x 2 root root      4096 Jul  4 12:46 META-INF
drwxr-xr-x 2 root root      4096 Jul  4 09:53 antrun
drwxr-xr-x 3 root root      4096 Jul  4 12:48 apache-atlas-2.0.0-bin
-rw-r--r-- 1 root root 699150132 Jul  4 12:48 apache-atlas-2.0.0-bin.tar.gz
drwxr-xr-x 3 root root      4096 Jul  4 12:47 apache-atlas-2.0.0-falcon-hook
-rw-r--r-- 1 root root   9203414 Jul  4 12:47 apache-atlas-2.0.0-falcon-hook.tar.gz
drwxr-xr-x 3 root root      4096 Jul  4 12:47 apache-atlas-2.0.0-hbase-hook
-rw-r--r-- 1 root root  11079854 Jul  4 12:47 apache-atlas-2.0.0-hbase-hook.tar.gz
drwxr-xr-x 3 root root      4096 Jul  4 12:47 apache-atlas-2.0.0-hive-hook
-rw-r--r-- 1 root root  16266900 Jul  4 12:47 apache-atlas-2.0.0-hive-hook.tar.gz
drwxr-xr-x 3 root root      4096 Jul  4 12:47 apache-atlas-2.0.0-kafka-hook
-rw-r--r-- 1 root root   9220158 Jul  4 12:47 apache-atlas-2.0.0-kafka-hook.tar.gz
drwxr-xr-x 3 root root      4096 Jul  4 12:48 apache-atlas-2.0.0-server
-rw-r--r-- 1 root root 594485525 Jul  4 12:47 apache-atlas-2.0.0-server.tar.gz
-rw-r--r-- 1 root root  11086709 Jul  4 12:49 apache-atlas-2.0.0-sources.tar.gz
drwxr-xr-x 3 root root      4096 Jul  4 12:47 apache-atlas-2.0.0-sqoop-hook
-rw-r--r-- 1 root root   9197108 Jul  4 12:47 apache-atlas-2.0.0-sqoop-hook.tar.gz
drwxr-xr-x 3 root root      4096 Jul  4 12:47 apache-atlas-2.0.0-storm-hook
-rw-r--r-- 1 root root  58913930 Jul  4 12:47 apache-atlas-2.0.0-storm-hook.tar.gz
drwxr-xr-x 2 root root      4096 Jul  4 12:47 archive-tmp
-rw-r--r-- 1 root root 678903737 Jul  4 12:47 atlas-distro-2.0.0.jar
drwxr-xr-x 2 root root      4096 Jul  4 12:46 bin
drwxr-xr-x 5 root root      4096 Jul  4 12:46 conf
drwxr-xr-x 7 root root      4096 Jul  4 09:53 hbase
drwxr-xr-x 3 root root      4096 Jul  4 09:53 hbase.temp
drwxr-xr-x 2 root root      4096 Jul  4 12:46 maven-archiver
drwxr-xr-x 3 root root      4096 Jul  4 09:53 maven-shared-archive-resources
-rw-r--r-- 1 root root      3868 Jul  4 09:53 rat.txt
drwxr-xr-x 9 root root      4096 Jul  4 12:46 solr
drwxr-xr-x 3 root root      4096 Jul  4 12:46 solr.temp
drwxr-xr-x 3 root root      4096 Jul  4 12:46 test-classes
[root@atlas target]# tar -zxf apache-atlas-2.0.0-bin.tar.gz -C /opt/software
总感觉解压这步好像很多余，因为上面那个好像就是解压好的，但是我没试，回头试完了再改这块的，也可能就不改了，哈哈哈哈

1.3 启动

先启动Solr服务，再启动Atlas服务，一定注意顺序，否则无法启动成功

[root@atlas apache-atlas-2.0.0]# solr/bin/solr start -c -z localhost:2181 -p 8983 -force
*** [WARN] *** Your open file limit is currently 1024.  
 It should be set to 65000 to avoid operational disruption. 
 If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
*** [WARN] ***  Your Max Processes Limit is currently 3802. 
 It should be set to 65000 to avoid operational disruption. 
 If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
Waiting up to 180 seconds to see Solr running on port 8983 [\]  
Started Solr server on port 8983 (pid=41271). Happy searching!
[root@atlas apache-atlas-2.0.0]# solr/bin/solr create -c fulltext_index -force -d conf/solr/
INFO  - 2020-07-05 01:31:41.473; org.apache.solr.util.configuration.SSLCredentialProviderFactory; Processing SSL Credential Provider chain: env;sysprop
Created collection 'fulltext_index' with 1 shard(s), 1 replica(s) with config-set 'fulltext_index'
[root@atlas apache-atlas-2.0.0]# solr/bin/solr create -c edge_index -force -d conf/solr/
INFO  - 2020-07-05 01:32:04.628; org.apache.solr.util.configuration.SSLCredentialProviderFactory; Processing SSL Credential Provider chain: env;sysprop
Created collection 'edge_index' with 1 shard(s), 1 replica(s) with config-set 'edge_index'
[root@atlas apache-atlas-2.0.0]# solr/bin/solr create -c vertex_index -force -d conf/solr/
INFO  - 2020-07-05 01:32:19.980; org.apache.solr.util.configuration.SSLCredentialProviderFactory; Processing SSL Credential Provider chain: env;sysprop
Created collection 'vertex_index' with 1 shard(s), 1 replica(s) with config-set 'vertex_index'
[root@atlas apache-atlas-2.0.0]# hbase/bin/start-hbase.sh
[root@atlas apache-atlas-2.0.0]# bin/atlas_start.py 
configured for local hbase.
hbase started.
configured for local solr.
solr started.
setting up solr collections...
starting atlas on host localhost
starting atlas on port 21000
..............................................................................................................................................................................................................................................................................................................
Apache Atlas Server started!!!

1.4 登录页面

atlas登录界面
默认用户名：admin
默认密码：admin
登录以后首页如下所示，Atlas2.1.0界面有调整，看到的首页可能会不同
atlas首页

1.5 导入数据

通过自带脚本导入hive的历史元数据信息，也可以自己实现其他数据源历史元数据的导入

2.Atlas独立部署

2.1Atlas编译

用mvn clean package -DskipTests -Pdist或者mvn clean package -DskipTests -Pdist,external-hbase-solr两种都可以

[root@atlas ~]# tar -zxf apache-atlas-2.0.0-sources.tar.gz -C /opt/software
[root@atlas ~]# cd /opt/software/apache-atlas-sources-2.0.0
[root@atlas apache-atlas-sources-2.0.0]# export MAVEN_OPTS="-Xms2g -Xmx2g"
[root@atlas apache-atlas-sources-2.0.0]# mvn clean package -DskipTests -Pdist

2.2集成Solr

首先确保Solr服务已经正确部署

修改atlas配置文件

修改atlas的配置文件atlas-application.properties

[root@atlas atlas-2.0.0]# vim conf/atlas-application.properties
atlas.graph.index.search.solr.zookeeper-url=zookeeper-01:2181,zookeeper-02:2181,zookeeper-03:2181
atlas.graph.index.search.solr.zookeeper-connect-timeout=60000
atlas.graph.index.search.solr.zookeeper-session-timeout=60000
atlas.graph.index.search.solr.wait-searcher=true

将Atlas目录下的solr配置文件拷贝到Solr服务安装目录下。如果Solr服务和Atlas服务不在一个节点就通过scp拷贝，注意，要拷贝到Solr服务的每个节点！！！

[root@atlas atlas-2.0.0]# scp -r conf/solr root@solr-01:/opt/software/solr-7.5.0/
[root@atlas atlas-2.0.0]# scp -r conf/solr root@solr-02:/opt/software/solr-7.5.0/
[root@atlas atlas-2.0.0]# scp -r conf/solr root@solr-03:/opt/software/solr-7.5.0/

启动Solr服务

[root@solr-01 solr-7.5.0]# bin/solr start -c -z zookeeper-01:2181,zookeeper-02:2181,zookeeper-03:2181 -p 8983 -force
[root@solr-02 solr-7.5.0]# bin/solr start -c -z zookeeper-01:2181,zookeeper-02:2181,zookeeper-03:2181 -p 8983 -force
[root@solr-03 solr-7.5.0]# bin/solr start -c -z zookeeper-01:2181,zookeeper-02:2181,zookeeper-03:2181 -p 8983 -force

创建collection

[root@solr-01 solr-7.5.0]# bin/solr create -c vertex_index -d /opt/software/atlas-2.0.0/conf/solr/ -shards 3 -replicationFactor 2 -force
[root@solr-01 solr-7.5.0]# bin/solr create -c edge_index -d /opt/software/atlas-2.0.0/conf/solr/ -shards 3 -replicationFactor 2 -force
[root@solr-01 solr-7.5.0]# bin/solr create -c fulltext_index -d /opt/software/atlas-2.0.0/conf/solr/ -shards 3 -replicationFactor 2 -force

如果需要删除collection可以执行bin/solr delete -c ${collection_name}