Apache Atlas 2.0.0 CDH环境部署问题及解决

最新推荐文章于 2024-04-26 03:30:00 发布

x_iaoa_o

最新推荐文章于 2024-04-26 03:30:00 发布

阅读量2.2k

点赞数 1

分类专栏： Atlas 元数据治理

原文链接：https://www.codenong.com/cs106258758/

版权

Atlas 同时被 2 个专栏收录

2 篇文章 1 订阅

订阅专栏

元数据治理

1 篇文章 0 订阅

订阅专栏

Apache Atlas部署FAQ
Atlas部署问题
Atlas启动报错(zookeeper启动失败导致)
1.问题描述
2.原因定位
3.解决方法
Atlas启动报错(solr启动失败导致)
1.问题描述
2.原因定位
3.解决方法
Atlas启动报错(HBase中没有Column)
1.问题描述
2.原因定位
3.解决方法
Atlas import-hive.sh报错
1.问题描述
2.原因定位
3.解决方法
Atlas部署问题

Atlas启动报错(zookeeper启动失败导致)

通过Atlas内嵌HBase和Solr启动Atlas服务

1.问题描述
无法访问访问Atlas页面，但提示Atlas启动成功

2.原因定位
检查Atlas启动日志，日志中提示在Zk中没有ZNode

[root@kafka-dev-01 logs]# vim /opt/software/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/logs/application.log

########################################################################################
                               Atlas Server (STARTUP)

        project.name:   apache-atlas
        project.description:    Metadata Management and Data Governance Platform over Hadoop
        build.user:     root
        build.epoch:    1589879105880
        project.version:        2.0.0
        build.version:  2.0.0
        vc.revision:    release
        vc.source.url:  scm:git:git://git.apache.org/atlas.git/atlas-webapp
######################################################################################## (Atlas:215)
2020-05-19 19:37:38,083 INFO  - [main:] ~ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (Atlas:216)
2020-05-19 19:37:38,084 INFO  - [main:] ~ Server starting with TLS ? false on port 21000 (Atlas:217)
2020-05-19 19:37:38,084 INFO  - [main:] ~ <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< (Atlas:218)
2020-05-19 19:37:38,580 INFO  - [main:] ~ No authentication method configured.  Defaulting to simple authentication (LoginProcessor:102)
2020-05-19 19:37:38,722 WARN  - [main:] ~ Unable to load native-hadoop library for your platform... using builtin-java classes where applicable (NativeCodeLoader:60)
2020-05-19 19:37:38,739 INFO  - [main:] ~ Logged in user root (auth:SIMPLE) (LoginProcessor:77)
2020-05-19 19:37:39,144 INFO  - [main:] ~ Not running setup per configuration atlas.server.run.setup.on.start. (SetupSteps$SetupRequired:189)
2020-05-19 19:37:40,100 WARN  - [main:] ~ Retrieve cluster id failed (ConnectionImplementation:551)
java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
        at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:549)
        at org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:287)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:219)
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:114)
        at org.janusgraph.diskstorage.hbase2.HBaseCompat2_0.createConnection(HBaseCompat2_0.java:46)
        at org.janusgraph.diskstorage.hbase2.HBaseStoreManager.<init>(HBaseStoreManager.java:314)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
        at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:476)
        at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:408)
        at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1254)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:111)
        at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:165)
        at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraph(AtlasJanusGraphDatabase.java:263)
        at org.apache.atlas.repository.graph.AtlasGraphProvider.getGraphInstance(AtlasGraphProvider.java:52)
        at org.apache.atlas.repository.graph.AtlasGraphProvider.get(AtlasGraphProvider.java:98)
        at org.apache.atlas.repository.graph.AtlasGraphProvider$$EnhancerBySpringCGLIB$$375c3cc3.CGLIB$get$1(<generated>)
        at org.apache.atlas.repository.graph.AtlasGraphProvider$$EnhancerBySpringCGLIB$$375c3cc3$$FastClassBySpringCGLIB$$86ea7f4c.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358)
        at org.apache.atlas.repository.graph.AtlasGraphProvider$$EnhancerBySpringCGLIB$$375c3cc3.get(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1181)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1075)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1201)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1103)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1201)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1103)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.aop.framework.autoproxy.BeanFactoryAdvisorRetrievalHelper.findAdvisorBeans(BeanFactoryAdvisorRetrievalHelper.java:92)
        at org.springframework.aop.framework.autoproxy.AbstractAdvisorAutoProxyCreator.findCandidateAdvisors(AbstractAdvisorAutoProxyCreator.java:102)
        at org.springframework.aop.aspectj.autoproxy.AspectJAwareAdvisorAutoProxyCreator.shouldSkip(AspectJAwareAdvisorAutoProxyCreator.java:103)
        at org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator.postProcessBeforeInstantiation(AbstractAutoProxyCreator.java:248)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInstantiation(AbstractAutowireCapableBeanFactory.java:1045)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveBeforeInstantiation(AbstractAutowireCapableBeanFactory.java:1019)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:473)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
        at org.apache.atlas.web.setup.KerberosAwareListener.contextInitialized(KerberosAwareListener.java:31)
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:843)
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
        at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:816)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.server.Server.start(Server.java:422)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.Server.doStart(Server.java:389)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:98)
        at org.apache.atlas.Atlas.main(Atlas.java:133)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:168)
        at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:323)
        at java.lang.Thread.run(Thread.java:748)
2020-05-19 19:40:09,645 INFO  - [pool-1-thread-1:] ~ ==> Shutdown of Atlas (Atlas$1:63)
2020-05-19 19:40:19,646 WARN  - [Thread-0:] ~ ShutdownHook '' timeout, java.util.concurrent.TimeoutException (ShutdownHookManager$1:71)
java.util.concurrent.TimeoutException
        at java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:68)
2020-05-19 19:40:29,647 ERROR - [Thread-0:] ~ ShutdownHookManger shutdown forcefully. (ShutdownHookManager$1:82)
2020-05-19 20:45:14,847 INFO  - [main:] ~ Loading atlas-application.properties from file:/opt/software/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/conf/atlas-application.properties (ApplicationProperties:123)

检查进程发现hbase未启动

[root@kafka-dev-01 logs]# jps
4159483 Jps

查看Hbase服务日志，发现日志中提示ZK端口被占用

2020-05-19 19:37:36,850 INFO  [main] server.ZooKeeperServer: Server environment:user.dir=/opt/software/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/server/webapp/atlas
2020-05-19 19:37:36,861 INFO  [main] server.ZooKeeperServer: Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /opt/software/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/data/hbase-zookeeper-data/zookeeper_0/version-2 snapdir /opt/software/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/data/hbase-zookeeper-data/zookeeper_0/version-2
2020-05-19 19:37:36,861 INFO  [main] server.ZooKeeperServer: minSessionTimeout set to -1
2020-05-19 19:37:36,862 INFO  [main] server.ZooKeeperServer: maxSessionTimeout set to -1
2020-05-19 19:37:36,868 INFO  [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181
2020-05-19 19:37:36,869 INFO  [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2182
2020-05-19 19:37:36,912 ERROR [main] server.ZooKeeperServer: ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2020-05-19 19:37:36,915 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182] server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:33280
2020-05-19 19:37:36,921 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182] server.ServerCnxn: The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, 2003003507=wchs, 2003003504=wchp, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}]
2020-05-19 19:37:36,921 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182] server.ServerCnxn: The list of enabled four letter word commands is : [[wchs, stat, stmk, conf, ruok, mntr, srvr, envi, srst, isro, dump, gtmk, crst, cons]]
2020-05-19 19:37:36,921 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182] server.NIOServerCnxn: Processing stat command from /127.0.0.1:33280
2020-05-19 19:37:36,923 INFO  [Thread-2] server.NIOServerCnxn: Stat command output
2020-05-19 19:37:36,923 INFO  [Thread-2] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:33280 (no session established for client)
2020-05-19 19:37:36,923 INFO  [main] zookeeper.MiniZooKeeperCluster: Started MiniZooKeeperCluster and ran successful 'stat' on client port=2182
2020-05-19 19:37:36,924 ERROR [main] master.HMasterCommandLine: Master exiting
java.io.IOException: Could not start ZK at requested port of 2181.  ZK was started at port: 2182.  Aborting as clients (e.g. shell) will not be able to find this ZK quorum.
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:217)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2964)

3.解决方法
检查占用2181端口的进程，杀掉该进程，重启hbase服务

[root@kafka-dev-01 logs]# netstat -nltp|grep 2181
tcp6       0      0 :::2181                 :::*                    LISTEN      4055296/java        
[root@kafka-dev-01 logs]# kill 4055296
[root@kafka-dev-01 logs]# /opt/software/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/hbase/bin/start-hbase.sh
running master, logging to /opt/software/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/hbase/bin/../logs/hbase-root-master-kafka-dev-01.out
[root@kafka-dev-01 bin]#

Atlas启动报错(solr启动失败导致)
通过Atlas内嵌HBase和Solr启动Atlas服务

1.问题描述
无法访问访问Atlas页面，但提示Atlas启动成功

2.原因定位
查看Atlas启动日志，发现Solr启动有问题

[root@kafka-dev-01 logs]# vim /opt/software/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/logs/application.log

2020-05-20 11:39:30,104 INFO  - [main:] ~ GraphTransaction intercept for org.apache.atlas.discovery.EntityLineageService.getAtlasLineageInfo (GraphTransactionAdvisor$1:41)
2020-05-20 11:39:30,106 INFO  - [main:] ~ GraphTransaction intercept for org.apache.atlas.discovery.EntityLineageService.getAtlasLineageInfo (GraphTransactionAdvisor$1:41)
2020-05-20 11:39:30,106 INFO  - [main:] ~ GraphTransaction intercept for org.apache.atlas.discovery.EntityLineageService.getSchemaForHiveTableByName (GraphTransactionAdvisor$1:41)
2020-05-20 11:39:30,107 INFO  - [main:] ~ GraphTransaction intercept for org.apache.atlas.discovery.EntityLineageService.getSchemaForHiveTableByGuid (GraphTransactionAdvisor$1:41)
2020-05-20 11:39:30,122 INFO  - [main:] ~ Creating indexes for graph. (GraphBackedSearchIndexer:248)
2020-05-20 11:39:31,215 INFO  - [main:] ~ Created index : vertex_index (GraphBackedSearchIndexer:253)
2020-05-20 11:39:31,234 INFO  - [main:] ~ Created index : edge_index (GraphBackedSearchIndexer:259)
2020-05-20 11:39:31,240 INFO  - [main:] ~ Created index : fulltext_index (GraphBackedSearchIndexer:265)
2020-05-20 11:39:31,386 ERROR - [main:] ~ GraphBackedSearchIndexer.initialize() failed (GraphBackedSearchIndexer:307)
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://10.90.0.94:8983/solr: Can not find the specified config set: vertex_index
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.janusgraph.diskstorage.solr.Solr6Index.createCollectionIfNotExists(Solr6Index.java:1086)
        at org.janusgraph.diskstorage.solr.Solr6Index.register(Solr6Index.java:307)
        at org.janusgraph.diskstorage.indexing.IndexTransaction.register(IndexTransaction.java:96)
        at org.janusgraph.graphdb.database.IndexSerializer.register(IndexSerializer.java:105)
        at org.janusgraph.graphdb.database.management.ManagementSystem.addIndexKey(ManagementSystem.java:529)
        at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphManagement.addMixedIndex(AtlasJanusGraphManagement.java:202)
        at org.apache.atlas.repository.graph.GraphBackedSearchIndexer.<init>(GraphBackedSearchIndexer.java:126)
        at org.apache.atlas.repository.graph.GraphBackedSearchIndexer.<init>(GraphBackedSearchIndexer.java:117)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:271)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1201)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1103)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1201)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1103)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
        at org.apache.atlas.web.setup.KerberosAwareListener.contextInitialized(KerberosAwareListener.java:31)
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:843)
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
        at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:816)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.server.Server.start(Server.java:422)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.Server.doStart(Server.java:389)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:98)
        at org.apache.atlas.Atlas.main(Atlas.java:133)
2020-05-20 12:00:33,545 INFO  - [pool-1-thread-1:] ~ ==> Shutdown of Atlas (Atlas$1:63)
2020-05-20 12:09:45,335 INFO  - [main:] ~ Loading atlas-application.properties from file:/opt/software/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/conf/atlas-application.properties (ApplicationProperties:123)

3.解决方法
在solr中创建，需要在启动zk的前提下创建

[root@kafka-dev-01 logs]# cd /opt/software/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/
[root@kafka-dev-01 apache-atlas-2.0.0]# solr/bin/solr create -c vertex_index -force -d conf/solr/
INFO  - 2020-05-20 12:07:24.129; org.apache.solr.util.configuration.SSLCredentialProviderFactory; Processing SSL Credential Provider chain: env;sysprop
Created collection 'vertex_index' with 1 shard(s), 1 replica(s) with config-set 'vertex_index'
[root@kafka-dev-01 apache-atlas-2.0.0]# solr/bin/solr create -c edge_index -force -d conf/solr/
INFO  - 2020-05-20 12:07:44.262; org.apache.solr.util.configuration.SSLCredentialProviderFactory; Processing SSL Credential Provider chain: env;sysprop
Created collection 'edge_index' with 1 shard(s), 1 replica(s) with config-set 'edge_index'
[root@kafka-dev-01 apache-atlas-2.0.0]# solr/bin/solr create -c fulltext_index -force -d conf/solr/
INFO  - 2020-05-20 12:08:01.040; org.apache.solr.util.configuration.SSLCredentialProviderFactory; Processing SSL Credential Provider chain: env;sysprop
Created collection 'fulltext_index' with 1 shard(s), 1 replica(s) with config-set 'fulltext_index'
[root@kafka-dev-01 apache-atlas-2.0.0]# solr/bin/solr start -c -z localhost:2181 -p 8983 -force    
*** [WARN] *** Your open file limit is currently 1024.  
 It should be set to 65000 to avoid operational disruption.
 If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh

Port 8983 is already being used by another process (pid: 4022294)
Please choose a different port using the -p option.

[root@kafka-dev-01 apache-atlas-2.0.0]# jps
4057719 Jps
4022294 jar
4055296 HMaster
[root@kafka-dev-01 apache-atlas-2.0.0]#

Atlas启动报错(HBase中没有Column)
Atlas与CDH集成，使用CDH Hive、HBase、Kafka

1.问题描述
无法访问访问Atlas页面，但提示Atlas启动成功

2.原因定位
查看Atlas启动日志

2020-05-20 19:32:34,642 WARN  - [main:] ~ Unexpected exception during getDeployment() (HBaseStoreManager:399)
java.lang.RuntimeException: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
        at org.janusgraph.diskstorage.hbase2.HBaseStoreManager.getDeployment(HBaseStoreManager.java:358)
        at org.janusgraph.diskstorage.hbase2.HBaseStoreManager.getFeatures(HBaseStoreManager.java:397)
        at org.janusgraph.diskstorage.Backend.getStandaloneGlobalConfiguration(Backend.java:441)
        at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1257)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:111)
        at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:165)
        at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraph.<init>(AtlasJanusGraph.java:95)
        at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:174)
        at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraph.<init>(AtlasJanusGraph.java:95)
        at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:174)
        at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraph.<init>(AtlasJanusGraph.java:95)
        at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:174)
        at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraph.<init>(AtlasJanusGraph.java:95)
...
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
        at org.janusgraph.diskstorage.hbase2.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:732)
        at org.janusgraph.diskstorage.hbase2.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:518)
        at org.janusgraph.diskstorage.hbase2.HBaseStoreManager.getDeployment(HBaseStoreManager.java:355)
        at org.janusgraph.diskstorage.hbase2.HBaseStoreManager.getFeatures(HBaseStoreManager.java:397)
        at org.janusgraph.diskstorage.common.AbstractStoreManager.getMetaDataSchema(AbstractStoreManager.java:58)
        at org.janusgraph.diskstorage.hbase2.HBaseKeyColumnValueStore.<init>(HBaseKeyColumnValueStore.java:103)
        at org.janusgraph.diskstorage.hbase2.HBaseStoreManager.openDatabase(HBaseStoreManager.java:457)
        at org.janusgraph.diskstorage.keycolumnvalue.KeyColumnValueStoreManager.openDatabase(KeyColumnValueStoreManager.java:43)
        at org.janusgraph.diskstorage.Backend.getStandaloneGlobalConfiguration(Backend.java:452)
        at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1257)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:111)
        at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:165)
        ... 1010 more
Caused by: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family table does not exist in region hbase:meta,,1.1588230740 in table 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', BLOOMFILTER => 'NONE', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}
        at org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:7941)
        at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6974)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2027)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)

        at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
        at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
        at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:279)
        at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:266)
        at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:129)
        at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:386)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:360)
        at org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1066)
        at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:389)
        at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:439)
        at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:436)
        at org.apache.hadoop.hbase.client.RpcRetryingCallable.call(RpcRetryingCallable.java:58)
        at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
        at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3078)
        at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3070)
        at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:436)
        at org.janusgraph.diskstorage.hbase2.HBaseAdmin2_0.tableExists(HBaseAdmin2_0.java:110)
        at org.janusgraph.diskstorage.hbase2.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:709)
        at org.janusgraph.diskstorage.hbase2.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:518)
        at org.janusgraph.diskstorage.hbase2.HBaseStoreManager.getDeployment(HBaseStoreManager.java:355)
        at org.janusgraph.diskstorage.hbase2.HBaseStoreManager.getFeatures(HBaseStoreManager.java:397)
        at org.janusgraph.diskstorage.common.AbstractStoreManager.getMetaDataSchema(AbstractStoreManager.java:58)
        at org.janusgraph.diskstorage.hbase2.HBaseKeyColumnValueStore.<init>(HBaseKeyColumnValueStore.java:103)
        at org.janusgraph.diskstorage.hbase2.HBaseStoreManager.openDatabase(HBaseStoreManager.java:457)
        at org.janusgraph.diskstorage.keycolumnvalue.KeyColumnValueStoreManager.openDatabase(KeyColumnValueStoreManager.java:43)
        at org.janusgraph.diskstorage.Backend.getStandaloneGlobalConfiguration(Backend.java:452)
        at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1257)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:111)
...
2020-05-20 19:32:34,852 WARN  - [main:] ~ JanusGraphException: Could not open global configuration (AtlasJanusGraphDatabase:167)
2020-05-20 19:32:34,853 WARN  - [main:] ~ Failed to obtain graph instance on attempt 3 of 3 (AtlasGraphProvider:118)
java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.hbase2.HBaseStoreManager
        at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:64)
        at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:476)
        at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:408)
        at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1254)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:111)

查看Atlas的版本依赖，发现Atlas2.0.0依赖版本过高，但是cdh5.16中集成的版本较低
在这里插入图片描述

3.解决方法
Atlas降版本，CDH-5.16可以用Atlas-1.2.0

Atlas import-hive.sh报错
1.问题描述

[root@cdh-cluster-01 bin]# ./import-hive.sh
Using Hive configuration directory [/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive/conf]
/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hive/conf:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*
Log file for import is /opt/software/apache-atlas-1.2.0/logs/import-hive.log
log4j:WARN No such property [maxFileSize] in org.apache.log4j.PatternLayout.
log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.PatternLayout.
Enter username for atlas :- admin
Enter password for atlas :-
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/jaxrs/json/JacksonJaxbJsonProvider
    at org.apache.atlas.AtlasBaseClient.getClient(AtlasBaseClient.java:270)
    at org.apache.atlas.AtlasBaseClient.initializeState(AtlasBaseClient.java:453)
    at org.apache.atlas.AtlasBaseClient.initializeState(AtlasBaseClient.java:448)
    at org.apache.atlas.AtlasBaseClient.<init>(AtlasBaseClient.java:132)
    at org.apache.atlas.AtlasClientV2.<init>(AtlasClientV2.java:82)
    at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:131)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.jaxrs.json.JacksonJaxbJsonProvider
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 6 more
Failed to import Hive Meta Data!!!
[root@cdh-cluster-01 bin]#

2.原因定位
通过查看Atlas对应版本源码，hive/hbase bridge模块，发现其相关依赖为hbase client原生的客户端，直接编译源码，则无法在HBase同步元数据，并且上述模块存在缺少jar的情况

3.解决方法
在编译前修改源码

#两个脚本中的内容也可以不添加，通过其他方式在编译以后解决
[root@kafka-dev-01 bin]# vim /root/apache-atlas-sources-1.2.0/addons/hive-bridge/src/bin/import-hive.sh

#在脚本中找到该行内容，并添加-Datlas.conf
"${JAVA_BIN}" ${JAVA_PROPERTIES} -Datlas.conf=/etc/hive/conf -cp "${CP}" org.apache.atlas.hive.bridge.HiveMetaStoreBridge $IMPORT_ARGS

[root@kafka-dev-01 bin]# vim /root/apache-atlas-sources-1.2.0/addons/hbase-bridge/src/bin/import-hbase.sh

#在脚本中找到该行内容，并添加-Datlas.conf
"${JAVA_BIN}" ${JAVA_PROPERTIES} -Datlas.conf=/etc/hbase/conf -cp "${CP}" org.apache.atlas.hbase.bridge.HBaseBridge $IMPORT_ARGS

添加依赖

[root@kafka-dev-01 hive-bridge]# vim /root/apache-atlas-sources-1.2.0/addons/hive-bridge/pom.xml

<!-- 加入这三个依赖，不加将看到如例一的报错，也可以直接从仓库下载对应的jar包，放到/opt/software/apache-atlas-1.2.0/hook/hive/atlas-hive-plugin-impl目录下 -->
        <dependency>
            <groupId>com.fasterxml.jackson.jaxrs</groupId>
            <artifactId>jackson-jaxrs-json-provider</artifactId>
            <version>${jackson.version}</version>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.module</groupId>
            <artifactId>jackson-module-jaxb-annotations</artifactId>
            <version>${jackson.version}</version>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.jaxrs</groupId>
            <artifactId>jackson-jaxrs-base</artifactId>
            <version>${jackson.version}</version>
        </dependency>

<!-- 修改原生的依赖，改为CDH的，不修改将看到例二的报错；还得再单独指定镜像仓库，阿里仓库没有，不修改将看到例三的报错 -->
    <properties>
        <hbase.version>1.2.0-cdh5.16.2</hbase.version>
        <calcite.version>0.9.2-incubating</calcite.version>
    </properties>
    <repositories>
        <repository>
          <id>cloudera</id>
          <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
        </repository>
    </repositories>

修改hbase bridge调用接口
hbase bridge采用的hbase-client原生jar，而在CDH中采用的jar包由于CDK对其中的方法进行了重写，所以需要进行修改
修改文件HBaseAtlasHook.java和HBaseBridge.java，将getKeepDeletedCells修改为getKeepDeletedCellsAsEnum()。

[root@kafka-dev-01 apache-atlas-sources-1.2.0]# cd addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/
[root@kafka-dev-01 bridge]# ll
total 64
-rw-r--r-- 1 root root 32542 Jun  3  2019 HBaseAtlasHook.java
-rw-r--r-- 1 root root 31320 Jun  3  2019 HBaseBridge.java
[root@kafka-dev-01 bridge]#

这里自己去实践了下单纯修改hbase bridge调用接口后还是会出例4的错误告诉我们有的抽象方法没有重写
这是我自己的操作虽然 atlas1.2编译成功了但是还没有放到集群上测试 hbase的数据能不能正常导入
进入 /apache-atlas-sources-1.2.0/addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/hook
修改文件 vim HBaseAtlasCoprocessor.java

//添加需要重写的10个方法
  public void preBalanceRSGroup(ObserverContext<MasterCoprocessorEnvironment> observerContext, String groupName) throws IOException  {

}


    public void postBalanceRSGroup(ObserverContext<MasterCoprocessorEnvironment> observerContext, String groupName, boolean balancerRan) throws IOException {

}

   public  void postRemoveRSGroup(ObserverContext<MasterCoprocessorEnvironment> observerContext, String name) throws IOException {

}

    public void postAddRSGroup(ObserverContext<MasterCoprocessorEnvironment> observerContext, String name) throws IOException {

}

    public void preRemoveRSGroup(ObserverContext<MasterCoprocessorEnvironment> observerContext, String name) throws IOException {

}

   public void  preAddRSGroup(ObserverContext<MasterCoprocessorEnvironment> observerContext, String name) throws IOException {

}

    public void postMoveTables(ObserverContext<MasterCoprocessorEnvironment> observerContext, Set<TableName> tables, String targetGroup) throws IOException {

}


   public void preMoveTables(ObserverContext<MasterCoprocessorEnvironment> observerContext, Set<TableName> tables, String targetGroup) throws IOException {

}


    public void postMoveServers(ObserverContext<MasterCoprocessorEnvironment> observerContext, Set<Address> servers, String targetGroup) throws IOException {

}

    public void preMoveServers(ObserverContext<MasterCoprocessorEnvironment> observerContext, Set<Address> servers, String targetGroup) throws IOException {

}

//再导入这2个包
import java.util.Set;
import org.apache.hadoop.hbase.net.Address;

例一：

Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/jaxrs/json/JacksonJaxbJsonProvider
 at org.apache.atlas.AtlasBaseClient.getClient(AtlasBaseClient.java:253)
 at org.apache.atlas.AtlasBaseClient.initializeState(AtlasBaseClient.java:425)
 at org.apache.atlas.AtlasBaseClient.initializeState(AtlasBaseClient.java:420)
 at org.apache.atlas.AtlasBaseClient.<init>(AtlasBaseClient.java:115)
 at org.apache.atlas.AtlasClientV2.<init>(AtlasClientV2.java:77)
 at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:131)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.jaxrs.json.JacksonJaxbJsonProvider
 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 ... 6 more

例二：

2020-05-21 11:46:02,586 INFO  - [main:] ~ checking HBase availability.. (HBaseBridge:204)
2020-05-21 11:46:02,932 ERROR - [main:] ~ ImportHBaseEntities failed (HBaseBridge:188)
java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
        at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:3172)
        at org.apache.atlas.hbase.bridge.HBaseBridge.<init>(HBaseBridge.java:206)
        at org.apache.atlas.hbase.bridge.HBaseBridge.main(HBaseBridge.java:148)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
        ... 5 more
Caused by: java.lang.NoSuchFieldError: HBASE_CLIENT_PREFETCH
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:721)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:654)
        ... 10 more

例三：

Downloading from repository.jboss.org-public: https://repository.jboss.org/nexus/content/groups/public/org/apache/hbase/hbase-hadoop2-compat/1.2.0-cdh5.16.2/hbase-hadoop2-compat-1.2.0-cdh5.16.2-tests.jar
Downloading from repository.jboss.org-public: https://repository.jboss.org/nexus/content/groups/public/org/apache/hbase/hbase-common/1.2.0-cdh5.16.2/hbase-common-1.2.0-cdh5.16.2.jar
Downloading from repository.jboss.org-public: https://repository.jboss.org/nexus/content/groups/public/org/apache/hbase/hbase-hadoop-compat/1.2.0-cdh5.16.2/hbase-hadoop-compat-1.2.0-cdh5.16.2-tests.jar
Downloading from typesafe: https://repo.typesafe.com/typesafe/releases/org/apache/hbase/hbase-server/1.2.0-cdh5.16.2/hbase-server-1.2.0-cdh5.16.2.jar
Downloading from typesafe: https://repo.typesafe.com/typesafe/releases/org/apache/hbase/hbase-server/1.2.0-cdh5.16.2/hbase-server-1.2.0-cdh5.16.2-tests.jar
Downloading from typesafe: https://repo.typesafe.com/typesafe/releases/org/apache/hbase/hbase-client/1.2.0-cdh5.16.2/hbase-client-1.2.0-cdh5.16.2.jar
Downloading from typesafe: https://repo.typesafe.com/typesafe/releases/org/apache/hbase/hbase-hadoop2-compat/1.2.0-cdh5.16.2/hbase-hadoop2-compat-1.2.0-cdh5.16.2-tests.jar
Downloading from typesafe: https://repo.typesafe.com/typesafe/releases/org/apache/hbase/hbase-common/1.2.0-cdh5.16.2/hbase-common-1.2.0-cdh5.16.2.jar
Downloading from typesafe: https://repo.typesafe.com/typesafe/releases/org/apache/hbase/hbase-hadoop-compat/1.2.0-cdh5.16.2/hbase-hadoop-compat-1.2.0-cdh5.16.2-tests.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Atlas Server Build Tools 1.0 ................ SUCCESS [  0.442 s]
[INFO] apache-atlas 1.2.0 ................................. SUCCESS [  2.714 s]
[INFO] Apache Atlas Test Utility Tools .................... SUCCESS [  4.101 s]
[INFO] Apache Atlas Integration ........................... SUCCESS [  3.813 s]
[INFO] Apache Atlas Common ................................ SUCCESS [  1.035 s]
[INFO] Apache Atlas Client ................................ SUCCESS [  0.107 s]
[INFO] atlas-client-common ................................ SUCCESS [  0.533 s]
[INFO] atlas-client-v1 .................................... SUCCESS [  0.705 s]
[INFO] Apache Atlas Server API ............................ SUCCESS [  0.838 s]
[INFO] Apache Atlas Notification .......................... SUCCESS [  1.486 s]
[INFO] atlas-client-v2 .................................... SUCCESS [  0.526 s]
[INFO] Apache Atlas Graph Database Projects ............... SUCCESS [  0.065 s]
[INFO] Apache Atlas Graph Database API .................... SUCCESS [  0.599 s]
[INFO] Graph Database Common Code ......................... SUCCESS [  0.630 s]
[INFO] Apache Atlas JanusGraph DB Impl .................... SUCCESS [ 46.304 s]
[INFO] Apache Atlas Graph Database Implementation Dependencies SUCCESS [  1.849 s]
[INFO] Shaded version of Apache hbase client .............. SUCCESS [  5.264 s]
[INFO] Shaded version of Apache hbase server .............. SUCCESS [ 13.347 s]
[INFO] Apache Atlas Authorization ......................... SUCCESS [  0.826 s]
[INFO] Apache Atlas Repository ............................ SUCCESS [  7.587 s]
[INFO] Apache Atlas UI .................................... SUCCESS [ 54.525 s]
[INFO] Apache Atlas Web Application ....................... SUCCESS [ 58.020 s]
[INFO] Apache Atlas Documentation ......................... SUCCESS [  7.783 s]
[INFO] Apache Atlas FileSystem Model ...................... SUCCESS [  1.850 s]
[INFO] Apache Atlas Plugin Classloader .................... SUCCESS [  0.480 s]
[INFO] Apache Atlas Hive Bridge Shim ...................... SUCCESS [  0.727 s]
[INFO] Apache Atlas Hive Bridge ........................... SUCCESS [  2.478 s]
[INFO] Apache Atlas Falcon Bridge Shim .................... SUCCESS [  0.760 s]
[INFO] Apache Atlas Falcon Bridge ......................... SUCCESS [  0.931 s]
[INFO] Apache Atlas Sqoop Bridge Shim ..................... SUCCESS [  0.113 s]
[INFO] Apache Atlas Sqoop Bridge .......................... SUCCESS [  1.350 s]
[INFO] Apache Atlas Storm Bridge Shim ..................... SUCCESS [ 22.128 s]
[INFO] Apache Atlas Storm Bridge .......................... SUCCESS [  1.684 s]
[INFO] Apache Atlas Hbase Bridge Shim ..................... SUCCESS [  1.297 s]
[INFO] Apache Atlas Hbase Bridge .......................... FAILURE [02:19 min]
[INFO] Apache Atlas Kafka Bridge .......................... SKIPPED
[INFO] Apache Atlas Distribution 1.2.0 .................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:27 min
[INFO] Finished at: 2020-05-21T00:41:44+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project hbase-bridge: Could not resolve dependencies for project org.apache.atlas:hbase-bridge:jar:1.2.0: The following artifacts could not be resolved: org.apache.hbase:hbase-server:jar:1.2.0-cdh5.16.2, org.apache.hbase:hbase-server:jar:tests:1.2.0-cdh5.16.2, org.apache.hbase:hbase-client:jar:1.2.0-cdh5.16.2, org.apache.hbase:hbase-common:jar:1.2.0-cdh5.16.2, org.apache.hbase:hbase-hadoop2-compat:jar:tests:1.2.0-cdh5.16.2, org.apache.hbase:hbase-hadoop-compat:jar:tests:1.2.0-cdh5.16.2: Could not find artifact org.apache.hbase:hbase-server:jar:1.2.0-cdh5.16.2 in alimaven (http://maven.aliyun.com/nexus/content/repositories/central/) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hbase-bridge
[root@kafka-dev-01 apache-atlas-sources-1.2.0]#

例四：


[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ hbase-bridge ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 6 source files to /root/apache-atlas-sources-1.2.0/addons/hbase-bridge/target/classes
[INFO] /root/apache-atlas-sources-1.2.0/addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/hook/HBaseAtlasCoprocessorBase.java: Some input files use or override a deprecated API.
[INFO] /root/apache-atlas-sources-1.2.0/addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/hook/HBaseAtlasCoprocessorBase.java: Recompile with -Xlint:deprecation for details.
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /root/apache-atlas-sources-1.2.0/addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/hook/HBaseAtlasCoprocessor.java:[37,8] org.apache.atlas.hbase.hook.HBaseAtlasCoprocessor is not abstract and does not override abstract method postBalanceRSGroup(org.apache.hadoop.hbase.coprocessor.ObserverContext<org.apache.hadoop.hbase.coprocessor.MasterCoprocessorEnvironment>,java.lang.String,boolean) in org.apache.hadoop.hbase.coprocessor.MasterObserver
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Atlas Server Build Tools 1.0 ................ SUCCESS [  0.372 s]
[INFO] apache-atlas 1.2.0 ................................. SUCCESS [  2.329 s]
[INFO] Apache Atlas Test Utility Tools .................... SUCCESS [  3.784 s]
[INFO] Apache Atlas Integration ........................... SUCCESS [  4.000 s]
[INFO] Apache Atlas Common ................................ SUCCESS [  1.143 s]
[INFO] Apache Atlas Client ................................ SUCCESS [  0.099 s]
[INFO] atlas-client-common ................................ SUCCESS [  0.474 s]
[INFO] atlas-client-v1 .................................... SUCCESS [  0.621 s]
[INFO] Apache Atlas Server API ............................ SUCCESS [  0.782 s]
[INFO] Apache Atlas Notification .......................... SUCCESS [  1.510 s]
[INFO] atlas-client-v2 .................................... SUCCESS [  0.534 s]
[INFO] Apache Atlas Graph Database Projects ............... SUCCESS [  0.081 s]
[INFO] Apache Atlas Graph Database API .................... SUCCESS [  0.596 s]
[INFO] Graph Database Common Code ......................... SUCCESS [  0.600 s]
[INFO] Apache Atlas JanusGraph DB Impl .................... SUCCESS [  4.628 s]
[INFO] Apache Atlas Graph Database Implementation Dependencies SUCCESS [  1.751 s]
[INFO] Shaded version of Apache hbase client .............. SUCCESS [  5.282 s]
[INFO] Shaded version of Apache hbase server .............. SUCCESS [ 13.297 s]
[INFO] Apache Atlas Authorization ......................... SUCCESS [  0.804 s]
[INFO] Apache Atlas Repository ............................ SUCCESS [  7.324 s]
[INFO] Apache Atlas UI .................................... SUCCESS [ 47.830 s]
[INFO] Apache Atlas Web Application ....................... SUCCESS [ 35.681 s]
[INFO] Apache Atlas Documentation ......................... SUCCESS [  3.309 s]
[INFO] Apache Atlas FileSystem Model ...................... SUCCESS [  1.833 s]
[INFO] Apache Atlas Plugin Classloader .................... SUCCESS [  0.406 s]
[INFO] Apache Atlas Hive Bridge Shim ...................... SUCCESS [  0.698 s]
[INFO] Apache Atlas Hive Bridge ........................... SUCCESS [  2.365 s]
[INFO] Apache Atlas Falcon Bridge Shim .................... SUCCESS [  0.817 s]
[INFO] Apache Atlas Falcon Bridge ......................... SUCCESS [  1.007 s]
[INFO] Apache Atlas Sqoop Bridge Shim ..................... SUCCESS [  0.104 s]
[INFO] Apache Atlas Sqoop Bridge .......................... SUCCESS [  1.232 s]
[INFO] Apache Atlas Storm Bridge Shim ..................... SUCCESS [  0.387 s]
[INFO] Apache Atlas Storm Bridge .......................... SUCCESS [  1.605 s]
[INFO] Apache Atlas Hbase Bridge Shim ..................... SUCCESS [  1.295 s]
[INFO] Apache Atlas Hbase Bridge .......................... FAILURE [  4.167 s]
[INFO] Apache Atlas Kafka Bridge .......................... SKIPPED
[INFO] Apache Atlas Distribution 1.2.0 .................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:33 min
[INFO] Finished at: 2020-05-21T12:42:57+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project hbase-bridge: Compilation failure
[ERROR] /root/apache-atlas-sources-1.2.0/addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/hook/HBaseAtlasCoprocessor.java:[37,8] org.apache.atlas.hbase.hook.HBaseAtlasCoprocessor is not abstract and does not override abstract method postBalanceRSGroup(org.apache.hadoop.hbase.coprocessor.ObserverContext<org.apache.hadoop.hbase.coprocessor.MasterCoprocessorEnvironment>,java.lang.String,boolean) in org.apache.hadoop.hbase.coprocessor.MasterObserver
[ERROR]
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hbase-bridge
[root@kafka-dev-01 apache-atlas-sources-1.2.0]#

本文转载于：https://www.codenong.com/cs106258758/

x_iaoa_o

关注

1
点赞
踩
6

收藏

觉得还不错? 一键收藏
7
评论
Apache Atlas 2.0.0 CDH环境部署问题及解决

Apache Atlas部署FAQAtlas部署问题Atlas启动报错(zookeeper启动失败导致)1.问题描述2.原因定位3.解决方法Atlas启动报错(solr启动失败导致)1.问题描述2.原因定位3.解决方法Atlas启动报错(HBase中没有Column)1.问题描述2.原因定位3.解决方法Atlas import-hive.sh报错1.问题描述2.原因定位3.解决方法Atlas部署问题Atlas启动报错(zookeeper启动失败导致)通过Atlas内嵌
复制链接

扫一扫