StreamSets从MySQL实时同步数据到Hbase问题总结

问题一：找不到zookeeper的/hbase/master

org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
        at org.apache.hadoop.hbase.client.ConnectionImplementation.isMasterRunning(ConnectionImplementation.java:585)
        at org.apache.hadoop.hbase.client.HBaseAdmin.available(HBaseAdmin.java:2355)
        at com.streamsets.pipeline.hbase.impl.HBaseProducer2_0.checkHBaseAvailable(HBaseProducer2_0.java:107)
        at com.streamsets.pipeline.stage.destination.hbase.HBaseTarget.lambda$init$0(HBaseTarget.java:132)
        at com.streamsets.pipeline.stage.destination.hbase.HBaseTarget$$Lambda$166/1813007165.run(Unknown Source)

问题解决：
首先进入到zookeeper-client下查看是否有/hbase/master
然后注释掉所有节点/etc/hosts中的前两行
最后尝试将ip地址修改为/etc/hosts中的映射主机名
重新启动集群

问题二：StreamSets无安全认证(kerberos)MySQL向有安全认证的Hbase中同步数据

org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Call to bioyuanb/172.16.1.175:16000 failed on local exception: org.apache.hadoop.hbase.ipc.FallbackDisallowedException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections.
	at org.apache.hadoop.hbase.client.ConnectionImplementation.isMasterRunning(ConnectionImplementation.java:585)
	at org.apache.hadoop.hbase.client.HBaseAdmin.available(HBaseAdmin.java:2355)
	at com.streamsets.pipeline.hbase.impl.HBaseProducer2_0.checkHBaseAvailable(HBaseProducer2_0.java:107)
	at com.streamsets.pipeline.stage.destination.hbase.HBaseTarget.lambda$init$0(HBaseTarget.java:132)
	at com.streamsets.pipeline.stage.destination.hbase.HBaseTarget$$Lambda$166/1813007165.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726)
	at com.streamsets.pipeline.stage.destination.hbase.HBaseTarget.init(HBaseTarget.java:130)
	at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48)
	at com.streamsets.pipeline.api.base.configurablestage.DStage.init(DStage.java:36)
	at com.streamsets.datacollector.runner.StageRuntime.lambda$init$0(StageRuntime.java:211)
	at com.streamsets.datacollector.runner.StageRuntime$$Lambda$139/631384246.get(Unknown Source)
	at com.streamsets.datacollector.util.LambdaUtil.withClassLoaderInternal(LambdaUtil.java:148)
	at com.streamsets.datacollector.util.LambdaUtil.withClassLoader(LambdaUtil.java:44)
	at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:209)
	at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:123)
	at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:47)
	at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:408)
	at com.streamsets.datacollector.runner.Pipeline.lambda$init$0(Pipeline.java:397)
	at com.streamsets.datacollector.runner.Pipeline$$Lambda$158/832547163.accept(Unknown Source)
	at com.streamsets.datacollector.runner.PipeRunner.forEach(PipeRunner.java:170)
	at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:394)
	at com.streamsets.datacollector.runner.preview.PreviewPipeline.run(PreviewPipeline.java:49)
	at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:219)
	at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer.lambda$start$0(AsyncPreviewer.java:95)
	at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer$$Lambda$136/33251279.call(Unknown Source)
	at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)

问题解决：在Hbase的配置中开启以下3个选项，然后重启，希望问题解决的朋友能给个赞，解决问题不容易，还有有什么问题可以评论区见。

问题三：在使用CDH版本的StreamSets引入External Libraries时报错如下：

com.streamsets.datacollector.util.RestException: REST_1003 - Failed to create directory: lib
	at com.streamsets.datacollector.restapi.StageLibraryResource.installExtras(StageLibraryResource.java:468)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)

问题解决：参考链接：https://www.pianshen.com/article/8717259858/
在集群的节点中做操作如下：

 mkdir /opt/cloudera/parcels/STREAMSETS_DATACOLLECTOR-3.13.0/sdc-extras 
 chown -R sdc:sdc  /opt/cloudera/parcels/STREAMSETS_DATACOLLECTOR-3.13.0/sdc-extras
 在CDH的StreamSets的配置sdc-env.sh中添加一下语句
 export STREAMSETS_LIBRARIES_EXTRA_DIR="/opt/cloudera/parcels/STREAMSETS_DATACOLLECTOR-3.13.0/sdc-extras"