CDH安装中遇到的问题解决

--------------------------cdh中重装hbase后之前的脚本残留解决---------------------

1.切换到zookeeper的bin目录；   //zookeeper-client status命令可以找到该bin目录

2.执行$sh zkCli.sh

输入 ls /

4.输入rmr /hbase

5.退出重启即可

-------------------------cdh重装过程中有datanode启动报错-----------------------------------

Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to archive.cloudera.com/192.168.1.203:8022. Exiting. 
java.io.IOException: All specified directories are failed to load.
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1394)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1355)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:228)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829)
	at java.lang.Thread.run(Thread.java:745)

there are 2 Possible Solutions to resolve

First:

Your namenode and datanode cluster ID does not match, make sure to make them the same.

In name node, change ur cluster id in the file located in:

$ nano HADOOP_FILE_SYSTEM/namenode/current/VERSION 

In data node you cluster id is stored in the file:

$ nano HADOOP_FILE_SYSTEM/datanode/current/VERSION

-------------------------cdh使用过程中kafka崩溃后重装清理残留的kafka步骤-----------------------------------

1.kafka-topics.sh --delete --zookeeper  host:port --topic topicname

提示marked for deletion

并没有真正删除，如果要真正删除

配置delete.topic.enable=true

配置文件在kafka\config目录server.properties

2、删除kafka存储目录（server.propertiewenjian log.dirs配置，默认为“/tmp/kafka-logs”）下对应的topic。(不同broker下存储的topic不一定相同，所有broker都要看一下)

3、进入zookeeper客户端删掉对应topic

        zkCli.sh  -server 127.0.0.1:42182

   找到topic目录

        ls   /brokers/topics

   删掉对应topic

    rmr  /brokers/topic/topic-name

   找到目录

        ls    /config/topics

   删掉对应topic

        rmr  /config/topics/topic-name  

-------------------------cdh集群中spark-shell 启动报错-----------------------------------

[root@archive bin]# ./spark-shell 
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
	at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:117)
	at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:117)
	at scala.Option.getOrElse(Option.scala:120)
	at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:117)
	at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:103)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:114)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	... 7 more

在CDH集群中启动spark-shell是默认以yarn模式启动的，缺少环境变量的配置。

1.如果要在本地模式启动可以加参数:spark-shell  --master  local[4]

2.如果要在集群模式启动的话，需要配置spark的环境变量 spark-conf/spark-env.sh 的 Spark 客户端高级配置代码段（安全阀）

<strong><span style="color:#ff0000;">export SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:/opt/cloudera/parcels/CDH/jars/htrace-core-3.1.0-incubating.jar</span></strong>

:/etc/hbase/conf:/usr/local/jars/elasticsearch-spark_2.10-2.2.0.jar:/usr/local/jars/joda-time-2.9.1.jar:/usr/local/jars/es-shaded-1.0-SNAPSHOT.jar