一：没有dataNode（Live Nodes的数量为0）或者两个nameNode都处于 standby状态？
可能的解决方案：
趟过的坑：clusterID不一致，集群的nameNode和dataNode必须保持一致！
如何使用bin/hadoop namenode -format格式化两次就会出现上面的bug
解决办法：
第一种：拷贝/home/hadoop/data(nameNode的路径)文件夹里的name/current文件夹下的VERSION到/home/hadoop/data/data/current文件夹下（注意其他节点也一样，clusterID要一致）
第二种：删除/home/hadoop/data下的data和name文件夹下的current文件夹（其他节点也需要删除），然后执行一次bin/hadoop namenode -format格式化，会重新生成clusterID。
Bug详情见http://blog.csdn.net/zhangt85/article/details/42078347
需要注意两点：
第一点
先启动 ./start-dfs.sh
再进行 格式化，则成功
第二点：
格式化的时候， 判断是否re_format filesystem 的时候 Y/N 一定要大写的 Y!!!
上面错误的常见异常：
17/06/13 17:17:06 INFO ipc.Client: Retrying connect to server: hadoop05/192.168.3.15:8485. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/06/13 17:17:06 INFO ipc.Client: Retrying connect to server: hadoop07/192.168.3.17:8485. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/06/13 17:17:07 INFO ipc.Client: Retrying connect to server: hadoop06/192.168.3.16:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/06/13 17:17:07 INFO ipc.Client: Retrying connect to server: hadoop05/192.168.3.15:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/06/13 17:17:07 WARN namenode.NameNode: Encountered exception during format:
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check if JNs are ready for formatting. 1 exceptions thrown:
192.168.3.16:8485: Call From hadoop01/192.168.3.11 to hadoop06:8485 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:232)
at org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:899)
at org.apache.hadoop.hdfs.server.namenode.FSImage.confirmFormat(FSImage.java:171)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:940)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1382)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1507)
17/06/13 17:17:07 INFO ipc.Client: Retrying connect to server: hadoop07/192.168.3.17:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/06/13 17:17:07 FATAL namenode.NameNode: Failed to start namenode.
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check if JNs are ready for formatting. 1 exceptions thrown:
192.168.3.16:8485: Call From hadoop01/192.168.3.11 to hadoop06:8485 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:232)
at org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:899)
at org.apache.hadoop.hdfs.server.namenode.FSImage.confirmFormat(FSImage.java:171)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:940)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1382)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1507)
2:hadoop 在window系统上运行报错？
java.io.IOException: Could not locate executable
null
\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:
356
)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:
371
)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:
364
)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:
80
)
at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:
611
)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:
272
)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:
260
)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:
790
)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:
760
)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:
633
)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$
1
.apply(Utils.scala:
2001
)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$
1
.apply(Utils.scala:
2001
)
at scala.Option.getOrElse(Option.scala:
120
)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:
2001
)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:
207
)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:
218
)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:
163
)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:
269
)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:
272
)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:
154
)
at SparkFromHbase$.main(SparkFromHbase.scala:
15
)
at SparkFromHbase.main(SparkFromHbase.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
57
)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
43
)
at java.lang.reflect.Method.invoke(Method.java:
606
)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:
134
)
很明显应该是HADOOP_HOME的问题。如果HADOOP_HOME为空，必然fullExeName为null\bin\winutils.exe。解决方法很简单，配置环境变量，不想重启电脑可以在程序里加上：
|
1
|
System.setProperty(
"hadoop.home.dir"
,
"E:\\Program Files\\hadoop-2.7.0"
);
|
注：E:\\Program Files\\hadoop-2.7.0是我本机解压的hadoop的路径。
稍后再执行，你可能还是会出现同样的错误，这个时候你可能会要怪我了。其实一开始我是拒绝的，因为你进入你的hadoop-x.x.x/bin目录下看，你会发现你压根就没有winutils.exe这个东东。
于是我告诉你，你可以去github下载一个，地球人都知道的地址发你一个。
地址：https://github.com/srccodes/hadoop-common-2.2.0-bin
下载好后，把winutils.exe加入你的hadoop-x.x.x/bin下,当然最好的方法还是配置环境变量：
A、将hadoop（windows平台下编译的hadoop）安装包解压一份到windows的任意一个目录下
B、在window系统中配置HADOOP_HOME指向你解压的安装包目录
C、在windows系统的path变量中加入HADOOP_HOME的bin目录
D、重启电脑
17/08/29 07:08:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/29 07:08:43 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
at selfSort.SelfSort$.main(SelfSort.scala:16)
at selfSort.SelfSort.main(SelfSort.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
17/08/29 07:08:43 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:394)
at selfSort.SelfSort$.main(SelfSort.scala:16)
at selfSort.SelfSort.main(SelfSort.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
从提示中可以看出找不到程序运行的master，此时需要配置环境变量。
传递给spark的master url可以有如下几种：
local 本地单线程
local[K] 本地多线程（指定K个内核）
local[*] 本地多线程（指定所有可用内核）
spark://HOST:PORT 连接到指定的 Spark standalone cluster master，需要指定端口。
mesos://HOST:PORT 连接到指定的 Mesos 集群，需要指定端口。
yarn-client客户端模式 连接到 YARN 集群。需要配置 HADOOP_CONF_DIR。
yarn-cluster集群模式 连接到 YARN 集群。需要配置 HADOOP_CONF_DIR。
点击edit configuration，在左侧点击该项目。在右侧VM options中输入“-Dspark.master=local”，指示本程序本地单线程运行，再次运行即可。
配置如图：
4:将jar包上传到Hadoop服务器并执行jar包的方法参数不对应导致报错output directory already exists
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://bi/sortedMR.txt already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:267)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:140)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1297)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1294)
扫一扫
2136
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
