Hadoop FAQ

===1, The ServiceName: mapreduce.shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers


A: Configure aux-services with mapreduce_shuffle instead of xxx.
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>


===2, WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable


A:
Set debug mode, execute command: $export HADOOP_ROOT_LOGGER=DEBUG,console
wrong ELF class: ELFCLASS32 (Possible cause: architecture word width mismatch)
$file /home/hadoop/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0
ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped


Found that its 32-bit not 64-bit, so package hadoop-2.2.0-src.tar.gz to get native 64-bit lib.


Install following package and program
Package: $sudo yum install zlib-devel lzo-devel gcc autoconf automake libtool ncurses-devel openssl-devel cmake 
Program: Ant Maven ProtocolBuffer findbugs


$mvn package -Pdist,native -DskipTests -Dtar


ERROR 1: Failed to package for can not download following jars: http://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.0-alpha-10/doxia-sink-api-1.0-alpha-10.jar
A: Use DNS 8.8.8.8 instead of 10.190.202.200


ERROR 2: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.2.0:protoc (compile-protoc) on project hadoop-common
A: Install protobuf https://code.google.com/p/protobuf/
$./configure
$make
$vi .bashrc
export PROTO_HOME=/home/hadoop/software/protobuf-2.5.0/src/.libs
export PATH=$PROTO_HOME:$PATH
$protoc --version
protoc: error while loading shared libraries: libprotobuf.so.8: cannot open shared object file: No such file or directory
$sudo cp -rp /home/hadoop/software/protobuf-2.5.0/src/.libs/libproto* /usr/lib64


ERROR 3: cannot access AbstractLifeCycle [ERROR] class file for org.mortbay.component.AbstractLifeCycle not found
A: Patch: https://issues.apache.org/jira/browse/HADOOP-10110
hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/pom.xml
<dependency>
  <groupId>org.mortbay.jetty</groupId>
  <artifactId>jetty-util</artifactId>
  <scope>test</scope>
</dependency>


===3, Error while run yarn task, "org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist"
A: Add following configurations to yarn-site.xml
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>


===4, Slave datanode log
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server
A:
1) Master namenode stop iptables.
$sudo /etc/init.d/iptables stop


2) $vi /etc/hosts
#127.0.0.1  localhost
#::1        localhost


===5, $hbase shell
Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available


A: Make hadoop with snappy
#mvn package -Drequire.snappy -Pdist,native,src -DskipTests -Dtar
Now got libsnappy.so libhdfs.so libhadoop.so, copy them to  hbase/lib/native/Linux-amd64-64


===6, WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hadoop/hd2home/hadoop/tmp/edits should be specified as a URI in configuration files. Please update hdfs configuration.


A: Delete dfs.namenode.edits.dir configuration item on hdfs-site.xml, it will use "file://${hadoop.tmp.dir}/dfs/name" as its default value.




===7. Execute command "$hadoop fs -ls /" on name node, data nodes have following error:


FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-140379527
-10.103.67.249-1389610212569 (storage id DS-1543890749-10.103.67.250-50010-1389607291301) service to hd2-master/10.103.67.249:9000
java.io.IOException: Incompatible clusterIDs in /home/hadoop/hd2home/hadoop/tmp/dfs/data: namenode clusterID = CID-d1dc6009-9383-426f-bae2-
3565fbe49787; datanode clusterID = CID-49e59c0e-5678-4885-a72d-433514fe2b84
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:391)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:191)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:219)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:837)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:808)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:280)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:222)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
        at java.lang.Thread.run(Thread.java:722)
2014-01-13 18:59:13,992 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-140379527-10.103
.67.249-1389610212569 (storage id DS-1543890749-10.103.67.250-50010-1389607291301) service to hd2-master/10.103.67.249:9000
2014-01-13 18:59:14,079 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool BP-140379527-10.103.67.249-1389610212569 (
storage id DS-1543890749-10.103.67.250-50010-1389607291301)
2014-01-13 18:59:16,080 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2014-01-13 18:59:16,083 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2014-01-13 18:59:16,087 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 


A:
Delete /home/hadoop/hd2home on namenode and all datanodes
Delete hadoop and hbase related files under /tmp on namenode and all datanodes
then reformat namenode.


$stopall; startall; hadoop namenode format


===8. Execute stopall on namenode, data node cannot be stopped gracefully, it was killed by the namenode SIGTERM signal, 
Namenode console have message:
hd2-s1: stopping nodemanager
hd2-s1: nodemanager did not stop gracefully after 5 seconds: killing with kill -9


Datanode have the following error:


2014-01-14 11:22:23,812 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
java.io.IOException: Failed on local exception: java.io.IOException: Connection reset by peer; Host Details : local host is: "hd2-s2/10.103.67.251"; destination host is: "hd2-master":9000; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
        at org.apache.hadoop.ipc.Client.call(Client.java:1351)
        at org.apache.hadoop.ipc.Client.call(Client.java:1300)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at com.sun.proxy.$Proxy9.sendHeartbeat(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy9.sendHeartbeat(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:167)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:445)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:525)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:676)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
        at sun.nio.ch.IOUtil.read(IOUtil.java:198)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
        at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
        at java.io.FilterInputStream.read(FilterInputStream.java:133)
        at java.io.FilterInputStream.read(FilterInputStream.java:133)
        at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995)
        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
2014-01-14 11:22:27,556 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hd2-master/10.103.67.249:9000. Already tried 0 time(s); retry policy is Retr
yUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2014-01-14 11:22:28,557 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hd2-master/10.103.67.249:9000. Already tried 1 time(s); retry policy is Retr
yUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2014-01-14 11:22:28,850 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: RECEIVED SIGNAL 15: SIGTERM
2014-01-14 11:22:28,852 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at hd2-s2/10.103.67.251
************************************************************/


A:




===9, Namenode have following error:
2014-01-14 11:55:04,220 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: trying to get DT with no secret manager running
2014-01-14 11:55:04,233 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user dr.who




===10, Start hadoop before formating namenode.
Master node have following error:
2014-01-14 23:29:24,024 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/hadoop/hd2home/hadoop/tmp/dfs/name is in an inconsistent state: storage directory does not exist or is 
not accessible.
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:292)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:787)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:568)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:443)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:491)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:684)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:669)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1254)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
2014-01-14 23:29:24,029 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1


Slave nodes have following error:
2014-01-14 23:29:43,252 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hd2-master/10.103.67.210:9000


A: Format namenode, then restart Hadoop
$hadoop namenode -format




===11, Run task have error
14/01/21 18:33:18 ERROR mapreduce.TableInputFormatBase: Cannot resolve the host name for hd2-s2/10.103.67.251 because of javax.naming.NameNotFoundException: DNS name not found [response code 3]; remaining name '251.67.103.10.in-addr.arpa'


===12, Eclipse tomcat run time error
org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [appServlet] in context with path [] threw exception [Handler processing failed; nested exception is java.lang.OutOfMemoryError: PermGen space] with root cause
A: D:\bin\apache-tomcat-7.0.35-p4\bin\catalina.sh
JAVA_OPTS="$JAVA_OPTS -server -XX:PermSize=256M -XX:MaxPermSize=512m








===13 Format hadoop have error
14/02/27 14:33:34 FATAL namenode.NameNode: Exception in namenode join
java.io.IOException: Cannot create directory /var/hadoop/data/namenode/current
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:301)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:523)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:544)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:147)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1213)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
        
A: Have no permission to create directory under var
Create hadoop directory on master and slave nodes.
$sudo mkdir -p /var/hadoop
$sudo chmod 755 /var/hadoop
$sudo chown hadoop:hadoop /var/hadoop