Hadoop常见问题及解决办法

The number of maps is usually driven by the number of DFS blocks in the input files. Although that causes people to adjust their DFS block size to adjust the number of maps. The right level of parallelism for maps seems to be around 10-100 maps/node, although we have taken it up to 300 or so for very cpu-light map tasks. Task setup takes awhile, so it is best if the maps take at least a minute to execute.

Actually controlling the number of maps is subtle. The mapred.map.tasks parameter is just a hint to the InputFormat for the number of maps. The default InputFormat behavior is to split the total number of bytes into the right number of fragments. However, in the default case the DFS block size of the input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapred.min.split.size. Thus, if you expect 10TB of input data and have 128MB DFS blocks, you’ll end up with 82k maps, unless your mapred.map.tasks is even larger. Ultimately the [WWW] InputFormat determines the number of maps.

The number of map tasks can also be increased manually using the JobConf’s conf.setNumMapTasks(int num). This can be used to increase the number of map tasks, but will not set the number below that which Hadoop determines via splitting the input data.

Number of Reduces

The right number of reduces seems to be 0.95 or 1.75 * (nodes * mapred.tasktracker.tasks.maximum). At 0.95 all of the reduces can launch immediately and start transfering map outputs as the maps finish. At 1.75 the faster nodes will finish their first round of reduces and launch a second round of reduces doing a much better job of load balancing.

Currently the number of reduces is limited to roughly 1000 by the buffer size for the output files (io.buffer.size * 2 * numReduces << heapSize). This will be fixed at some point, but until it is it provides a pretty firm upper bound.

The number of reduces also controls the number of output files in the output directory, but usually that is not important because the next map/reduce step will split them into even smaller splits for the maps.

The number of reduce tasks can also be increased in the same way as the map tasks, via JobConf’s conf.setNumReduceTasks(int num).

自己的理解：

mapper个数的设置：跟input file 有关系，也跟filesplits有关系，filesplits的上线为dfs.block.size，下线可以通过mapred.min.split.size设置，最后还是由InputFormat决定。

较好的建议：

The right number of reduces seems to be 0.95 or 1.75 multiplied by (<no. of nodes> * mapred.tasktracker.reduce.tasks.maximum).increasing the number of reduces increases the framework overhead, but increases load balancing and lowers the cost of failures.

mapred.tasktracker.reduce.tasks.maximum

2

The maximum number of reduce tasks that will be run

simultaneously by a task tracker.

单个node新加硬盘

1.修改需要新加硬盘的node的dfs.data.dir，用逗号分隔新、旧文件目录

2.重启dfs

同步hadoop 代码

hadoop-env.sh

# host:path where hadoop code should be rsync’d from.  Unset by default.

# export HADOOP_MASTER=master:/home/$USER/src/hadoop

用命令合并HDFS小文件

hadoop fs -getmerge

重启reduce job方法

Introduced recovery of jobs when JobTracker restarts. This facility is off by default.

Introduced config parameters “mapred.jobtracker.restart.recover”, “mapred.jobtracker.job.history.block.size”, and “mapred.jobtracker.job.history.buffer.size”.

还未验证过。

IO写操作出现问题

0-1246359584298, infoPort=50075, ipcPort=50020):Got exception while serving blk_-5911099437886836280_1292 to /172.16.100.165:

java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/

172.16.100.165:50010 remote=/172.16.100.165:50930]

at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:185)

at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)

at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)

at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:293)

at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:387)

at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:179)

at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:94)

at java.lang.Thread.run(Thread.java:619)

It seems there are many reasons that it can timeout, the example given in

HADOOP-3831 is a slow reading client.

解决办法：在hadoop-site.xml中设置dfs.datanode.socket.write.timeout=0试试；

My understanding is that this issue should be fixed in Hadoop 0.19.1 so that

we should leave the standard timeout. However until then this can help

resolve issues like the one you’re seeing.

HDFS退服节点的方法

目前版本的dfsadmin的帮助信息是没写清楚的，已经file了一个bug了，正确的方法如下：

1. 将 dfs.hosts 置为当前的 slaves，文件名用完整路径，注意，列表中的节点主机名要用大名，即 uname -n 可以得到的那个。

2. 将 slaves 中要被退服的节点的全名列表放在另一个文件里，如 slaves.ex，使用 dfs.host.exclude 参数指向这个文件的完整路径

3. 运行命令 bin/hadoop dfsadmin -refreshNodes

4. web界面或 bin/hadoop dfsadmin -report 可以看到退服节点的状态是 Decomission in progress，直到需要复制的数据复制完成为止

5. 完成之后，从 slaves 里（指 dfs.hosts 指向的文件）去掉已经退服的节点

附带说一下 -refreshNodes 命令的另外三种用途：

2. 添加允许的节点到列表中（添加主机名到 dfs.hosts 里来）

3. 直接去掉节点，不做数据副本备份（在 dfs.hosts 里去掉主机名）

4. 退服的逆操作——停止 exclude 里面和 dfs.hosts 里面都有的，正在进行 decomission 的节点的退服，也就是把 Decomission in progress 的节点重新变为 Normal （在 web 界面叫 in service)

hadoop 学习借鉴

1. 解决hadoop OutOfMemoryError问题：

mapred.child.java.opts

-Xmx800M -server

With the right JVM size in your hadoop-site.xml , you will have to copy this

to all mapred nodes and restart the cluster.

或者：hadoop jar jarfile [main class] -D mapred.child.java.opts=-Xmx800M

2. Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.

when i use nutch1.0,get this error:

Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.

这个也很好解决：

可以删除conf/log4j.properties，然后可以看到详细的错误报告

我这儿出现的是out of memory

解决办法是在给运行主类org.apache.nutch.crawl.Crawl加上参数：-Xms64m -Xmx512m

你的或许不是这个问题，但是能看到详细的错误报告问题就好解决了

distribute cache使用

类似一个全局变量，但是由于这个变量较大，所以不能设置在config文件中，转而使用distribute cache

具体使用方法：(详见《the definitive guide》,P240)

1. 在命令行调用时：调用-files，引入需要查询的文件(可以是local file, HDFS file(使用hdfs://xxx?)), 或者 -archives (JAR,ZIP, tar等)

% hadoop jar job.jar MaxTemperatureByStationNameUsingDistributedCacheFile /

-files input/ncdc/metadata/stations-fixed-width.txt input/ncdc/all output

2. 程序中调用：

public void configure(JobConf conf) {

metadata = new NcdcStationMetadata();

try {

metadata.initialize(new File(“stations-fixed-width.txt”));

} catch (IOException e) {

throw new RuntimeException(e);

}

}

另外一种间接的使用方法：在hadoop-0.19.0中好像没有

调用addCacheFile()或者addCacheArchive()添加文件，

使用getLocalCacheFiles() 或 getLocalCacheArchives() 获得文件

hadoop的job显示web

There are web-based interfaces to both the JobTracker (MapReduce master) and NameNode (HDFS master) which display status pages about the state of the entire system. By default, these are located at [WWW] http://job.tracker.addr:50030/ and [WWW] http://name.node.addr:50070/.

hadoop监控

OnlyXP(52388483) 131702

用nagios作告警，ganglia作监控图表即可

status of 255 error

错误类型：

java.io.IOException: Task process exit with nonzero status of 255.

at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:424)

错误原因：

Set mapred.jobtracker.retirejob.interval and mapred.userlog.retain.hours to higher value. By default, their values are 24 hours. These might be the reason for failure, though I’m not sure

split size

FileInputFormat input splits: (详见 《the definitive guide》P190)

mapred.min.split.size: default=1, the smallest valide size in bytes for a file split.

mapred.max.split.size: default=Long.MAX_VALUE, the largest valid size.

dfs.block.size: default = 64M, 系统中设置为128M。

如果设置 minimum split size > block size, 会增加块的数量。(猜想从其他节点拿去数据的时候，会合并block，导致block数量增多)

如果设置maximum split size < block size, 会进一步拆分block。

split size = max(minimumSize, min(maximumSize, blockSize));

其中 minimumSize < blockSize < maximumSize.

sort by value

hadoop 不提供直接的sort by value方法，因为这样会降低mapreduce性能。

但可以用组合的办法来实现，具体实现方法见《the definitive guide》, P250

基本思想：

1. 组合key/value作为新的key；

2. 重载partitioner，根据old key来分割；

conf.setPartitionerClass(FirstPartitioner.class);

3. 自定义keyComparator：先根据old key排序，再根据old value排序；

conf.setOutputKeyComparatorClass(KeyComparator.class);

4. 重载GroupComparator, 也根据old key 来组合；  conf.setOutputValueGroupingComparator(GroupComparator.class);

small input files的处理

对于一系列的small files作为input file，会降低hadoop效率。

有3种方法可以将small file合并处理：

1. 将一系列的small files合并成一个sequneceFile，加快mapreduce速度。

详见WholeFileInputFormat及SmallFilesToSequenceFileConverter,《the definitive guide》, P194

2. 使用CombineFileInputFormat集成FileinputFormat，但是未实现过；

3. 使用hadoop archives(类似打包)，减少小文件在namenode中的metadata内存消耗。(这个方法不一定可行，所以不建议使用)

方法：

将/my/files目录及其子目录归档成files.har，然后放在/my目录下

bin/hadoop archive -archiveName files.har /my/files /my

查看files in the archive:

bin/hadoop fs -lsr har://my/files.har

skip bad records

JobConf conf = new JobConf(ProductMR.class);

conf.setJobName(“ProductMR”);

conf.setOutputKeyClass(Text.class);

conf.setOutputValueClass(Product.class);

conf.setMapperClass(Map.class);

conf.setReducerClass(Reduce.class);

conf.setMapOutputCompressorClass(DefaultCodec.class);

conf.setInputFormat(SequenceFileInputFormat.class);

conf.setOutputFormat(SequenceFileOutputFormat.class);

String objpath = “abc1”;

SequenceFileInputFormat.addInputPath(conf, new Path(objpath));

SkipBadRecords.setMapperMaxSkipRecords(conf, Long.MAX_VALUE);

SkipBadRecords.setAttemptsToStartSkipping(conf, 0);

SkipBadRecords.setSkipOutputPath(conf, new Path(“data/product/skip/”));

String output = “abc”;

SequenceFileOutputFormat.setOutputPath(conf, new Path(output));

JobClient.runJob(conf);

For skipping failed tasks try : mapred.max.map.failures.percent

restart 单个datanode

如果一个datanode 出现问题，解决之后需要重新加入cluster而不重启cluster，方法如下：

bin/hadoop-daemon.sh start datanode

bin/hadoop-daemon.sh start jobtracker

reduce exceed 100%

"Reduce Task Progress shows > 100% when the total size of map outputs (for a

single reducer) is high "

造成原因：

在reduce的merge过程中，check progress有误差，导致status > 100%，在统计过程中就会出现以下错误：java.lang.ArrayIndexOutOfBoundsException: 3

at org.apache.hadoop.mapred.StatusHttpServer$TaskGraphServlet.getReduceAvarageProgresses(StatusHttpServer.java:228)

at org.apache.hadoop.mapred.StatusHttpServer$TaskGraphServlet.doGet(StatusHttpServer.java:159)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)

at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)

at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)

at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)

at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)

at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)

at org.mortbay.http.HttpServer.service(HttpServer.java:954)

jira地址：

counters

3中counters：

1. built-in counters: Map input bytes, Map output records…

2. enum counters

调用方式：

enum Temperature {

MISSING,

MALFORMED

}

reporter.incrCounter(Temperature.MISSING, 1)

结果显示：

09/04/20 06:33:36 INFO mapred.JobClient:   Air Temperature Recor

09/04/20 06:33:36 INFO mapred.JobClient:     Malformed=3

09/04/20 06:33:36 INFO mapred.JobClient:     Missing=66136856

3. dynamic countes:

调用方式：

reporter.incrCounter(“TemperatureQuality”, parser.getQuality(),1);

结果显示：

09/04/20 06:33:36 INFO mapred.JobClient:   TemperatureQuality

09/04/20 06:33:36 INFO mapred.JobClient:     2=1246032

09/04/20 06:33:36 INFO mapred.JobClient:     1=973422173

09/04/20 06:33:36 INFO mapred.JobClient:     0=1

7: Namenode in safe mode

解决方法

bin/hadoop dfsadmin -safemode leave

8:java.net.NoRouteToHostException: No route to host

j解决方法：

sudo /etc/init.d/iptables stop

9：更改namenode后，在hive中运行select 依旧指向之前的namenode地址

这是因为：When youcreate a table, hive actually stores the location of the table (e.g.

hdfs://ip:port/user/root/…) in the SDS and DBS tables in the metastore . So when I bring up a new cluster the master has a new IP, but hive’s metastore is still pointing to the locations within the old

cluster. I could modify the metastore to update with the new IP everytime I bring up a cluster. But the easier and simpler solution was to just use an elastic IP for the master

所以要将metastore中的之前出现的namenode地址全部更换为现有的namenode地址

10：Your DataNode is started and you can create directories with bin/hadoop dfs -mkdir, but you get an error message when you try to put files into the HDFS (e.g., when you run a command like bin/hadoop dfs -put).

解决方法：

Go to the HDFS info web page (open your web browser and go to http://namenode:dfs_info_port where namenode is the hostname of your NameNode and dfs_info_port is the port you chose dfs.info.port; if followed the QuickStart on your personal computer then this URL will be http://localhost:50070). Once at that page click on the number where it tells you how many DataNodes you have to look at a list of the DataNodes in your cluster.

If it says you have used 100% of your space, then you need to free up room on local disk(s) of the DataNode(s).

If you are on Windows then this number will not be accurate (there is some kind of bug either in Cygwin’s df.exe or in Windows). Just free up some more space and you should be okay. On one Windows machine we tried the disk had 1GB free but Hadoop reported that it was 100% full. Then we freed up another 1GB and then it said that the disk was 99.15% full and started writing data into the HDFS again. We encountered this bug on Windows XP SP2.

11：Your DataNodes won’t start, and you see something like this in logs/*datanode*:

Incompatible namespaceIDs in /tmp/hadoop-ross/dfs/data

原因：

Your Hadoop namespaceID became corrupted. Unfortunately the easiest thing to do reformat the HDFS.

解决方法：

You need to do something like this:

bin/stop-all.sh

rm -Rf /tmp/hadoop-your-username/*

bin/hadoop namenode -format

12：You can run Hadoop jobs written in Java (like the grep example), but your HadoopStreaming jobs (such as the Python example that fetches web page titles) won’t work.

原因：

You might have given only a relative path to the mapper and reducer programs. The tutorial originally just specified relative paths, but absolute paths are required if you are running in a real cluster.

解决方法：

Use absolute paths like this from the tutorial:

bin/hadoop jar contrib/hadoop-0.15.2-streaming.jar /

-mapper  $HOME/proj/hadoop/multifetch.py         /

-reducer $HOME/proj/hadoop/reducer.py            /

-input   urls/*                                  /

-output  titles

13： 2009-01-08 10:02:40,709 ERROR metadata.Hive (Hive.java:getPartitions(499)) - javax.jdo.JDODataStoreException: Required table missing : ““PARTITIONS”” in Catalog “” Schema “”. JPOX requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable “org.jpox.autoCreateTables”

原因：就是因为在 hive-default.xml 里把 org.jpox.fixedDatastore 设置成 true 了

starting namenode, logging to /home/hadoop/HadoopInstall/hadoop/bin/…/logs/hadoop-hadoop-namenode-hadoop.out

localhost: starting datanode, logging to /home/hadoop/HadoopInstall/hadoop/bin/…/logs/hadoop-hadoop-datanode-hadoop.out

localhost: starting secondarynamenode, logging to /home/hadoop/HadoopInstall/hadoop/bin/…/logs/hadoop-hadoop-secondarynamenode-hadoop.out

localhost: Exception in thread “main” java.lang.NullPointerException

localhost:      at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:130)

localhost:      at org.apache.hadoop.dfs.NameNode.getAddress(NameNode.java:116)

localhost:      at org.apache.hadoop.dfs.NameNode.getAddress(NameNode.java:120)

localhost:      at org.apache.hadoop.dfs.SecondaryNameNode.initialize(SecondaryNameNode.java:124)

localhost:      at org.apache.hadoop.dfs.SecondaryNameNode.(SecondaryNameNode.java:108)

localhost:      at org.apache.hadoop.dfs.SecondaryNameNode.main(SecondaryNameNode.java:460)

14：09/08/31 18:25:45 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:Bad connect ack with firstBadLink 192.168.1.11:50010

> 09/08/31 18:25:45 INFO hdfs.DFSClient: Abandoning block blk_-8575812198227241296_1001

> 09/08/31 18:25:51 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:

Bad connect ack with firstBadLink 192.168.1.16:50010

> 09/08/31 18:25:51 INFO hdfs.DFSClient: Abandoning block blk_-2932256218448902464_1001

> 09/08/31 18:25:57 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:

Bad connect ack with firstBadLink 192.168.1.11:50010

> 09/08/31 18:25:57 INFO hdfs.DFSClient: Abandoning block blk_-1014449966480421244_1001

> 09/08/31 18:26:03 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException:

Bad connect ack with firstBadLink 192.168.1.16:50010

> 09/08/31 18:26:03 INFO hdfs.DFSClient: Abandoning block blk_7193173823538206978_1001

> 09/08/31 18:26:09 WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable

to create new block.

>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2731)

>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1996)

>         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2182)

>

> 09/08/31 18:26:09 WARN hdfs.DFSClient: Error Recovery for block blk_7193173823538206978_1001

bad datanode[2] nodes == null

> 09/08/31 18:26:09 WARN hdfs.DFSClient: Could not get block locations. Source file “/user/umer/8GB_input”

- Aborting…

> put: Bad connect ack with firstBadLink 192.168.1.16:50010

解决方法：

I have resolved the issue:

What i did:

1. ‘/etc/init.d/iptables stop’ -->stopped firewall

2. SELINUX=disabled in ‘/etc/selinux/config’ file.–>disabled selinux

I worked for me after these two changes

解决jline.ConsoleReader.readLine在Windows上不生效问题方法

在 CliDriver.java的main()函数中，有一条语句reader.readLine，用来读取标准输入，但在Windows平台上该语句总是 返回null，这个reader是一个实例jline.ConsoleReader实例，给Windows Eclipse调试带来不便。

我们可以通过使用java.util.Scanner.Scanner来替代它，将原来的

while ((line=reader.readLine(curPrompt+"> ")) != null)

复制代码

替换为：

Scanner sc = new Scanner(System.in);

while ((line=sc.nextLine()) != null)

复制代码

重新编译发布，即可正常从标准输入读取输入的SQL语句了。

Windows eclispe调试hive报does not have a scheme错误可能原因

1、Hive配置文件中的“hive.metastore.local”配置项值为false，需要将它修改为true，因为是单机版

2、没有设置HIVE_HOME环境变量，或设置错误

3、 “does not have a scheme”很可能是因为找不到“hive-default.xml”。使用Eclipse调试Hive时，遇到找不到hive- default.xml的解决方法：http://bbs.hadoopor.com/thread-292-1-1.html

1、中文问题

从url中解析出中文,但hadoop中打印出来仍是乱码?我们曾经以为hadoop是不支持中文的，后来经过查看源代码，发现hadoop仅仅是不支持以gbk格式输出中文而己。

这是TextOutputFormat.class中的代码，hadoop默认的输出都是继承自FileOutputFormat来 的，FileOutputFormat的两个子类一个是基于二进制流的输出，一个就是基于文本的输出TextOutputFormat。

public class TextOutputFormat<K, V> extends FileOutputFormat<K, V> {

protected static class LineRecordWriter<K, V>

implements RecordWriter<K, V> {

private static final String utf8 = “UTF-8″;//这里被写死成了utf-8

private static final byte[] newline;

static {

try {

newline = “/n”.getBytes(utf8);

} catch (UnsupportedEncodingException uee) {

throw new IllegalArgumentException(”can’t find ” + utf8 + ” encoding”);

}

}

…

public LineRecordWriter(DataOutputStream out, String keyValueSeparator) {

this.out = out;

try {

this.keyValueSeparator = keyValueSeparator.getBytes(utf8);

} catch (UnsupportedEncodingException uee) {

throw new IllegalArgumentException(”can’t find ” + utf8 + ” encoding”);

}

}

…

private void writeObject(Object o) throws IOException {

if (o instanceof Text) {

Text to = (Text) o;

out.write(to.getBytes(), 0, to.getLength());//这里也需要修改

最后

学习视频：

大厂面试真题：

gException uee) {

throw new IllegalArgumentException(”can’t find ” + utf8 + ” encoding”);

}

}

…

public LineRecordWriter(DataOutputStream out, String keyValueSeparator) {

this.out = out;

try {

this.keyValueSeparator = keyValueSeparator.getBytes(utf8);

} catch (UnsupportedEncodingException uee) {

throw new IllegalArgumentException(”can’t find ” + utf8 + ” encoding”);

}

}

…

private void writeObject(Object o) throws IOException {

if (o instanceof Text) {

Text to = (Text) o;

out.write(to.getBytes(), 0, to.getLength());//这里也需要修改

最后

学习视频：

[外链图片转存中…(img-cYWp7Skh-1716575217984)]

大厂面试真题：

[外链图片转存中…(img-wrb1YYbQ-1716575217985)]