前几天线上Sqoop的一个Job(从MySQL抽取数据到Hadoop)突然报了OOME,后重跑并做java trace发现内存占用都是byte[],同时cpu top 3的方法都是com.mysql.jdbc.ByteArraryBuffer.getBytes即内存都是被数据消耗掉了;很奇怪,为什么在option里指定了fetch-size=100怎么会OOME呢(平均记录长度不到1kb);
再看昨天成功的发现100W条记录,发现占用了860MB内存,明显是fetch-size没有生效
+---------+---------+------------+----------+-------------+--------------+
| type | status | host | cpusec | mrinput_rec | memory_mb |
+---------+---------+------------+----------+-------------+--------------+
| CLEANUP | SUCCESS | A | 0.3400 | NULL | 191.84765625 |
| MAP | SUCCESS | A | 335.6400 | 1006942 | 862.39843750 |
| SETUP | SUCCESS | B | 0.2000 | NULL | 179.34765625 |
+---------+---------+------------+----------+-------------+--------------+
没办法,把sqoop源码翻出来终于发现RC了:fetchsize被忽略掉了
protected void initOptionDefaults() {
if (options.getFetchSize() == null) {
LOG.info("Preparing to use a MySQL streaming resultset.");
options.setFetchSize(Integer.MIN_VALUE);
} else if (
!options.getFetchSize().equals(Integer.MIN_VALUE)
&& !options.getFetchSize().equals(0)) {
LOG.info("Argument '--fetch-size " + options.getFetchSize()
+ "' will probably get ignored by MySQL JDBC driver.");
}
}
究其原因是MySQL提供的API只支持row-by-row和all模式:
By default, ResultSets are completely retrieved and stored in memory. In most cases this is the most efficient way to operate, and due to the design of the MySQL network protocol is easier to implement. If you are working with ResultSets that have a large number of rows or large values, and cannot allocate heap space in your JVM for the memory required, you can tell the driver to stream the results back one row at a time.
http://dev.mysql.com/doc/refman/5.5/en/connector-j-reference-implementation-notes.html
最后把fetchsize给去掉了,Job执行成功,700W行占用内存400MB;
+---------+---------+------------+----------+-------------+--------------+
| type | status | host | cpusec | mrinput_rec | memory_mb |
+---------+---------+------------+----------+-------------+--------------+
| CLEANUP | SUCCESS | A | 0.4200 | NULL | 183.49218750 |
| MAP | FAILED | B | NULL | NULL | NULL |
| MAP | SUCCESS | A | 377.1200 | 7195560 | 408.08593750 |
| SETUP | SUCCESS | C | 0.2900 | NULL | 188.64843750 |
+---------+---------+------------+----------+-------------+--------------+
[@more@]来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/11676357/viewspace-1060920/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/11676357/viewspace-1060920/