五种 SQL on Hadoop 的全面分析，星环让去IOE再非空谈(哈哈)

Hadoop的诞生是划时代的数据变革，但关系数据库时代的存留也为Hadoop真正占领数据库领域埋下了许多的障碍。对SQL(尤其是PL/SQL)的支持一直是Hadoop大数据平台在接手旧数据时代时亟待解决的问题。Hadoop对SQL数据库的支持度一直是企业用户最关心的诉求点之一，也是他们选择的Hadoop平台的重要标准。

　　本次的中国Hadoop技术峰会在SQL on Hadoop领域设有专门的分论坛，表示出了主办方及业界对这一问题的重视。幸运的是，在七月二十三日下午的分论坛上，我们也的确看到了当前中国Hadoop技术在这方面已取得了突破性的进展。

　　此次的分论坛，Facebook、惠普、MapR、星环科技等多家业内巨头在SQL on Hadoop领域进行业内的分享与比拼，这也是这些公司首次在SQL领域的研究上同台竞技。

　　当日下午，星环科技首席工程师陈振强博士题为《 A comparison study of SQL on Hadoop engines》对当前五种SQL on Hadoop引擎进行了性能与兼容性各方面的比较，一时间高下立判，引来众多关注。

　　陈博士比较的五种引擎分别是Cloudera的Impala、Hontonworks的Tez、星环科技的Inceptor、Spark SQL和Pivotal的Greenplum。

　　首先在对SQL的支持上，Inceptor和Greenplum对于SQL的支持都超过了其余三种引擎，星环的Inceptor引擎则是达到了SQL2003和PL/SQL的全面兼容。

　　随后他又展示了在TPC-DS系统测试基准的99个测试案例中，星环Inceptor与Pivotal Greenplum理论通过数都是99个，处于领先地位。但在实际运行中，Inceptor全数通过，Pivotal Greenplum则有4个出错，实际通过数为95个。星环Inceptor引擎改进了内存计算引擎Spark，内存的使用更加合理，所以稳定性高，TPC-DS 99个SQL的连续测试中都能够正确运行，而不会象其它引擎一样出现Out of memory的情况。

　　在性能优化方面，星环Inceptor引擎实现了CBO(基于代价的优化)，并为之添加了几百种优化规则，如Join顺序优化、过滤条件前置等等，通过这些优化规则Inceptor引擎可以得到非常高率的逻辑执行计划。陈博士以TPC-DS中的Query3为例介绍了CBO的优化效果，Query3的原始的Join顺序为date_dim(小表)，store_sales(大表)，item(小表)。store_sales与date_dim的过滤率为1/6左右，store_sales与item的过滤率为1/1000左右。通过CBO优化后的Join顺序为store_sales， item，date_dim，比没有CBO优化性能提升5倍以上。

　　星环Inceptor引擎还实现了代码生成器(Code Generator)，其中实现了Constant folding、loop unrolling、loop invariant host、CSE等传统编译器中的优化方法，使得表达式的计算更加高效。在TPC-DS测试中，Inceptor引擎表达式的计算性能与基于C++实现的Impala引擎相当，有些SQL性能甚至更好。

　　此外，星环Inceptor引擎基于Spark的DAG进行了减少shuffle的优化，使得计算的stage更少、效率更高。

　　正是有了如此多的优化，在TPC-DS基准测试中，星环Inceptor引擎的性能比类MPP引擎Impala提升2.7倍，比Map/Reduce改进引擎Tez提升4.7倍，比开源的SparkSQL提升5.5倍，比MPP引擎GP提升3.3倍。

领军去IOE时代，自主研发先行者

 

height="250" marginheight="0" border="0" src="http://g.163.com/r?site=netease&affiliate=news&cat=article&type=logo300x250&location=12" frameborder="no" width="300" marginwidth="0" scrolling="no">

　　总结来说，在SQL支持上，星环Inceptor遥遥领先，而计算引擎框架渐渐趋同，优化效果各有不同。最后，陈振强博士还提到，国内政策目前对中国IT行业去IOE有着迫切的要求，但在技术和引擎上的弱势或者极差的兼容性都将对去IOE造成阻碍。而星环科技自主研发Inceptor，在兼容效果和性能表现上的优异都可与国外同行比肩，可以说，为国内IT业去IOE的自主化研发带来的希望与成功的曙光。

http://bjdx.wuhunews.cn/jksh/10177411871.shtml

http://bjdx.wuhunews.cn/gzjk/10177411102.shtml

http://bjdx.wuhunews.cn/rdgz/10177411392.shtml

http://bjdx.wuhunews.cn/shzt/10177411504.shtml

http://bjdx.wuhunews.cn/jjcs/10177410496.shtml

http://bjdx.wuhunews.cn/jkbk/10177410623.shtml

http://bjdx.wuhunews.cn/shys/10177410878.shtml

http://bjdx.wuhunews.cn/ydys/10177410183.shtml

http://bjdx.wuhunews.cn/ysjk/10177409918.shtml

http://bjdx.wuhunews.cn/sjys/10177409507.shtml

http://bjdx.wuhunews.cn/jkzl/10177408871.shtml

http://bjdx.wuhunews.cn/jkkd/10177409027.shtml

http://bjdx.wuhunews.cn/shcs/10177409229.shtml

http://bjdx.wuhunews.cn/jkxcs/10177408529.shtml

http://bjdx.wuhunews.cn/shkt/10177408316.shtml

http://bjdx.wuhunews.cn/jkys/10177408092.shtml

http://bjdx.wuhunews.cn/shzt/10177407646.shtml

http://bjdx.wuhunews.cn/jksh/10177407851.shtml

http://bjdx.wuhunews.cn/rdgz/10177407323.shtml

http://bjdx.wuhunews.cn/gzjk/10177407111.shtml

http://bjdx.wuhunews.cn/shys/10177406827.shtml

http://bjdx.wuhunews.cn/ysjk/10177406102.shtml

http://bjdx.wuhunews.cn/ydys/10177406327.shtml

http://bjdx.wuhunews.cn/jjcs/10177406567.shtml

http://bjdx.wuhunews.cn/shcs/10177405638.shtml

http://bjdx.wuhunews.cn/jkkd/10177405475.shtml

http://bjdx.wuhunews.cn/sjys/10177405804.shtml

http://bjdx.wuhunews.cn/jkzl/10177405198.shtml

http://bjdx.wuhunews.cn/jkxcs/10177404721.shtml

http://bjdx.wuhunews.cn/shkt/10177404571.shtml

http://bjdx.wuhunews.cn/jkys/10177404229.shtml

http://bjdx.wuhunews.cn/jksh/10177404027.shtml

http://bjdx.wuhunews.cn/shys/10177403196.shtml

http://bjdx.wuhunews.cn/gzjk/10177403382.shtml

http://bjdx.wuhunews.cn/rdgz/10177403568.shtml

http://bjdx.wuhunews.cn/shzt/10177403742.shtml

http://bjdx.wuhunews.cn/jjcs/10177402623.shtml

http://bjdx.wuhunews.cn/ydys/10177402482.shtml

http://bjdx.wuhunews.cn/ysjk/10177402296.shtml

http://bjdx.wuhunews.cn/jkbk/10177402829.shtml

http://bjdx.wuhunews.cn/sjys/10177401608.shtml

http://bjdx.wuhunews.cn/jkzl/10177400977.shtml

http://bjdx.wuhunews.cn/jkkd/10177401102.shtml

http://bjdx.wuhunews.cn/shcs/10177401359.shtml

http://bjdx.wuhunews.cn/jkxcs/10177400681.shtml

http://bjdx.wuhunews.cn/shkt/10177400421.shtml

http://bjdx.wuhunews.cn/jkys/10177400157.shtml

http://bjdx.wuhunews.cn/shzt/10177399775.shtml

http://bjdx.wuhunews.cn/jksh/10177399982.shtml

http://bjdx.wuhunews.cn/rdgz/10177399432.shtml

http://bjdx.wuhunews.cn/gzjk/10177399255.shtml

http://bjdx.wuhunews.cn/shys/10177398955.shtml

http://bjdx.wuhunews.cn/jkbk/10177398662.shtml

http://bjdx.wuhunews.cn/ydys/10177398152.shtml

http://bjdx.wuhunews.cn/jjcs/10177398373.shtml

http://bjdx.wuhunews.cn/ysjk/10177397887.shtml

http://bjdx.wuhunews.cn/shcs/10177396453.shtml

http://bjdx.wuhunews.cn/sjys/10177396604.shtml

http://bjdx.wuhunews.cn/jkkd/10177395567.shtml

http://bjdx.wuhunews.cn/jkxcs/10177395052.shtml

http://bjdx.wuhunews.cn/jkzl/10177395212.shtml

http://bjdx.wuhunews.cn/jkys/10177394478.shtml

http://bjdx.wuhunews.cn/shkt/10177394781.shtml

http://bjdx.wuhunews.cn/gzjk/10177393630.shtml

http://bjdx.wuhunews.cn/rdgz/10177393852.shtml

http://bjdx.wuhunews.cn/shzt/10177394002.shtml

http://bjdx.wuhunews.cn/jksh/10177394251.shtml

http://bjdx.wuhunews.cn/ysjk/10177392722.shtml

http://bjdx.wuhunews.cn/jjcs/10177393127.shtml

http://bjdx.wuhunews.cn/ydys/10177392971.shtml

http://bjdx.wuhunews.cn/shys/10177393359.shtml

http://bjdx.wuhunews.cn/sjys/10177392402.shtml

http://bjdx.wuhunews.cn/jkzl/10177391723.shtml

http://bjdx.wuhunews.cn/jkxcs/10177391521.shtml

http://bjdx.wuhunews.cn/jkkd/10177391957.shtml

http://bjdx.wuhunews.cn/shcs/10177392159.shtml

http://bjdx.wuhunews.cn/shkt/10177391271.shtml

http://bjdx.wuhunews.cn/jkys/10177390907.shtml

http://bjdx.wuhunews.cn/shzt/10177390508.shtml

http://bjdx.wuhunews.cn/rdgz/10177390341.shtml

http://bjdx.wuhunews.cn/jksh/10177390721.shtml

http://bjdx.wuhunews.cn/jkbk/10177389478.shtml

http://bjdx.wuhunews.cn/shys/10177389702.shtml

http://bjdx.wuhunews.cn/gzjk/10177389959.shtml

http://bjdx.wuhunews.cn/ydys/10177389043.shtml

http://bjdx.wuhunews.cn/jjcs/10177389229.shtml

http://bjdx.wuhunews.cn/ysjk/10177388821.shtml

http://bjdx.wuhunews.cn/sjys/10177388336.shtml

http://bjdx.wuhunews.cn/shcs/10177388004.shtml

http://bjdx.wuhunews.cn/jkkd/10177387866.shtml

http://bjdx.wuhunews.cn/jkxcs/10177387211.shtml

http://bjdx.wuhunews.cn/jkzl/10177387402.shtml

http://bjdx.wuhunews.cn/jkys/10177386776.shtml

http://bjdx.wuhunews.cn/shzt/10177386278.shtml

http://bjdx.wuhunews.cn/jksh/10177386585.shtml

http://bjdx.wuhunews.cn/shkt/10177386904.shtml

http://bjdx.wuhunews.cn/rdgz/10177386029.shtml

http://bjdx.wuhunews.cn/gzjk/10177385866.shtml

http://bjdx.wuhunews.cn/shys/10177385652.shtml

http://bjdx.wuhunews.cn/jjcs/10177385121.shtml

http://bjdx.wuhunews.cn/ydys/10177384936.shtml

http://bjdx.wuhunews.cn/jkbk/10177385348.shtml

http://bjdx.wuhunews.cn/ysjk/10177384662.shtml

http://bjdx.wuhunews.cn/jkkd/10177383947.shtml

http://bjdx.wuhunews.cn/shcs/10177384138.shtml

http://bjdx.wuhunews.cn/sjys/10177384387.shtml

http://bjdx.wuhunews.cn/jkys/10177383022.shtml

http://bjdx.wuhunews.cn/jkzl/10177383627.shtml

http://bjdx.wuhunews.cn/jkxcs/10177383476.shtml

http://bjdx.wuhunews.cn/shkt/10177383227.shtml

http://bjdx.wuhunews.cn/jksh/10177382871.shtml

http://bjdx.wuhunews.cn/gzjk/10177382136.shtml

http://bjdx.wuhunews.cn/rdgz/10177382357.shtml

http://bjdx.wuhunews.cn/shzt/10177382507.shtml

http://bjdx.wuhunews.cn/jkbk/10177381671.shtml

http://bjdx.wuhunews.cn/shys/10177381821.shtml

http://bjdx.wuhunews.cn/jjcs/10177380007.shtml

http://bjdx.wuhunews.cn/ydys/10177378462.shtml

http://bjdx.wuhunews.cn/ysjk/10177378096.shtml

http://bjdx.wuhunews.cn/jkzl/10177377166.shtml

http://bjdx.wuhunews.cn/shcs/10177377582.shtml

http://bjdx.wuhunews.cn/jkkd/10177377316.shtml

http://bjdx.wuhunews.cn/sjys/10177377737.shtml

http://bjdx.wuhunews.cn/jkys/10177376416.shtml

http://bjdx.wuhunews.cn/jkxcs/10177376859.shtml

http://bjdx.wuhunews.cn/shkt/10177376627.shtml

http://bjdx.wuhunews.cn/shzt/10177375918.shtml

http://bjdx.wuhunews.cn/rdgz/10177375713.shtml

http://bjdx.wuhunews.cn/jksh/10177376167.shtml

http://bjdx.wuhunews.cn/jkbk/10177374947.shtml

http://bjdx.wuhunews.cn/shys/10177375138.shtml

http://bjdx.wuhunews.cn/gzjk/10177375312.shtml

http://bjdx.wuhunews.cn/ysjk/10177374287.shtml

http://bjdx.wuhunews.cn/ydys/10177374473.shtml

http://bjdx.wuhunews.cn/jjcs/10177374659.shtml

http://bjdx.wuhunews.cn/sjys/10177373992.shtml

http://bjdx.wuhunews.cn/jkxcs/10177373091.shtml

http://bjdx.wuhunews.cn/jkzl/10177373206.shtml

http://bjdx.wuhunews.cn/jkkd/10177373436.shtml

http://bjdx.wuhunews.cn/shcs/10177373627.shtml

http://bjdx.wuhunews.cn/jkys/10177372538.shtml

http://bjdx.wuhunews.cn/jksh/10177372330.shtml

http://bjdx.wuhunews.cn/shkt/10177372729.shtml

http://bjdx.wuhunews.cn/gzjk/10177371646.shtml

http://bjdx.wuhunews.cn/shys/10177371496.shtml

http://bjdx.wuhunews.cn/rdgz/10177371876.shtml

http://bjdx.wuhunews.cn/shzt/10177372053.shtml

http://bjdx.wuhunews.cn/jjcs/10177370981.shtml

http://bjdx.wuhunews.cn/ydys/10177370768.shtml

http://bjdx.wuhunews.cn/jkbk/10177371167.shtml

http://bjdx.wuhunews.cn/shcs/10177369966.shtml

http://bjdx.wuhunews.cn/sjys/10177370196.shtml

http://bjdx.wuhunews.cn/jkys/10177368877.shtml

http://bjdx.wuhunews.cn/jkkd/10177369659.shtml

http://bjdx.wuhunews.cn/jkzl/10177369492.shtml

http://bjdx.wuhunews.cn/jkxcs/10177369238.shtml

http://bjdx.wuhunews.cn/shkt/10177369008.shtml

http://bjdx.wuhunews.cn/shzt/10177368398.shtml

http://bjdx.wuhunews.cn/rdgz/10177368132.shtml

http://bjdx.wuhunews.cn/shys/10177367716.shtml

http://bjdx.wuhunews.cn/gzjk/10177367966.shtml

http://bjdx.wuhunews.cn/jksh/10177368512.shtml

http://bjdx.wuhunews.cn/jkbk/10177367404.shtml

http://bjdx.wuhunews.cn/jjcs/10177367213.shtml

http://bjdx.wuhunews.cn/ysjk/10177367030.shtml

http://bjdx.wuhunews.cn/shcs/10177366516.shtml

http://bjdx.wuhunews.cn/sjys/10177366737.shtml

http://bjdx.wuhunews.cn/jkys/10177365460.shtml

http://bjdx.wuhunews.cn/jkkd/10177366287.shtml

http://bjdx.wuhunews.cn/jkxcs/10177365876.shtml

http://bjdx.wuhunews.cn/jkzl/10177366098.shtml

http://bjdx.wuhunews.cn/jksh/10177365247.shtml

http://bjdx.wuhunews.cn/shkt/10177365662.shtml

http://bjdx.wuhunews.cn/jkbk/10177364122.shtml

http://bjdx.wuhunews.cn/gzjk/10177364543.shtml