大数据-玩转数据-MaxCompute SQL

已于 2022-04-17 17:00:13 修改
阅读量5.7k 收藏 22
点赞数 2
文章标签： 阿里云 sql 数据仓库 etl工程师 数据库开发
于 2022-04-17 16:47:57 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/s_unbo/article/details/124087448
版权

一、说明

本文为您介绍MaxCompute SQL常见使用场景，让您快速掌握SQL的写法。

二、准备数据集

本文以emp表和dept表为示例数据集。您可以自行在MaxCompute项目上创建表并上传数据。数据导入请参见概述。
下载emp表数据文件和dept表数据文件。
创建emp表。

CREATE TABLE IF NOT EXISTS emp (
  EMPNO STRING,
  ENAME STRING,
  JOB STRING,
  MGR BIGINT,
  HIREDATE DATETIME,
  SAL DOUBLE,
  COMM DOUBLE,
  DEPTNO BIGINT);

创建dept表。

CREATE TABLE IF NOT EXISTS dept (
  DEPTNO BIGINT,
  DNAME STRING,
  LOC STRING);

三、SQL示例

示例1：查询员工人数大于零的所有部门。
为了避免数据量太大，此场景下建议您使用JOIN子句。

SELECT d.*
FROM dept d
JOIN (
    SELECT DISTINCT deptno AS no
    FROM emp
) e
ON d.deptno = e.no;

示例2： 查询薪金比SMITH高的所有员工。
此场景为MAPJOIN的典型场景。

SELECT /*+ MapJoin(a) */ e.empno
    , e.ename
    , e.sal
FROM emp e
JOIN (
    SELECT MAX(sal) AS sal
    FROM `emp`
    WHERE `ENAME` = 'SMITH'
) a
ON e.sal > a.sal;

MAPJION会把小表全部加载到内存中，在map阶段直接拿另外一个表的数据和内存中表数据做匹配，由于在map端是进行了join操作，省去了reduce运行的时间，算是一种优化
前提b表是一张小表，默认25m的表是小表

注意事项
使用mapjoin时，一次性加载到内存中的表最多是8张，如果超过8张小表，应该嵌套一层子循环，将多余的表在外层中写入mapjion里面，代码实现如下：

select /*+ mapjoin(b1,b2)*/
b.*
from
(select /*+ mapjoin(a1,a2,a3,a4,a5,a6,a7,a8)*/
 a.*
from a
join a1 on a.id1=a1.id1
join a2 on a.id2=a2.id2
join a3 on a.id3=a3.id3
join a4 on a.id4=a4.id4
join a5 on a.id5=a5.id5
join a6 on a.id6=a6.id6
join a7 on a.id7=a7.id7
join a8 on a.id8=a8.id8) b
join b1 on b.id11=b1.id11
join b2 on b.id11=b2.id12

示例3： 查询所有员工的姓名及其直接上级的姓名。
此场景为等值连接。

SELECT a.ename
    , b.ename
FROM emp a
LEFT OUTER JOIN emp b
ON b.empno = a.mgr;

示例4：查询基本薪金大于1500的所有工作。
此场景下需要使用HAVING子句。

SELECT emp.`JOB`
    , MIN(emp.sal) AS sal
FROM `emp`
GROUP BY emp.`JOB`
HAVING MIN(emp.sal) > 1500;

示例5： 查询在每个部门工作的员工数量、平均工资和平均服务期限。
此场景为使用内建函数的典型场景。

SELECT COUNT(empno) AS cnt_emp
    , ROUND(AVG(sal), 2) AS avg_sal
    , ROUND(AVG(datediff(getdate(), hiredate, 'dd')), 2) AS avg_hire
FROM `emp`
GROUP BY `DEPTNO`;

示例6： 查询每个部门的薪水前3名的人员的姓名以及其排序。
此场景为典型的Top N场景。

SELECT *
FROM (
  SELECT deptno
    , ename
    , sal
    , ROW_NUMBER() OVER (PARTITION BY deptno ORDER BY sal DESC) AS nums
  FROM emp
) emp1
WHERE emp1.nums < 4;

**示例7：**查询每个部门的人数以及该部门中办事员（CLERK）人数的占比。

SELECT deptno
    , COUNT(empno) AS cnt
    , ROUND(SUM(CASE 
      WHEN job = 'CLERK' THEN 1
      ELSE 0
    END) / COUNT(empno), 2) AS rate
FROM `EMP`
GROUP BY deptno;

四、注意事项

  • 使用GROUP BY时，SELECT部分必须是分组项或聚合函数。
  • ORDER BY后面必须加LIMIT N。
  • SELECT表达式中不能用子查询，可以改写为JOIN。
  • JOIN不支持笛卡尔积，可以使用MAPJOIN替代。
  • UNION All需要改成子查询的格式。
  • IN/NOT IN语句对应的子查询只能有一列，而且返回的行数不能超过1000，否则也需要改成JOIN操作执行。

五、与标准SQL的主要区别及解决方法

MaxCompute SQL与标准SQL的基本区别
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述
DDL与DML的区别及解决方法
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

六、不兼容SQL重写

背景信息
MaxCompute 2.0完全拥抱开源生态，支持更多的语言功能，拥有更快的运行速度。但是MaxCompute 2.0会执行更严格的语法检测，一些在旧版本编译器下正常执行的不严谨的语法在MaxCompute 2.0下执行会报错。

group.by.with.star
说明：即select * …group by…语句。
MaxCompute 2.0版本中，要求Group By列表是源表中所有的列，否则执行报错。
旧版MaxCompute中，即使Group By列表不覆盖源表中所有的列，也支持select * from group by key语法。
示例
场景1：Group By Key中不包含所有列。
错误写法
select * from t group by key;
报错信息
FAILED: ODPS-0130071:[1,8] Semantic analysis exception - column reference t.value should appear in GROUP BY key
正确写法
select distinct key from t;
场景2：group by key包含所有列。
如下写法不推荐。
select * from t group by key, value; -- t has columns key and value
虽然MaxCompute2.0不会报错，但推荐改为如下。
select distinct key, value from t;
bad.escape
说明：错误的escape序列问题。

按照MaxCompute规定，在String literal中应该用反斜线加三位8进制数字表示从0到127的ASCII字符。例如：使用“\001”、“\002”表示0、1。但\01、\0001也被当作\001处理了。

这种方式会给新用户带来困扰，比如需要用“\0001”表示“\000”+“1”，便没有办法实现。同时对于从其他系统迁移而来的用户而言，会导致正确性错误。
说明 \000后面再加数字，如\0001 - \0009或\00001的写法可能会返回错误。
MaxCompute 2.0会解决此问题，对脚本中错误的序列进行修改。
错误写法
select split(key, "\01"), value like "\0001" from t;
报错信息
FAILED: ODPS-0130161:[1,19] Parse exception - unexpected escape sequence: 01
ODPS-0130161:[1,38] Parse exception - unexpected escape sequence: 0001
正确改法
select split(key, "\001"), value like "\001" from t;
column.repeated.in.creation
说明：如果创建表时列名重复，MaxCompute 2.0将会报错。

示例
错误写法
create table t (a BIGINT, b BIGINT, a BIGINT);
报错信息
FAILED: ODPS-0130071:[1,37] Semantic analysis exception - column repeated in creation: a
正确写法
create table t (a BIGINT, b BIGINT);
string.join.double
说明：写JOIN条件时，等号的左右两边分别是STRING和DOUBLE类型。
旧版MaxCompute会把两边都转成BIGINT类型，会导致严重的精度损失问题，例如：1.1=“1”在连接条件中会被认为是相等的。
MaxCompute 2.0会与Hive兼容转为DOUBLE类型。
示例
不推荐写法
select * from t1 join t2 on t1.double_value = t2.string_value;
Warning信息
WARNING:[1,48]  implicit conversion from STRING to DOUBLE, potential data loss, use CAST function to suppress
推荐改法
select * from t1 join t2 on t.double_value = cast(t2.string_value as double);
window.ref.prev.window.alias
说明：Window Function引用同级select List中的其他Window Function Alias的问题。

示例
如果rn在t1中不存在，错误写法如下。
select row_number() over (partition by c1 order by c1) rn,
row_number() over (partition by c1 order by rn) rn2
from t1;
报错信息
FAILED: ODPS-0130071:[2,45] Semantic analysis exception - column rn cannot be resolved
正确改法
select row_number() over (partition by c1 order by rn) rn2
from
(select c1, row_number() over (partition by c1 order by c1) rn
from t1
) tmp;
select.invalid.token.after.star
说明：select列表里面允许用户使用星号（*）代表选择某张表的全部列，但星号（*）后面不允许加alias，即使星号（*）展开之后只有一列也不允许，新一代编译器将会对类似语法进行报错。

示例
错误写法
select * as alias from table_test;
报错信息
FAILED: ODPS-0130161:[1,10] Parse exception - invalid token 'as'
正确改法
select * from table_test;
agg.having.ref.prev.agg.alias
说明：有Having子句的情况下，select List可以出现前面Aggregate Function Alias的问题。

示例
错误写法
select count(c1) cnt,
sum(c1) / cnt avg
from t1
group by c2
having cnt > 1;
报错信息
FAILED: ODPS-0130071:[2,11] Semantic analysis exception - column cnt cannot be resolved
ODPS-0130071:[2,11] Semantic analysis exception - column reference cnt should appear in GROUP BY key
其中s、cnt在源表t1中都不存在，但因为有Having子句，旧版MaxCompute并未报错，MaxCompute 2.0则会提示column cannot be resolve，并报错。

正确改法
select cnt, s, s/cnt avg
from
(
select count(c1) cnt,
sum(c1) s
from t1
group by c2
having count(c1) > 1
) tmp;
order.by.no.limit
说明：MaxCompute默认order by后需要增加limit限制数量，因为order by是全量排序，没有limit时执行性能较低。

示例
错误写法
select * from (select *
from (select cast(login_user_cnt as int) as uv, '3' as shuzi
from test_login_cnt where type = 'device' and type_name = 'mobile') v
order by v.uv desc) v
order by v.shuzi limit 20;
报错信息
FAILED: ODPS-0130071:[4,1] Semantic analysis exception - ORDER BY must be used with a LIMIT clause
在子查询order by v.uv desc中增加limit。

另外，MaxCompute 1.0对于view的检查不够严格。比如在一个不需要检查limit的Project（odps.sql.validate.orderby.limit=false）中，创建了一个View。
create view table_view as select id from table_view order by id;
若访问此View：

select * from table_view;
MaxCompute 1.0不会报错，而MaxCompute 2.0会报如下错误信息：

FAILED: ODPS-0130071:[1,15] Semantic analysis exception - while resolving view xdj.xdj_view_limit - ORDER BY must be used with a LIMIT clause
generated.column.name.multi.window
说明：使用自动生成的alias的问题。

旧版MaxCompute会为select语句中的每个表达式自动生成一个alias，这个alias会最后显示在Console上。但是，它并不承诺这个alias的生成规则，也不承诺这个alias的生成规则会保持不变，所以不建议用户使用自动生成的alias。

MaxCompute 2.0会对使用自动生成alias的情况给予警告，由于牵涉面较广，暂时无法直接给予禁止。

对于某些情况，MaxCompute的不同版本间生成的alias规则存在已知的变动，但因为已有一些线上作业依赖于此类alias，这些查询在 MaxCompute版本升级或者回滚时可能会失败，存在此问题的用户，请修改您的查询，对于感兴趣的列，显式地指定列的别名。

示例
不推荐写法
select _c0 from (select count(*) from table_name) t;
建议改法：
select c from (select count(*) c from table_name) t;
non.boolean.filter
使用了非BOOLEAN过滤条件的问题。

MaxCompute不允许布尔类型与其他类型之间的隐式转换，但旧版MaxCompute会允许用户在某些情况下使用BIGINT作为过滤条件。MaxCompute 2.0将不再允许，如果您的脚本中存在这样的过滤条件，请及时修改。示例如下：

错误写法：

select id, count(*) from table_name group by id having id;
报错信息：

FAILED: ODPS-0130071:[1,50] Semantic analysis exception - expect a BOOLEAN expression
正确改法：

select id, count(*) from table_name group by id having id <> 0;
post.select.ambiguous
在order by、cluster by、distribute by、sort by等语句中，引用了名字冲突的列的问题。

旧版MaxCompute中，系统会默认选取Select列表中的后一列作为操作对象，MaxCompute 2.0将会进行报错，请及时修改。示例如下：

错误写法：

select a, b as a from t order by a limit 10;
报错信息：

FAILED: ODPS-0130071:[1,34] Semantic analysis exception - a is ambiguous, can be both t.a or null.a
正确改法：

select a as c, b as a from t order by a limit 10;
本次推送修改会包括名字冲突但语义一样的情况，虽然不会出现歧义，但是考虑到这种情况容易导致错误，作为一个警告，希望用户进行修改。

duplicated.partition.column
在query中指定了同名的partition的问题。

旧版MaxCompute在用户指定同名partition key时并未报错，而是后一个的值直接覆盖了前一个，容易产生混乱。MaxCompute2.0将会对此情况进行报错，示例如下：

错误写法一：

insert overwrite table partition (ds = '1', ds = '2'）select ... ;
实际上，在运行时ds = ‘1’被忽略。

正确改法：

insert overwrite table partition (ds = '2'）select ... ;
错误写法二：

create table t (a bigint, ds string) partitioned by (ds string);
正确改法：

create table t (a bigint) partitioned by (ds string);
order.by.col.ambiguous
select list中alias重复，之后的order by子句引用到重复的alias的问题。

错误写法：

select id, id
from table_test 
order by id;
正确改法：

select id, id id2
from table_name 
order by id;
需要去掉重复的alias，order by子句再进行引用。

in.subquery.without.result
colx in subquery没有返回任何结果，则colx在源表中不存在的问题。

错误写法：

select * from table_name
where not_exist_col in (select id from table_name limit 0);
报错信息：

FAILED: ODPS-0130071:[2,7] Semantic analysis exception - column not_exist_col cannot be resolved
ctas.if.not.exists
目标表语法错误问题。

如果目标表已经存在，旧版MaxCompute不会做任何语法检查，MaxCompute 2.0则会做正常的语法检查，这种情况会出现很多错误信息，示例如下：

错误写法：

create table if not exists table_name
as
select * from not_exist_table;
报错信息：

FAILED: ODPS-0130131:[1,50] Table not found - table meta_dev.not_exist_table cannot be resolved
worker.restart.instance.timeout
旧版MaxCompute UDF每输出一条记录，便会触发一次对分布式文件系统的写操作，同时会向Fuxi发送心跳，如果UDF 10分钟没有输出任何结果，会得到如下错误提示：

FAILED: ODPS-0123144: Fuxi job failed - WorkerRestart errCode:252,errMsg:kInstanceMonitorTimeout, usually caused by bad udf performance.
MaxCompute 2.0的Runtime框架支持向量化，一次会处理某一列的多行来提升执行效率。但向量化可能导致原来不会报错的语句（2条记录的输出时间间隔不超过10分钟），因为一次处理多行，没有及时向Fuxi发送心跳而导致超时。

遇到这个错误，建议首先检查UDF是否有性能问题，每条记录需要数秒的处理时间。如果无法优化UDF性能，可以尝试手动设置batch row大小来绕开（默认为1024）：

set odps.sql.executionengine.batch.rowcount=16;
divide.nan.or.overflow
旧版MaxCompute不会做除法常量折叠的问题。

比如如下语句，旧版MaxCompute对应的物理执行计划如下：

explain
select if(false, 0/0, 1.0)
from table_name;
in task M1_Stg1:
    Data source: meta_dev.table_name
    TS: alias: table_name
      SEL: If(False, Divide(UDFToDouble(0), UDFToDouble(0)), 1.0)
        FS: output: None
由此可以看出，IF和Divide函数仍然被保留，运行时因为IF第一个参数为false，第二个参数Divide的表达式不需要求值，所以不会出现除零异常。

而MaxCompute 2.0支持除法常量折叠，所以会报错。如下所示：

错误写法：

select IF(FALSE, 0/0, 1.0)
from table_name;
报错信息：

FAILED: ODPS-0130071:[1,19] Semantic analysis exception - encounter runtime exception while evaluating function /, detailed message: DIVIDE func result NaN, two params are 0.000000 and 0.000000
除了上述的错误，还可能遇到overflow错误，比如：

错误写法：

select if(false, 1/0, 1.0)
from table_name;
报错信息：

FAILED: ODPS-0130071:[1,19] Semantic analysis exception - encounter runtime exception while evaluating function /, detailed message: DIVIDE func result overflow, two params are 1.000000 and 0.000000
正确改法：

建议去掉/0的用法，换成合法常量。

CASE WHEN常量折叠也有类似问题，比如：CASE WHEN TRUE THEN 0 ELSE 0/0，MaxCompute 2.0常量折叠时所有子表达式都会求值，导致除0错误。

CASE WHEN可能涉及更复杂的优化场景，比如：

select case when key = 0 then 0 else 1/key end
from (
select 0 as key from src
union all
select key from src) r;
优化器会将除法下推到子查询中，转换类似于：

M (
select case when 0 = 0 then 0 else 1/0 end c1 from src
UNION ALL
select case when key = 0 then 0 else 1/key end c1 from src) r;
报错信息：

FAILED: ODPS-0130071:[0,0] Semantic analysis exception - physical plan generation failed: java.lang.ArithmeticException: DIVIDE func result overflow, two params are 1.000000 and 0.000000
其中UNION ALL第一个子句常量折叠报错，建议将SQL中的 CASE WHEN挪到子查询中，并去掉无用的CASE WHEN和去掉/0用法：

select c1 end
from (
select 0 c1 end from src
union all
select case when key = 0 then 0 else 1/key end) r;
small.table.exceeds.mem.limit
旧版MaxCompute支持Multi-way Join优化，多个Join如果有相同Join Key，会合并到一个Fuxi Task中执行，比如下面例子中的J4_1_2_3_Stg1：

explain
select t1.*
from t1 join t2 on t1.c1 = t2.c1
join t3 on t1.c1 = t3.c1;
旧版MaxCompute物理执行计划：

In Job job0:
root Tasks: M1_Stg1, M2_Stg1, M3_Stg1
J4_1_2_3_Stg1 depends on: M1_Stg1, M2_Stg1, M3_Stg1

In Task M1_Stg1:
    Data source: meta_dev.t1

In Task M2_Stg1:
    Data source: meta_dev.t2

In Task M3_Stg1:
    Data source: meta_dev.t3

In Task J4_1_2_3_Stg1:
    JOIN: t1 INNER JOIN unknown INNER JOIN unknown
        SEL: t1._col0, t1._col1, t1._col2
            FS: output: None
如果增加MapJoin hint，旧版MaxCompute物理执行计划不会改变。也就是说对于旧版MaxCompute优先应用Multi-way Join优化，并且可以忽略用户指定MapJoin hint。

explain
select /* +mapjoin(t1) */ t1.*
from t1 join t2 on t1.c1 = t2.c1
join t3 on t1.c1 = t3.c1;
旧版MaxCompute物理执行计划同上。

MaxCompute 2.0 Optimizer会优先使用用户指定的MapJoin hint，对于上述例子，如果t1比较大的话，会遇到类似错误：

FAILED: ODPS-0010000:System internal error - SQL Runtime Internal Error: Hash Join Cursor HashJoin_REL… small table exceeds, memory limit(MB) 640, fixed memory used …, variable memory used …
对于这种情况，如果MapJoin不是期望行为，建议去掉MapJoin hint。

sigkill.oom
同small.table.exceeds.mem.limit，如果用户指定了MapJoin hint，并且用户本身所指定的小表比较大。在旧版MaxCompute下有可能被优化成Multi-way Join从而成功。但在MaxCompute 2.0下，用户可能通过设定odps.sql.mapjoin.memory.max来避免小表超限的错误，但每个MaxCompute worker有固定的内存限制，如果小表本身过大，则MaxCompute worker会由于内存超限而被杀掉，错误类似于：

Fuxi job failed - WorkerRestart errCode:9,errMsg:SigKill(OOM), usually caused by OOM(outof memory).
这里建议您去掉MapJoin hint，使用Multi-way Join。

wm_concat.first.argument.const
聚合函数中关于WM_CONCAT的说明，一直要求WM_CONCAT第一个参数为常量，旧版MaxCompute检查不严格，比如源表没有数据，就算WM_CONCAT第一个参数为ColumnReference，也不会报错。

函数声明：
string wm_concat(string separator, string str)
参数说明：
separator：String类型常量，分隔符。其他类型或非常量将引发异常。
MaxCompute 2.0会在plan阶段便检查参数的合法性，假如WM_CONCAT的第一个参数不是常量，会立即报错。示例如下：

错误写法：

select wm_concat(value, ',') FROM src group by value;
报错信息：

FAILED: ODPS-0130071:[0,0] Semantic analysis exception - physical plan generation failed: com.aliyun.odps.lot.cbo.validator.AggregateCallValidator$AggregateCallValidationException: Invalid argument type - The first argument of WM_CONCAT must be constant string.
pt.implicit.convertion.failed
srcpt是一个分区表，并有两个分区：

create table srcpt(key STRING, value STRING) partitioned by (pt STRING);
alter table srcpt add partition (pt='pt1');
alter table srcpt add partition (pt='pt2');
对于以上SQL，String类型pt列，INT类型常量，都会转为DOUBLE进行比较。即使Project设置了odps.sql.udf.strict.mode=true，旧版MaxCompute不会报错，所有pt都会过滤掉，而MaxCompute 2.0会直接报错。示例如下：

错误写法：

select key from srcpt where pt in (1, 2);
报错信息：

FAILED: ODPS-0130071:[0,0] Semantic analysis exception - physical plan generation failed: java.lang.NumberFormatException: ODPS-0123091:Illegal type cast - In function cast, value 'pt1' cannot be casted from String to Double.
建议避免STRING分区列和INT类型常量比较，将INT类型常量改成STRING类型。

having.use.select.alias
SQL规范定义Group by + Having子句是Select子句之前阶段，所以Having中不应该使用Select子句生成的Column alias。

示例
错误写法：
select id id2 from table_name group by id having id2 > 0;
报错信息：
FAILED: ODPS-0130071:[1,44] Semantic analysis exception - column id2 cannot be resolvedODPS-0130071:[1,44] Semantic analysis exception - column reference id2 should appear in GROUP BY key
其中id2为Select子句中新生成的Column alias，不应该在Having子句中使用。

dynamic.pt.to.static
说明：MaxCompute2.0动态分区某些情况会被优化器转换成静态分区处理。

示例
insert overwrite table srcpt partition(pt) select id, 'pt1' from table_name;
会被转化成

insert overwrite table srcpt partition(pt='pt1') select id from table_name;
如果用户指定的分区值不合法，比如错误地使用了’${bizdate}’，MaxCompute 2.0语法检查阶段便会报错。详情请参见分区。

错误写法：

insert overwrite table srcpt partition(pt) select id, '${bizdate}' from table_name limit 0;
报错信息：

FAILED: ODPS-0130071:[1,24] Semantic analysis exception - wrong columns count 2 in data source, requires 3 columns (includes dynamic partitions if any)
旧版MaxCompute因为LIMIT 0，SQL最终没有输出任何数据，动态分区不会创建，所以最终不报错。

lot.not.in.subquery
说明：In subquery中NULL值的处理问题。

在标准SQL的IN 运算中，如果后面的值列表中出现NULL，则返回值不会出现false，只可能是NULL或者true。如1 in (null, 1, 2, 3) 为true，而1 in (null, 2, 3) 为NULL，null in (null, 1, 2, 3) 为NULL。同理not in操作在列表中有NULL的情况下，只会返回false或者NULL，不会出现true。

MaxCompute 2.0会用标准的行为进行处理，收到此提醒的用户请注意检查您的查询，IN操作中的子查询中是否会出现空值，出现空值时行为是否与您预期相符，如果不符合预期请做相应的修改。

示例
select * from t where c not in (select accepted from c_list);
若accepted中不会出现NULL值，则此问题可忽略。若出现空值，则c not in (select accepted from c_list)原先返回true，则新版本返回NULL。

正确写法
select * from t where c not in (select accepted from c_list where accepted is not null)

七、导出SQL的运行结果

您可以通过以下方法导出SQL的运行结果：

  • 如果数据比较少，请使用SQL Task得到全部的查询结果。
  • 如果需要导出某个表或者分区，请使用Tunnel直接导出查询结果。
  • 如果SQL比较复杂，请使用Tunnel和SQL相互配合导出查询结果。

DataWorks 可以方便地帮您运行SQL，同步数据，并支持定时调度，配置任务依赖的功能。
开源工具DataX可以帮助您方便地把MaxCompute中的数据导出到目标数据源，详情请参见DataX概述。

（1） 、 SQLTask方式导出 SQLTask使用SDK方法，直接调用MaxCompute
SQL的接口SQLTask.getResult(i)，可以很方便地运行SQL并获得其返回结果。使用方法请参见SQLTask。

使用SQLTask时，请注意： SQLTask.getResult(i)用于导出SELECT查询结果，不适用于导出show tables;等其他MaxCompute命令操作结果。

SELECT语句返回给客户端的数据条数可以通过READ_TABLE_MAX_ROW进行设置，详情请参见设置项目空间属性。

SELECT语句最多返回1万条数据至客户端。即如果在客户端（包括SQLTask）直接执行SELECT语句，相当于在SELECT语句最后加了Limit N。

如果使用CREATE TABLE XX AS SELECT或者INSERT INTO/OVERWRITE TABLE把结果固化到具体的表里则不受此限制。

（2）、Tunnel方式导出
如果您需要导出的查询结果是某张表的全部内容（或者是具体的某个分区的全部内容），可以通过Tunnel来实现，详情请参见命令行工具 和基于SDK编写的Tunnel SDK。

此处提供一个Tunnel命令行导出数据的简单示例，Tunnel SDK的编写适用于Tunnel命令行无法支持的场景，详情请参见批量数据通道概述。

tunnel d wc_out c:\wc_out.dat;
2016-12-16 19:32:08 - new session: 201612161932082d3c9b0a012f68e7 total lines: 3
2016-12-16 19:32:08 - file [0]: [0, 3), c:\wc_out.dat
downloading 3 records into 1 file
2016-12-16 19:32:08 - file [0] start
2016-12-16 19:32:08 - file [0] OK. total: 21 bytes
download OK

（3）、SQLTask配合Tunnel方式导出
SQLTask不能处理超过1万条数据，而Tunnel方式可以，两者可以互补，因此可以基于两者实现超过1万条数据的导出。

代码实现的示例如下。

private static final String accessId = "userAccessId";
    private static final String accessKey = "userAccessKey";
    private static final String endPoint = "http://service.cn-shanghai.maxcompute.aliyun.com/api";
    private static final String project = "userProject";
    private static final String sql = "userSQL";
    private static final String table = "Tmp_" + UUID.randomUUID().toString().replace("-", "_");//用随机字符串作为临时表的名称。
    private static final Odps odps = getOdps();

    public static void main(String[] args) {
        System.out.println(table);
        runSql();
        tunnel();
    }

    /*
     * 下载SQLTask的结果。
     * */
    private static void tunnel() {
        TableTunnel tunnel = new TableTunnel(odps);
        try {
            DownloadSession downloadSession = tunnel.createDownloadSession(
                    project, table);
            System.out.println("Session Status is : "
                    + downloadSession.getStatus().toString());
            long count = downloadSession.getRecordCount();
            System.out.println("RecordCount is: " + count);
            RecordReader recordReader = downloadSession.openRecordReader(0,
                    count);
            Record record;
            while ((record = recordReader.read()) != null) {
                consumeRecord(record, downloadSession.getSchema());
            }
            recordReader.close();
        } catch (TunnelException e) {
            e.printStackTrace();
        } catch (IOException e1) {
            e1.printStackTrace();
        }
    }

    /*
     * 保存数据。
     * 数据量少时直接打印后拷贝也是可行的。实际场景可以用Java.io写到本地文件，或者写到远端存储上保存起来。
     * */
    private static void consumeRecord(Record record, TableSchema schema) {
        System.out.println(record.getString("username")+","+record.getBigint("cnt"));
    }

    /*
     * 运行SQL，把查询结果保存成临时表。
     * 此处保存数据的生命周期为1天，即使删除步骤出了问题，也不会太浪费存储空间。
     * */
    private static void runSql() {
        Instance i;
        StringBuilder sb = new StringBuilder("Create Table ").append(table)
                .append(" lifecycle 1 as ").append(sql);
        try {
            System.out.println(sb.toString());
            i = SQLTask.run(getOdps(), sb.toString());
            i.waitForSuccess();

        } catch (OdpsException e) {
            e.printStackTrace();
        }
    }

    /*
     * 初始化MaxCompute的连接信息。
     * */
    private static Odps getOdps() {
        Account account = new AliyunAccount(accessId, accessKey);
        Odps odps = new Odps(account);
        odps.setEndpoint(endPoint);
        odps.setDefaultProject(project);
        return odps;
    }

（4）、DataWorks数据同步方式导出
DataWorks支持运行SQL并配置数据同步任务，以完成数据生成和导出需求。

  • 登录DataWorks控制台。
  • 在左侧导航栏，单击工作空间列表。
  • 单击相应工作空间后的进入数据开发。
  • 新建业务流程。
  • 右键单击业务流程，选择新建业务流程
  • 输入业务名称。
  • 单击新建。
  • 创建SQL节点。
  • 右键单击业务流程，选择新建 > MaxCompute > ODPS SQL。
  • 填写节点名称为runsql，单击提交。
  • 配置ODPS SQL节点，配置完成后单击保存。
  • 创建数据同步节点。
  • 右键单击业务流程，选择新建 > 数据集成 > 离线同步。
  • 填写节点名称为sync2mysql，单击提交。
  • 选择数据来源以及去向。
  • 配置字段映射。
  • 配置通道控制。
  • 单击保存。
  • 将数据同步节点和ODPS SQL节点连线配置成依赖关系，ODPS SQL节点作为数据的产出节点，数据同步节点作为数据的导出节点。
  • 工作流调度配置完成后（可以直接使用默认配置），单击运行。数据同步的运行日志，如下所示。
2016-12-17 23:43:46.394 [job-15598025] INFO JobContainer - 
任务启动时刻 : 2016-12-17 23:43:34
任务结束时刻 : 2016-12-17 23:43:46
任务总计耗时 : 11s
任务平均流量 : 31.36KB/s
记录写入速度 : 1668rec/s
读出记录总数 : 16689
读写失败总数 : 0
执行如下SQL语句查看数据同步的结果。
select count(*) from result_in_db;

八、分区剪裁合理性评估

本文为您介绍如何评估分区剪裁合理性。

MaxCompute分区表是指在创建表时指定分区空间，即指定表内的几个字段作为分区列。使用数据时，如果指定了需要访问的分区名称，则只会读取相应的分区，避免全表扫描，提高处理效率，降低费用。

分区剪裁是指对分区列指定过滤条件，使得SQL执行时只用读取表分区的部分数据，避免全表扫描引起的数据错误及资源浪费。但是分区失效的情况会经常发生。

本文将从以下两方面介绍分区剪裁：

  • 判断分区剪裁是否生效。
  • 分区剪裁失效的场景分析。
  • 判断分区剪裁是否生效
  • 通过EXPLAIN命令查看SQL执行计划，用于判断SQL中的分区剪裁是否生效。
    （1）、 分区剪裁未生效。
explain
select seller_id
from xxxxx_trd_slr_ord_1d
where ds=rand();

从执行计划中可见，SQL读取了表的1344个分区，即该表的所有分区。

（2）、分区剪裁生效

explain
select seller_id
from xxxxx_trd_slr_ord_1d
where ds='20150801';

从执行计划中可见，SQL只读取了表的20150801分区。

（3）、分区剪裁失效的场景分析
自定义函数导致分区剪裁失效
当分区剪裁的条件中使用了用户自定义函数（或者部分内建函数）时，分区剪裁失效。所以，对于分区值的限定，如果使用了非常规函数，建议您使用Explain命令查看执行计划，确定分区剪裁是否生效。

explain
select ...
from xxxxx_base2_brd_ind_cw
where ds = concat(SPLIT_PART(bi_week_dim(' ${bdp.system.bizdate}'), ',', 1), SPLIT_PART(bi_week_dim(' ${bdp.system.bizdate}'), ',', 2))

说明 UDF已支持分区裁剪，详情请参见WHERE子句过滤（where_condition）文中的说明。

使用JOIN时分区剪裁失效
在SQL语句中使用JOIN进行关联时：
如果分区剪裁条件放在WHERE子句中，则分区剪裁会生效。
如果分区剪裁条件放在ON子句中，从表的分区剪裁会生效，主表则不会生效。

下面针对三种JOIN具体说明：
LEFT OUTER JOIN
分区剪裁条件均放在ON子句中

set odps.sql.allow.fullscan=true;
explain
select a.seller_id
    ,a.pay_ord_pbt_1d_001
from xxxxx_trd_slr_ord_1d a
left outer join
     xxxxx_seller b
on a.seller_id=b.user_id
and a.ds='20150801'
and b.ds='20150801';

由上图可见，左表进行了全表扫描，只有右表的分区裁剪有效果。

分区剪裁条件均放在WHERE子句中

set odps.sql.allow.fullscan=true;
explain
select a.seller_id
    ,a.pay_ord_pbt_1d_001
from xxxxx_trd_slr_ord_1d a
left outer join
    xxxxx_seller b
on a.seller_id=b.user_id
where a.ds='20150801'
and b.ds='20150801';

由上图可见，两张表的分区裁剪都有效果。

RIGHT OUTER JOIN
与LEFT OUTER JOIN类似，如果分区剪裁条件放在ON子句中则只有RIGHT OUTER JOIN的左表生效。如果分区剪裁条件放在WHERE中，则两张表都会生效。

FULL OUTER JOIN
分区剪裁条件只有都放在WHERE子句中才会生效，放在ON子句中都不会生效。

注意事项
分区剪裁如果失效影响较大，且不容易发现。因此，建议在代码提交时关注分区剪裁失效问题。
自定义函数中使用分区剪裁时，需要修改类或者在SQL语句前设置set odps.sql.udf.ppr.deterministic = true;。详情请参见WHERE子句过滤（where_condition）。

九、分组取出每组数据的前N条

示例数据
目前的数据，如下表所示。

empno	ename	job	sal
7369	SMITH	CLERK	800.0
7876	SMITH	CLERK	1100.0
7900	JAMES	CLERK	950.0
7934	MILLER	CLERK	1300.0
7499	ALLEN	SALESMAN	1600.0
7654	MARTIN	SALESMAN	1250.0
7844	TURNER	SALESMAN	1500.0
7521	WARD	SALESMAN	1250.0

实现方法
您可以通过以下两种方法实现：
取出每条数据的行号，再用 where 语句进行过滤。

SELECT * FROM (
  SELECT empno
  , ename
  , sal
  , job
  , ROW_NUMBER() OVER (PARTITION BY job ORDER BY sal) AS rn
  FROM emp
) tmp
WHERE rn < 10;

使用 UDTF 实现 Split 函数。
详情请参见 MaxCompute 学习计划 中最后的示例。此例可以更迅速地判断当前的序号，如果是已经超过预定的条数（例如10 条），便不再处理，从而提高计算效率。

十、多行数据合并为一行数据

示例数据

class	gender	name
1	M	LiLei
1	F	HanMM
1	M	Jim
1	M	HanMM
2	F	Kate
2	M	Peter

使用示例
示例1：将class相同的names合并为一行，并对names去重。去重操作可通过嵌套子查询实现。

SELECT class, wm_concat(distinct ',', name) FROM students GROUP BY class;

说明 wm_concat是字符拼接函数，详情请参见WM_CONCAT。
输出结果如下。

class	names
1	LiLei,HanMM,Jim
2	Kate,Peter

示例2：统计不同class对应的男女人数。

SELECT 
class
,SUM(CASE WHEN gender = 'M' THEN 1 ELSE 0 END) AS cnt_m
,SUM(CASE WHEN gender = 'F' THEN 1 ELSE 0 END) AS cnt_f
FROM students
GROUP BY class;

输出结果如下。

class	cnt_m	cnt_f
1	2	1
2	1	1

十一、行转列及列转行最佳实践

本段基于示例为您介绍如何使用SQL实现行转列、列转行需求。
将多行数据转换成一行显示，或将一列数据转换成多列显示。

列转行
将一行数据转换成多行显示，或将多列数据转换成一列显示。
示例数据
创建用于实现行转列的源表并插入数据，命令示例如下。

create table rowtocolumn (name string, subject string, result bigint);
insert into table rowtocolumn values 
('张三' , '语文' , 74),
('张三' , '数学' , 83),
('张三' , '物理' , 93),
('李四' , '语文' , 74),
('李四' , '数学' , 84),
('李四' , '物理' , 94);

创建用于实现列转行的源表并插入数据，命令示例如下。

create table columntorow (name string, chinese bigint, mathematics bigint, physics bigint);

insert into table columntorow values 
('张三' , 74, 83, 93),
('李四' , 74, 84, 94);

行转列示例
您可以通过如下两种方法实现行转列：
方法一：使用case when表达式，灵活提取各科目（subject）的值作为单独的列，命令示例如下。

select name as 姓名,
       max(case subject when '语文' then result end) as 语文,
       max(case subject when '数学' then result end) as 数学,
       max(case subject when '物理' then result end) as 物理 
from rowtocolumn 
group by name;

返回结果如下。

+--------+------------+------------+------------+
| 姓名   | 语文      | 数学     | 物理      |
+--------+------------+------------+------------+
| 张三   | 74       | 83       | 93        |
| 李四   | 74       | 84       | 94        |
+--------+------------+------------+------------+

方法二：借助MaxCompute提供的内建函数实现，先基于CONCAT和WM_CONCAT函数合并科目和成绩为一列，然后通过KEYVALUE函数解析科目（subject）的值作为单独的列。命令示例如下。

select name as 姓名,
       keyvalue(subject, '语文') as 语文,
       keyvalue(subject, '数学') as 数学,
       keyvalue(subject, '物理') as 物理
from(
     select name, wm_concat(';',concat(subject,':',result))as subject 
     from rowtocolumn
     group by name);

返回结果如下。
+--------+------------+------------+------------+
| 姓名   | 语文      | 数学      | 物理      |
+--------+------------+------------+------------+
| 张三   | 74       | 83        | 93        |
| 李四   | 74       | 84        | 94        |
+--------+------------+------------+------------+

列转行示例
您可以通过如下两种方法实现列转行：
方法一：使用union all，将各科目（chinese、mathematics、physics）整合为一列，命令示例如下。
–解除order by必须带limit的限制，方便列转行SQL命令对结果按照姓名排序。

set odps.sql.validate.orderby.limit=false;

–列转行SQL。

select name as 姓名, subject as 科目, result as 成绩 
from(
     select name, '语文' as subject, chinese as result from columntorow 
     union all 
     select name, '数学' as subject, mathematics as result from columntorow 
     union all 
     select name, '物理' as subject, physics as result from columntorow) 
order by name;

返回结果如下。

+--------+--------+------------+
| 姓名   | 科目   | 成绩       |
+--------+--------+------------+
| 张三   | 语文   | 74         |
| 张三   | 数学   | 83         |
| 张三   | 物理   | 93         |
| 李四   | 语文   | 74         |
| 李四   | 数学   | 84         |
| 李四   | 物理   | 94         |
+--------+--------+------------+

方法二：借助MaxCompute提供的内建函数实现，先基于CONCAT函数拼接各科目和成绩，然后基于TRANS_ARRAY和SPLIT_PART函数逐层拆解科目和成绩作为单独的列。命令示例如下。

select name as 姓名,
       split_part(subject,':',1) as 科目,
       split_part(subject,':',2) as 成绩
from(
       select trans_array(1,';',name,subject) as (name,subject) 
       from(
            select name,
        concat('语文',':',chinese,';','数学',':',mathematics,';','物理',':',physics) as subject 
            from columntorow)tt)tx;

返回结果如下。

+--------+--------+------------+
| 姓名   | 科目   | 成绩       |
+--------+--------+------------+
| 张三   | 语文   | 74         |
| 张三   | 数学   | 83         |
| 张三   | 物理   | 93         |
| 李四   | 语文   | 74         |
| 李四   | 数学   | 84         |
| 李四   | 物理   | 94         |
+--------+--------+------------+

十二、MaxCompute SQL中的关联操作

本文为您介绍MaxCompute SQL中的关联（JOIN）操作。

类型 说明

  • INNER JOIN 输出符合关联条件的数据。
  • LEFT JOIN 输出左表的所有记录，以及右表中符合关联条件的数据。右表中不符合关联条件的行，输出NULL。
  • RIGHT JOIN 输出右表的所有记录，以及左表中符合关联条件的数据。左表中不符合关联条件的行，输出NULL。
  • FULL JOIN 输出左表和右表的所有记录，对于不符合关联条件的数据，未关联的另一侧输出NULL。
  • LEFT SEMI JOIN 对于左表中的一条数据，如果右表存在符合关联条件的行，则输出左表。
  • LEFT ANTI JOIN 对于左表中的一条数据，如果右表中不存在符合关联条件的数据，则输出左表。

SQL语句中，同时存在JOIN和WHERE子句时，如下所示。

(SELECT * FROM A WHERE {subquery_where_condition} A) A
JOIN
(SELECT * FROM B WHERE {subquery_where_condition} B) B
ON {on_condition}
WHERE {where_condition}

计算顺序如下：

  • 子查询中的WHERE子句（即{subquery_where_condition}）。
  • JOIN子句中的关联条件（即{on_condition}）。
  • JOIN结果集中的WHERE子句（即{where_condition}）。

因此，对于不同的JOIN类型，过滤条件在{subquery_where_condition}、{on_condition}和{where_condition}中时，查询结果可能一致，也可能不一致。详情请参见场景说明。

示例数据
表A
建表语句如下。

CREATE TABLE A AS SELECT * FROM VALUES (1, 20180101),(2, 20180101),(2, 20180102) t (key, ds);

示例数据如下。

key	ds
1	20180101
2	20180101
2	20180102

表B
建表语句如下。

CREATE TABLE B AS SELECT * FROM VALUES (1, 20180101),(3, 20180101),(2, 20180102) t (key, ds);

示例数据如下。

key	ds
1	20180101
3	20180101
2	20180102

表A和表B的笛卡尔乘积如下。

a.key	a.ds	b.key	b.ds
1	20180101	1	20180101
1	20180101	3	20180101
1	20180101	2	20180102
2	20180101	1	20180101
2	20180101	3	20180101
2	20180101	2	20180102
2	20180102	1	20180101
2	20180102	3	20180101
2	20180102	2	20180102

场景说明
INNER JOIN
INNER JOIN对左右表执行笛卡尔乘积，然后输出满足ON表达式的行。

结论：过滤条件在{subquery_where_condition}、{on_condition}和{where_condition}中时，查询结果是一致的。
情况1：过滤条件在子查询{subquery_where_condition}中。

SELECT A.*, B.*
FROM
(SELECT * FROM A WHERE ds='20180101') A
JOIN
(SELECT * FROM B WHERE ds='20180101') B
ON a.key = b.key;

结果如下。

a.key	a.ds	b.key	b.ds
1	20180101	1	20180101

情况2：过滤条件在JOIN的关联条件{on_condition}中。

SELECT A.*, B.*
FROM A JOIN B
ON a.key = b.key and A.ds='20180101' and B.ds='20180101';

笛卡尔积结果为9条，满足关联条件的结果只有1条，如下。

a.key	a.ds	b.key	b.ds
1	20180101	1	20180101

情况3：过滤条件在JOIN结果集的WHERE子句中。

SELECT A.*, B.*
FROM A JOIN B
ON a.key = b.key
WHERE A.ds='20180101' and B.ds='20180101';

笛卡尔积的结果为9条，满足关联条件的结果有3条，如下。

a.key	a.ds	b.key	b.ds
1	20180101	1	20180101
2	20180102	2	20180102
2	20180101	2	20180102

对上述结果执行JOIN结果集中的过滤条件A.ds=‘20180101’ and B.ds=‘20180101’，结果只有1条，如下。

a.key	a.ds	b.key	b.ds
1	20180101	1	20180101

LEFT JOIN
LEFT JOIN对左右表执行笛卡尔乘积，输出满足ON表达式的行。对于左表中不满足ON表达式的行，输出左表，右表输出NULL。

结论：过滤条件在{subquery_where_condition}、{on_condition}和{where_condition}中时，查询结果不一致。
左表的过滤条件在{subquery_where_condition}和{where_condition}中时，查询结果是一致的。
右表的过滤条件在{subquery_where_condition}和{on_condition}中时，查询结果是一致的。
情况1：过滤条件在子查询
{subquery_where_condition}中。

SELECT A.*, B.*
FROM
(SELECT * FROM A WHERE ds='20180101') A
LEFT JOIN
(SELECT * FROM B WHERE ds='20180101') B
ON a.key = b.key;

结果如下。

a.key	a.ds	b.key	b.ds
1	20180101	1	20180101
2	20180101	NULL	NULL

情况2：过滤条件在JOIN的关联条件{on_condition}中。

SELECT A.*, B.*
FROM A LEFT JOIN B
ON a.key = b.key and A.ds='20180101' and B.ds='20180101';

笛卡尔积的结果有9条，满足关联条件的结果只有1条。左表输出剩余不满足关联条件的两条记录，右表输出NULL。

a.key	a.ds	b.key	b.ds
1	20180101	1	20180101
2	20180101	NULL	NULL
2	20180102	NULL	NULL

情况3：过滤条件在JOIN结果集的WHERE子句中。

SELECT A.*, B.*
FROM A LEFT JOIN B
ON a.key = b.key
WHERE A.ds='20180101' and B.ds='20180101';

笛卡尔积的结果为9条，满足ON条件的结果有3条。
a.key a.ds b.key b.ds

1	20180101	1	20180101
2	20180101	2	20180102
2	20180102	2	20180102

对上述结果执行JOIN结果集中的过滤条件A.ds=‘20180101’ and B.ds=‘20180101’，结果只有1条。

a.key	a.ds	b.key	b.ds
1	20180101	1	20180101

RIGHT JOIN
RIGHT JOIN和LEFT JOIN是类似的，只是左右表的区别。
过滤条件在{subquery_where_condition}、{on_condition}和{where_condition}时，查询结果不一致。
右表的过滤条件，在{subquery_where_condition}和{where_condition}中时，查询结果一致。
左表的过滤条件，放在{subquery_where_condition}和{on_condition}中时，查询结果一致。
FULL JOIN
FULL JOIN对左右表执行笛卡尔乘积，然后输出满足关联条件的行。对于左右表中不满足关联条件的行，输出有数据表的行，无数据的表输出NULL。

结论：过滤条件在{subquery_where_condition}、{on_condition}和{where_condition}时，查询结果不一致。

情况1：过滤条件在子查询{subquery_where_condition}中。

SELECT A.*, B.*
FROM
(SELECT * FROM A WHERE ds='20180101') A
FULL JOIN
(SELECT * FROM B WHERE ds='20180101') B
ON a.key = b.key;

结果如下。

a.key	a.ds	b.key	b.ds
1	20180101	1	20180101
2	20180101	NULL	NULL
NULL	NULL	3	20180101

情况2：过滤条件在JOIN的关联条件{on_condition}中。

SELECT A.*, B.*
FROM A FULL JOIN B
ON a.key = b.key and A.ds='20180101' and B.ds='20180101';

笛卡尔积的结果有9条，满足关联条件的结果只有1条。对于左表不满足关联条件的两条记录输出左表数据，右表输出NULL。对于右表不满足关联条件的两条记录输出右表数据，左表输出NULL。

a.key	a.ds	b.key	b.ds
1	20180101	1	20180101
2	20180101	NULL	NULL
2	20180102	NULL	NULL
NULL	NULL	3	20180101
NULL	NULL	2	20180102

情况3：过滤条件在JOIN结果集的WHERE子句中。

SELECT A.*, B.*
FROM A FULL JOIN B
ON a.key = b.key
WHERE A.ds='20180101' and B.ds='20180101';

笛卡尔积的结果有9条，满足关联条件的结果有3条。

a.key	a.ds	b.key	b.ds
1	20180101	1	20180101
2	20180101	2	20180102
2	20180102	2	20180102

对于不满足关联条件的表输出数据，另一表输出NULL。

a.key	a.ds	b.key	b.ds
1	20180101	1	20180101
2	20180101	2	20180102
2	20180102	2	20180102
NULL	NULL	3	20180101

对上述结果执行JOIN结果集中的过滤条件A.ds=‘20180101’ and B.ds=‘20180101’，结果只有1条。

a.key	a.ds	b.key	b.ds
1	20180101	1	20180101

LEFT SEMI JOIN
LEFT SEMI JOIN将左表的每一条记录，和右表进行匹配。如果匹配成功，则输出左表。如果匹配不成功，则跳过。由于只输出左表，所以JOIN后的WHERE条件中不涉及右表。

结论：过滤条件在{subquery_where_condition}、{on_condition}和{where_condition}中时，查询结果是一致的。

情况1：过滤条件在子查询{subquery_where_condition}中。

SELECT A.*
FROM
(SELECT * FROM A WHERE ds='20180101') A
LEFT SEMI JOIN
(SELECT * FROM B WHERE ds='20180101') B
ON a.key = b.key;

结果如下。

a.key	a.ds
1	20180101

情况2：过滤条件在JOIN的关联条件{on_condition}中。

SELECT A.*
FROM A LEFT SEMI JOIN B
ON a.key = b.key and A.ds='20180101' and B.ds='20180101';

结果如下。

a.key	a.ds
1	20180101

情况3：过滤条件在JOIN结果集的WHERE子句中。

SELECT A.*
FROM A LEFT SEMI JOIN
(SELECT * FROM B WHERE ds='20180101') B
ON a.key = b.key
WHERE A.ds='20180101';

符合关联条件的结果如下。

a.key	a.ds
1	20180101

对上述结果执行JOIN结果集中的过滤条件A.ds=‘20180101’，结果如下。

a.key	a.ds
1	20180101

LEFT ANTI JOIN
LEFT ANTI JOIN将左表的每一条记录，和右表进行匹配。如果右表中的记录不匹配，则输出左表。由于只输出左表，所以JOIN后的WHERE条件中不能涉及右表。LEFT ANTI JOIN常常用来实现NOT EXISTS语义。

结论：过滤条件在{subquery_where_condition}、{on_condition}和{where_condition}中时，查询结果不一致。
左表的过滤条件在{subquery_where_condition}和{where_condition}中时，查询结果是一致的。
右表的过滤条件在{subquery_where_condition}和{on_condition}中时，查询结果是一致的。
情况1：过滤条件在子查询{subquery_where_condition}中。

SELECT A.*
FROM
(SELECT * FROM A WHERE ds='20180101') A
LEFT ANTI JOIN
(SELECT * FROM B WHERE ds='20180101') B
ON a.key = b.key;

结果如下。

a.key	a.ds
2	20180101

情况2：过滤条件在JOIN的关联条件{on_condition}中。

SELECT A.*
FROM A LEFT ANTI JOIN B
ON a.key = b.key and A.ds='20180101' and B.ds='20180101';

结果如下。

a.key	a.ds
2	20180101
2	20180102

情况3：过滤条件在JOIN结果集的WHERE子句中。

SELECT A.*
FROM A LEFT ANTI JOIN
(SELECT * FROM B WHERE ds='20180101') B
ON a.key = b.key
WHERE A.ds='20180101';

左表中符合关联条件的数据如下。

a.key	a.ds
2	20180101
2	20180102

对上述结果执行JOIN结果集中的过滤条件A.ds=‘20180101’，结果如下。

a.key	a.ds
2	20180101

注意事项

  • INNER JOIN/LEFT SEMI JOIN左右表的过滤条件不受限制。
  • LEFT JOIN/LEFT ANTI
    JOIN左表的过滤条件需放在{subquery_where_condition}或{where_condition}中，
  • 右表的过滤条件需放在{subquery_where_condition}或{on_condition}中
  • RIGHT JOIN和LEFT
    JOIN相反，右表的过滤条件需放在{subquery_where_condition}或{where_condition}中，左表的过滤条件需放在{subquery_where_condition}或{on_condition}。
  • FULL OUTER JOIN的过滤条件只能放在{subquery_where_condition}中。

阿里云参考文档：
https://help.aliyun.com/product/27797.html

人猿宇宙
关注
  • 2
    点赞
  • 踩
  • 22
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
大数据开发-> SQL语法基础
09-20
大数据开发——> SQL语法基础 适合大数据初学者以及SQL初学者
大数据--SQL and pySpark
08-09
大数据--SQL and pySpark 对比分析
大数据-分布式大数据SQL查询可视化界面设计.zip
03-03
大数据_分布式大数据SQL查询可视化界面设计
大数据-squirrel-sql-3.9.1_大数据
09-25
大数据-squirrel-sql-3.9.1
解决方法：FAILED: ODPS-0130071:[0,0] Semantic analysis exception - physical plan generation failed: java.
数据搬运工
08-06 1万+
**产生原因：**上游数据中的字段类型为decimal（38,18），下游数据类型也为decimal **解决方法：**将两表数据类型统一。上游数据的类型为decimal，下游数据类型也为decimal
SQL学习《SQL必会知识》-GROUP BY
u013378092的博客
07-02 501
GROUP BY子句必须在WHERE子句之后，ORDER BY子句之前； 使用HAVING过滤分组，不能使用WHERE过滤是因为WHERE过滤的是行，而不是分组，并且WHERE并没有分组的概念。 HAVING和WHERE子句的用法是及其类似的，基本可以相互替换使用，例：SELECT COUNT(*) AS ct FROM products GROUP BY products.vend_id H
Hive_优化/问题
weixin_43875878的博客
09-23 1755
②提高每个Join Worker的内存大小,odps.stage.joiner.mem,256 MB~12288 MB,默认值为1024 MB.②设置mapjoin时小表的最大文件大小,默认为25000000(25M):set hive.mapjoin.smalltable.filesize.①开启Mapjoin,默认开启:set hive.auto.convert.join = true.①提高每个join Worker的数量,odps.stage.joiner.num.
Hive报错"Expression not in GROUP BY key"的解决方案
热门推荐
chinamcafee的专栏
11-30 4万+
执行Hive语句遇到"Expression not in GROUP BY key"的快速解决方法
group by column中的column与前面条件发生的错误的解决方案
duancsdn的博客
11-24 2598
出现问题的sql语句：select a.*,b.relevant_state,b.filename,b.uploadtime,b.province_code,b.city_code,b.team_code,b.filesize,b.nasid,b.filetime,b.filepath,b.id,b.filetype,c.realname,d.nas_name from cases_files_key a left join files b on a.fileid = b.id left join user
ODPS-0130071:[0,0] Semantic analysis exception - physical plan generation failed
山中有石为玉
05-14 2万+
使用maxcompute sql时报错： FAILED: ODPS-0130071:[0,0] Semantic analysis exception - physical plan generation failed: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: Node [rel#416195775:Subset#5....
微软内部资料-SQL性能优化5
11-27
Contents Overview 1 Lesson 1: Index Concepts 3 Lesson 2: Concepts – Statistics 29 Lesson 3: Concepts – Query Optimization 37 Lesson 4: Information Collection and Analysis 61 Lesson 5: Formulating and Implementing Resolution 75 Module 6: Troubleshooting Query Performance Overview At the end of this module, you will be able to:  Describe the different types of indexes and how indexes can be used to improve performance.  Describe what statistics are used for and how they can help in optimizing query performance.  Describe how queries are optimized.  Analyze the information collected from various tools.  Formulate resolution to query performance problems. Lesson 1: Index Concepts Indexes are the most useful tool for improving query performance. Without a useful index, Microsoft® SQL Server™ must search every row on every page in table to find the rows to return. With a multitable query, SQL Server must sometimes search a table multiple times so each page is scanned much more than once. Having useful indexes speeds up finding individual rows in a table, as well as finding the matching rows needed to join two tables. What You Will Learn After completing this lesson, you will be able to:  Understand the structure of SQL Server indexes.  Describe how SQL Server uses indexes to find rows.  Describe how fillfactor can impact the performance of data retrieval and insertion.  Describe the different types of fragmentation that can occur within an index. Recommended Reading  Chapter 8: “Indexes”, Inside SQL Server 2000 by Kalen Delaney  Chapter 11: “Batches, Stored Procedures and Functions”, Inside SQL Server 2000 by Kalen Delaney Finding Rows without Indexes With No Indexes, A Table Must Be Scanned SQL Server keeps track of which pages belong to a table or index by using IAM pages. If there is no clustered index, there is a sysindexes row for the table with an indid value of 0, and that row will keep track of the address of the first IAM for the table. The IAM is a giant bitmap, and every 1 bit indicates that the corresponding extent belongs to the table. The IAM allows SQL Server to do efficient prefetching of the table’s extents, but every row still must be examined. General Index Structure All SQL Server Indexes Are Organized As B-Trees Indexes in SQL Server store their information using standard B-trees. A B-tree provides fast access to data by searching on a key value of the index. B-trees cluster records with similar keys. The B stands for balanced, and balancing the tree is a core feature of a B-tree’s usefulness. The trees are managed, and branches are grafted as necessary, so that navigating down the tree to find a value and locate a specific record takes only a few page accesses. Because the trees are balanced, finding any record requires about the same amount of resources, and retrieval speed is consistent because the index has the same depth throughout. Clustered and Nonclustered Indexes Both Index Types Have Many Common Features An index consists of a tree with a root from which the navigation begins, possible intermediate index levels, and bottom-level leaf pages. You use the index to find the correct leaf page. The number of levels in an index will vary depending on the number of rows in the table and the size of the key column or columns for the index. If you create an index using a large key, fewer entries will fit on a page, so more pages (and possibly more levels) will be needed for the index. On a qualified select, update, or delete, the correct leaf page will be the lowest page of the tree in which one or more rows with the specified key or keys reside. A qualified operation is one that affects only specific rows that satisfy the conditions of a WHERE clause, as opposed to accessing the whole table. An index can have multiple node levels An index page above the leaf is called a node page. Each index row in node pages contains an index key (or set of keys for a composite index) and a pointer to a page at the next level for which the first key value is the same as the key value in the current index row. Leaf Level contains all key values In any index, whether clustered or nonclustered, the leaf level contains every key value, in key sequence. In SQL Server 2000, the sequence can be either ascending or descending. The sysindexes table contains all sizing, location and distribution information Any information about size of indexes or tables is stored in sysindexes. The only source of any storage location information is the sysindexes table, which keeps track of the address of the root page for every index, and the first IAM page for the index or table. There is also a column for the first page of the table, but this is not guaranteed to be reliable. SQL Server can find all pages belonging to an index or table by examining the IAM pages. Sysindexes contains a pointer to the first IAM page, and each IAM page contains a pointer to the next one. The Difference between Clustered and Nonclustered Indexes The main difference between the two types of indexes is how much information is stored at the leaf. The leaf levels of both types of indexes contain all the key values in order, but they also contain other information. Clustered Indexes The Leaf Level of a Clustered Index Is the Data The leaf level of a clustered index contains the data pages, not just the index keys. Another way to say this is that the data itself is part of the clustered index. A clustered index keeps the data in a table ordered around the key. The data pages in the table are kept in a doubly linked list called the page chain. The order of pages in the page chain, and the order of rows on the data pages, is the order of the index key or keys. Deciding which key to cluster on is an important performance consideration. When the index is traversed to the leaf level, the data itself has been retrieved, not simply pointed to. Uniqueness Is Maintained In Key Values In SQL Server 2000, all clustered indexes are unique. If you build a clustered index without specifying the unique keyword, SQL Server forces uniqueness by adding a uniqueifier to the rows when necessary. This uniqueifier is a 4-byte value added as an additional sort key to only the rows that have duplicates of their primary sort key. You can see this extra value if you use DBCC PAGE to look at the actual index rows the section on indexes internal. . Finding Rows in a Clustered Index The Leaf Level of a Clustered Index Contains the Data A clustered index is like a telephone directory in which all of the rows for customers with the same last name are clustered together in the same part of the book. Just as the organization of a telephone directory makes it easy for a person to search, SQL Server quickly searches a table with a clustered index. Because a clustered index determines the sequence in which rows are stored in a table, there can only be one clustered index for a table at a time. Performance Considerations Keeping your clustered key value small increases the number of index rows that can be placed on an index page and decreases the number of levels that must be traversed. This minimizes I/O. As we’ll see, the clustered key is duplicated in every nonclustered index row, so keeping your clustered key small will allow you to have more index fit per page in all your indexes. Note The query corresponding to the slide is: SELECT lastname, firstname FROM member WHERE lastname = ‘Ota’ Nonclustered Indexes The Leaf Level of a Nonclustered Index Contains a Bookmark A nonclustered index is like the index of a textbook. The data is stored in one place and the index is stored in another. Pointers indicate the storage location of the indexed items in the underlying table. In a nonclustered index, the leaf level contains each index key, plus a bookmark that tells SQL Server where to find the data row corresponding to the key in the index. A bookmark can take one of two forms:  If the table has a clustered index, the bookmark is the clustered index key for the corresponding data row. This clustered key can be multiple column if the clustered index is composite, or is defined to be non-unique.  If the table is a heap (in other words, it has no clustered index), the bookmark is a RID, which is an actual row locator in the form File#:Page#:Slot#. Finding Rows with a NC Index on a Heap Nonclustered Indexes Are Very Efficient When Searching For A Single Row After the nonclustered key at the leaf level of the index is found, only one more page access is needed to find the data row. Searching for a single row using a nonclustered index is almost as efficient as searching for a single row in a clustered index. However, if we are searching for multiple rows, such as duplicate values, or keys in a range, anything more than a small number of rows will make the nonclustered index search very inefficient. Note The query corresponding to the slide is: SELECT lastname, firstname FROM member WHERE lastname BETWEEN ‘Master’ AND ‘Rudd’ Finding Rows with a NC Index on a Clustered Table A Clustered Key Is Used as the Bookmark for All Nonclustered Indexes If the table has a clustered index, all columns of the clustered key will be duplicated in the nonclustered index leaf rows, unless there is overlap between the clustered and nonclustered key. For example, if the clustered index is on (lastname, firstname) and a nonclustered index is on firstname, the firstname value will not be duplicated in the nonclustered index leaf rows. Note The query corresponding to the slide is: SELECT lastname, firstname, phone FROM member WHERE firstname = ‘Mike’ Covering Indexes A Covering Index Provides the Fastest Data Access A covering index contains ALL the fields accessed in the query. Normally, only the columns in the WHERE clause are helpful in determining useful indexes, but for a covering index, all columns must be included. If all columns needed for the query are in the index, SQL Server never needs to access the data pages. If even one column in the query is not part of the index, the data rows must be accessed. The leaf level of an index is the only level that contains every key value, or set of key values. For a clustered index, the leaf level is the data itself, so in reality, a clustered index ALWAYS covers any query. Nevertheless, for most of our optimization discussions, we only consider nonclustered indexes. Scanning the leaf level of a nonclustered index is almost always faster than scanning a clustered index, so covering indexes are particular valuable when we need ALL the key values of a particular nonclustered index. Example: Select an aggregate value of a column with a clustered index. Suppose we have a nonclustered index on price, this query is covered: SELECT avg(price) from titles Since the clustered key is included in every nonclustered index row, the clustered key can be included in the covering. Suppose you have a nonclustered index on price and a clustered index on title_id; then this query is covered: SELECT title_id, price FROM titles WHERE price between 10 and 20 Performance Considerations In general, you do want to keep your indexes narrow. However, if you have a critical query that just is not giving you satisfactory performance no matter what you do, you should consider creating an index to cover it, or adding one or two extra columns to an existing index, so that the query will be covered. The leaf level of a nonclustered index is like a ‘mini’ clustered index, so you can have most of the benefits of clustering, even if there already is another clustered index on the table. The tradeoff to adding more, wider indexes for covering queries are the added disk space, and more overhead for updating those columns that are now part of the index. Bug In general, SQL Server will detect when a query is covered, and detect the possible covering indexes. However, in some cases, you must force SQL Server to use a covering index by including a WHERE clause, even if the WHERE clause will return ALL the rows in the table. This is SHILOH bug #352079 Steps to reproduce 1. Make copy of orders table from Northwind: USE Northwind CREATE TABLE [NewOrders] ( [OrderID] [int] NOT NULL , [CustomerID] [nchar] (5) NULL , [EmployeeID] [int] NULL , [OrderDate] [datetime] NULL , [RequiredDate] [datetime] NULL , [ShippedDate] [datetime] NULL , [ShipVia] [int] NULL , [Freight] [money] NULL , [ShipName] [nvarchar] (40) NULL, [ShipAddress] [nvarchar] (60) , [ShipCity] [nvarchar] (15) NULL, [ShipRegion] [nvarchar] (15) NULL, [ShipPostalCode] [nvarchar] (10) NULL, [ShipCountry] [nvarchar] (15) NULL ) INSERT into NewOrders SELECT * FROM Orders 2. Build nc index on OrderDate: create index dateindex on neworders(orderdate) 3. Test Query by looking at query plan: select orderdate from NewOrders The index is being scanned, as expected. 4. Build an index on orderId: create index orderid_index on neworders(orderID) 5. Test Query by looking at query plan: select orderdate from NewOrders Now the TABLE is being scanned, instead of the original index! Index Intersection Multiple Indexes Can Be Used On A Single Table In versions prior to SQL Server 7, only one index could be used for any table to process any single query. The only exception was a query involving an OR. In current SQL Server versions, multiple nonclustered indexes can each be accessed, retrieving a set of keys with bookmarks, and then the result sets can be joined on the common bookmarks. The optimizer weighs the cost of performing the unindexed join on the intermediate result sets, with the cost of only using one index, and then scanning the entire result set from that single index. Fillfactor and Performance Creating an Index with a Low Fillfactor Delays Page Splits when Inserting DBCC SHOWCONTIG will show you a low value for “Avg. Page Density” when a low fillfactor has been specified. This is good for inserts and updates, because it will delay the need to split pages to make room for new rows. It can be bad for scans, because fewer rows will be on each page, and more pages must be read to access the same amount of data. However, this cost will be minimal if the scan density value is good. Index Reorganization DBCC SHOWCONTIG Provides Lots of Information Here’s some sample output from running a basic DBCC SHOWCONTIG on the order details table in the Northwind database: DBCC SHOWCONTIG scanning 'Order Details' table... Table: 'Order Details' (325576198); index ID: 1, database ID:6 TABLE level scan performed. - Pages Scanned................................: 9 - Extents Scanned..............................: 6 - Extent Switches..............................: 5 - Avg. Pages per Extent........................: 1.5 - Scan Density [Best Count:Actual Count].......: 33.33% [2:6] - Logical Scan Fragmentation ..................: 0.00% - Extent Scan Fragmentation ...................: 16.67% - Avg. Bytes Free per Page.....................: 673.2 - Avg. Page Density (full).....................: 91.68% By default, DBCC SHOWCONTIG scans the page chain at the leaf level of the specified index and keeps track of the following values:  Average number of bytes free on each page (Avg. Bytes Free per Page)  Number of pages accessed (Pages scanned)  Number of extents accessed (Extents scanned)  Number of times a page had a lower page number than the previous page in the scan (This value for Out of order pages is not displayed, but is used for additional computations.)  Number of times a page in the scan was on a different extent than the previous page in the scan (Extent switches) SQL Server also keeps track of all the extents that have been accessed, and then it determines how many gaps are in the used extents. An extent is identified by the page number of its first page. So, if extents 8, 16, 24, 32, and 40 make up an index, there are no gaps. If the extents are 8, 16, 24, and 40, there is one gap. The value in DBCC SHOWCONTIG’s output called Extent Scan Fragmentation is computed by dividing the number of gaps by the number of extents, so in this example the Extent Scan Fragmentation is ¼, or 25 percent. A table using extents 8, 24, 40, and 56 has three gaps, and its Extent Scan Fragmentation is ¾, or 75 percent. The maximum number of gaps is the number of extents - 1, so Extent Scan Fragmentation can never be 100 percent. The value in DBCC SHOWCONTIG’s output called Logical Scan Fragmentation is computed by dividing the number of Out of order pages by the number of pages in the table. This value is meaningless in a heap. You can use either the Extent Scan Fragmentation value or the Logical Scan Fragmentation value to determine the general level of fragmentation in a table. The lower the value, the less fragmentation there is. Alternatively, you can use the value called Scan Density, which is computed by dividing the optimum number of extent switches by the actual number of extent switches. A high value means that there is little fragmentation. Scan Density is not valid if the table spans multiple files; therefore, it is less useful than the other values. SQL Server 2000 allows online defragmentation You can choose from several methods for removing fragmentation from an index. You could rebuild the index and have SQL Server allocate all new contiguous pages for you. To rebuild the index, you can use a simple DROP INDEX and CREATE INDEX combination, but in many cases using these commands is less than optimal. In particular, if the index is supporting a constraint, you cannot use the DROP INDEX command. Alternatively, you can use DBCC DBREINDEX, which can rebuild all the indexes on a table in one operation, or you can use the drop_existing clause along with CREATE INDEX. The drawback of these methods is that the table is unavailable while SQL Server is rebuilding the index. When you are rebuilding only nonclustered indexes, SQL Server takes a shared lock on the table, which means that users cannot make modifications, but other processes can SELECT from the table. Of course, those SELECT queries cannot take advantage of the index you are rebuilding, so they might not perform as well as they would otherwise. If you are rebuilding a clustered index, SQL Server takes an exclusive lock and does not allow access to the table, so your data is temporarily unavailable. SQL Server 2000 lets you defragment an index without completely rebuilding it. DBCC INDEXDEFRAG reorders the leaf-level pages into physical order as well as logical order, but using only the pages that are already allocated to the leaf level. This command does an in-place ordering, which is similar to a sorting technique called bubble sort (you might be familiar with this technique if you've studied and compared various sorting algorithms). In-place ordering can reduce logical fragmentation to 2 percent or less, making an ordered scan through the leaf level much faster. DBCC INDEXDEFRAG also compacts the pages of an index, based on the original fillfactor. The pages will not always end up with the original fillfactor, but SQL Server uses that value as a goal. The defragmentation process attempts to leave at least enough space for one average-size row on each page. In addition, if SQL Server cannot obtain a lock on a page during the compaction phase of DBCC INDEXDEFRAG, it skips the page and does not return to it. Any empty pages created as a result of compaction are removed. The algorithm SQL Server 2000 uses for DBCC INDEXDEFRAG finds the next physical page in a file belonging to the index's leaf level and the next logical page in the leaf level to swap it with. To find the next physical page, the algorithm scans the IAM pages belonging to that index. In a database spanning multiple files, in which a table or index has pages on more than one file, SQL Server handles pages on different files separately. SQL Server finds the next logical page by scanning the index's leaf level. After each page move, SQL Server drops all locks and saves the last key on the last page it moved. The next iteration of the algorithm uses the last key to find the next logical page. This process lets other users update the table and index while DBCC INDEXDEFRAG is running. Let us look at an example in which an index's leaf level consists of the following pages in the following logical order: 47 22 83 32 12 90 64 The first key is on page 47, and the last key is on page 64. SQL Server would have to scan the pages in this order to retrieve the data in sorted order. As its first step, DBCC INDEXDEFRAG would find the first physical page, 12, and the first logical page, 47. It would then swap the pages, using a temporary buffer as a holding area. After the first swap, the leaf level would look like this: 12 22 83 32 47 90 64 The next physical page is 22, which is also the next logical page, so no work would be necessary. DBCC INDEXDEFRAG would then swap the next physical page, 32, with the next logical page, 83: 12 22 32 83 47 90 64 After the next swap of 47 with 83, the leaf level would look like this: 12 22 32 47 83 90 64 Then, the defragmentation process would swap 64 with 83: 12 22 32 47 64 90 83 and 83 with 90: 12 22 32 47 64 83 90 At the end of the DBCC INDEXDEFRAG operation, the pages in the table or index are not contiguous, but their logical order matches their physical order. Now, if the pages were accessed from disk in sorted order, the head would need to move in only one direction. Keep in mind that DBCC INDEXDEFRAG uses only pages that are already part of the index's leaf level; it allocates no new pages. In addition, defragmenting a large table can take quite a while, and you will get a report every 5 minutes about the estimated percentage completed. However, except for the locks on the pages being switched, this command needs no additional locks. All the table's other pages and indexes are fully available for your applications to use during the defragmentation process. If you must completely rebuild an index because you want a new fillfactor, or if simple defragmentation is not enough because you want to remove all fragmentation from your indexes, another SQL Server 2000 improvement makes index rebuilding less of an imposition on the rest of the system. SQL Server 2000 lets you create an index in parallel—that is, using multiple processors—which drastically reduces the time necessary to perform the rebuild. The algorithm SQL Server 2000 uses, allows near-linear scaling with the number of processors you use for the rebuild, so four processors will take only one-fourth the time that one processor requires to rebuild an index. System availability increases because the length of time that a table is unavailable decreases. Note that only the SQL Server 2000 Enterprise Edition supports parallel index creation. Indexes on Views and Computed Columns Building an Index Gives the Data Physical Existence Normally, views are only logical and the rows comprising the view’s data are not generated until the view is accessed. The values for computed columns are typically not stored anywhere in the database; only the definition for the computation is stored and the computation is redone every time a computed column is accessed. The first index on a view must be a clustered index, so that the leaf level can hold all the actual rows that make up the view. Once that clustered index has been build, and the view’s data is now physical, additional (nonclustered) indexes can be built. An index on a computed column can be nonclustered, because all we need to store is the index key values. Common Prerequisites for Indexed Views and Indexes on Computed Columns In order for SQL Server to create use these special indexes, you must have the seven SET options correctly specified: ARITHABORT, CONCAT_NULL_YIELDS_NULL, QUOTED_IDENTIFIER, ANSI_NULLS, ANSI_PADDING, ANSI_WARNING must be all ON NUMERIC_ROUNDABORT must be OFF Only deterministic expressions can be used in the definition of Indexed Views or indexes on Computed Columns. See the BOL for the list of deterministic functions and expressions. Property functions are available to check if a column or view meets the requirements and is indexable. SELECT OBJECTPROPERTY (Object_id, ‘IsIndexable’) SELECT COLUMNPROPERTY (Object_id, column_name , ‘IsIndexable’ ) Schema Binding Guarantees That Object Definition Won’t Change A view can only be indexed if it has been built with schema binding. The SQL Server Optimizer Determines If the Indexed View Can Be Used The query must request a subset of the data contained in the view. The ability of the optimizer to use the indexed view even if the view is not directly referenced is available only in SQL Server 2000 Enterprise Edition. In Standard edition, you can create indexed views, and you can select directly from them, but the optimizer will not choose to use them if they are not directly referenced. Examples of Indexed Views: The best candidates for improvement by indexed views are queries performing aggregations and joins. We will explain how the useful indexed views may be created for these two major groups of queries. The considerations are valid also for queries and indexed views using both joins and aggregations. -- Example: USE Northwind -- Identify 5 products with overall biggest discount total. -- This may be expressed for example by two different queries: -- Q1. select TOP 5 ProductID, SUM(UnitPrice*Quantity)- SUM(UnitPrice*Quantity*(1.00-Discount)) Rebate from [order details] group by ProductID order by Rebate desc --Q2. select TOP 5 ProductID, SUM(UnitPrice*Quantity*Discount) Rebate from [order details] group by ProductID order by Rebate desc --The following indexed view will be used to execute Q1. create view Vdiscount1 with schemabinding as select SUM(UnitPrice*Quantity) SumPrice, SUM(UnitPrice*Quantity*(1.00-Discount)) SumDiscountPrice, COUNT_BIG(*) Count, ProductID from dbo.[order details] group By ProductID create unique clustered index VDiscountInd on Vdiscount1 (ProductID) However, it will not be used by the Q2 because the indexed view does not contain the SUM(UnitPrice*Quantity*Discount) aggregate. We can construct another indexed view create view Vdiscount2 with schemabinding as select SUM(UnitPrice*Quantity) SumPrice, SUM(UnitPrice*Quantity*(1.00-Discount)) SumDiscountPrice, SUM(UnitPrice*Quantity*Discount) SumDiscoutPrice2, COUNT_BIG(*) Count, ProductID from dbo.[order details] group By ProductID create unique clustered index VDiscountInd on Vdiscount2 (ProductID) This view may be used by both Q1 and Q2. Observe that the indexed view Vdiscount2 will have the same number of rows and only one more column compared to Vdiscount1, and it may be used by more queries. In general, try to design indexed views that may be used by more queries. The following query asking for the order with the largest total discount -- Q3. select TOP 3 OrderID, SUM(UnitPrice*Quantity*Discount) OrderRebate from dbo.[order details] group By OrderID Q3 can use neither of the Vdiscount views because the column OrderID is not included in the view definition. To address this variation of the discount analysis query we may create a different indexed view, similar to the query itself. An attempt to generalize the previous indexed view Vdiscount2 so that all three queries Q1, Q2, and Q3 can take advantage of a single indexed view would require a view with both OrderID and ProductID as grouping columns. Because the OrderID, ProductID combination is unique in the original order details table the resulting view would have as many rows as the original table and we would see no savings in using such view compared to using the original table. Consider the size of the resulting indexed view. In the case of pure aggregation, the indexed view may provide no significant performance gains if its size is close to the size of the original table. Complex aggregates (STDEV, VARIANCE, AVG) cannot participate in the index view definition. However, SQL Server may use an indexed view to execute a query containing AVG aggregate. Query containing STDEV or VARIANCE cannot use indexed view to pre-compute these values. The next example shows a query producing the average price for a particular product -- Q4. select ProductName, od.ProductID, AVG(od.UnitPrice*(1.00-Discount)) AvgPrice, SUM(od.Quantity) Units from [order details] od, Products p where od.ProductID=p.ProductID group by ProductName, od.ProductID This is an example of indexed view that will be considered by the SQL Server to answer the Q4 create view v3 with schemabinding as select od.ProductID, SUM(od.UnitPrice*(1.00-Discount)) Price, COUNT_BIG(*) Count, SUM(od.Quantity) Units from dbo.[order details] od group by od.ProductID go create UNIQUE CLUSTERED index iv3 on v3 (ProductID) go Observe that the view definition does not contain the table Products. The indexed view does not need to contain all tables used in the query that uses the indexed view. In addition, the following query (same as above Q4 only with one additional search condition) will use the same indexed view. Observe that the added predicate references only columns from tables not present in the v3 view definition. -- Q5. select ProductName, od.ProductID, AVG(od.UnitPrice*(1.00-Discount)) AvgPrice, SUM(od.Quantity) Units from [order details] od, Products p where od.ProductID=p.ProductID and p.ProductName like '%tofu%' group by ProductName, od.ProductID The following query cannot use the indexed view because the added search condition od.UnitPrice>10 contains a column from the table in the view definition and the column is neither grouping column nor the predicate appears in the view definition. -- Q6. select ProductName, od.ProductID, AVG(od.UnitPrice*(1.00-Discount)) AvgPrice, SUM(od.Quantity) Units from [order details] od, Products p where od.ProductID=p.ProductID and od.UnitPrice>10 group by ProductName, od.ProductID To contrast the Q6 case, the following query will use the indexed view v3 since the added predicate is on the grouping column of the view v3. -- Q7. select ProductName, od.ProductID, AVG(od.UnitPrice*(1.00-Discount)) AvgPrice, SUM(od.Quantity) Units from [order details] od, Products p where od.ProductID=p.ProductID and od.ProductID in (1,2,13,41) group by ProductName, od.ProductID -- The previous query Q6 will use the following indexed view V4: create view V4 with schemabinding as select ProductName, od.ProductID, SUM(od.UnitPrice*(1.00-Discount)) AvgPrice, SUM(od.Quantity) Units, COUNT_BIG(*) Count from dbo.[order details] od, dbo.Products p where od.ProductID=p.ProductID and od.UnitPrice>10 group by ProductName, od.ProductID create unique clustered index VDiscountInd on V4 (ProductName, ProductID) The same index on the view V4 will be used also for a query where a join to the table Orders is added, for example -- Q8. select ProductName, od.ProductID, AVG(od.UnitPrice*(1.00-Discount)) AvgPrice, SUM(od.Quantity) Units from dbo.[order details] od, dbo.Products p, dbo.Orders o where od.ProductID=p.ProductID and o.OrderID=od.OrderID and od.UnitPrice>10 group by ProductName, od.ProductID We will show several modifications of the query Q8 and explain why such modifications cannot use the above view V4. -- Q8a. select ProductName, od.ProductID, AVG(od.UnitPrice*(1.00-Discount)) AvgPrice, SUM(od.Quantity) Units from dbo.[order details] od, dbo.Products p, dbo.Orders o where od.ProductID=p.ProductID and o.OrderID=od.OrderID and od.UnitPrice>25 group by ProductName, od.ProductID 8a cannot use the indexed view because of the where clause mismatch. Observe that table Orders does not participate in the indexed view V4 definition. In spite of that, adding a predicate on this table will disallow using the indexed view because the added predicate may eliminate additional rows participating in the aggregates as it is shown in Q8b. -- Q8b. select ProductName, od.ProductID, AVG(od.UnitPrice*(1.00-Discount)) AvgPrice, SUM(od.Quantity) Units from dbo.[order details] od, dbo.Products p, dbo.Orders o where od.ProductID=p.ProductID and o.OrderID=od.OrderID and od.UnitPrice>10 and o.OrderDate>'01/01/1998' group by ProductName, od.ProductID Locking and Indexes In General, You Should Let SQL Server Control the Locking within Indexes The stored procedure sp_indexoption lets you manually control the unit of locking within an index. It also lets you disallow page locks or row locks within an index. Since these options are available only for indexes, there is no way to control the locking within the data pages of a heap. (But remember that if a table has a clustered index, the data pages are part of the index and are affected by the sp_indexoption setting.) The index options are set for each table or index individually. Two options, Allow Rowlocks and AllowPageLocks, are both set to TRUE initially for every table and index. If both of these options are set to FALSE for a table, only full table locks are allowed. As described in Module 4, SQL Server determines at runtime whether to initially lock rows, pages, or the entire table. The locking of rows (or keys) is heavily favored. The type of locking chosen is based on the number of rows and pages to be scanned, the number of rows on a page, the isolation level in effect, the update activity going on, the number of users on the system needing memory for their own purposes, and so on. SAP databases frequently use sp_indexoption to reduce deadlocks Setting vs. Querying In SQL Server 2000, the procedure sp_indexoption should only be used for setting an index option. To query an option, use the INDEXPROPERTY function. Lesson 2: Concepts – Statistics Statistics are the most important tool that the SQL Server query optimizer has to determine the ideal execution plan for a query. Statistics that are out of date or nonexistent seriously jeopardize query performance. SQL Server 2000 computes and stores statistics in a completely different format that all earlier versions of SQL Server. One of the improvements is an increased ability to determine which values are out of the normal range in terms of the number of occurrences. The new statistics maintenance routines are particularly good at determining when a key value has a very unusual skew of data. What You Will Learn After completing this lesson, you will be able to:  Define terms related to statistics collected by SQL Server.  Describe how statistics are maintained by SQL Server.  Discuss the autostats feature of SQL Server.  Describe how statistics are used in query optimization. Recommended Reading  Statistics Used by the Query Optimizer in Microsoft SQL Server 2000 http://msdn.microsoft.com/library/techart/statquery.htm Definitions Cardinality The cardinality means how many unique values exist in the data. Density For each index and set of column statistics, SQL Server keeps track of details about the uniqueness (or density) of the data values encountered, which provides a measure of how selective the index is. A unique index, of course, has the lowest density —by definition, each index entry can point to only one row. A unique index has a density value of 1/number of rows in the table. Density values range from 0 through 1. Highly selective indexes have density values of 0.10 or lower. For example, a unique index on a table with 8345 rows has a density of 0.00012 (1/8345). If a nonunique nonclustered index has a density of 0.2165 on the same table, each index key can be expected to point to about 1807 rows (0.2165 × 8345). This is probably not selective enough to be more efficient than just scanning the table, so this index is probably not useful. Because driving the query from a nonclustered index means that the pages must be retrieved in index order, an estimated 1807 data page accesses (or logical reads) are needed if there is no clustered index on the table and the leaf level of the index contains the actual RID of the desired data row. The only time a data page doesn’t need to be reaccessed is when the occasional coincidence occurs in which two adjacent index entries happen to point to the same data page. In general, you can think of density as the average number of duplicates. We can also talk about the term ‘join density’, which applies to the average number of duplicates in the foreign key column. This would answer the question: in this one-to-many relationship, how many is ‘many’? Selectivity In general selectivity applies to a particular data value referenced in a WHERE clause. High selectivity means that only a small percentage of the rows satisfy the WHERE clause filter, and a low selectivity means that many rows will satisfy the filter. For example, in an employees table, the column employee_id is probably very selective, and the column gender is probably not very selective at all. Statistics Statistics are a histogram consisting of an even sampling of values for a column or for an index key (or the first column of the key for a composite index) based on the current data. The histogram is stored in the statblob field of the sysindexes table, which is of type image. (Remember that image data is actually stored in structures separate from the data row itself. The data row merely contains a pointer to the image data. For simplicity’s sake, we’ll talk about the index statistics as being stored in the image field called statblob.) To fully estimate the usefulness of an index, the optimizer also needs to know the number of pages in the table or index; this information is stored in the dpages column of sysindexes. During the second phase of query optimization, index selection, the query optimizer determines whether an index exists for a columns in your WHERE clause, assesses the index’s usefulness by determining the selectivity of the clause (that is, how many rows will be returned), and estimates the cost of finding the qualifying rows. Statistics for a single column index consist of one histogram and one density value. The multicolumn statistics for one set of columns in a composite index consist of one histogram for the first column in the index and density values for each prefix combination of columns (including the first column alone). The fact that density information is kept for all columns helps the optimizer decide how useful the index is for joins. Suppose, for example, that an index is composed of three key fields. The density on the first column might be 0.50, which is not too useful. However, as you look at more key columns in the index, the number of rows pointed to is fewer than (or in the worst case, the same as) the first column, so the density value goes down. If you are looking at both the first and second columns, the density might be 0.25, which is somewhat better. Moreover, if you examine three columns, the density might be 0.03, which is highly selective. It does not make sense to refer to the density of only the second column. The lead column density is always needed. Statistics Maintenance Statistics Information Tracks the Distribution of Key Values SQL Server statistics is basically a histogram that contains up to 200 values of a given key column. In addition to the histogram, the statblob field contains the following information:  The time of the last statistics collection  The number of rows used to produce the histogram and density information  The average key length  Densities for other combinations of columns In the statblob column, up to 200 sample values are stored; the range of key values between each sample value is called a step. The sample value is the endpoint of the range. Three values are stored along with each step: a value called EQ_ROWS, which is the number of rows that have a value equal to that sample value; a value called RANGE_ROWS, which specifies how many other values are inside the range (between two adjacent sample values); and the number of distinct values, or RANGE_DENSITY of the range. DBCC SHOW_STATISTICS The DBCC SHOW_STATISTICS output shows us the first two of these three values, but not the range density. The RANGE_DENSITY is instead used to compute two additional values:  DISTINCT_RANGE_ROWS—the number of distinct rows inside this range (not counting the RANGE_HI_KEY value itself. This is computed as 1/RANGE_DENSITY.  AVG_RANGE_ROWS—the average number of rows per distinct value, computed as RANGE_DENSITY * RANGE_ROWS. In addition to statistics on indexes, SQL Server can also keep track of statistics on columns with no indexes. Knowing the density, or the likelihood of a particular value occurring, can help the optimizer determine an optimum processing strategy, even if SQL Server can’t use an index to actually locate the values. Statistics on Columns Column statistics can be useful for two main purposes  When the SQL Server optimizer is determining the optimal join order, it frequently is best to have the smaller input processed first. By ‘input’ we mean table after all filters in the WHERE clause have been applied. Even if there is no useful index on a column in the WHERE clause, statistics could tell us that only a few rows will quality, and those the resulting input will be very small.  The SQL Server query optimizer can use column statistics on non-initial columns in a composite nonclustered index to determine if scanning the leaf level to obtain the bookmarks will be an efficient processing strategy. For example, in the member table in the credit database, the first name column is almost unique. Suppose we have a nonclustered index on (lastname, firstname), and we issue this query: select * from member where firstname = 'MPRO' In this case, statistics on the firstname column would indicate very few rows satisfying this condition, so the optimizer will choose to scan the nonclustered index, since it is smaller than the clustered index (the table). The small number of bookmarks will then be followed to retrieve the actual data. Manually Updating Statistics You can also manually force statistics to be updated in one of two ways. You can run the UPDATE STATISTICS command on a table or on one specific index or column statistics, or you can also execute the procedure sp_updatestats, which runs UPDATE STATISTICS against all user-defined tables in the current database. You can create statistics on unindexed columns using the CREATE STATISTICS command or by executing sp_createstats, which creates single-column statistics for all eligible columns for all user tables in the current database. This includes all columns except computed columns and columns of the ntext, text, or image datatypes, and columns that already have statistics or are the first column of an index. Autostats By Default SQL Server Will Update Statistics on Any Index or Column as Needed Every database is created with the database options auto create statistics and auto update statistics set to true, but you can turn either one off. You can also turn off automatic updating of statistics for a specific table in one of two ways:  UPDATE STATISTICS In addition to updating the statistics, the option WITH NORECOMPUTE indicates that the statistics should not be automatically recomputed in the future. Running UPDATE STATISTICS again without the WITH NORECOMPUTE option enables automatic updates.  sp_autostats This procedure sets or unsets a flag for a table to indicate that statistics should or should not be updated automatically. You can also use this procedure with only the table name to find out whether the table is set to automatically have its index statistics updated. ' However, setting the database option auto update statistics to FALSE overrides any individual table settings. In other words, no automatic updating of statistics takes place. This is not a recommended practice unless thorough testing has shown you that you do not need the automatic updates or that the performance overhead is more than you can afford. Trace Flags Trace flag 205 – reports recompile due to autostats. Trace flag 8721 – writes information to the errorlog when AutoStats has been run. For more information, see the following Knowledge Base article: Q195565 “INF: How SQL Server 7.0 Autostats Work.” Statistics and Performance The Performance Penalty of NOT Having Up-To-Date Statistics Far Outweighs the Benefit of Avoiding Automatic Updating Autostats should be turned off only after thorough testing shows it to be necessary. Because autostats only forces a recompile after a certain number or percentage of rows has been changed, you do not have to make any adjustments for a read-only database. Lesson 3: Concepts – Query Optimization What You Will Learn After completing this lesson, you will be able to:  Describe the phases of query optimization.  Discuss how SQL Server estimates the selectivity of indexes and column and how this estimate is used in query optimization. Recommended Reading  Chapter 15: “The Query Processor”, Inside SQL Server 2000 by Kalen Delaney  Chapter 16: “Query Tuning”, Inside SQL Server 2000 by Kalen Delaney  Whitepaper about SQL Server Query Processor Architecture by Hal Berenson and Kalen Delaney http://msdn.microsoft.com/library/backgrnd/html/sqlquerproc.htm Phases of Query Optimization Query Optimization Involves several phases Trivial Plan Optimization Optimization itself goes through several steps. The first step is something called Trivial Plan Optimization. The whole idea of trivial plan optimization is that cost based optimization is a bit expensive to run. The optimizer can try a great many possible variations trying to find the cheapest plan. If SQL Server knows that there is only one really viable plan for a query, it could avoid a lot of work. A prime example is a query that consists of an INSERT with a VALUES clause. There is only one possible plan. Another example is a SELECT where all the columns are in a unique covering index, and that index is the only one that is useable. There is no other index that has that set of columns in it. These two examples are cases where SQL Server should just generate the plan and not try to find something better. The trivial plan optimizer finds the really obvious plans, which are typically very inexpensive. In fact, all the plans that get through the autoparameterization template result in plans that the trivial plan optimizer can find. Between those two mechanisms, the plans that are simple tend to be weeded out earlier in the process and do not pay a lot of the compilation cost. This is a good thing, because the number of potential plans in 7.0 went up astronomically as SQL Server added hash joins, merge joins and index intersections, to its list of processing techniques. Simplification and Statistics Loading If a plan is not found by the trivial plan optimizer, SQL Server can perform some simplifications, usually thought of as syntactic transformations of the query itself, looking for commutative properties and operations that can be rearranged. SQL Server can do constant folding, and other operations that do not require looking at the cost or analyzing what indexes are, but that can result in a more efficient query. SQL Server then loads up the metadata including the statistics information on the indexes, and then the optimizer goes through a series of phases of cost based optimization. Cost Based Optimization Phases The cost based optimizer is designed as a set of transformation rules that try various permutations of indexes and join strategies. Because of the number of potential plans in SQL Server 7.0 and SQL Server 2000, if the optimizer just ran through all the combinations and produced a plan, the optimization process would take a very long time to run. Therefore, optimization is broken up into phases. Each phase is a set of rules. After each phase is run, the cost of any resulting plan is examined, and if SQL Server determines that the plan is cheap enough, that plan is kept and executed. If the plan is not cheap enough, the optimizer runs the next phase, which is another set of rules. In the vast majority of cases, a good plan will be found in the preliminary phases. Typically, if the plan that a query would have had in SQL Server 6.5 is also the optimal plan in SQL Server 7.0 and SQL Server 2000, the plan will tend to be found either by the trivial plan optimizer or by the first phase of the cost based optimizer. The rules were intentionally organized to try to make that be true. The plan will probably consist of using a single index and using nested loops. However, every once in a while, because of lack of statistical information, or some other nuance, the optimizer will have to proceed with the later phases of optimization. Sometimes this is because there is a real possibility that the optimizer could find a better plan. When a plan is found, it becomes the optimizer’s output, and then SQL Server goes through all the caching mechanisms that we have already discussed in Module 5. Full Optimization At some point, the optimizer determines that it has gone through enough preliminary phases, and it reverts to a phase called full optimization. If the optimizer goes through all the preliminary phases, and still has not found a cheap plan, it examines the cost for the plan that it has so far. If the cost is above the threshold, the optimizer goes into a phase called full optimization. This threshold is configurable, as the configuration option ‘cost threshold for parallelism’. The full optimization phase assumes that this plan should be run this in parallel. If the machine is very busy, the plan will end up running it in serial, but the optimizer has a goal to produce a good parallel. If the cost is below the threshold (or a single processor machine), the full optimization phase just uses a brute force method to find a serial plan. Selectivity Estimation Selectivity Is One of The Most Important Pieces of Information One of the most import things the optimizer needs to know is the number of rows from any table that will meet all the conditions in the query. If there are no restrictions on a table, and all the rows will be needed, the optimizer can determine the number of rows from the sysindexes table. This number is not absolutely guaranteed to be accurate, but it is the number the optimizer uses. If there is a filter on the table in a WHERE clause, the optimizer needs statistics information. Indexes automatically maintain statistics, and the optimizer will use these values to determine the usefulness of the index. If there is no index on the column involved in the filter, then column statistics can be used or generated. Optimizing Search Arguments In General, the Filters in the WHERE Clause Determine Which Indexes Will Be Useful If an indexed column is referenced in a Search Argument (SARG), the optimizer will analyze the cost of using that index. A SARG has the form:  column <operator> value  value <operator> column  Operator must be one of =, >, >= <, <= The value can be a constant, an operation, or a variable. Some functions also will be treated as SARGs. These queries have SARGs, and a nonclustered index on firstname will be used in most cases: select * from member where firstname < 'AKKG' select * from member where firstname = substring('HAAKGALSFJA', 2,5) select * from member where firstname = 'AA' + 'KG' declare @name char(4) set @name = 'AKKG' select * from member where firstname < @name Not all functions can be used in SARGs. select * from charge where charge_amt < 2*2 select * from charge where charge_amt < sqrt(16) Compare these queries to ones using = instead of <. With =, the optimizer can use the density information to come up with a good row estimate, even if it’s not going to actually perform the function’s calculations. A filter with a variable is usually a SARG The issue is, can the optimizer come up with useful costing information? A filter with a variable is not a SARG if the variable is of a different datatype, and the column must be converted to the variable’s datatype For more information, see the following Knowledge Base article: Q198625 Enter Title of KB Article Here Use credit go CREATE TABLE [member2] ( [member_no] [smallint] NOT NULL , [lastname] [shortstring] NOT NULL , [firstname] [shortstring] NOT NULL , [middleinitial] [letter] NULL , [street] [shortstring] NOT NULL , [city] [shortstring] NOT NULL , [state_prov] [statecode] NOT NULL , [country] [countrycode] NOT NULL , [mail_code] [mailcode] NOT NULL ) GO insert into member2 select member_no, lastname, firstname, middleinitial, street, city, state_prov, country, mail_code from member alter table member2 add constraint pk_member2 primary key clustered (lastname, member_no, firstname, country) declare @id int set @id = 47 update member2 set city = city + ' City', state_prov = state_prov + ' State' where lastname = 'Barr' and member_no = @id and firstname = 'URQYJBFVRRPWKVW' and country = 'USA' These queries don’t have SARGs, and a table scan will be done: select * from member where substring(lastname, 1,2) = ‘BA’ Some non-SARGs can be converted select * from member where lastname like ‘ba%’ In some cases, you can rewrite your query to turn a non-SARG into a SARG; for example, you can rewrite the substring query above and the LIKE query that follows it. Join Order and Types of Joins Join Order and Strategy Is Determined By the Optimizer The execution plan output will display the join order from top to bottom; i.e. the table listed on top is the first one accessed in a join. You can override the optimizer’s join order decision in two ways:  OPTION (FORCE ORDER) applies to one query  SET FORCEPLAN ON applies to entire session, until set OFF If either of these options is used, the join order is determined by the order the tables are listed in the query’s FROM clause, and no optimizer on JOIN ORDER is done. Forcing the JOIN order may force a particular join strategy. For example, in most outer join operations, the outer table is processed first, and a nested loops join is done. However, if you force the inner table to be accessed first, a merge join will need to be done. Compare the query plan for this query with and without the FORCE ORDER hint: select * from titles right join publishers on titles.pub_id = publishers.pub_id -- OPTION (FORCE ORDER) Nested Loop Join A nested iteration is when the query optimizer constructs a set of nested loops, and the result set grows as it progresses through the rows. The query optimizer performs the following steps. 1. Finds a row from the first table. 2. Uses that row to scan the next table. 3. Uses the result of the previous table to scan the next table. Evaluating Join Combinations The query optimizer automatically evaluates at least four or more possible join combinations, even if those combinations are not specified in the join predicate. You do not have to add redundant clauses. The query optimizer balances the cost and uses statistics to determine the number of join combinations that it evaluates. Evaluating every possible join combination is inefficient and costly. Evaluating Cost of Query Performance When the query optimizer performs a nested join, you should be aware that certain costs are incurred. Nested loop joins are far superior to both merge joins and hash joins when executing small transactions, such as those affecting only a small set of rows. The query optimizer:  Uses nested loop joins if the outer input is quite small and the inner input is indexed and quite large.  Uses the smaller input as the outer table.  Requires that a useful index exist on the join predicate for the inner table.  Always uses a nested loop join strategy if the join operation uses an operator other than an equality operator. Merge Joins The columns of the join conditions are used as inputs to process a merge join. SQL Server performs the following steps when using a merge join strategy: 1. Gets the first input values from each input set. 2. Compares input values. 3. Performs a merge algorithm. • If the input values are equal, the rows are returned. • If the input values are not equal, the lower value is discarded, and the next input value from that input is used for the next comparison. 4. Repeats the process until all of the rows from one of the input sets have been processed. 5. Evaluates any remaining search conditions in the query and returns only rows that qualify. Note Only one pass per input is done. The merge join operation ends after all of the input values of one input have been evaluated. The remaining values from the other input are not processed. Requires That Joined Columns Are Sorted If you execute a query with join operations, and the joined columns are in sorted order, the query optimizer processes the query by using a merge join strategy. A merge join is very efficient because the columns are already sorted, and it requires fewer page I/O. Evaluates Sorted Values For the query optimizer to use the merge join, the inputs must be sorted. The query optimizer evaluates sorted values in the following order: 1. Uses an existing index tree (most typical). The query optimizer can use the index tree from a clustered index or a covered nonclustered index. 2. Leverages sort operations that the GROUP BY, ORDER BY, and CUBE clauses use. The sorting operation only has to be performed once. 3. Performs its own sort operation in which a SORT operator is displayed when graphically viewing the execution plan. The query optimizer does this very rarely. Performance Considerations Consider the following facts about the query optimizer's use of the merge join:  SQL Server performs a merge join for all types of join operations (except cross join or full join operations), including UNION operations.  A merge join operation may be a one-to-one, one-to-many, or many-to-many operation. If the merge join is a many-to-many operation, SQL Server uses a temporary table to store the rows. If duplicate values from each input exist, one of the inputs rewinds to the start of the duplicates as each duplicate value from the other input is processed.  Query performance for a merge join is very fast, but the cost can be high if the query optimizer must perform its own sort operation. If the data volume is large and the desired data can be obtained presorted from existing Balanced-Tree (B-Tree) indexes, merge join is often the fastest join algorithm.  A merge join is typically used if the two join inputs have a large amount of data and are sorted on their join columns (for example, if the join inputs were obtained by scanning sorted indexes).  Merge join operations can only be performed with an equality operator in the join predicate. Hashing is a strategy for dividing data into equal sets of a manageable size based on a given property or characteristic. The grouped data can then be used to determine whether a particular data item matches an existing value. Note Duplicate data or ranges of data are not useful for hash joins because the data is not organized together or in order. When a Hash Join Is Used The query optimizer uses a hash join option when it estimates that it is more efficient than processing queries by using a nested loop or merge join. It typically uses a hash join when an index does not exist or when existing indexes are not useful. Assigns a Build and Probe Input The query optimizer assigns a build and probe input. If the query optimizer incorrectly assigns the build and probe input (this may occur because of imprecise density estimates), it reverses them dynamically. The ability to change input roles dynamically is called role reversal. Build input consists of the column values from a table with the lowest number of rows. Build input creates a hash table in memory to store these values. The hash bucket is a storage place in the hash table in which each row of the build input is inserted. Rows from one of the join tables are placed into the hash bucket where the hash key value of the row matches the hash key value of the bucket. Hash buckets are stored as a linked list and only contain the columns that are needed for the query. A hash table contains hash buckets. The hash table is created from the build input. Probe input consists of the column values from the table with the most rows. Probe input is what the build input checks to find a match in the hash buckets. Note The query optimizer uses column or index statistics to help determine which input is the smaller of the two. Processing a Hash Join The following list is a simplified description of how the query optimizer processes a hash join. It is not intended to be comprehensive because the algorithm is very complex. SQL Server: 1. Reads the probe input. Each probe input is processed one row at a time. 2. Performs the hash algorithm against each probe input and generates a hash key value. 3. Finds the hash bucket that matches the hash key value. 4. Accesses the hash bucket and looks for the matching row. 5. Returns the row if a match is found. Performance Considerations Consider the following facts about the hash joins that the query optimizer uses:  Similar to merge joins, a hash join is very efficient, because it uses hash buckets, which are like a dynamic index but with less overhead for combining rows.  Hash joins can be performed for all types of join operations (except cross join operations), including UNION and DIFFERENCE operations.  A hash operator can remove duplicates and group data, such as SUM (salary) GROUP BY department. The query optimizer uses only one input for both the build and probe roles.  If join inputs are large and are of similar size, the performance of a hash join operation is similar to a merge join with prior sorting. However, if the size of the join inputs is significantly different, the performance of a hash join is often much faster.  Hash joins can process large, unsorted, non-indexed inputs efficiently. Hash joins are useful in complex queries because the intermediate results: • Are not indexed (unless explicitly saved to disk and then indexed). • Are often not sorted for the next operation in the execution plan.  The query optimizer can identify incorrect estimates and make corrections dynamically to process the query more efficiently.  A hash join reduces the need for database denormalization. Denormalization is typically used to achieve better performance by reducing join operations despite redundancy, such as inconsistent updates. Hash joins give you the option to vertically partition your data as part of your physical database design. Vertical partitioning represents groups of columns from a single table in separate files or indexes. Subquery Performance Joins Are Not Inherently Better Than Subqueries Here is an example showing three different ways to update a table, using a second table for lookup purposes. The first uses a JOIN with the update, the second uses a regular introduced with IN, and the third uses a correlated subquery. All three yield nearly identical performance. Note Note that performance comparisons cannot just be made based on I/Os. With HASHING and MERGING techniques, the number of reads may be the same for two queries, yet one may take a lot longer and use more memory resources. Also, always be sure to monitor statistics time. Suppose you want to add a 5 percent discount to order items in the Order Details table for which the supplier is Exotic Liquids, whose supplierid is 1. -- JOIN solution BEGIN TRAN UPDATE OD SET discount = discount + 0.05 FROM [Order Details] AS OD JOIN Products AS P ON OD.productid = P.productid WHERE supplierid = 1 ROLLBACK TRAN -- Regular subquery solution BEGIN TRAN UPDATE [Order Details] SET discount = discount + 0.05 WHERE productid IN (SELECT productid FROM Products WHERE supplierid = 1) ROLLBACK TRAN -- Correlated Subquery Solution BEGIN TRAN UPDATE [Order Details] SET discount = discount + 0.05 WHERE EXISTS(SELECT supplierid FROM Products WHERE [Order Details].productid = Products.productid AND supplierid = 1) ROLLBACK TRAN Internally, Your Join May Be Rewritten SQL Server’s query processor had many different ways of resolving your JOIN expressions. Subqueries may be converted to a JOIN with an implied distinct, which may result in a logical operator of SEMI JOIN. Compare the plans of the first two queries: USE credit select member_no from member where member_no in (select member_no from charge) select distinct m.member_no from member m join charge c on m.member_no = c.member_no The second query uses a HASH MATCH as the final step to remove the duplicates. The first query only had to do a semi join. For these queries, although the I/O values are the same, the first query (with the subquery) runs much faster (almost twice as fast). Another similar looking join is
大数据实战Demo系统-MaxCompute数据仓库数据转换实践.zip
最新发布
03-22
《大数据实战Demo系统-MaxCompute数据仓库数据转换实践》是一份专业的大数据解决方案文档，它详细阐述了如何利用阿里云的MaxCompute服务进行高效的数据转换操作。MaxCompute是一个强大的数据仓库服务，提供了海量...
must appear in the GROUP BY clause or be used in an aggregate function
ThomasXiao2013的博客
11-09 2万+
今天在分组统计的时候pgsql报错 must appear in the GROUP BY clause or be used in an aggregate function，在mysql里面是可以的，但是pgsql报错，我去stackoverflow查询了一下，发现有人遇到过和我一样的问题，这是pgsql一个常见的聚合问题，在SQL3标准以前，选择显示的字段必须出现在在 GROUP ...
column " " must appear in the GROUP BY clause or be used in an aggregate function
weixin_42612033的博客
10-30 2万+
用了聚集函数 前面要select的列必须出现在聚集函数或者groupby里，不然它不知道怎么分配 查询大于平均WEIGHT的零件，列出他们的供应商代码（SNO），零件代码（PNO），工程代码（JNO），供应数量（QTY） insert into P(Pno,Pname,color,weight) values ('P1','螺母','红',12), ('P2','螺栓','绿',17), ('P3...
Dataworks常见错误记录
编程笔记
05-24 7288
文章目录一、Semantic analysis exception - column xxx cannot be resolved二、Semantic analysis exception - column reference @cte_1621825676877.id is ambiguous 一、Semantic analysis exception - column xxx cannot be resolved 该列无法解析。意思就是源表中找不到对应的字段。 二、Semantic analysis e
MaxCompute常见问题总结以及排查方案
weixin_34348174的博客
03-07 1721
计费相关 存储计费：按照存储在 MaxCompute 的数据的容量大小进行阶梯计费。 计算计费：MaxCompute 分按量后付费和按 CU 预付费两种计算计费方式。 按量后付费：按量后付费方式针对 SQL 任务和 MapReduce 任务进行计费。 SQL 任务按量后付费：即 SQL 任务按 I/O 后付费。 MapReduce 按量后付费：即 Map...
查询ODPS数据报ORDER BY must be used with a LIMIT clause, please set odps.sql.validate.orderby.limit=false
qq_43265673的博客
02-02 3127
在使用Java JDBC查询ODPS上的数据时，若使用了Order By，会遇到如下错误： failed: ODPS-0130071:[1,307] Semantic analysis exception - ORDER BY must be used with a LIMIT clause, please set odps.sql.validate.orderby.limit=false to use it. 出错原因：没有加Limit，添加LIMIT 10000（limit的值根据业务自由设置）后
【MaxCompute】学习笔记操作表sql
Wan_Yuan的博客
12-23 1635
创建表 命令格式： # 建表 CREATE TABLE [IF NOT EXISTS] table_name [(col_name data_type [COMMENT col_comment], ...)] [COMMENT table_comment] [PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)] [AS select_statement] # 复制表 CREATE TABLE [IF NOT EXIS
HiveSQL常见问题总结
lyxuan1208的博客
06-01 2319
1.column repeated in creation: member_id 在所有之前的select里，对应字段如果有名字一样的，必须加表名防混淆 2.去重（完全重复的行） select distinct * from ( ） 如果是要删除表中存在的重复记录，那就逻辑处理，如下： select Test from Table group by Test having cou
spark大数据-structured streaming
06-06
Spark大数据中的Structured Streaming是一种基于Spark SQL引擎的流处理框架，它可以将流数据视为一张表，实现流数据的实时处理和分析。Structured Streaming支持各种数据源，包括Kafka、Flume、HDFS等，同时也支持各种输出方式，如控制台输出、文件输出、Kafka输出等。Structured Streaming还提供了丰富的API和函数库，方便用户进行数据处理和转换。相比传统的流处理框架，Structured Streaming具有更高的容错性和可靠性，能够保证数据的准确性和一致性。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值