一张6亿条数据表引发的事故

最新推荐文章于 2022-08-26 10:18:33 发布

zbdba

最新推荐文章于 2022-08-26 10:18:33 发布

阅读量2.3k

点赞数 1

分类专栏： Oracle Performance 文章标签： oracle sql tuning 绑定变量窥探 dbms_shared_pool

本文链接：https://blog.csdn.net/zbdba/article/details/47317909

版权

Oracle Performance 专栏收录该内容

7 篇文章 0 订阅

订阅专栏

业务人员告诉我某系统磁盘IO持续高达300MB/s，系统平台为AIX，遂 topas 查看果然如此。

用下面脚本到Oracle数据库中看了一下：

SELECT Disk_Reads DiskReads, Executions, SQL_ID, SQL_Text SQLText, 
   SQL_FullText SQLFullText 
FROM
(
   SELECT Disk_Reads, Executions, SQL_ID, LTRIM(SQL_Text) SQL_Text, 
      SQL_FullText, Operation, Options, 
      Row_Number() OVER 
         (Partition By sql_text ORDER BY Disk_Reads * Executions DESC) 
         KeepHighSQL
   FROM
   (
       SELECT Avg(Disk_Reads) OVER (Partition By sql_text) Disk_Reads, 
          Max(Executions) OVER (Partition By sql_text) Executions, 
          t.SQL_ID, sql_text, sql_fulltext, p.operation,p.options
       FROM v$sql t, v$sql_plan p
       WHERE t.hash_value=p.hash_value AND p.operation='TABLE ACCESS' 
       AND p.options='FULL' AND p.object_owner NOT IN ('SYS','SYSTEM')
       AND t.Executions > 1
   ) 
   ORDER BY DISK_READS * EXECUTIONS DESC
)
WHERE KeepHighSQL = 1
AND rownum <=5;

这里当时没有保存记录，总之跟后来AWR收集的是一样的sql，如下所示。

可以看到第一个sql的物理读非常高。遂看了一下执行计划

SYS@zbdba>select * from table(dbms_xplan.display_cursor('54043712',null,'advanced'));

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
HASH_VALUE  54043712, child number 0
------------------------------------
SELECT D.MODEL_ID, D.OBJ_ID, D.OBJ_TYPE, D.DATA_TYPE, D.DATA_DATE, D.DATA_FROM_DATE, D.DATA_TO_DATE,
D.DATA_FLAG  FROM zbdba1 D, zbdba2 C  WHERE D.MODEL_ID = C.MODEL_ID AND C.COLLECT_ID =
:COLLECT_ID        AND D.DATA_DATE = :DATA_DATE AND D.DATA_TYPE = :DATA_TYPE AND D.VALUE_FLAG = 0

Plan hash value: 1780662521

-------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                    | Name                     | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
-------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |                          |       |       |     9 (100)|          |       |       |
|   1 |  NESTED LOOPS                |                          |     1 |    61 |     9   (0)| 00:00:01 |       |       |
|   2 |   PARTITION RANGE SINGLE     |                          |     1 |    51 |     8   (0)| 00:00:01 |   KEY |   KEY |
|*  3 |    TABLE ACCESS FULL         | zbdba1 |     1 |    51 |     8   (0)| 00:00:01 |   KEY |   KEY |
|*  4 |   TABLE ACCESS BY INDEX ROWID| zbdba2             |     1 |    10 |     1   (0)| 00:00:01 |       |       |
|*  5 |    INDEX UNIQUE SCAN         | PK_zbdba2          |     1 |       |     0   (0)|          |       |       |
-------------------------------------------------------------------------------------------------------------------------

Query Block Name / Object Alias (identified by operation id):
-------------------------------------------------------------

   1 - SEL$1
   3 - SEL$1 / D@SEL$1
   4 - SEL$1 / C@SEL$1
   5 - SEL$1 / C@SEL$1

Outline Data
-------------

  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('10.2.0.5')
      ALL_ROWS
      OUTLINE_LEAF(@"SEL$1")
      FULL(@"SEL$1" "D"@"SEL$1")
      INDEX_RS_ASC(@"SEL$1" "C"@"SEL$1" ("zbdba2"."MODEL_ID"))
      LEADING(@"SEL$1" "D"@"SEL$1" "C"@"SEL$1")
      USE_NL(@"SEL$1" "C"@"SEL$1")
      END_OUTLINE_DATA
  */

Peeked Binds (identified by position):
--------------------------------------

   1 - :COLLECT_ID (VARCHAR2(30), CSID=852): '70350'
   2 - :DATA_DATE (DATE): 06/01/15 00:00:00
   3 - :DATA_TYPE (VARCHAR2(30), CSID=852): '02'

Predicate Information (identified by operation id):
---------------------------------------------------

   3 - filter(("D"."DATA_TYPE"=:DATA_TYPE AND "D"."DATA_DATE"=:DATA_DATE AND "D"."VALUE_FLAG"=0))
   4 - filter("C"."COLLECT_ID"=TO_NUMBER(:COLLECT_ID))
   5 - access("D"."MODEL_ID"="C"."MODEL_ID")

Column Projection Information (identified by operation id):
-----------------------------------------------------------

   1 - "D"."MODEL_ID"[NUMBER,22], "D"."OBJ_ID"[NUMBER,22], "D"."OBJ_TYPE"[VARCHAR2,2],
       "D"."DATA_TYPE"[VARCHAR2,2], "D"."DATA_DATE"[DATE,7], "D"."DATA_FROM_DATE"[DATE,7], "D"."DATA_TO_DATE"[DATE,7],
       "D"."DATA_FLAG"[NUMBER,22]
   2 - "D"."MODEL_ID"[NUMBER,22], "D"."OBJ_ID"[NUMBER,22], "D"."OBJ_TYPE"[VARCHAR2,2],
       "D"."DATA_TYPE"[VARCHAR2,2], "D"."DATA_DATE"[DATE,7], "D"."DATA_FROM_DATE"[DATE,7], "D"."DATA_TO_DATE"[DATE,7],
       "D"."DATA_FLAG"[NUMBER,22]
   3 - "D"."MODEL_ID"[NUMBER,22], "D"."OBJ_ID"[NUMBER,22], "D"."OBJ_TYPE"[VARCHAR2,2],
       "D"."DATA_TYPE"[VARCHAR2,2], "D"."DATA_DATE"[DATE,7], "D"."DATA_FROM_DATE"[DATE,7], "D"."DATA_TO_DATE"[DATE,7],
       "D"."DATA_FLAG"[NUMBER,22]
   5 - "C".ROWID[ROWID,10]

HASH_VALUE  54043712, child number 1

SELECT D.MODEL_ID, D.OBJ_ID, D.OBJ_TYPE, D.DATA_TYPE, D.DATA_DATE, D.DATA_FROM_DATE, D.DATA_TO_DATE,
D.DATA_FLAG  FROM zbdba1 D, zbdba2 C  WHERE D.MODEL_ID = C.MODEL_ID AND C.COLLECT_ID =
:COLLECT_ID        AND D.DATA_DATE = :DATA_DATE AND D.DATA_TYPE = :DATA_TYPE AND D.VALUE_FLAG = 0

NOTE: cannot fetch plan for HASH_VALUE: 54043712, CHILD_NUMBER: 1
      Please verify value of HASH_VALUE and CHILD_NUMBER;
      It could also be that the plan is no longer in cursor cache (check v$sql_plan)


81 rows selected.

查看表行数：

SYS@zbdba>select num_rows,last_analyzed from dba_tables where table_name='zbdba1';

  NUM_ROWS LAST_ANALYZE
---------- ------------
659764063 31-JUL-15

SYS@zbdba>select num_rows,last_analyzed from dba_tables where table_name='zbdba2';

  NUM_ROWS LAST_ANALYZE
---------- ------------
     76513 14-JUL-15

explan plan for 该sql查看执行计划：
SYS@zbdba>select * from table(dbms_xplan.display());

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Plan hash value: 2057366878

----------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                          | Name                        | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
----------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                   |                             |     3 |   183 |    11   (0)| 00:00:01 |       |       |
|   1 |  TABLE ACCESS BY GLOBAL INDEX ROWID| zbdba1    |     1 |    51 |     4   (0)| 00:00:01 | ROWID | ROWID |
|   2 |   NESTED LOOPS                     |                             |     3 |   183 |    11   (0)| 00:00:01 |       |       |
|   3 |    TABLE ACCESS BY INDEX ROWID     | zbdba2                |     2 |    20 |     4   (0)| 00:00:01 |       |       |
|*  4 |     INDEX RANGE SCAN               | IDX_zbdba2COLLECTFLAG |     2 |       |     2   (0)| 00:00:01 |       |       |
|*  5 |    INDEX RANGE SCAN                | IDX_C_COLLECT_MD_001        |     1 |       |     3   (0)| 00:00:01 |       |       |
----------------------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   4 - access("C"."COLLECT_ID"=TO_NUMBER(:COLLECT_ID))
   5 - access("D"."MODEL_ID"="C"."MODEL_ID" AND "D"."DATA_DATE"=:DATA_DATE AND "D"."DATA_TYPE"=:DATA_TYPE AND
              "D"."VALUE_FLAG"=0)

19 rows selected.

发现走了索引，但是当时为什么没有走索引呢。并且在二节点走了全表扫描，一节点走的是索引范围扫描。

该oracle数据库的版本为10.2.0.5，在11g之前没有引入ACS（Adaptive Cursor Sharing），所以这里CBO在第一次进行硬解析的时候才会窥视变量的值，并且生成执行计划，之后一直使用相同的执行计划。
这里我猜想，在2节点。第一次使用绑定变量的时候，CBO认为应该使用全表扫描效率更高，所以在以后一直使用该执行计划。然而在一节点，第一次使用绑定变量的时候，CBO认为走范围扫描效率更高。所以这里导致1节点和2节点的执行计划不一样。

找到原因了，就好办了。业务人员怕数据库负载过大导致宕机，遂叫我把该sql的相关进程（发现30个进程）全部kill掉。kill完进程后磁盘IO瞬间降到50MB。

还没有完，改sql以后还会有这种选择。我们怎么去避免？
既然了解了CBO的做法，那就触发它再一次去执行一次硬解析获得正确的执行计划。有如下4种方法：
1、alter system flush shared_pool（想跪就跑这个）
2、对相关表做DDL操作
3、重新收集统计信息
4、dbms_shared_pool.purge

前面三种方案对生产系统都影响比较大，所以利用第四种方法。

SQL> select address,hash_value,executionsfrom v$sql where hash_value=54043712
 
ADDRESS          HASH_VALUE EXECUTIONS
 
---------------- ---------- ---------- -----------
0000040229F039E0 54043712          1           

SQL> alter session set events '5614566 trace name context forever';

SQL> exec dbms_shared_pool.purge('0000040229F039E0,54043712','C');

重新利用合适的绑定变量跑出正确的执行计划即可。

如何永久保持不变呢？
加hint，强制走索引。

explain plan for SELECT /*+ index(zbdba1,IDX_C_COLLECT_MD_001)* / D.MODEL_ID,
       D.OBJ_ID,
       D.OBJ_TYPE,
       D.DATA_TYPE,
       D.DATA_DATE,
       D.DATA_FROM_DATE,
       D.DATA_TO_DATE,
       D.DATA_FLAG
  FROM EIC2.zbdba1 D, EIC2.zbdba2 C
WHERE D.MODEL_ID = C.MODEL_ID
   AND C.COLLECT_ID = '70350'
   AND D.DATA_DATE = to_date('06/01/15','MM/DD/YY')
   AND D.DATA_TYPE = '02'
   AND D.VALUE_FLAG = 0;

如果以后该索引有变化，也将会失效。

当然优化没有一劳永逸的事情，针对大小超过50GB，数据量高达6亿条的表还是要定期检查它相关sql的执行计划。