1.联通现场状态
时间：2012年8月17日
数据库版本：Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bi
操作系统平台：HP-UX
告警日志： more alert_mdsoss.log
2.定位问题
报错现象：
Fri Aug 17 13:37:39 EAT 2012
ORA-1652: unable to extend temp segment by 128 in tablespace MDSTEMP 显示不能扩展临时段，说明临时表空间已经被使用满了，空间不够。
说明：从metalink上官方解释，没有更多的空闲区分给这个临时段了，可以给表空间添加数据文件的方式来解决此问题，表面上是这样，我们更加的深入了解，是什么原因导致的临时段没有空间了呢，我们都知道临时段是记录排序和数据迁移的，现在深层次问题不是空间不够，过一会再执行sql可能就不报错了。是sql语句不够优化。因为当sql在批量DML操作的时候，会突发性占用大量临时空间排序，就会报临时段不够用，新数据此时不能入库！过一会空间释放后又可以入库了，要想解决此问题就需要sql优化。
The below is from metalink:
Error: ORA-1652
Text: unable to extend temp segment by %s in tablespace %s
------- -----------------------------------------------------------------------
Cause: Failed to allocate an extent for temp segment in tablespace.
Action: Use ALTER TABLESPACE ADD DATAFILE statement to add one or more
files to the tablespace indicated or create the object in another
tablespace.
select * from gnwebbrw12081720; 此时是有数据的，说明空间已经释放了
colfile_name for a35
selectfile_name,file_id,bytes/1024/1024,status,autoextensible TABLESPACE_NAME from DBA_TEMP_FILES;
FILE_NAME FILE_ID BYTES/1024/1024 STATUS TAB
----------------------------------- ---------- --------------- --------- ---
/oradata/mdsoss/temp01.dbf 1 24671 AVAILABLE YES
/oradata/mdsoss/mdstmp.dbf 2 20000 AVAILABLE NO MDSTEMP 不是自动扩展，如果是就没有上述问题了，但我们不建议使用数据文件自动扩展功能，不容易监控。看 24G + 20G 空间是没有问题的，一般都是sql写的不够好导致不必要排序。
3.解决方案
（1）重启实例，7*24 重启实例 smon进程可以释放sort段，但我们的库是不能down的
（2）增加数据文件，我的空间很紧张，不可以
（3）配置合理sort_area大小 已经配置完毕了，现PGA 4G sort_area_size 208M
（4）sql optimization 最佳方案
（5）总结哪些操作会导致临时表空间暴涨呢
什么操作在使用temp
- 索引创建或重建.
- ORDER BY or GROUP BY
- DISTINCT 操作.
- UNION & INTERSECT & MINUS
- Sort-Merge joins.
- Analyze 操作
- 有些异常将会引起temp暴涨
当处理以上操作时候呢，dba需要加倍关注temp使用情况？我们现在来看看谁使用这些临时段。
（5）临时表空间使用情况
select tablespace_name,current_users,total_blocks,used_blocks,free_blocks from v$sort_segment;
TABLESPACE_NAME CURRENT_USERS TOTAL_BLOCKS USED_BLOCKS FREE_BLOCKS
------------------- ------------- ------------ ----------- -----------
TEMP 1 3157760 128 3157632
MDSTEMP 24 2559872 2337152 222720 已经使用了92%
（6）谁在使用这些sort段
select username,session_addr,sqladdr,sqlhash from v$sort_usage;
USERNAME SESSION_ADDR SQLADDR SQLHASH
------------------------------ ---------------- ---------------- ----------
MDSOSS C0000008483ECFB8 C0000008512150B8 3342809064
SABOCOUSR C00000084B405E50 C00000033F867510 141205382
MDSOSS C00000084740E988 C0000008508AB1C0 409467952
MDSOSS C0000008483DE390 C00000033B8914F0 2951877480
MDSOSS C00000084A404460 C0000003404007A0 2584373469
MDSOSS C0000008483F5088 C00000033FA63E18 2245874020
MDSOSS C0000008483FFC48 C00000084D5B5F98 3000467390
MDSOSS C0000008483F5088 C00000033FA63E18 2245874020
MDSOSS C000000852404A60 C00000084DD6F598 1491833069
MDSOSS C0000008483EBA40 C00000084DE28990 1530468420
MDSOSS C0000008483FD158 C00000084E13A648 3459409074
MDSOSS C000000852404A60 C00000084DD6F598 1491833069
MDSOSS C000000852404A60 C00000084DD6F598 1491833069
MDSOSS C000000852404A60 C00000084DD6F598 1491833069
MDSOSS C0000008523DDBC8 C000000850AE75E0 2349095001
MDSOSS C0000008474068B8 C00000084CCC7790 2982079417
MDSOSS C0000008494501E0 C00000085098B400 3836056470
MDSOSS C00000084B3E5B10 C00000084DA9B990 719858768
MDSOSS C0000008523F3348 C00000084D96F830 2434343698
MDSOSS C00000084740D410 C0000008174B3540 2103182003
USERNAME SESSION_ADDR SQLADDR SQLHASH
------------------------------ ---------------- ---------------- ----------
MDSOSS C00000084A3FD908 C00000084C944888 3846639713
MDSOSS C0000008523D5AF8 C00000084DC6BB30 3158920754
MDSOSS C0000008483FA668 C00000032FCDF3D8 1691040305
MDSOSS C00000084943BFD8 C00000084CE4CA70 2036150049
MDSOSS C00000084A4170F0 C00000084D75FF68 2058192145
MDSOSS C0000008483CF768 C00000084DC096B8 3375505524
MDSOSS C000000849452CD0 C00000031E5A32F8 346930689
MDSOSS C000000849433F08 C000000340106C48 2344782277
MDSOSS C0000008484067A0 C000000344414D88 4097260861
MDSOSS C00000084B3C57D0 C000000850BF8528 3690121153
MDSOSS C00000084740D410 C0000008174B3540 2103182003
MDSOSS C0000008523DDBC8 C000000850AE75E0 2349095001
MDSOSS C0000008483EBA40 C00000084DE28990 1530468420
MDSOSS C0000008483FD158 C00000084E13A648 3459409074
MDSOSS C00000084B3CC328 C00000084D2185C8 497018778
MDSOSS C0000008523F73B0 C00000034046B670 3944536657
MDSOSS C00000084B3CC328 C00000084D2185C8 497018778
MDSOSS C00000084B3CC328 C00000084D2185C8 497018778
MDSOSS C0000008473E3A88 C00000084F13BD98 1450489357
MDSOSS C0000008473E2510 C0000008518A3DA0 2446627624
MDSOSS C0000008473E2510 C0000008518A3DA0 2446627624
USERNAME SESSION_ADDR SQLADDR SQLHASH
------------------------------ ---------------- ---------------- ----------
MDSOSS C0000008473E2510 C0000008518A3DA0 2446627624
MDSOSS C0000008523DDBC8 C000000850AE75E0 2349095001
MDSOSS C0000008523DDBC8 C000000850AE75E0 2349095001
MDSOSS C0000008473E2510 C0000008518A3DA0 2446627624
MDSOSS C0000008473E2510 C0000008518A3DA0 2446627624
MDSOSS C00000084B3CC328 C00000084D2185C8 497018778
MDSOSS C00000084A3F98A0 C00000030871D7E0 1117602678
MDSOSS C00000084941A720 C00000084F2EB838 1495689429
MDSOSS C00000084A3F98A0 C00000030871D7E0 1117602678
MDSOSS C00000084A3F98A0 C00000030871D7E0 1117602678
MDSOSS C00000084941A720 C00000084F2EB838 1495689429
MDSOSS C00000084941A720 C00000084F2EB838 1495689429
MDSOSS C00000084B3CC328 C00000084D2185C8 497018778
MDSOSS C00000084941A720 C00000084F2EB838 1495689429
MDSOSS C00000084941A720 C00000084F2EB838 1495689429
MDSOSS C00000084A3F98A0 C00000030871D7E0 1117602678
MDSOSS C00000084A3F98A0 C00000030871D7E0 1117602678
MDSOSS C00000084A3F8328 C000000851A28740 87302580
MDSOSS C00000084A3F8328 C000000851A28740 87302580
MDSOSS C00000084A3F8328 C000000851A28740 87302580
61 rows selected
此时我们知道MDSOSS用户的sql是导致临时段爆满的罪魁祸首
7）找出哪些SQL语句在使用sort段，利用多表关联查询
select se.username,se.sid,su.extents,su.blocks*to_number(rtrim(p.value)) as Space,tablespace,segtype,sql_text from v$sort_usage su,v
$parameter p,v$session se,v$sql s where p.name='db_block_size' and su.session_addr=se.saddr and s.hash_value=su.sqlhash and
s.address=su.sqladdr order by se.username,se.sid;
USERNAME SID EXTENTS SPACE TABLESPACE SEGTYPE SQL_TEXT
MDSOSS 840 1 1048576 MDSTEMP DATA insert into tmp1768202 select * from
TPA_F_EMAIL_SMTP_SUM_5 where first_result
MDSOSS 840 1 1048576 MDSTEMP INDEX insert into tmp1768202 select * from
TPA_F_EMAIL_SMTP_SUM_5 where first_result
MDSOSS 840 1 1048576 MDSTEMP INDEX insert into tmp1768202 select * from
TPA_F_EMAIL_SMTP_SUM_5 where first_result
MDSOSS 840 1 1048576 MDSTEMP DATA insert into tmp1768202 select * from
TPA_F_EMAIL_SMTP_SUM_5 where first_result
MDSOSS 840 1 1048576 MDSTEMP DATA insert into tmp1768202 select * from
TPA_F_EMAIL_SMTP_SUM_5 where first_result
MDSOSS 877 14 14680064 MDSTEMP DATA CREATE GLOBAL TEMPORARY TABLE tmp708304 AS SELECT * from
TPA_S_SP_SUM_5 where
MDSOSS 879 21 22020096 MDSTEMP DATA insert into tmp521803
(ne_id,ne_type,first_result,sum_level,compress_date,regio
MDSOSS 879 1 1048576 MDSTEMP DATA insert into tmp521803
(ne_id,ne_type,first_result,sum_level,compress_date,regio
MDSOSS 879 1 1048576 MDSTEMP INDEX insert into tmp521803
(ne_id,ne_type,first_result,sum_level,compress_date,regio
MDSOSS 879 1 1048576 MDSTEMP DATA insert into tmp521803
(ne_id,ne_type,first_result,sum_level,compress_date,regio
MDSOSS 922 389 407896064 MDSTEMP DATA insert into tmp2489701 select * from GNWEBBRW12081720 s
where capturetime >= to
MDSOSS 946 6 6291456 MDSTEMP DATA select * from dual
MDSOSS 948 417 437256192 MDSTEMP DATA insert into tmp2462001 SELECT s.*,decode(m.flag,1,0,1)
result1_flag FROM HTTP_
MDSOSS 1028 1 1048576 MDSTEMP DATA CREATE GLOBAL TEMPORARY TABLE tmp1804004 AS SELECT * from
TPA_S_SP_SUM_5 where
MDSOSS 1050 28 29360128 MDSTEMP DATA insert into tmp2407302 select substr(imei,1,8)
imei,useragent brand from GnWebb
MDSOSS 1050 434 455081984 MDSTEMP DATA insert into tmp2407302 select substr(imei,1,8)
imei,useragent brand from GnWebb
MDSOSS 1066 1 1048576 MDSTEMP INDEX update tmp1737203 t set apn= nvl(apn,'-1'),lac= nvl
(lac,'-1'),ci= nvl(ci,'-1')
MDSOSS 1066 2 2097152 MDSTEMP DATA update tmp1737203 t set apn= nvl(apn,'-1'),lac= nvl
(lac,'-1'),ci= nvl(ci,'-1')
MDSOSS 1066 1 1048576 MDSTEMP DATA update tmp1737203 t set apn= nvl(apn,'-1'),lac= nvl
(lac,'-1'),ci= nvl(ci,'-1')
MDSOSS 1066 2 2097152 MDSTEMP DATA update tmp1737203 t set apn= nvl(apn,'-1'),lac= nvl
(lac,'-1'),ci= nvl(ci,'-1')
MDSOSS 1066 1 1048576 MDSTEMP INDEX update tmp1737203 t set apn= nvl(apn,'-1'),lac= nvl
(lac,'-1'),ci= nvl(ci,'-1')
MDSOSS 1074 1 1048576 MDSTEMP DATA insert into tmp2336801 SELECT s.*,decode(m.flag,1,0,1)
result1_flag FROM WAP_R
MDSOSS 1074 34 35651584 MDSTEMP DATA insert into tmp2336801 SELECT s.*,decode(m.flag,1,0,1)
result1_flag FROM WAP_R
MDSOSS 1086 4 4194304 MDSTEMP DATA insert into tmp1261403
(ne_id,ne_type,first_result,sum_level,compress_date,regi
MDSOSS 1086 4 4194304 MDSTEMP DATA insert into tmp1261403
(ne_id,ne_type,first_result,sum_level,compress_date,regi
MDSOSS 1086 2 2097152 MDSTEMP INDEX insert into tmp1261403
(ne_id,ne_type,first_result,sum_level,compress_date,regi
MDSOSS 1086 900 943718400 MDSTEMP DATA insert into tmp1261403
(ne_id,ne_type,first_result,sum_level,compress_date,regi
我把占用sort段空间最多的几个SQL列举出来，请看SID：922 948 434 会话ID 不外乎都与gnwebbrw http XDR数据有关，当用gnweb数据插入临时表（数据分析）时产生了大量的排序从而占用大量排序区，此时我们要分析了这种排序是否是有必要的。
（8）排序区分配
排序区域的分配 - 专用服务器分配sort area，排序区域在PGA！ - 共享服务器分配sort area，排序区域在UGA. (UGA在shared_pool中分配).
我们采用的全是dedicated server 模式，pga_aggregate_target参数决定sort_area的大于,这时sort_area应该是pga总内存的5%，我们PGA=4G，
sort_area=4G*5%=204.8M 和我们从spotlight上监控的结果一样。
小结,如何从根本上降低临时表空间的膨胀呢？方法有2个：
1 设置合理的pga或sort_area_size
2 优化引起disk sort的sql语句
最后我把发现占用sort段较多的process kill掉，来临时缓解排序段的使用，提交给研发，调整SQL
selectsid,paddr from v$session where sid=1177; 查询进程地址
select p.spid,se.sid,se.username,se.machine from v$sort_usage su,v$process p,v$session se,v$sql s where se.paddr='C0000008482BF9B0' and p.ADDR='C0000008482BF9B0'; 通过进程地址查询进程号，kill 进程
Oracle@TJGRDB:[/oracle/admin/mdsoss/bdump] kill -9 29944
oracle@TJGRDB:[/oracle/admin/mdsoss/bdump] ps -ef | grep 15072
oracle 10587 8420 1 22:58:58 pts/tb 0:00 grep 15072
oracle 15072 1 250 20:40:04 ? 127:20 oraclemdsoss (LOCAL=NO)
oracle@TJGRDB:[/oracle/admin/mdsoss/bdump] kill -9 15072
oracle@TJGRDB:[/oracle/admin/mdsoss/bdump] ps -ef | grep 23259
oracle 11625 8420 1 23:02:02 pts/tb 0:00 grep 23259
oracle 23259 1 253 21:00:15 ? 53:31 oraclemdsoss (LOCAL=NO)
oracle@TJGRDB:[/oracle/admin/mdsoss/bdump] kill -9 23259
2926
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
