APPLIES TO:
Oracle Database - Standard Edition - Version 11.2.0.2 and laterOracle Database - Enterprise Edition - Version 11.2.0.2 and later
Oracle Database - Personal Edition - Version 11.2.0.2 and later
Information in this document applies to any platform.
SYMPTOMS
The key symptoms in these cases are typically the following:
- DBMS_STATS is being executed in some form
- ORA-20011, ORA-29913 and ORA-29400 Errors are seen, ORA-29913 specifying external function "ODCIEXTTABLEOPEN" signifying that there was an error in the callout by the external function "ODCIEXTTABLEOPEN " while trying to open an external table.
Some specific examples follow:
- When attempting to run GATHER_STATS_JOB the following errors occur:
ORA-20011: Approximate NDV failed:ORA-29913: error in executing ODCIEXTTABLEOPEN callout ORA-29400: data cartridge error
usually accompanied with varying other errors such as:KUP-11024: This external table can only be accessed from within a Data Pump job
- The alert log may also show following errors:
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x5400000000] [PC:0xFFFFFFFF7CB19EFC, memcpy%sun4u-opl()+236] [flags: 0x0, count: 1]
ERROR: Unable to normalize symbol name for the following short stack (at offset 200):
dbgexProcessError()+176<-dbgePostErrorKGE()+1348<-dbkePostKGE_kgsf()+48<-kgeade()+NOTE: With cases where the file was not found in the directory specified raising:KUP-04040: file %s in %s not foundSee:
Document 150737.1 ORA-29913, ORA-29400, KUP-04040 While Selecting from External Table
Document 1290722.1 "KUP-04040: file timezdif.csv in TIMEZDIF_DIR not found " logged by GATHER_STATS_JOB in alert.log after upgrade
Document 1305127.1 KUP-04040: file TRCA_CONTROL.txt in TRCA$STAGE not found - Alternatively you may get text in the trace file like:
*** 2012-09-10 05:00:11.921 DBMS_STATS: GATHER_STATS_JOB: GATHER_TABLE_STATS('"SH"','"SALES_TRANSACTIONS_EXT"','""', ...) DBMS_STATS: ORA-20011: Approximate NDV failed: ORA-29913: error in executing ODCIEXTTABLEOPEN callout ORA-29400: data cartridge error error opening file ../demo/schema/log/ext_1v3.log
CAUSE
The primary cause of this issue is that an OS file for an "external table" existed at some point in time but does not now. However, the database still believes the OS file for the table exists since the dictionary information about the object has not been modified to reflect the change. When DBMS_STATS is run against the table in question, it makes a call out to the external table which fails because the object is not there.
There are many reasons that an external table may not exist including:
-
Temporary Datapump external tables have not been cleaned up properly. The dictionary information should have been dropped when the DataPump jobs completed.
-
An OS file for an External table has been removed without clearing up the corresponding data dictionary information. For example: Oracle Demo Schema Tables such as the external table “SALES_TRANSACTIONS_EXT” may have been removed but the dictionary has not been updated to reflect this. The "SALES_TRANSACTIONS_EXT" table is an external table in the "SH" schema which is one of Demo Schema provided by Oracle.
SOLUTION
Essentially the solution to this issue is to clean up the orphaned dictionary entries. Different solutions are appropriate dependent on the exact nature of the primary cause and these are outlined below:
Temporary Datapump External Table
- Ensure that there are no DataPump jobs running at the same time as the DBMS_STATS job (this is to avoid any potential complications associated with cleaning up at the same time as someting else is running).
- Check and clean up orphaned DataPump jobs. The following article addresses a case where DBMS_WORKLOAD_CAPTURE does not drop external tables (causing ORA-20011 from DBMS_STATS):
Document 10327346.8 Bug 10327346 - DBMS_WORKLOAD_CAPTURE does not drop external tables (causing ORA-20011 from DBMS_STATS)
Both above steps can be done by following Document:
Document 336014.1 How To Cleanup Orphaned DataPump Jobs In DBA_DATAPUMP_JOBS ? - Identify external tables. To do this, Run the following as SYSDBA in SQL*Plus
spool obj.out
set linesize 200 trimspool on
set pagesize 2000
col owner form a30
col created form a25
col last_ddl_time form a25
col object_name form a30
col object_type form a25
select OWNER,OBJECT_NAME,OBJECT_TYPE, status,
to_char(CREATED,'dd-mon-yyyy hh24:mi:ss') created
,to_char(LAST_DDL_TIME , 'dd-mon-yyyy hh24:mi:ss') last_ddl_time
from dba_objects
where object_name like 'ET$%'
/
select owner, TABLE_NAME, DEFAULT_DIRECTORY_NAME, ACCESS_TYPE
from dba_external_tables
order by 1,2
/
spool off
Correlate the information from DBA_OBJECTS and DBA_EXTERNAL TABLES above to identify the temporary external tables that belong to the DataPump. - Drop the temporary external tables that belong to the DataPump. eg:
SQL> drop table system.&1 purge;
Enter value for 1: ET$00654E1E0001
old 1: drop table system.&1 purge
new 1: drop table system.ET$00654E1E0001 purge
Other External Table
With cases where specific External tables (be they Demo Schema tables or other tables) are missing, the process for handling them is much the same and can be resolved by following the procedures below. For example, if the additional error is 'error opening file ../demo/schema/log/ext_1v3.log', then this indicates that there is a problem opening or locating the log file from the demo schema directory. The implication is that the demo tables have not been cleared up correctly:
- Try to locate the files for these tables in their directory.
To confirm if a file is present at the expected location for the external file, the following query might be helpful.
undefine owner
undefine table_pattern
select el.table_name, el.owner, dir.directory_path||'/'||dir.directory_name "path"
from dba_external_locations el
, dba_directories dir
where el.table_name like '%&&table_pattern%'
and el.owner like '%&&owner%'
and el.directory_owner = dir.owner
and el.directory_name = dir.directory_name
order by 1, 2;
It may be that the files still exist but they have just been renamed or re-located. If that is the case you can correct the location to avoid the problem. If the file has been removed then follow either the following steps:
- Lock the statistics on these tables by using the following command:
DBMS_STATS.LOCK_TABLE_STATS ('ownname','tabname');(This step prevents DBMS_STATS from gathering against the missing table)
AND/OR
Remove the dictionary object for the external table. For example if the os file for the "SALES_TRANSACTIONS_EXT" Demo table is missing you would use :DROP TABLE SALES_TRANSACTIONS_EXT;