Index WRH$_SGASTAT_U has status UNUSABLE after upgrade to 12.2

Update 2016-04-26: this is new bug 25954054 – WRH$_SGASTAT_U BECOMING UNUSABLE STATE IN UPGRADED DB.

After upgrading our OMR and AWRW instances from 12.1 to 12.2, our monitoring revealed that index SYS.WRH$_SGASTAT_U was becoming unusable at regular intervals.
The first couple of times, we just rebuilt it in the hope that the problem would just go away(!); of course it didn’t, so I had to do a little bit of investigation.

Reviewing the alert log identified the exact time that the problem had occurred:

Some indexes or index [sub]partitions of table SYS.WRH$_SGASTAT have been marked unusable

Grepping for “SGASTAT” in the trace directory, also revealed that the index being unusable was causing inserts into AWR had been failing:

*** 2017-04-12T12:00:18.338061+08:00
*** SESSION ID:(289.8864) 2017-04-12T12:00:18.338092+08:00
*** CLIENT ID:() 2017-04-12T12:00:18.338103+08:00
*** SERVICE NAME:(SYS$BACKGROUND) 2017-04-12T12:00:18.338108+08:00
*** MODULE NAME:(MMON_SLAVE) 2017-04-12T12:00:18.338119+08:00
*** ACTION NAME:(Auto-Flush Slave Action) 2017-04-12T12:00:18.338125+08:00
*** CLIENT DRIVER:() 2017-04-12T12:00:18.338132+08:00

*** KEUUXS - encountered error: (ORA-01502: index 'SYS.WRH$_SGASTAT_U' or partition of such index is in unusable state
  *** SQLSTR: total-len=464, dump-len=464,
      STR={insert into  wrh$_sgastat  (dbid, per_pdb, con_dbid, snap_id, instance_number, pool, name, bytes)  select distinct  :dbid, 0 per_pdb,  :srcdbid  con_dbid,  :snap_id, :instance_number, pool, name, bytes FROM     (select con_id, pool, name, bytes,             100*(bytes) / (sum(bytes) over (partition by pool)) part_pct        from  v$sgastat      ) sgastat  WHERE (1 = 1)   and part_pct >= 1      or pool is null      or name = 'free memory' or name = 'row cache' }
*** KEWRAFM1: Error=13509 encountered by kewrfteh

As we still had the redo logs for the period of time identified by the alert log, I fired up logminer, and right after a bunch of inserts into WRH$_SGASTAT, I found the following statement:

alter table WRH$_SGASTAT drop partition WRH$_SGASTA_1521710113_17057;

I am guessing this is part of the standard mechanism for purging AWR data based on configured retention settings.
A quick check reveals that of the partitioned AWR base tables, 3 have indexes that are not partitioned:

SQL> SELECT                                                                                                                                                                                                          
  2     partitioned,                                                                                                                                                                                                 
  3     COUNT(*)                                                                                                                                                                                                     
  4  FROM                                                                                                                                                                                                            
  5     dba_indexes                                                                                                                                                                                                  
  6  WHERE                                                                                                                                                                                                           
  7     index_type = 'NORMAL'                                                                                                                                                                                        
  8  AND                                                                                                                                                                                                             
  9     table_name IN (                                                                                                                                                                                              
 10        SELECT                                                                                                                                                                                                    
 11           table_name                                                                                                                                                                                             
 12        FROM                                                                                                                                                                                                      
 13           dba_tables                                                                                                                                                                                             
 14        WHERE                                                                                                                                                                                                     
 15           table_name LIKE 'WRH$\_%' ESCAPE '\'                                                                                                                                                                   
 16        AND                                                                                                                                                                                                       
 17           partitioned = 'YES'                                                                                                                                                                                    
 18     )                                                                                                                                                                                                            
 19  GROUP BY                                                                                                                                                                                                        
 20     partitioned;                                                                                                                                                                                                 
PAR   COUNT(*)                                                                                                                                                                                                       
--- ----------                                                                                                                                                                                                       
NO           3                                                                                                                                                                                                       
YES         35                                                                                                                                                                                                       

SQL> SELECT                                                                                                                                                                                                          
  2     table_owner,                                                                                                                                                                                                 
  3     table_name,                                                                                                                                                                                                  
  4     index_name                                                                                                                                                                                                   
  5  FROM                                                                                                                                                                                                            
  6     dba_indexes                                                                                                                                                                                                  
  7  WHERE                                                                                                                                                                                                           
  8     index_type = 'NORMAL'                                                                                                                                                                                        
  9  AND                                                                                                                                                                                                             
 10     table_name IN (                                                                                                                                                                                              
 11        SELECT                                                                                                                                                                                                    
 12           table_name                                                                                                                                                                                             
 13        FROM                                                                                                                                                                                                      
 14           dba_tables                                                                                                                                                                                             
 15        WHERE                                                                                                                                                                                                     
 16           table_name LIKE 'WRH$\_%' ESCAPE '\'                                                                                                                                                                   
 17        AND                                                                                                                                                                                                       
 18           partitioned = 'YES'                                                                                                                                                                                    
 19     )                                                                                                                                                                                                            
 20  AND partitioned = 'NO';                                                                                                                                                                                         
TABLE_OWNER                    TABLE_NAME                     INDEX_NAME                                                                                                                                             
------------------------------ ------------------------------ ------------------------------                                                                                                                         
SYS                            WRH$_SGASTAT                   WRH$_SGASTAT_U                                                                                                                                         
SYS                            WRH$_CON_SYSMETRIC_HISTORY     WRH$_CON_SYSMET_HISTORY_INDEX                                                                                                                          
SYS                            WRH$_SYSMETRIC_HISTORY         WRH$_SYSMETRIC_HISTORY_INDEX                                                                                                                           

Running the same query on both 11.2 and 12.1 instances reveals no such mis-configured indexes We are following up with Oracle Support, but I would be interested to know if anyone else can check my findings on their 12.2 instances.

Working with Oracle Support we have worked around the issue using the following script:

ALTER TABLE wrh$_sgastat DROP CONSTRAINT wrh$_sgastat_u;

ALTER TABLE wrh$_sgastat ADD CONSTRAINT wrh$_sgastat_u UNIQUE ( dbid,snap_id,instance_number,name,pool,con_dbid )
   TABLESPACE sysaux;

In addition, I believe I have found the bug in file $ORACLE_HOME/rdbms/admin/c1201000.sql at line 3036:

alter table wrh$_sgastat drop constraint wrh$_sgastat_u;

alter table wrh$_sgastat add constraint wrh$_sgastat_u
        unique (dbid, snap_id, instance_number, name, pool, con_dbid, stattype);

TNS-12599: TNS:cryptographic checksum mismatch

Checking the alert log of some of one of the instances we have recently upgraded to 12.2 I noticed the following error message occurring frequently:

NI cryptographic checksum mismatch error: 12599.

TNS for Linux: Version - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version - Production
TCP/IP NT Protocol Adapter for Linux: Version - Production
Time: 06-APR-2017 11:38:05
Tracing to file:
Tns error struct:
ns main err code: 12599

TNS-12599: TNS:cryptographic checksum mismatch
ns secondary err code: 12656
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0

Comparing with entries in listener.log file, I identified that these corresponded to connections from Enterprise Manager. Reproducing the issue was easy, I navigated to the database in Enterprise Manager, went through the menus “Oracle Database->Target Setup->Monitoring Configuration” and clicked on the “Test Connection button”.

Every time the test succeeded, but the message was appended to the alert log.

MOS note 1927120.1 goes into some detail about the error message, including the fact that is is safe to ignore. It should be resolved when Enterprise Manager is updated to include latest JDBC driver.

One option listed to prevent the issue appearing in the alert log is to set DIAG_ADR_ENABLED=OFF in sqlnet.ora (it will be then instead be logged to sqlnet.log).

I found that setting the following two parameters in sqlnet.ora also meant the warning did not occur (setting either parameter by itself was not sufficient).


Upgrade Recovery Catalog to 12cR2 fails and leaves catalog an unusable state

After upgrading our AWR Warehouse instance to 12.2 our standard RMAN scripts were failing because the Recovery Catalog required upgrading. Attempting the upgrade failed, and left the Recovery Catalog unable. Fortunately we were able to restore the catalog from weekly export.

I was able to reproduce the problem by importing into a new schema, below are the results. Note I enable tracing to dig into exactly what was failing:

[oracle@vmhkoemawrbpd1 ~]$ rman catalog TST_RCAT/TST_RCAT debug trace=/tmp/upgrade.trc

Recovery Manager: Release - Production on Tue Mar 21 17:16:31 2017

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.

RMAN-06008: connected to recovery catalog database
RMAN-06186: PL/SQL package TST_RCAT.DBMS_RCVCAT version in RCVCAT database is too old


RMAN-06435: recovery catalog owner is TST_RCAT
RMAN-06442: enter UPGRADE CATALOG command again to confirm catalog upgrade


RMAN-06444: error creating init_grsp_pdb_key
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-06004: ORACLE error from recovery catalog database: ORA-01422: exact fetch returns more than requested number of rows

The following extract from the trace file identifies the failing section of code:

DBGSQL:         RCVCAT> declare
DBGSQL:            cursor grsp_c is
DBGSQL:               select distinct grsp.rowid rid, dbinc.db_key db_key
DBGSQL:                 from grsp, dbinc
DBGSQL:                where grsp.pdb_key is null
DBGSQL:                  and grsp.dbinc_key = dbinc.dbinc_key
DBGSQL:                order by dbinc.db_key;
DBGSQL:            prev_db_key   number := 0;
DBGSQL:            local_pdb_key number;
DBGSQL:         begin
DBGSQL:            for r in grsp_c loop
DBGSQL:              if (r.db_key  prev_db_key) then
DBGSQL:                 prev_db_key := r.db_key;
DBGSQL:                 select pdb_key into local_pdb_key
DBGSQL:                   from pdb
DBGSQL:                  where pdb.con_id in (1, 0);
DBGSQL:              end if;
DBGSQL:              update grsp set pdb_key = local_pdb_key where rowid = r.rid;
DBGSQL:            end loop;
DBGSQL:         end;
DBGSQL:            sqlcode = 1422
DBGSQL:         error: ORA-01422: exact fetch returns more than requested number of rows (krmkosqlerr)

Table PDB seems to hold information about Pluggable DataBases (a feature of multitenant).  In our catalog we had 30 records with con_id equals to 0 or 1.  Obviously the upgrade code was not expecting this.

Attempts to use the partially upgraded catalog now fail due to the mismatch of components within the catalog.

Recovery Manager: Release - Production on Tue Mar 21 17:32:53 2017

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

connected to recovery catalog database
PL/SQL package TST_RCAT.DBMS_RCVCAT version in RCVCAT database is too new
RMAN must be upgraded to version to work with this package
RMAN> backup database;

Starting backup at 17:35:44 21/03/2017
PL/SQL package TST_RCAT.DBMS_RCVCAT version in RCVCAT database is too new
RMAN must be upgraded to version to work with this package
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 03/21/2017 17:35:44
RMAN-03014: implicit resync of recovery catalog failed
RMAN-06429: RCVCAT database is not compatible with this version of RMAN

Oracle support have published a document about this issue:

UPGRADE CATALOG command from 12.1 to 12.2 Fails With RMAN-6004 and ORA-1422 (Doc ID 2252894.1)