Recovering Oracle Block corruption

Occurence of Problem:
While trying to login to our application running on Oracle 10g R2 (10.2.0.4) we received the following error:

ORA-01115: IO error reading block from file 1 (block # 94697)
ORA-01110: data file 1: '(...)\SYSTEM01.DBF'
ORA-27091: unable to queue I/O
ORA-27070: async read/write failed
OSD-04006: ReadFile() failure, unable to read from file
O/S-Error: (OS 23) Data error (cyclic redundancy check).

Investigation:

• Oracle's Metalink pointed out that the problem was a block corruption on the disk.


• It is a 500GB Seagate SATA disk. The Event Viewer of Windows Server 2008 also gave the 'disk' block corrupt error.


• A DBVERIFY was done on the corrupted datafile and it returned an O/S Error 0RA-01115.


• A complete Check Disk was run on the disk and the corrupted block were fixed.


• DBVERIFY now gave the following error for the datafile
  

  
  Corrupt block relative dba: 0x004171e8 (file 1, block 94696)
  Fractured block found during buffer read
  Data in bad block:
  type: 6 format: 2 rdba: 0x004171e8
  last change scn: 0x0000.01c61503 seq: 0x1 flg: 0x06
  spare1: 0x0 spare2: 0x0 spare3: 0x0
  consistency value in tail: 0x00000000
  check value in block header: 0xa06b
  computed block checksum: 0xdfa7
  Reread of rdba: 0x004171e8 (file 1, block 94696) found same corrupted data
  

  
  Same error also appears in the alert.log file


• Attempting to login into the application gave a block corruption error in Oracle in the datafile.
  

  
  ORA-01578: ORACLE data block corrupted (file # 1, block # 94696)
  ORA-01110: data file 1: '(...)\SYSTEM01.DBF'
  

  
  Do note that it is the previous block ID from last error message


• Alert.log contained the following information
  

  
  Corrupt Block Found
  TSN = 0, TSNAME = SYSTEM
  RFN = 1, BLK = 94696, RDBA = 4289000
  OBJN = 331, OBJD = 331, OBJECT = I_PROCEDUREPLSQL$, SUBOBJECT =
  SEGMENT OWNER = SYS, SEGMENT TYPE = Index Segment
  

  
  

Resolution:
Refer to Oracle's documentation on Detecting and Repairing Data Block Corruption.

1. Logon to SYS user and create a REPAIR Table:
   

   
   BEGIN
   DBMS_REPAIR.ADMIN_TABLES (
   TABLE_NAME => 'REPAIR_TABLE',
   TABLE_TYPE => dbms_repair.repair_table,
   ACTION => dbms_repair.create_action,
   TABLESPACE => 'USERS');
   END;
   



2. Check the Object for corruption
   

   
   DECLARE num_corrupt INT;
   BEGIN
   num_corrupt := 0;
   DBMS_REPAIR.CHECK_OBJECT (
   SCHEMA_NAME => 'SYS',
   OBJECT_NAME => 'I_PROCEDUREPLSQL$',
   OBJECT_TYPE => dbms_repair.index_object,
   REPAIR_TABLE_NAME => 'REPAIR_TABLE',
   CORRUPT_COUNT => num_corrupt);
   DBMS_OUTPUT.PUT_LINE('number corrupt: ' || TO_CHAR (num_corrupt));
   END;
   

   
   Note: If parameter Object Type is ignored, it takes the default as TABLE_OBJECT and the query will return a table or view not found error. The object turned out to be an INDEX OBJECT.


3. Command - "select * from repair_object" will show the corrupt blocks and associated information


4. Fix the errors using the following code:
   

   
   DECLARE num_fix INT;
   BEGIN
   num_fix := 0;
   DBMS_REPAIR.FIX_CORRUPT_BLOCKS (
   SCHEMA_NAME => 'SYS',
   OBJECT_NAME=> 'I_PROCEDUREPLSQL$',
   OBJECT_TYPE => dbms_repair.INDEX_OBJECT,
   REPAIR_TABLE_NAME => 'REPAIR_TABLE',
   FIX_COUNT=> num_fix);
   DBMS_OUTPUT.PUT_LINE('num fix: ' || TO_CHAR(num_fix));
   END;
   

   
   num fix: 0 was returned.
   Corrupt block could not be fixed


5. It was a UNIQUE index on procedures. The index was dropped!


6. Trying to recreate the index failed due to duplicate records


7. From the object ID, the problematic procedure (from our application) was located and dropped.


8. Index creation was successful.


9. Application procedure which was dropped was re-created.


10. Login to the application was successful.


11. DB Verify still gives the same error informing that a block has been marked as corrupt and is not usable.

Using AWE to improve Oracle performance

Problem:
Hardware:

1. 4 x Dual core Intel Xeon processors


2. 12 GB of physical RAM


3. 2 x 74 GB SATA Disks configured as RAID-0 (RAID-1 removed to increase performance. Data security done using archiving and automated backups).

Despite having a high performance configuration, Oracle Database 10g R2 (Version 10.2.0.4) was reporting problems with speed. Typical user complaints were as follows

• Startup time for login screen is high.


• Transition between screens is slow.


• Execution of programs within a screen is fast.


• Time lag (Freeze) experienced (anything between 0.5 second to 5 seconds) while traversing between multiple records in one screen.

Analysis:
The Oracle Database was tuned manually. All the values for Pool size and SGA, PGA targets were provided manually in the SPFILE during installation.
Some of them were as follows:

• _db_cache_size=989855744


• _large_pool_size=25165824


• java_pool_size=512165824


• shared_pool_size=233554432


• sga_max_size=2500016000


• sga_target=2500016000


• pga_aggregate_target=262144000

The /3gb switch was turned on for the Operating system. oracle.exe process was able to acquire upto 2.5 GB physical memory. But the performance problems continued. The complete 12GB of physical RAM was also unutilised most of the time.

Solution:
Windows Address Windowing Extension (AWE) was used to increase the Oracle Data Buffer size. The Windows AWE is actually a set of extensions that allows applications like Oracle Database to manipulate physical memory greater than 4GB on 32-bit Operating systems. AWE allows fast memory management.

How To:
To use the Windows AWE, the following was done.

1. Turn on /PAE (Physical Address Extension) and /3gb switch on the Operating System so that more than 2GB of addressable space is available to applications.


2. Set the parameter AWE_WINDOW_MEMORY to a suitable value in the registry.
   Parameter is set in HKEY_LOCAL_MACHINE\Software\Oracle\(Oracle_Home)
   Value used: 536870912


3. In the init.ora set USE_INDIRECT_DATA_BUFFERS=TRUE


4. In the init.ora set DB_BLOCK_BUFFERS to a suitable value.
   Value used: 850000


5. The above should be sufficient for most databases. But we use poi Java utilities to import/export data to excel that requires a minimum of 500 MB Java Pool size.
   Hence, in init.ora JAVA_POOL_SIZE=512165824


6. An error was encountered on starting up the database that said that there was insufficient Streams Pool size. For which in init.ora, we set
   STREAMS_POOL_SIZE=100M

Note: Oracle's Metalink note - 225349.1 has a detailed description on using AWE and the calculations needed to arrive at appropriate values for AWE_WINDOW_MEMORY and DB_BLOCK_BUFFERS.

On starting up the database, we can see that Oracle acquired upto 9GB of physical memory and all the performance related problems from users were addressed.

Problems encountered:

• If the DB_BLOCK_BUFFERS or AWE_WINDOW_MEMORY value is incorrect, we get an
  ORA-27102: out of memory Error followed by
  O/S-Error: (OS 8) Not enough storage is available to process this command

The init.ora used is attached herewith:

_db_block_lru_latches=32
audit_file_dest='C:\admin\(db_name)\adump'
background_dump_dest='C:\admin\(db_name)\bdump'
compatible='10.2.0.3.0'
control_files='D:\(db_name)\CONTROLFILE\(controlfilename).CTL'
core_dump_dest='C:\admin\(db_name)\cdump'
db_block_buffers=850000
db_block_size=8192
db_create_file_dest='D:\(db_name)'
db_domain=''
db_file_multiblock_read_count=16
db_name=(db_name)
dispatchers='(protocol=TCP)'
java_pool_size=512165824
job_queue_processes=10
log_buffer=8388608
open_cursors=3000
processes=150
remote_login_passwordfile='EXCLUSIVE'
shared_pool_size=1468006400
shared_servers=5
streams_pool_size=100M
undo_management='AUTO'
undo_tablespace='UNDOTBS1'
use_indirect_data_buffers=TRUE
user_dump_dest='C:\admin\(db_name)\udump'
log_archive_format=Log_%s_%t_%r.arc
log_archive_dest='D:\(db_name)\ARCHIVE'

Recovering Database from Ora 01194 error

Problem:
The Oracle Database (10.2.0.4) instance would start, database would get mounted, but it would not shift to open state prompting an error:

ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: [Database File Location]

Actions performed before problem

1. Attempt was being made to put the database into Archive mode. But the command "alter database archivelog;" would give an error mentioning some database file was corrupt. Recovery had to be performed on the file. (Error number was not noted)


2. Windows update installation required a restart of the Server. Restart was done without explicitly shutting the Oracle Database down.


3. To put the database in Archive Mode, Shutdown abort was done once. (Instead of normal shutdown or shutdown immediate)

Actions performed to recover database

We first tried,

recover database until cancel;

The command then suggested certain archive files in the archive destination mentioned in our init.ora. However, since Archiving was never turned on earlier, those files did not exist. The command failed with an OS Error which said that the file was not found.

We removed the archive log parameters from init.ora and tried the same command

recover database until cancel;

The command suggested Archive files in the folder [Oracle-Home]/RDBMS which did not exist either.

Recover Database from backup control file was attempted

recover database using backup controlfile until cancel;

But we kept getting the same error as we did not have any archive log file.

In all the above cases, we could not open the database using command.

alter database open resetlogs;

as it gave the error that the SYSTEM.DBF still needed more recovery.

We tried searching Oracle's Metalink support for the problem and then tried to ignore the suggestions made by recovery. Type CANCEL when file is suggested. Even that failed.

We finally decided to point to the current redo log files of the database to check if the database could be recovered and that worked.

recover database using backup controlfile until cancel;

Oracle suggests filename. Do not click enter, instead type the location of the redo log file. Starting from Redo log file 1, then 2 and then the 3rd until all recovery is complete.
Database recovered.