Tuesday, 10 September 2013

Bug Fixes on RAC Database

How to fix the Bugs on Oracle RAC & Non-RAC Database

This following document describes how you can install the one-off patch to fix bugs on Oracle Database 10g Release 10.2.0.4.0
Now I am going to show you how to fix the ORA-00600: INTERNAL ERROR CODE, ARGUMENTS: [KTFBTGEX-7] bug on RAC environment.
Note:
Before going to apply any patches please read and must be follow the README file from patch file.
Bug Example:
The bug is: ORA-00600: INTERNAL ERROR CODE, ARGUMENTS: [KTFBTGEX-7]
According to the ORA-600 [ktfbtgex-7] can occur against a Locally Managed Tablespace for various specific scenarios.
Cause: Problem can occur if a datafile's end is right on a bitmap block's end.In this case the "first free bit" field may get advanced past the end of the datafile.
Note:
Before apply please ensure that the Oracle Database on which you are installing the patch is Oracle Database 10g Release 10.2.0.4.0 version.
Oracle recommends you to use the latest version of OPatch.
If you do not have the latest version, then follow the instructions outlined in the My Oracle Support note 224346.1
And available at:  https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=224346.1
Pre-Installation Steps:
1.      Download the Bug fix patch (p6324052_10204_Linux-x86) from metalink.
2.      Change the owner of the patch file to oracle user.
$ Chown –R oracle: install p6324052_10204_Linux-x86
3.      Extract the contents of the patch ZIP file to the location.
$ unzip -d p6324052_10204_Linux-x86
$ Cd /6324052
4.      Set the PATH variable to locate the opatch utility.
$ export PATH=$PATH: $ORACLE_HOME/OPatch
5.      Check whether this patch can be applied online or it’s an offline patch.
[oracle@rac1 6324052]$ opatch query -all
Invoking OPatch 10.2.0.4.2
Oracle Interim Patch Installer version 10.2.0.4.2
Copyright (c) 2007, Oracle Corporation.  All rights reserved.
Oracle Home               : /oracle/product/10.2.0/db_1
Central Inventory        : /oracle/product/10.2.0/oraInventory
From                            : /etc/oraInst.loc
OPatch version             : 10.2.0.4.2
OUI version                 : 10.2.0.4.0
OUI location               : /oracle/product/10.2.0/db_1/oui
Log file location: /oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2012-02-26_11-17-30AM.log
--------------------------------------------------------------------------------
 Patch created on 5 Jul 2011, 06:39:04 hrs PST8PDT
 Need to shutdown Oracle instances: true
 Patch is roll-backable: true
 Patch is a rolling patch: true
 Patch has sql related actions: false
 Patch is an online patch: false
 Patch is a portal patch: false
 List of platforms supported:
 46: Linux Intel
 List of bugs to be fixed:
 6324052: PART: NT: ORA-00600: INTERNAL ERROR CODE, ARGUMENTS: [KTFBTGEX-7]
 List of executables affected:
 ORACLE_HOME/bin/oracle
 List of optional components:
 oracle.rdbms:  10.2.0.4.0
 List of optional actions:
 Update /oracle/product/10.2.0/db_1/lib/libserver10.a with /ktfb.o
 cd /oracle/product/10.2.0/db_1/rdbms/lib
 ; make -f ins_rdbms.mk ioracle ORACLE_HOME=/oracle/product/10.2.0/db_1
Note: Review the above query, its states that the oracle instance has to be shutdown.
If the Need to shutdown Oracle instances:  false, no need to down the database instance.
But the above query has given result is: Need to shutdown Oracle instances: true
So we have to shutdown the Oracle Instance,
Note: In my environment I have 2-node Cluster database, so the opatch has to apply one by one.
6.         Ensure that you shut down all the services running from the Oracle Home.
For a Non-RAC environment (Single-Instance):
Shut down all the services running from the Oracle home.
Install the patch in single-instance, follow these steps:
a)      $ Sqlplus /nologg
Sql>connect / as sysdba
Sql> shut Immediate  
Sql> exit
b)      Stop the Listener
$ lsnrctl stop
c)      Now Apply the Optach
[oracle@rac1 6324052]$ opatch apply
d)     Check the status, whether the opatch was successfully applied
 [oracle@rac1 6324052]$ opatch lsinventory
e)      Finally Start the Database Instance and Listener
$ lsnrctl start
$sqlplus / as sysdba
Sql> startup
Sql>exit
Note: The opatch was successfully applied on Single Database Instance.
For a RAC environment:
Shut down all the services (database, ASM, listeners, Nodeapps, and CRS daemons) running from the Oracle home of the node you want to patch. After you patch this node, start the services on this node. Repeat this process for each of the other nodes of the Oracle RAC system.
OPatch is used on only one node at a time.
If this patch is not rolling RAC installable treat it as a non-RAC environment and shut down all services.
Syntax:
·         Shutdown The Oracle Instance on Node 1
·         Apply the Opatch to the Oracle Instance on Node 1
·         Start the Oracle Instance on Node 1
·         Shutdown the Oracle Instance on Node 2
·         Apply the Opatch to the Oracle Instance on Node2
·         Start the oracle Instance on Node2
Install the patch in RAC environment, follow these steps:
a)      Stop the Database Instance, ASM, Nodeapps and CRS daemons on node1
$ srvctl stop instance –d racdb –i racdb1
$ srvctl stop asm –n rac1
$ srvctl stop nodeapps –n rac1
$ crsctl stop crs
b)      Now Apply the Optach
Note:
When OPatch starts, it validates the patch and ensures that there are no conflicts with the software already installed in the ORACLE_HOME of the Oracle Database.
OPatch categorizes conflicts into the following types:
-Conflicts with a patch already applied to the ORACLE_HOME - In this case, stop the patch installation and contact Oracle Support Services.
-Conflicts with a patch already applied to the ORACLE_HOME that is a subset of the patch you are trying to apply.
- In this case, continue with the patch installation because the new patch contains all the fixes from the existing patch in the ORACLE_HOME. The subset patch will automatically be rolled back prior to the installation of the new patch.
[oracle@rac1 6324052]$ opatch apply
(Oracle Home = '/oracle/product/10.2.0/db_1')
Is the local system ready for patching? [y|n]
Y
Running makes for target ioracle
Apply Session adding interim patch '6324052' to inventory
Verifying the update...
Inventory check OK: Patch ID 6324052 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 6324052 are present in Oracle Home.
The local system has been patched.  You can restart Oracle instances on it.
Note:
When the apply finishes on node1, opatch asks for confirmation before going on to apply the patch to node 2.
We must start the Instance1 which has just we patched (node1) and need to shut down the Instance 2 (rac2) as per as opatch recomends.
Patching in rolling mode.
The node 'rac2' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on 'rac2'.
(Oracle Home = '/oracle/product/10.2.0/db_1')
Is the node ready for patching? [y|n]
Note:
Before saying Y.      (Make sure start instance 1 and shut down the instance2)
Start node1 and shut downing node2:
On node1:
$crsctl start crs
$srvctl start nodeapps –n rac1
$srvctl start asm –n rac1
$srvctl start instance –d racdb –i racdb1
$crs_stat –t
On node2:
$srvctl stop instance -d racdb -i racdb2
$srvctl stop asm –n rac2
$srvctl stop nodeapps –n rac2
$crs_relocate ora.racdb.ora      (If database on racdb2 instance, the crs_relocate option will relocate to instance 2)
$crsctl stop crs
Now press Y to continue the rolling patch on instance 2.
(Oracle Home = '/oracle/product/10.2.0/db_1')
Is the node ready for patching? [y|n]
Y
Propagating directories to remote nodes...
Running command on remote node 'rac2':
The node 'rac2' has been patched.  You can restart Oracle instances on it.
There were relinks on remote nodes.
 Remember to check the binary size and timestamp on the nodes ‘rac2’.
The following make commands were invoked on remote nodes:
'cd /oracle/product/10.2.0/db_1/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/oracle/product/10.2.0/db_1
………………………….
c)   Check the whether the patch was successfully applied or not on both nodes:
On node1:
$ opatch lsinventory
Interim patches (2):
Patch 6324052      : applied on Tue Feb 28 08:44:05 AST 2012
Created on 5 Jul 2011, 06:39:04 hrs PST8PDT
Bugs fixed:      6324052
On node2:
$ opatch lsinventory
Interim patches (2):
Patch 6324052      : applied on Tue Feb 28 08:44:05 AST 2012
Created on 5 Jul 2011, 06:39:04 hrs PST8PDT
Bugs fixed:     6324052
Post Installation Steps:
Restart the Databases on both nodes. (for bug: 6324052)
Note:
The Post Installation steps are different for different bug; it depends on the type of bugs. May be some bugs fixes need to run some scripts on database levels.
The below example is not for 6324053 bug
Example:
After you install the patch, reload the packages into the Oracle Database.
On RAC-environment, reload the packages into the RAC database on Each Node.
cd rdbms/admin
run sqlplus as sysdba
sql>@catnomtt.sql
sql>@catnomta.sql
sql>@dbmsmeta.sql
sql>@dbmsmeti.sql ………..
……….. etc…
-- Recompile all invalid objects using utlrp.sql
Sql>@utlrp.sql
The opatch was successfully applied on both nodes without any errors...
Patch Deinstallation Instructions:
1. Make sure to follow the same pre-install steps when deinstalling a patch.  This includes verifying the inventory and shutting down any services running from the ORACLE_HOME / machine before rolling
the patch back.
2. Change to the directory where the patch was unzipped.
cd /6324052
3. Run OPatch to deinstall the patch.
 opatch rollback -id 6324052

No comments: