Removing a Node From Oracle10g RAC Cluster

Removing a Node From a 10g RAC Cluster 

The most three important steps that need to be followed are:

a). Remove the Instance using DBCA or Command Line.

b). Remove the node from the cluster.

c). Reconfigure the OS and remaining hardware.

Before going removing steps we need to check the following :

Oracle recommends verifying that you have a good backup of the OCR (Oracle Configuration       Repository) and voting disks using ocrconfig -showbackup.

If you do not have, take the backups for OCR and Voting disks. 

1.  Remove the Instance

Method1. Removing the Instance using SRVCTL command line:

a.    Stop and remove related Instance  

# srvctl stop instance  -d racdb  -i racdb2

# srvctl remove instance  -d racdb  -i racdb2

Remove instance racdb2 for the database racdb? (Y/ [n]) y

If archivelog error occur, then see If for any reason the redo threads are not disabled then disable the thread:

SQL> SELECT group#, thread#, status FROM v$log;

SQL> alter database disable public thread 2;

Check the status of cluster with crs_stat output

#srvctl config database -d racdb

Let’s see the method for deleting node with dbca option:

Method2: Removing Instance using DBCA:

Run DBCA from one of the nodes you are going to keep (rac1).  Leave the database up and also leave the departing instance up and running.

On RAC1:

We can see the the crs_stat command shows us all the services are up and running.

Run DBCA from one of the nodes you are going to keep.

Leave the database up and also leave the departing instance up and running.

$dbca

Click Next and Choose "Instance Management"

Choose "Delete an instance"

On the next screen, select the cluster database from which you will delete an instance.

provide the system privilege username and password.

On the next screen, a list of cluster database instances will appear.  Highlight the instance you would like to delete then click next.

Click Finish, it will appear the following message box.

Click OK button.

Click No button:

Note:

If your database is in archive log mode you may encounter the Following errors:

ORA-350 

ORA-312 

This may occur because the DBCA cannot drop the current log, as it needs archiving.

This issue is fixed in the 10.1.0.3 patchset.

But previous to this patchset you should click the ignore button and when the DBCA completes, manually archive the logs for the deleted instance and dropt the log group.

        SQL> alter system archive log all;

        SQL>SELECT group#, thread#, status FROM v$log;

If the belonging  thread is not disabled then disables the thread. 

        SQL> alter database disable thread 2;

--Verify that the instance was removed from the OCR with the following commands:

#srvctl config database -d

#srvctl config database -d racdb

#cd /bin

#./crs_stat –t

Note:

The above command shows, the node2 (rac2) Instance was removed from the cluster repository list.

The database resource should not be running on the node which is going to be removed from the cluster.

In my example, the crs_stat output shows the database resource “ora.rac.db” is running on the node2, so that we need to be relocate to another running node1 (rac1).

#./crs_relocate  ora.racdb.db

Now check with crs_stat -t command, we will find the "ora.rac.db" is running on the Node1 (rac1).         

Remember don’t forget to relocate the database resource, if database resource running on removed node.

2.Remove ASM Instance:

Stop and Remove  ASM with “srvctl remove” command on node2

# srvctl stop asm  -n rac2

# srvctl remove asm  -n rac2

#crs_stat -t

Confirm the ASM was removed successfully

# srvctl config  asm  -n rac2

# srvctl config asm –n rac1

3.  Removing Node from Cluster:

Once the instance has been deleted, the process of removing the node from the cluster is a manual process.

This is accomplished by running scripts on the deleted node to remove the CRS install, as well as scripts on the remaining nodes to update the node list. 

The following steps assume that the node to be removed is still functioning.

$ $ORA_CRS_HOME/bin/olsnodes -n

Rac1    1

Rac2    2

Before removing node from cluster, we must delete the listener for deleted node.

3. (a) Deleting Listener:

After deleting the ASM instance on node2(rac2), remove the LISTENER running on deleted node using netca utility.

On RAC2:

1. Run the NETCA utility to remove the network configuration: 
2.  $ netca

Select only removing node, i.e. rac2 and click next

Select the delete option and next

Select the Listener Option:

Click on Yes button:

Note: 

If any error occurs while removing Listener on deleting node (rac2), you needed to login to deleting node (rac2) and manually kill the process ID for the listener process.

Run the crs_stat command to verify that all database resources are running on nodes that are going to be available node.

#crs_stat -t

The above command shows us, there is no listener is tuning for deleted node (rac2).

3. (b) Stope and remove the Nodeapps for deleting node

#srvctl stop nodeapps –n rac2

#srvctl remove nodeapps  -n rac2

Please confirm that you intend to remove the node-level applications on rac2 (y/[n]) y

Now check the nodeapps are deleted sucessfully with crs_stat command.

#crs_stat -t

The above command shows us there is no Nodeapps, ASM and Database services for cluster list.

4.Remove the Database HOME from node2 Clusterware:

In my example I did not have any separate ASM HOME. If ASM HOME configured, just repeat the same

following procedure for the ASM HOME.

Next as the Oracle user run the installer with the updateNodeList option on any remaining node (current

running nodes, i.e RAC1) in the cluster.

Sytanx:

$ORACLE_HOME/oui/bin/runInstaller  -updateNodeList  ORACLE_HOME=CLUSTER_NODES=,,

Example:

On Node1 with Oracle user:

rac1$ echo $ORACLE_HOME

rac1 $ORACLE_HOME/oui/bin/runInstaller –updateNodeList

ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1 CLUSTER_NODES=rac1

Note:

That the command above will produce the following error which can safely be ignored:

PRKC-1002: All the submitted commands did not execute successfully And with the above command we are defining the RDBMS $ORACLE_HOME’s that now a part of the cluster in the Oracle inventory on node1.

5.Remove CRS (Clusterware) from rac2

Change to the root user deleted node to finish the removal on a node that is being removed.

This command will stop the CRS stack and delete the ocr.loc file on the node to be removed. The “nosharedvar” option assumes the ocr.loc file is not on a shared file sytem.  If it does exist on a shared file system then specify sharedvar instead.

The “nosharedhome” option specifies that the CRS_HOME is on a local filesystem.

If the CRS_HOME is on a shared file system, specify “sharedhome” instead of “nosharedhome”.

Run the rootdelete.sh script from /install.

5.(b).Example:

On node2 with root user:

# cd /install

# ./rootdelete.sh local nosharedvar nosharedhome

After successful above command, just try to check the crs_stat command

On rac2:

Crs_stat -t

Crs-0184: Cannot communicate with the CRS daemon.

5 (b). 

From any of the remaining nodes other than the one that is being deleted, execute the rootdeletenode.sh script from /install directory as a root user to remove the node2 from the OCR.

It needs node name and node number which can be obtained by running olsnodes -n command line utility.

On rac1:

# olsnodes -n

rac1       1

rac2       2

# cd /install

# ./rootdeletenode.sh ,2

In the above command, Do NOT put a space after the comma between the two

Example:

On rac1 with root user:

# cd CRS_HOME/install

# ./rootdeletenode.sh  rac2,2

6.Update the Inventory for CRS on the remaining Nodes:

Now switch back to the oracle user account on current running nodes (node1) and run the same runInstaller command as before on existing node1  (rac1).

Run it this time from the instead of the ORACLE_HOME and specify all of the remaining nodes.

      This step needs to be performed from once per home (Clusterware, ASM and RDBMS homes).

Syntax:

For Clusterware:

$ /oui/bin/runInstaller -updateNodeList      ORACLE_HOME=CLUSTER_NODES=, , CRS=TRUE

For ASM:

$ /oui/bin/runInstaller -updateNodeList      ORACLE_HOME=CLUSTER_NODES=, , CRS=TRUE

For RDBMS:

$ /oui/bin/runInstaller -updateNodeList      ORACLE_HOME=CLUSTER_NODES=, , CRS=TRUE

Example:

For Cluster ware:

On node1 (rac1) with oracle user:

Rac1$ /oracle/product/10.2.0/crs/oui/bin/runInstaller –updateNodeList

ORACLE_HOME=/oracle/product/10.2.0/crs CLUSTER_NODES=rac1 CRS=TRUE

With this command we are defining the CRS HOME's that now are part of the cluster in the Oracle inventory. 

For ASM Home:

Rac1$ /oracle/product/10.2.0/crs/oui/bin/runInstaller –updateNodeList   ORACLE_HOME=/oracle/product/10.2.0/asm CLUSTER_NODES=rac1 CRS=TRUE

For RDBMS Home:

Rac1 $/oracle/product/10.2.0/crs/oui/bin/runInstaller –updateNodeList  

ORACLE_HOME=/oracle/product/10.2.0/db_1 CLUSTER_NODES=rac1 CRS=TRUE

Manually Deleting the Oracle homes and CRS home etc:

Once the node updates are done you will need to manually delete the $ORACLE_HOME and $CRS_HOME from the node to be expunged, unless, of course, either of these is on a shared file system that is still being used.

         #ORACLE_HOME>: rm -rf *

        #CRS_HOME> : rm -rf *   (as root)

Next, as root, from the deleted node, verify that all init scripts and soft links are removed:

rm -f /etc/init.d/init.cssd

rm -f /etc/init.d/init.crs

rm -f /etc/init.d/init.crsd

rm -f /etc/init.d/init.evmd

rm -f /etc/rc2.d/K96init.crs

rm -f /etc/rc2.d/S96init.crs

rm -f /etc/rc3.d/K96init.crs

rm -f /etc/rc3.d/S96init.crs

rm -f /etc/rc5.d/K96init.crs

rm -f /etc/rc5.d/S96init.crs

rm -Rf /etc/oracle/scls_scr

Now you can also remove the /etc/oracle directory, the /etc/oratab file, and the Oracle inventory (your wish)

Removing a Node from a 10gR1 RAC Cluster [ID 269320.1]